Resource outage when moving physical disc when using DPM to backup an HyperV Cluster.

Microsoft just released a critical fix for hyper cluster that have been backed up by System Center DPM

“Physical Disk resource move during the backup of a Cluster Shared Volume (CSV) may cause resource outage”.

If you have configured a Windows Server 2012-based Hyper-V failover cluster CSV and your backup software perform the backup by using snapshots and the Physical Disk resource is moved to another node in the cluster, the Physical Disk resource may fail to come online if the backup of the CSV is in progress. As a result, virtual machines that rely on the CSV may crash.

A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing this specific problem.

More info :http://support.microsoft.com/kb/2869923

  • Windows Server 2012 Datacenter
  • Windows Server 2012 Essentials
  • Windows Server 2012 Foundation
  • Windows Server 2012 Standard

Hyper-V Windows 2012 : High Availability and Resiliency : new enhancements

There are a number of new enhancements that ensure key workloads are resilient, and protected.

Hyper-V ReplicaAsynchronous, application-consistent virtual machine replication and it does not depend on any hardware vendor. You can establish an Hyper-V Replica between 2 separated physical locations without a storage. It permits asynchronous replication of Hyper-V virtual machines between two locations for business continuity and failure recovery.

Incremental BackupsTrue differential disk backups of virtual hard disks to help ensure that the data is backed up and restored when necessary. It also reduces storage costs because it backs up only what has changed, not the entire disk.

NIC TeamingProvides increased reliability and performance for virtual machines and now does not depends on manufacturer drivers.

Hyper-V Clustering Enhancements – Unmatched scale and flexibility for virtualized infrastructures:

Unmatched Scale – Windows Server 2012 support up to 64 physical nodes and up to 4,000 virtual machines in a single cluster providing scalability and flexibility for key virtualized workloads.

Flexible Virtual Machine Guest Clustering – Provides not only iSCSI guest clustering support, including MPIO, but also enables the use of Virtual Fibre Channel adapters within the virtual machine allowing workloads access to storage area networks using fiber channel fabric. In addition, a virtual fibre channel enables IT to cluster guest operating systems over Fibre Channel providing HA for workloads within VMs and utilize the built-in Windows multi-path I/O (MPIO) for high-availability and load balancing on the storage path.  By employing MPIO and Failover Clustering together as complimentary technologies, users are able to mitigate the risk of a system outage at both the hardware and application levels.

Highly Secure Clustered Storage – Hyper-V, Failover Clustering and BitLocker now work in concert to create the ideal and secure platform for private cloud infrastructure. Windows Server 2012 Cluster disks that are encrypted using BitLocker Drive Encryption enable better physical security for deployments outside secure data centers, providing a critical safeguard for the cloud and helping protect against inadvertent data leaks

Enhanced Cluster Shared Volumes – Cluster Shared Volume 2.0 (CSV). CSV has been greatly enhanced in a number of ways. From a usability standpoint, CSV is now a core Failover Clustering feature, with simplified administration and management.  To support up to 64 nodes in a cluster, CSV has been improved in aspects of both performance and scalability. In terms of integrating with our partners, CSV has specifically been enhanced to work out of the box with storage filter drivers such as those used by: anti-virus, data protection, backup and storage replication ensuring a more seamless integration with existing investments.

3 Levels of Availability – Bringing higher availability to workloads that do not support clustering. It does this by providing a light-weight, simple solution to monitor applications running in the VMs and integrating with the host. By monitoring services and event logs inside the virtual machine, Hyper-V and Failover Clustering can detect whether the key services that a virtual machine provides are healthy and provide automatic corrective action such as restarting the virtual machine or restarting a service within the VM.  This is in addition to the already existing virtual machine failover capabilities should a host fail, or the virtual machine itself become unresponsive.

Cluster-Aware Updating – An in-box end-to-end solution for updating Windows Server 2012 Hyper-V Failover Clusters, helping customers to preview, apply, and report on updates, all with zero downtime to the virtual machines.

Virtual Machine Failover Prioritization –  Virtual machine priorities can now be configured to control the order in which specific virtual machines failover or start. This ensures higher priority virtual machines are given the resources they need and lower priority virtual machines are given resources as they are available.

Affinity (and Anti-Affinity) Virtual Machine Rules – Administrators can now configure partnered virtual machines so that at failover, the partnered machines are migrated simultaneously. For example, administrators can configure their SharePoint virtual machine and the partnered SQL Server virtual machine to always failover together to the same node. Administrators can also specify that two specific virtual machines cannot coexist on the same node in a failover scenario.

How Does VmWare compare?

Capability Windows   Server 2012 Hyper-V

VMware   ESXi 5.0

VMware   vSphere 5.0 Enterprise Plus
 Incremental   Backups

 Yes

No

Yes

 VM   Replication

 Yes

 No

vCenter SRM

 NIC   Teaming

 Yes

 Yes

Yes

 Integrated   High Availability

 Yes

 No

 Yes

 Guest OS   Application Monitoring

 Yes

N/A

 No

 Cluster-Aware   Updating

 Yes

N/A

Yes

Failover   Prioritization

Yes

 N/A

 Yes

Affinity   & Anti-Affinity Rules

Yes

N/A

Yes

What are you waiting for? start today your own POC of Windows 2012 !

Hyper-V : Network Design, Configuration and Prioritization : Guidance

There is a lot of posts regarding Hyper-V and network, but I found a lot people still don’t get it.

1. Network Design. How many nic’s we need for production environment for High Availiability:

  • 1 for Management. Microsoft recommends a dedicated network adapter for Hyper-V server management.
  • 2 ( Teamed )  for Virtual machines.Virtual network configurations of the external type require a minimum of one network adapter.
  • 2 ( MPIO ) for SCSI.Microsoft recommends that IP storage communication have a dedicated network, so one adapter is required and two or more are necessary to support multipathing.
  • 1 for Failover cluster.Windows® failover cluster requires a private network.
  • 1 for Live migration.This new Hyper-V R2 feature supports the migration of running virtual machines between Hyper-V servers. Microsoft recommends configuring a dedicated physical network adapter for live migration traffic. This network should be separate from the network for private communication between the cluster nodes, from the network for the virtual machine, and from the network for storage
  • 1 for CSV. Microsoft recommends a dedicated network to support the communications traffic created by this new Hyper-V R2 feature. In the network adapter properties, Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks must be enabled to support SMB

But how about production environments when the blades have only 4 Physical NIC’s?

Option 1. If your vendor does support NPAR technology(Broadcom, QLogic), you will be able to create up to 4 “Virtual Logical NIC’s” per physical NIC ( VLAN/QoS ). Although this solution is not supported by MS, it’s the best solution in terms of performance and it is supported by the vendors. This solution will provide you 100% HA as you can have up to 16 Logical NIC’s.

Option 2. Supported by MS. Allocate 2(two) NIC’sfor the iSCSI using MPIO and then :

Host configuration Virtual machine access Management Cluster and Cluster Shared Volumes Live migration Comments
2 network adapters with 10 Gbps Virtual network adapter 1 Virtual network adapter 1 with bandwidth capped at 1% Network adapter 2 Network adapter 2 with bandwidth capped at 50% Supported

Note that the QoS configuration is based on “per port”  and Windows only allows you to cap specify caps – not reserves. This solution, although supported by MS, dos not give you 100% HA.

2. Network Configuration. What need to be enabled/disabled?

Usage Number of Network Cards Comments
Management Network(Parent Partition) 1 Network Card
  • Make sure this card is listed first in the Adapter and Bindings connection order.
  • In Failover Cluster Manager make sure that the NIC is configured to allow cluster network communication on this network. This will act as a secondary connection for the Heartbeat.
Storage ISCSI 2 Network Cards – Not Teamed
  • Enable MPIO.
  • Disable NetBIOS on these interfaces
  • Do not configure a Gateway
  • Do not configure a DNS server
  • Make sure that each NIC is NOT set to register its connection in DNS
  • Remove File and Printer sharing
  • Do not remove Client from Microsoft networks if using Netapp Snapdrive with RPC authentication
  • In Failover Cluster Manager select- Do not allow cluster network communication on this network
VM Network
(Parent Partition)
2 Network cards :
1 for Dynamic IP’s
1 for Reserved IP’s
  • Disable NetBIOS on these interfaces
  • Do not configure a Gateway
  • Do not configure a DNS server
  • Make sure that each NIC is NOT set to register its connection in DNS
  • Remove File and Printer sharing and Client from Microsoft networks
  • In Failover Cluster Manager select – Do not allow cluster network communication on this network.
Cluster Heartbeat 1 Network Card
  • Disable NetBIOS on this interface
  • Do not configure a Gateway
  • Do not configure a DNS server
  • Make sure that this NIC is NOT set to register its connection in DNS
  • Make sure that Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks are enabled to support Server Message Block (SMB), which is required for CSV.
  • In Failover Cluster Manager make sure that the NIC is configured to allow cluster network communication on this network.
  • In Failover Cluster Manager remove the tick box for Allow Clients Connect through this network. This setting has nothing to do with the host/parent partition. This setting is used to control over what NICs the Cluster Resources can be accessed.
Cluster Shared Volume (CSV) 1 Network Card
  • Disable NetBIOS on this interface
  • Make sure that this NIC is NOT set to register its connection in DNS
  • Make sure that Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks are enabled to support Server Message Block (SMB), which is required for CSV.
  • In Failover Cluster Manager remove the tick box for Allow Clients Connect through this network. This setting has nothing to do with the host/parent partition. This setting is used to control over what NICs the Cluster Resources can be accessed. This is more relevant for other workloads e.g. File Cluster. It has no impact on the communication with the host partition or for the VM’s themselves.
  • By default the cluster will automatically choose the NIC to be used for CSV communication. We will change this later.
  • This traffic is not routable and has to be on the same subnet for all nodes.
Live Migration 1 Network Card
  • Disable NetBIOS on this interface
  • Make sure that this NIC is NOT set to register its connection in DNS.
  • In Failover Cluster Manager remove the tick box for Allow Clients Connect through this network. This setting has nothing to do with the host/parent partition. This setting is used to control over what NICs the Cluster Resources can be accessed. This is more relevant for other workloads e.g. File Cluster. It has no impact on the communication with the host partition or for the VM’s themselves.
  • By default the cluster will automatically choose the NIC to be used for Live-Migration. You can select multiple networks for LM and give them a preference.

 

2. Network Prioritization. What need to be enabled/disabled?

By default, all internal cluster network have a metric value starting at 1000 and incrementing by 100.  The first internal network which the cluster sees when it first comes online has a metric of 1000, the second has a metric of 1100, etc.

When you create CSV’s,  the failover cluster automatically chooses the network that appears to be the best for CSV communication. The lowest Metric value designates the network for Cluster and CSV traffic. The second lowest value designates the network for live migration. Additional networks with a metric below 10000 will be used as backup networks if the “Cluster & CSV Traffic” or “Live Migration Traffic” networks fail.  The lowest network with a value of at least 10000 will be used for “Public Traffic”. Consider giving the highest possible values to the networks which you do not want any cluster or public traffic to go through, such as for “ISCSI Traffic”, so that they are never used, or only used when no other networks at all are available.

To view the networks, their metric values, and if they were automatically or manually configured, run the clustering PowerShell cmdlet:
PS > Get-ClusterNetwork | ft Name, Metric, AutoMetric

To change the value of a network metric, run:
PS >Get-ClusterNetwork “Live Migration” ).Metric =800

If you want the cluster to start automatically assigning the Metric setting again for the network named “Live Migration”:
PS > Get-ClusterNetwork “Live Migration” ).AutoMetric = $true

How to override Network Prioritization Behavior?

Option 1. Change the network’s properties. If you select “Do not allow cluster network communication on this network”, then it will not be possible to send any “Cluster & CSV Traffic” or “Live Migration Traffic” through this network, even if the network has the lowest metric values.  The cluster will honor this override and find the network with the next lowest value to send this type of traffic :

  1. In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.
  2.  Select Properties
  3. Change the radio buttons or checkboxes.

Option 2 (exclusively for “Live Migration Traffic”) :

To configure a cluster network for live migration:

  1. In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.
  2. Expand Services and applications.
  3. In the console tree (on the left), select the clustered virtual machine for which you want to configure the network for live migration.
  4. Right-click the virtual machine resource displayed in the center pane (not on the left), and then click Properties.
  5. Click the Network for live migration tab, and select one or more cluster networks to use for live migration. Use the buttons on the right to move the cluster networks up or down to ensure that a private cluster network is the most preferred. The default preference order is as follows: networks that have no default gateway should be located first; networks that are used by cluster shared volumes and cluster traffic should be located last.Live migration will be attempted in the order of the networks specified in the list of cluster networks. If the connection to the destination node using the first network is not successful, the next network in the list is used until the complete list is exhausted, or there is a successful connection to the destination node using one of the networks.

Note : You don’t need to perform this action as per VM basis. When you configure a network for live migration for a specific virtual machine, the setting is global and therefore applies to all virtual machines.

Some other interesting articles:

http://technet.microsoft.com/en-us/library/dd446679(WS.10).aspx

http://www.hyper-v.nu/archives/hvredevoort/2011/03/windows-server-2008-r2-sp1-and-hp-network-teaming-testing-results/

http://blogs.technet.com/b/vishwa/archive/2011/02/01/tuning-scvmm-for-vdi-deployments.aspx

http://blogs.msdn.com/b/clustering/archive/2011/06/17/10176338.aspx

http://technet.microsoft.com/en-us/library/dd446679.aspx

I am Speaking at Teched Australia 2011

I am absolutely thrilled to announce I will be presenting the following two sessions at Tech.Ed Australia 2011 :

SCVMM 2012: Deployment, Planning, Upgrade

This session provides a scenario rich detailed walk through of VMM 2012 deployment, planning, and upgrade scenarios. Come and learn how to best plan your next VMM rollout

SCVMM 2012 Fabric Lifecycle: Networking and Storage

This session provides a scenario rich detailed walk through of  new and more robust networking and storage features in VMM 2012. In this session you will learn how to discover, configure, and provision networking
and storage fabric for use with the private cloud

Came along! It will be an excellent session.

Tech.Ed Australia 2011 is on the Gold Coast between the 30th August and the 2nd September, registrations are now open. Find out more at http://australia.msteched.com/

MS Virtualization for VMware Pros : Jump Start

Exclusive Jump Start virtual training event – “Microsoft Virtualization for VMware Professionals”  FREE – on TechNet Edge

Where do I go for this great training?

The HD-quality video recordings of this course are on TechNet Edge. If you’re interested in one specific topic, I’ve included links to each module as well.

 ·   Entire course on TechNet Edge: “Microsoft Virtualization for VMware Professionals” Jump Start

o   Virtualization Jump Start (01): Virtualization Overview

o   Virtualization Jump Start (02): Differentiating Microsoft & VMware

o   Virtualization Jump Start (03a): Hyper-V Deployment Options & Architecture | Part 1

o   Virtualization Jump Start (03b): Hyper-V Deployment Options & Architecture | Part 2

o   Virtualization Jump Start (04): High-Availability & Clustering

o   Virtualization Jump Start (05): System Center Suite Overview with focus on DPM

o   Virtualization Jump Start (06): Automation with Opalis, Service Manager & PowerShell

o   Virtualization Jump Start (07): System Center Virtual Machine Manager 2012

o   Virtualization Jump Start (08): Private Cloud Solutions, Architecture & VMM Self-Service Portal 2.0

o   Virtualization Jump Start (09): Virtual Desktop Infrastructure (VDI) Architecture | Part 1

o   Virtualization Jump Start (10): Virtual Desktop Infrastructure (VDI) Architecture | Part 2

o   Virtualization Jump Start (11): v-Alliance Solution Overview

o   Virtualization Jump Start (12): Application Delivery for VDI

·  Links to course materials on Born to Learn

Hyper-V CSV protection with DPM 2010

  DPM design for Hyper-V :

  •  Incremental Backups Only – full only once
  •  No More Backup Window – online backups
  • Application consistency via VSS
  • Protect Live Migration VMs in CSV clusters
  • Protect whole VM, recover individual items
  • Auto Protection of new VMs
  • Routine backups – nightly or more frequently

Strategy to Deply DPM 2010 in Hyper-V/CSV environment :

1. Plan:

  • Identify Project Requirements and scope
  • Simulate using Hyper-V Calculator
  • Review Hyper-V design guidelines & Best Practices
  • Acquire VSS Hardware Provider & check environment with SDPM SAN Testing Tool:
  • Validate a Pilot Environment
  • Ready for Deployment
    • Integration components should be updated on both VMs.
    • Ensure that Live Migration is functional

  2. Deploy : Protected CSV server

  • Install all recommended prerequisites:
General Prerequisite Install the integration components on the guests. For more information, see Install a Guest Operating System (http://go.microsoft.com/fwlink/?LinkId=165041).
Microsoft Windows Server 2008 SP2 Install Microsoft Windows Server 2008 SP2 OR Install the Microsoft Hyper-V prerequisites (http://go.microsoft.com/fwlink/?LinkID=133781).
Microsoft Windows Server 2008 R2 Install the following on the host computer:KB975354 (http://support.microsoft.com/kb/975354)KB975921 (http://support.microsoft.com/kb/975921)
  • Failover Cluster Validation
  • Enable Automount : Enable Auto mount on the host for successful child partition backups :  MOUNTVOL /E
  • Install Hardware Provider:Ensure that VSS hardware providers are installed : vssadmin list providers
  • Validate Hardware snapshots
  • Use DSConfig Tool to serialize backups (for software snapshots providers only) : 
    ·   Problem:With software snapshots, simultaneous triggering of VM backups (especially scale deployments) will cause backup jobs to time out and fail.
    ·   Enable Per CSV LUN Serialization:This serialization limits the number of virtual machine backups happening on a single CSV LUN.Create a DataSourceGroups.xml file by running the DSConfig.ps1 script on any one node of a cluster containing CSV.Place the file in the DPM server at %PROGRAMFILES%\Microsoft DPM\DPM\Config.Repeat step 1 for every cluster that is protected by a DPM server.The DataSourceGroups.xml file needs to be updated only when virtual machines are added, deleted, or modified in the cluster and protection is configured for them.

 3. Deploy  : DPM 2010 server

  • Enable Hyper-V Role on DPM Server :DPM supports item-level recovery (ILR), which allows you to do granular recovery of files, folders, volumes, and virtual hard disks (VHDs) from a host-level backup of Hyper-V virtual machines to a network share or a volume on a DPM protected server. You must have the Hyper-V role enabled on the DPM server to perform item-level recoveries. During item-level recovery, DPM has to mount the VHDs of the protected virtual machines
  • Install DPM server pre-requisites
  • Install DPM 2010
  • Turn off “TCP Chimney offload”For better throughput, TCP Chimney Offload should be turned off.
  • Allocate Storage Pool & Deploy Agents
  • Datasourcegroups.xml file:
    • Merge the DatasourceGroups.XML file for all CSVs protected by DPM server. This is applicable when software providers are used on the CSV cluster.
    •  Copy the <Group> tags from all the DataSourceGroup.xml files generated and add the text between the <DataSourceGroup> tags. The DataSourceGroups.xml file will now contain one <header> tag, one <DataSourceGroup> tag, and <Group> tags from all CSV clusters.

 

 

Follow

Get every new post delivered to your Inbox.

Join 988 other followers