Archive
Hyper-V. CSV Volumes pause states 5120: Workaround
Microsoft is aware that after installing KB3126593 (MS16-014) there may be an issue that causes loss of network packets.
This may cause Cluster Shared Volumes (CSV) on the nodes Failover Clusters to going into a paused state with an event ID 5120 in the System event log what indicates “Status c000020c – STATUS_CONNECTION_DISCONNECTED”.
You will also find that, on the node that owns the CSV volume(s), there will be an event 7031 with the source: “Service Control Manager”, which indicates “The Windows Firewall Service terminated unexpectedly”.
Troubleshooting Cluster Shared Volume Auto-Pauses – Event 5120. https://blogs.msdn.microsoft.com/clustering/2014/12/08/troubleshooting-cluster-shared-volume-auto-pauses-event-5120/
Microsoft is aware of this problem and are working on a fix. But until it is release here is what you could do to get around this issue:
Option 1: Disable the Firewall LOGGING for all profiles (domain, private, public). (just the logging, not the firewall . You don’t want to be un-protected)
1. Start the Windows Firewall with Advanced Security management console (wf.msc)
2. Right-click Windows Firewall with Advanced Security on Local Computer and select Properties
3. For all profiles (Domain, Private, Public), under Logging, click Customize. Set Log Dropped Packets and Log Successful connections to NO.
Option 2. Uninstall https://support.microsoft.com/en-us/kb/3126593 from the systems.
I will post an update once Microsoft releases the Hotfix.
Resource outage when moving physical disc when using DPM to backup an HyperV Cluster.
Microsoft just released a critical fix for hyper cluster that have been backed up by System Center DPM
“Physical Disk resource move during the backup of a Cluster Shared Volume (CSV) may cause resource outage”.
If you have configured a Windows Server 2012-based Hyper-V failover cluster CSV and your backup software perform the backup by using snapshots and the Physical Disk resource is moved to another node in the cluster, the Physical Disk resource may fail to come online if the backup of the CSV is in progress. As a result, virtual machines that rely on the CSV may crash.
A supported hotfix is available from Microsoft. However, this hotfix is intended to correct only the problem that is described in this article. Apply this hotfix only to systems that are experiencing this specific problem.
More info :http://support.microsoft.com/kb/2869923
- Windows Server 2012 Datacenter
- Windows Server 2012 Essentials
- Windows Server 2012 Foundation
- Windows Server 2012 Standard
Hyper-V Windows 2012 : High Availability and Resiliency : new enhancements
There are a number of new enhancements that ensure key workloads are resilient, and protected.
• Hyper-V Replica – Asynchronous, application-consistent virtual machine replication and it does not depend on any hardware vendor. You can establish an Hyper-V Replica between 2 separated physical locations without a storage. It permits asynchronous replication of Hyper-V virtual machines between two locations for business continuity and failure recovery.
• Incremental Backups – True differential disk backups of virtual hard disks to help ensure that the data is backed up and restored when necessary. It also reduces storage costs because it backs up only what has changed, not the entire disk.
• NIC Teaming – Provides increased reliability and performance for virtual machines and now does not depends on manufacturer drivers.
• Hyper-V Clustering Enhancements – Unmatched scale and flexibility for virtualized infrastructures:
• Unmatched Scale – Windows Server 2012 support up to 64 physical nodes and up to 4,000 virtual machines in a single cluster providing scalability and flexibility for key virtualized workloads.
• Flexible Virtual Machine Guest Clustering – Provides not only iSCSI guest clustering support, including MPIO, but also enables the use of Virtual Fibre Channel adapters within the virtual machine allowing workloads access to storage area networks using fiber channel fabric. In addition, a virtual fibre channel enables IT to cluster guest operating systems over Fibre Channel providing HA for workloads within VMs and utilize the built-in Windows multi-path I/O (MPIO) for high-availability and load balancing on the storage path. By employing MPIO and Failover Clustering together as complimentary technologies, users are able to mitigate the risk of a system outage at both the hardware and application levels.
• Highly Secure Clustered Storage – Hyper-V, Failover Clustering and BitLocker now work in concert to create the ideal and secure platform for private cloud infrastructure. Windows Server 2012 Cluster disks that are encrypted using BitLocker Drive Encryption enable better physical security for deployments outside secure data centers, providing a critical safeguard for the cloud and helping protect against inadvertent data leaks
• Enhanced Cluster Shared Volumes – Cluster Shared Volume 2.0 (CSV). CSV has been greatly enhanced in a number of ways. From a usability standpoint, CSV is now a core Failover Clustering feature, with simplified administration and management. To support up to 64 nodes in a cluster, CSV has been improved in aspects of both performance and scalability. In terms of integrating with our partners, CSV has specifically been enhanced to work out of the box with storage filter drivers such as those used by: anti-virus, data protection, backup and storage replication ensuring a more seamless integration with existing investments.
• 3 Levels of Availability – Bringing higher availability to workloads that do not support clustering. It does this by providing a light-weight, simple solution to monitor applications running in the VMs and integrating with the host. By monitoring services and event logs inside the virtual machine, Hyper-V and Failover Clustering can detect whether the key services that a virtual machine provides are healthy and provide automatic corrective action such as restarting the virtual machine or restarting a service within the VM. This is in addition to the already existing virtual machine failover capabilities should a host fail, or the virtual machine itself become unresponsive.
• Cluster-Aware Updating – An in-box end-to-end solution for updating Windows Server 2012 Hyper-V Failover Clusters, helping customers to preview, apply, and report on updates, all with zero downtime to the virtual machines.
• Virtual Machine Failover Prioritization – Virtual machine priorities can now be configured to control the order in which specific virtual machines failover or start. This ensures higher priority virtual machines are given the resources they need and lower priority virtual machines are given resources as they are available.
• Affinity (and Anti-Affinity) Virtual Machine Rules – Administrators can now configure partnered virtual machines so that at failover, the partnered machines are migrated simultaneously. For example, administrators can configure their SharePoint virtual machine and the partnered SQL Server virtual machine to always failover together to the same node. Administrators can also specify that two specific virtual machines cannot coexist on the same node in a failover scenario.
How Does VmWare compare?
| Capability | Windows Server 2012 Hyper-V |
VMware ESXi 5.0 |
VMware vSphere 5.0 Enterprise Plus |
| Incremental Backups |
Yes |
No |
Yes |
| VM Replication |
Yes |
No |
vCenter SRM |
| NIC Teaming |
Yes |
Yes |
Yes |
| Integrated High Availability |
Yes |
No |
Yes |
| Guest OS Application Monitoring |
Yes |
N/A |
No |
| Cluster-Aware Updating |
Yes |
N/A |
Yes |
| Failover Prioritization |
Yes |
N/A |
Yes |
| Affinity & Anti-Affinity Rules |
Yes |
N/A |
Yes |
What are you waiting for? start today your own POC of Windows 2012 !
Hyper-V : Network Design, Configuration and Prioritization : Guidance
There is a lot of posts regarding Hyper-V and network, but I found a lot people still don’t get it.
1. Network Design. How many nic’s we need for production environment for High Availiability:
- 1 for Management. Microsoft recommends a dedicated network adapter for Hyper-V server management.
- 2 ( Teamed ) for Virtual machines.Virtual network configurations of the external type require a minimum of one network adapter.
- 2 ( MPIO ) for SCSI.Microsoft recommends that IP storage communication have a dedicated network, so one adapter is required and two or more are necessary to support multipathing.
- 1 for Failover cluster.Windows® failover cluster requires a private network.
- 1 for Live migration.This new Hyper-V R2 feature supports the migration of running virtual machines between Hyper-V servers. Microsoft recommends configuring a dedicated physical network adapter for live migration traffic. This network should be separate from the network for private communication between the cluster nodes, from the network for the virtual machine, and from the network for storage
- 1 for CSV. Microsoft recommends a dedicated network to support the communications traffic created by this new Hyper-V R2 feature. In the network adapter properties, Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks must be enabled to support SMB
But how about production environments when the blades have only 4 Physical NIC’s?
Option 1. If your vendor does support NPAR technology(Broadcom, QLogic), you will be able to create up to 4 “Virtual Logical NIC’s” per physical NIC ( VLAN/QoS ). Although this solution is not supported by MS, it’s the best solution in terms of performance and it is supported by the vendors. This solution will provide you 100% HA as you can have up to 16 Logical NIC’s.
Option 2. Supported by MS. Allocate 2(two) NIC’sfor the iSCSI using MPIO and then :
| Host configuration | Virtual machine access | Management | Cluster and Cluster Shared Volumes | Live migration | Comments |
| 2 network adapters with 10 Gbps | Virtual network adapter 1 | Virtual network adapter 1 with bandwidth capped at 1% | Network adapter 2 | Network adapter 2 with bandwidth capped at 50% | Supported |
Note that the QoS configuration is based on “per port” and Windows only allows you to cap specify caps – not reserves. This solution, although supported by MS, dos not give you 100% HA.
2. Network Configuration. What need to be enabled/disabled?
| Usage | Number of Network Cards | Comments |
| Management Network(Parent Partition) | 1 Network Card |
|
| Storage ISCSI | 2 Network Cards – Not Teamed |
|
| VM Network (Parent Partition) |
2 Network cards : 1 for Dynamic IP’s 1 for Reserved IP’s |
|
| Cluster Heartbeat | 1 Network Card |
|
| Cluster Shared Volume (CSV) | 1 Network Card |
|
| Live Migration | 1 Network Card |
|
2. Network Prioritization. What need to be enabled/disabled?
By default, all internal cluster network have a metric value starting at 1000 and incrementing by 100. The first internal network which the cluster sees when it first comes online has a metric of 1000, the second has a metric of 1100, etc.
When you create CSV’s, the failover cluster automatically chooses the network that appears to be the best for CSV communication. The lowest Metric value designates the network for Cluster and CSV traffic. The second lowest value designates the network for live migration. Additional networks with a metric below 10000 will be used as backup networks if the “Cluster & CSV Traffic” or “Live Migration Traffic” networks fail. The lowest network with a value of at least 10000 will be used for “Public Traffic”. Consider giving the highest possible values to the networks which you do not want any cluster or public traffic to go through, such as for “ISCSI Traffic”, so that they are never used, or only used when no other networks at all are available.
To view the networks, their metric values, and if they were automatically or manually configured, run the clustering PowerShell cmdlet:
PS > Get-ClusterNetwork | ft Name, Metric, AutoMetric
To change the value of a network metric, run:
PS >Get-ClusterNetwork “Live Migration” ).Metric =800
If you want the cluster to start automatically assigning the Metric setting again for the network named “Live Migration”:
PS > Get-ClusterNetwork “Live Migration” ).AutoMetric = $true
How to override Network Prioritization Behavior?
Option 1. Change the network’s properties. If you select “Do not allow cluster network communication on this network”, then it will not be possible to send any “Cluster & CSV Traffic” or “Live Migration Traffic” through this network, even if the network has the lowest metric values. The cluster will honor this override and find the network with the next lowest value to send this type of traffic :
- In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.
- Select Properties
- Change the radio buttons or checkboxes.
Option 2 (exclusively for “Live Migration Traffic”) :
To configure a cluster network for live migration:
- In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.
- Expand Services and applications.
- In the console tree (on the left), select the clustered virtual machine for which you want to configure the network for live migration.
- Right-click the virtual machine resource displayed in the center pane (not on the left), and then click Properties.
- Click the Network for live migration tab, and select one or more cluster networks to use for live migration. Use the buttons on the right to move the cluster networks up or down to ensure that a private cluster network is the most preferred. The default preference order is as follows: networks that have no default gateway should be located first; networks that are used by cluster shared volumes and cluster traffic should be located last.Live migration will be attempted in the order of the networks specified in the list of cluster networks. If the connection to the destination node using the first network is not successful, the next network in the list is used until the complete list is exhausted, or there is a successful connection to the destination node using one of the networks.
Note : You don’t need to perform this action as per VM basis. When you configure a network for live migration for a specific virtual machine, the setting is global and therefore applies to all virtual machines.
Some other interesting articles:
http://technet.microsoft.com/en-us/library/dd446679(WS.10).aspx
http://blogs.technet.com/b/vishwa/archive/2011/02/01/tuning-scvmm-for-vdi-deployments.aspx
http://blogs.msdn.com/b/clustering/archive/2011/06/17/10176338.aspx
I am Speaking at Teched Australia 2011
I am absolutely thrilled to announce I will be presenting the following two sessions at Tech.Ed Australia 2011 :
| SCVMM 2012: Deployment, Planning, Upgrade
This session provides a scenario rich detailed walk through of VMM 2012 deployment, planning, and upgrade scenarios. Come and learn how to best plan your next VMM rollout |
| SCVMM 2012 Fabric Lifecycle: Networking and Storage
This session provides a scenario rich detailed walk through of new and more robust networking and storage features in VMM 2012. In this session you will learn how to discover, configure, and provision networking |
Came along! It will be an excellent session.
Tech.Ed Australia 2011 is on the Gold Coast between the 30th August and the 2nd September, registrations are now open. Find out more at http://australia.msteched.com/
MS Virtualization for VMware Pros : Jump Start
Exclusive Jump Start virtual training event – “Microsoft Virtualization for VMware Professionals” FREE – on TechNet Edge
Where do I go for this great training?
The HD-quality video recordings of this course are on TechNet Edge. If you’re interested in one specific topic, I’ve included links to each module as well.
· Entire course on TechNet Edge: “Microsoft Virtualization for VMware Professionals” Jump Start
o Virtualization Jump Start (01): Virtualization Overview
o Virtualization Jump Start (02): Differentiating Microsoft & VMware
o Virtualization Jump Start (03a): Hyper-V Deployment Options & Architecture | Part 1
o Virtualization Jump Start (03b): Hyper-V Deployment Options & Architecture | Part 2
o Virtualization Jump Start (04): High-Availability & Clustering
o Virtualization Jump Start (05): System Center Suite Overview with focus on DPM
o Virtualization Jump Start (06): Automation with Opalis, Service Manager & PowerShell
o Virtualization Jump Start (07): System Center Virtual Machine Manager 2012
o Virtualization Jump Start (08): Private Cloud Solutions, Architecture & VMM Self-Service Portal 2.0
o Virtualization Jump Start (09): Virtual Desktop Infrastructure (VDI) Architecture | Part 1
o Virtualization Jump Start (10): Virtual Desktop Infrastructure (VDI) Architecture | Part 2
o Virtualization Jump Start (11): v-Alliance Solution Overview
o Virtualization Jump Start (12): Application Delivery for VDI
· Links to course materials on Born to Learn
Hyper-V CSV protection with DPM 2010
DPM design for Hyper-V :
- Incremental Backups Only – full only once
- No More Backup Window – online backups
- Application consistency via VSS
- Protect Live Migration VMs in CSV clusters
- Protect whole VM, recover individual items
- Auto Protection of new VMs
- Routine backups – nightly or more frequently
Strategy to Deply DPM 2010 in Hyper-V/CSV environment :
1. Plan:
- Identify Project Requirements and scope
- Simulate using Hyper-V Calculator
- Review Hyper-V design guidelines & Best Practices
- Acquire VSS Hardware Provider & check environment with SDPM SAN Testing Tool:
- Verify that the SAN supports hardware snapshots & is certified for DPM.
- http://blogs.technet.com/dpm/archive/2010/02/05/tested-hardware-vss-provider-table.aspx
- Install the VSS hardware provider
- Self Certifying your SAN. If your SAN is not in the certified list & you have a SAN, run the SCDPM SAN Testing Tool : StartCSVTests.cmd /run ( to be realeased yet )
- Validate a Pilot Environment
- Ready for Deployment
- Integration components should be updated on both VMs.
- Ensure that Live Migration is functional
2. Deploy : Protected CSV server
- Install all recommended prerequisites:
| General Prerequisite | Install the integration components on the guests. For more information, see Install a Guest Operating System (http://go.microsoft.com/fwlink/?LinkId=165041). |
| Microsoft Windows Server 2008 SP2 | Install Microsoft Windows Server 2008 SP2 OR Install the Microsoft Hyper-V prerequisites (http://go.microsoft.com/fwlink/?LinkID=133781). |
| Microsoft Windows Server 2008 R2 | Install the following on the host computer:KB975354 (http://support.microsoft.com/kb/975354)KB975921 (http://support.microsoft.com/kb/975921) |
- Failover Cluster Validation
- Enable Automount : Enable Auto mount on the host for successful child partition backups : MOUNTVOL /E
- Install Hardware Provider:Ensure that VSS hardware providers are installed : vssadmin list providers
- Validate Hardware snapshots
- Use DSConfig Tool to serialize backups (for software snapshots providers only) :
· Problem:With software snapshots, simultaneous triggering of VM backups (especially scale deployments) will cause backup jobs to time out and fail.
· Enable Per CSV LUN Serialization:This serialization limits the number of virtual machine backups happening on a single CSV LUN.Create a DataSourceGroups.xml file by running the DSConfig.ps1 script on any one node of a cluster containing CSV.Place the file in the DPM server at %PROGRAMFILES%\Microsoft DPM\DPM\Config.Repeat step 1 for every cluster that is protected by a DPM server.The DataSourceGroups.xml file needs to be updated only when virtual machines are added, deleted, or modified in the cluster and protection is configured for them.
3. Deploy : DPM 2010 server
- Enable Hyper-V Role on DPM Server :DPM supports item-level recovery (ILR), which allows you to do granular recovery of files, folders, volumes, and virtual hard disks (VHDs) from a host-level backup of Hyper-V virtual machines to a network share or a volume on a DPM protected server. You must have the Hyper-V role enabled on the DPM server to perform item-level recoveries. During item-level recovery, DPM has to mount the VHDs of the protected virtual machines
- Install DPM server pre-requisites
- Install DPM 2010
- Turn off “TCP Chimney offload”For better throughput, TCP Chimney Offload should be turned off.
- Allocate Storage Pool & Deploy Agents
- Datasourcegroups.xml file:
- Merge the DatasourceGroups.XML file for all CSVs protected by DPM server. This is applicable when software providers are used on the CSV cluster.
- Copy the <Group> tags from all the DataSourceGroup.xml files generated and add the text between the <DataSourceGroup> tags. The DataSourceGroups.xml file will now contain one <header> tag, one <DataSourceGroup> tag, and <Group> tags from all CSV clusters.
- Configure Protection Groups
- DPM2010 provides Datasource caching for faster enquiry.
- Remember to clear cache if new VMs have been added
- Enable AutoProtection:
Script to Auto-add VMs : http://blogs.technet.com/dpm/archive/2009/12/03/hyper-v-protection-with-dpm-2010-beta-how-to-automatically-protect-new-virtual-machines.aspx
Recommended Books
System Center 2012 R2 Virtual Machine
System Center Virtual Machine Manager 2012 Cookbook
Cloudtidings.com 