Archive
Hyper-V. CSV Volumes pause states 5120: Workaround
Microsoft is aware that after installing KB3126593 (MS16-014) there may be an issue that causes loss of network packets.
This may cause Cluster Shared Volumes (CSV) on the nodes Failover Clusters to going into a paused state with an event ID 5120 in the System event log what indicates “Status c000020c – STATUS_CONNECTION_DISCONNECTED”.
You will also find that, on the node that owns the CSV volume(s), there will be an event 7031 with the source: “Service Control Manager”, which indicates “The Windows Firewall Service terminated unexpectedly”.
Troubleshooting Cluster Shared Volume Auto-Pauses – Event 5120. https://blogs.msdn.microsoft.com/clustering/2014/12/08/troubleshooting-cluster-shared-volume-auto-pauses-event-5120/
Microsoft is aware of this problem and are working on a fix. But until it is release here is what you could do to get around this issue:
Option 1: Disable the Firewall LOGGING for all profiles (domain, private, public). (just the logging, not the firewall . You don’t want to be un-protected)
1. Start the Windows Firewall with Advanced Security management console (wf.msc)
2. Right-click Windows Firewall with Advanced Security on Local Computer and select Properties
3. For all profiles (Domain, Private, Public), under Logging, click Customize. Set Log Dropped Packets and Log Successful connections to NO.
Option 2. Uninstall https://support.microsoft.com/en-us/kb/3126593 from the systems.
I will post an update once Microsoft releases the Hotfix.
What’s New in Hyper-V vNext ? Check out at the Infrastructure Saturday event in Brisbane
Saturday 22nd November
Infrastructure Saturday is a day long event for south east Queensland based IT Professional that work with Microsoft products. This event is an educational, informative & lively day filled with sessions about Microsoft technologies.
Location: Microsoft office, Brisbane, QLD. http://www.infrastructuresaturday.com/
Topics covered in my Session: What’s New in Hyper-V vNext?
- New Virtual Machine Upgrade Process
- New Integration Components installation method
- Secure Boot for Linux
- Distributed Storage Quality of Service (QoS)
- Hyper-V Backup
- Hyper-V Virtual Machine Configuration
- Cluster OS Rolling Upgrade
Windows Server 2012 Update – released today
Today is a big day with the release of Windows Server 2012 R2 Update!
Windows Server 2012 R2 Update is a cumulative update of every hotfix released for Win2012 R2 to date. This just radically simplified our set of recommended hotfixes.
New Win2012 R2 Cluster Recommended hotfixes:
1. Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 Update April, 2014
http://support.microsoft.com/kb/2919355
2. CSV snapshot file is corrupted when you create some files on the live volume
Update to improve cloud service provider resiliency in Windows Server 2012
Microsoft released an update to improve the cloud service provider in Windows 2012
Note : The KB2870270 roll-up hotfix replaces KB2848344. This hotfix contains the fix for Trim as well as ODX. It is no longer necessary to disable Trim or ODX with this fix applied
The following issues are also resolved with this update:
– An Windows 2012 Hyper-V Cluster that uses CSV is moved or is toggled between directed access and redirected access when the performance monitor is running. In this situation, the performance counters no longer function.
– An Windows 2012 Hyper-V Cluster and the Virtual Machines VHD files are stored on a scale-out file share and the server is under a heavy load. In this scenario, the server may crash, and you may receive a Stop error message that resembles: STOP: 0x000000C2 (parameter1, parameter2, parameter3, parameter4) – Notes: The parameters in the Stop error message vary, depending on the configuration of the computer and not all “Stop 0x000000C2” errors are caused by this issue.
– When you use the Windows thin provisioning feature on a Windows Server 2012-based computer, you experience the same I/O delays or I/O stalls that occur during the volume-unmounting process in NTF volumes.
Plus the issues that are described in the following Microsoft Knowledge Base (KB) articles are resolved:
Hyper-V Windows 2012 : High Availability and Resiliency : new enhancements
There are a number of new enhancements that ensure key workloads are resilient, and protected.
• Hyper-V Replica – Asynchronous, application-consistent virtual machine replication and it does not depend on any hardware vendor. You can establish an Hyper-V Replica between 2 separated physical locations without a storage. It permits asynchronous replication of Hyper-V virtual machines between two locations for business continuity and failure recovery.
• Incremental Backups – True differential disk backups of virtual hard disks to help ensure that the data is backed up and restored when necessary. It also reduces storage costs because it backs up only what has changed, not the entire disk.
• NIC Teaming – Provides increased reliability and performance for virtual machines and now does not depends on manufacturer drivers.
• Hyper-V Clustering Enhancements – Unmatched scale and flexibility for virtualized infrastructures:
• Unmatched Scale – Windows Server 2012 support up to 64 physical nodes and up to 4,000 virtual machines in a single cluster providing scalability and flexibility for key virtualized workloads.
• Flexible Virtual Machine Guest Clustering – Provides not only iSCSI guest clustering support, including MPIO, but also enables the use of Virtual Fibre Channel adapters within the virtual machine allowing workloads access to storage area networks using fiber channel fabric. In addition, a virtual fibre channel enables IT to cluster guest operating systems over Fibre Channel providing HA for workloads within VMs and utilize the built-in Windows multi-path I/O (MPIO) for high-availability and load balancing on the storage path. By employing MPIO and Failover Clustering together as complimentary technologies, users are able to mitigate the risk of a system outage at both the hardware and application levels.
• Highly Secure Clustered Storage – Hyper-V, Failover Clustering and BitLocker now work in concert to create the ideal and secure platform for private cloud infrastructure. Windows Server 2012 Cluster disks that are encrypted using BitLocker Drive Encryption enable better physical security for deployments outside secure data centers, providing a critical safeguard for the cloud and helping protect against inadvertent data leaks
• Enhanced Cluster Shared Volumes – Cluster Shared Volume 2.0 (CSV). CSV has been greatly enhanced in a number of ways. From a usability standpoint, CSV is now a core Failover Clustering feature, with simplified administration and management. To support up to 64 nodes in a cluster, CSV has been improved in aspects of both performance and scalability. In terms of integrating with our partners, CSV has specifically been enhanced to work out of the box with storage filter drivers such as those used by: anti-virus, data protection, backup and storage replication ensuring a more seamless integration with existing investments.
• 3 Levels of Availability – Bringing higher availability to workloads that do not support clustering. It does this by providing a light-weight, simple solution to monitor applications running in the VMs and integrating with the host. By monitoring services and event logs inside the virtual machine, Hyper-V and Failover Clustering can detect whether the key services that a virtual machine provides are healthy and provide automatic corrective action such as restarting the virtual machine or restarting a service within the VM. This is in addition to the already existing virtual machine failover capabilities should a host fail, or the virtual machine itself become unresponsive.
• Cluster-Aware Updating – An in-box end-to-end solution for updating Windows Server 2012 Hyper-V Failover Clusters, helping customers to preview, apply, and report on updates, all with zero downtime to the virtual machines.
• Virtual Machine Failover Prioritization – Virtual machine priorities can now be configured to control the order in which specific virtual machines failover or start. This ensures higher priority virtual machines are given the resources they need and lower priority virtual machines are given resources as they are available.
• Affinity (and Anti-Affinity) Virtual Machine Rules – Administrators can now configure partnered virtual machines so that at failover, the partnered machines are migrated simultaneously. For example, administrators can configure their SharePoint virtual machine and the partnered SQL Server virtual machine to always failover together to the same node. Administrators can also specify that two specific virtual machines cannot coexist on the same node in a failover scenario.
How Does VmWare compare?
| Capability | Windows Server 2012 Hyper-V |
VMware ESXi 5.0 |
VMware vSphere 5.0 Enterprise Plus |
| Incremental Backups |
Yes |
No |
Yes |
| VM Replication |
Yes |
No |
vCenter SRM |
| NIC Teaming |
Yes |
Yes |
Yes |
| Integrated High Availability |
Yes |
No |
Yes |
| Guest OS Application Monitoring |
Yes |
N/A |
No |
| Cluster-Aware Updating |
Yes |
N/A |
Yes |
| Failover Prioritization |
Yes |
N/A |
Yes |
| Affinity & Anti-Affinity Rules |
Yes |
N/A |
Yes |
What are you waiting for? start today your own POC of Windows 2012 !
W2008R2 Failover Cluster : New hotfix to improve performance enables faster failover times in a SQL 2012
After you enable the AlwaysOn Availability Group feature in Microsoft SQL Server 2012 on a Windows Server 2008 R2-based failover cluster, using local and remote Availability Group database replicas, the failover time to the local replica takes longer than expected.
Note This issue occurs because of inter-node communication within the cluster.
To improve the performance of the “AlwaysOn Availability Group” feature in SQL Server 2012 is available for Windows Server 2008 R2.
Important : You must restart the computer after you apply this hotfix
To download: http://support.microsoft.com/kb/2687741/en-us
Windows 2008R2 – Hyper-V / Cluster. Post SP1 patches
Here is the list of required and optional post Windows 2008R2 SP1 hotfixes.
Please keep in mind that it all depends on which hardware and software combinations are used:
Note: Windows Server 2008 R2 SP1 should be installed at a minimum plus the required hotfixes.
1 Windows & Hyper-V : Required Hotfixes
Validate SCSI Device Vital Product Data (VPD) test fails after you install Windows Server 2008 R2 SP1
http://support.microsoft.com/kb/2531907(required for 3+ node Hyper-V clusters)
The network connection of a running Hyper-V virtual machine may be lost under heavy outgoing network traffic on a computer that is running Windows Server 2008 R2 SP1
http://support.microsoft.com/kb/2263829
The Cluster service stops unexpectedly on a Windows Server 2008 R2 failover cluster node when you perform multiple backup operations in parallel on a cluster shared volume
http://support.microsoft.com/kb/2494162(required for Veeam backup software)
MPIO failover fails on a computer that is running Windows Server 2008 R2
http://support.microsoft.com/kb/2460971
The MPIO driver fails over all paths incorrectly when a transient single failure occurs in Windows Server 2008 or in Windows Server 2008 R2
http://support.microsoft.com/kb/2522766
Performance decreases in Windows Server 2008 R2 when the Hyper-V role is installed on a computer that uses Intel Westmere or Sandy Bridge processors
http://support.microsoft.com/kb/2517329 (for example : Intel Xeon 5600 series processors )
Stop error 0x0000007a occurs on a virtual machine that is running on a Windows Server 2008 R2-based failover cluster with a cluster shared volume, and the state of the CSV is switched to redirected access.
http://support.microsoft.com/kb/2494016 (required for Veeam backup software)
Optional Hotfixes
An update is available for Hyper-V Best Practices Analyzer for Windows Server 2008 R2
http://support.microsoft.com/kb/2485986
“0x0000009E” Stop error when you add an extra storage disk to a failover cluster in Windows Server 2008 R2
http://support.microsoft.com/kb/2520235
A virtual machine online backup fails in Windows Server 2008 R2 when the SAN policy is set to “Offline All”
http://support.microsoft.com/kb/2521348
Cluster node cannot rejoin the cluster after the node is restarted or removed from the cluster in Windows Server 2008 R2
http://support.microsoft.com/kb/2549472
Cluster service stops when an error occurs in the registry replication process of a failover cluster in Windows Server 2008 R2 or in Windows Server 2008
http://support.microsoft.com/kb/2496034
“0x20001” Stop error when you start a Linux VM in Windows Server 2008 R2 SP1
http://support.microsoft.com/kb/2550569
A heap memory leak occurs when an application or service queries the MSCluster_Resource WMI class in Windows Server 2008 R2
http://support.microsoft.com/kb/2580360
Cluster service initiates a failover after a delay of about 80 seconds when you shutdown the active node in Windows Server 2008 R2
http://support.microsoft.com/kb/2575625/en-us?sd=rss&spid=14134
New registration entries are added to the Persistent Reservation table when the physical disk resource that is associated with the CSV is taken offline on a Windows Server 2008 R2-based Failover Cluster
http://support.microsoft.com/kb/2579052/en-us?sd=rss&spid=14134
A transient communication failure causes a Windows Server 2008 R2 failover cluster to stop working
http://support.microsoft.com/kb/2550886
Cluster service leaks memory when the service handles state change notifications in Windows Server 2008 R2 or Windows Server 2008
http://support.microsoft.com/kb/2550894
Hyper-V Export function consumes all available memory in Windows Server 2008 or in Windows Server 2008 R2
http://support.microsoft.com/kb/2547551
Microcode update for Intel processors in Windows 7 or in Windows Server 2008 R2
http://support.microsoft.com/kb/2493989
Corrupted VSS snapshot
http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=975688&kbln=en-us
FIX: The guest operating system may crash (STOP 0xd) when you perform a live migration of Hyper-V virtual machines in a Windows Server 2008 R2 environment
http://support.microsoft.com/kb/2636573
————-
A BIG Thanks to Hans Vredevoort
free training: Failover Clustering with Hyper-V: Designing a Highly-Available Infrastructure for the Private Cloud
Tuesday, December 6th from 8am-11am PST. Microsoft is are offering a free training: Failover Clustering with Hyper-V: Designing a Highly-Available Infrastructure for the Private Cloud
Register here: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=287
Join Symon Perriman (Technical Evangelist for Private Cloud) and Elden Christensen (Principal Program Manager Lead for Clustering) to understand how Windows Server 2008 R2 Failover Clustering with Hyper-V provides critical infrastructure for any datacenter with VM high-availability and mobility. This event will provide an introduction to clustering, then focus on the design, deployment and management considerations for your virtualized datacenter or Private Cloud. Topics include hardware, validation, deployment, host clustering, guest clustering, virtualization, live migration, multi-site clustering, System Center Virtual Machine Manager 2008 & 2012, and offers a wealth of best practices. No prior clustering knowledge is needed.
Agenda:
· Introduction to Clustering – Learn the basics of Windows Server 2008 R2 Failover Clustering
· Hyper-V High-Availability – Dive into best practices, tips and tricks to provide VM high-availability and mobility
· Private Cloud High-Availability – Enhance the datacenter with integration from SCVMM (2008 R2 and 2012) and multi-site clustering
Register here: http://mctreadiness.com/MicrosoftCareerConferenceRegistration.aspx?pid=287
The Great Big Hyper-V Survey of 2011
The Hyper-V MVP’s Aida Finn, Damian Flynn (another Hyper-V MVP), and Hans Vredevoort (Failover Clustering MVP), have joined forces to bring you the …
GREAT BIG HYPER-V SURVEY of 2011
The goals are:
- We learn a bit more about what everyone is up to
- We can share the findings with everyone so you can learn what everyone else is up to
This survey will run from this morning until 5th of September. We want to publish the results later that week, which just so happens to be the week before the Build Windows conference. We’ll be publishing the percentages breakdowns, and also trying to figure out trends.
In the survey, we ask about:
- Your Hyper-V project/environment
- Your Hyper-V installation
- Systems management
- What you considering to do in 2012
There is no personal information, no company specific information. Microsoft has zero involvement. They’ll see/read the results the same way you do, on the blogs of myself, Damian, and Hans (Hyper-V.nu).
The whole thing will take just 5 minutes; the more people that contribute, the more we will all learn about what people are up to, and the more we’ll be able to tweak blog posts, speaking, training, writing, etc, to what is really being done. If this goes well, we’ll do another one in 2012, 2013, and so on.
So come on …. give the greater community 5 minutes of your time.
Share it : http://kwiksurveys.com/?u=BigHyperVSurvey2011. let’s make it The Great Big Hyper-V Survey of 2011
Hyper-V : Network Design, Configuration and Prioritization : Guidance
There is a lot of posts regarding Hyper-V and network, but I found a lot people still don’t get it.
1. Network Design. How many nic’s we need for production environment for High Availiability:
- 1 for Management. Microsoft recommends a dedicated network adapter for Hyper-V server management.
- 2 ( Teamed ) for Virtual machines.Virtual network configurations of the external type require a minimum of one network adapter.
- 2 ( MPIO ) for SCSI.Microsoft recommends that IP storage communication have a dedicated network, so one adapter is required and two or more are necessary to support multipathing.
- 1 for Failover cluster.Windows® failover cluster requires a private network.
- 1 for Live migration.This new Hyper-V R2 feature supports the migration of running virtual machines between Hyper-V servers. Microsoft recommends configuring a dedicated physical network adapter for live migration traffic. This network should be separate from the network for private communication between the cluster nodes, from the network for the virtual machine, and from the network for storage
- 1 for CSV. Microsoft recommends a dedicated network to support the communications traffic created by this new Hyper-V R2 feature. In the network adapter properties, Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks must be enabled to support SMB
But how about production environments when the blades have only 4 Physical NIC’s?
Option 1. If your vendor does support NPAR technology(Broadcom, QLogic), you will be able to create up to 4 “Virtual Logical NIC’s” per physical NIC ( VLAN/QoS ). Although this solution is not supported by MS, it’s the best solution in terms of performance and it is supported by the vendors. This solution will provide you 100% HA as you can have up to 16 Logical NIC’s.
Option 2. Supported by MS. Allocate 2(two) NIC’sfor the iSCSI using MPIO and then :
| Host configuration | Virtual machine access | Management | Cluster and Cluster Shared Volumes | Live migration | Comments |
| 2 network adapters with 10 Gbps | Virtual network adapter 1 | Virtual network adapter 1 with bandwidth capped at 1% | Network adapter 2 | Network adapter 2 with bandwidth capped at 50% | Supported |
Note that the QoS configuration is based on “per port” and Windows only allows you to cap specify caps – not reserves. This solution, although supported by MS, dos not give you 100% HA.
2. Network Configuration. What need to be enabled/disabled?
| Usage | Number of Network Cards | Comments |
| Management Network(Parent Partition) | 1 Network Card |
|
| Storage ISCSI | 2 Network Cards – Not Teamed |
|
| VM Network (Parent Partition) |
2 Network cards : 1 for Dynamic IP’s 1 for Reserved IP’s |
|
| Cluster Heartbeat | 1 Network Card |
|
| Cluster Shared Volume (CSV) | 1 Network Card |
|
| Live Migration | 1 Network Card |
|
2. Network Prioritization. What need to be enabled/disabled?
By default, all internal cluster network have a metric value starting at 1000 and incrementing by 100. The first internal network which the cluster sees when it first comes online has a metric of 1000, the second has a metric of 1100, etc.
When you create CSV’s, the failover cluster automatically chooses the network that appears to be the best for CSV communication. The lowest Metric value designates the network for Cluster and CSV traffic. The second lowest value designates the network for live migration. Additional networks with a metric below 10000 will be used as backup networks if the “Cluster & CSV Traffic” or “Live Migration Traffic” networks fail. The lowest network with a value of at least 10000 will be used for “Public Traffic”. Consider giving the highest possible values to the networks which you do not want any cluster or public traffic to go through, such as for “ISCSI Traffic”, so that they are never used, or only used when no other networks at all are available.
To view the networks, their metric values, and if they were automatically or manually configured, run the clustering PowerShell cmdlet:
PS > Get-ClusterNetwork | ft Name, Metric, AutoMetric
To change the value of a network metric, run:
PS >Get-ClusterNetwork “Live Migration” ).Metric =800
If you want the cluster to start automatically assigning the Metric setting again for the network named “Live Migration”:
PS > Get-ClusterNetwork “Live Migration” ).AutoMetric = $true
How to override Network Prioritization Behavior?
Option 1. Change the network’s properties. If you select “Do not allow cluster network communication on this network”, then it will not be possible to send any “Cluster & CSV Traffic” or “Live Migration Traffic” through this network, even if the network has the lowest metric values. The cluster will honor this override and find the network with the next lowest value to send this type of traffic :
- In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.
- Select Properties
- Change the radio buttons or checkboxes.
Option 2 (exclusively for “Live Migration Traffic”) :
To configure a cluster network for live migration:
- In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.
- Expand Services and applications.
- In the console tree (on the left), select the clustered virtual machine for which you want to configure the network for live migration.
- Right-click the virtual machine resource displayed in the center pane (not on the left), and then click Properties.
- Click the Network for live migration tab, and select one or more cluster networks to use for live migration. Use the buttons on the right to move the cluster networks up or down to ensure that a private cluster network is the most preferred. The default preference order is as follows: networks that have no default gateway should be located first; networks that are used by cluster shared volumes and cluster traffic should be located last.Live migration will be attempted in the order of the networks specified in the list of cluster networks. If the connection to the destination node using the first network is not successful, the next network in the list is used until the complete list is exhausted, or there is a successful connection to the destination node using one of the networks.
Note : You don’t need to perform this action as per VM basis. When you configure a network for live migration for a specific virtual machine, the setting is global and therefore applies to all virtual machines.
Some other interesting articles:
http://technet.microsoft.com/en-us/library/dd446679(WS.10).aspx
http://blogs.technet.com/b/vishwa/archive/2011/02/01/tuning-scvmm-for-vdi-deployments.aspx
http://blogs.msdn.com/b/clustering/archive/2011/06/17/10176338.aspx
Recommended Books
System Center 2012 R2 Virtual Machine
System Center Virtual Machine Manager 2012 Cookbook
Cloudtidings.com 