Cluster Shared Volume stays in redirected mode

I recently had a perplexing problem on one of my lab servers, which took a lot of head-scratching to solve.  Fortunately I had some time to burn so I managed to get to the bottom of it.

Symptom

If I moved a disk or a CSV to a specific node in my Hyper-V failover cluster it would put the CSV in redirected mode and log the following to the System log

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Event ID:      5125
Task Category: Cluster Shared Volume
Level:         Warning
Keywords:     
User:          SYSTEM
Description:
Cluster Shared Volume '\\?\Volume{0bf0b229-9b0e-11e1-8a3a-e4115ba98410}\' ('') has identified one or more active filter drivers on this device stack that could interfere with CSV operations. I/O access will be redirected to the storage device over the network through another Cluster node. This may result in degraded performance. Please contact the filter driver vendor to verify interoperability with Cluster Shared Volumes.
Active filter drivers found:
aksdf (Encryption)

Cause

After a fair bit of head-scratching, rolling back actions and research with Sysinternals Process Monitor I pinpointed the problem to NetApp Single Mailbox Restore for Exchange.  During installation it installs the aksdf.sys device driver.  A quick google search showed it to be a driver used for USB dongle licensing.  Weird, since SMBR does not require a dongle.  Anyhow, this device driver conflicts with the CSV and forces it to run in redirected mode

Solution

The solution is simple – navigate to the HKLM\SYSTEM\CurrentControlSet\Services\akdsf registry key and set the Start key to have a value of four (4), as per the below screenshot.

This is not documented anywhere on the NetApp support site, so I will file a bug report.  In mitigation, I cannot see that one will actually run SMBR on one of your production cluster nodes.  Still, it should be trivial for NetApp to patch their installation routine to not install the aksdf.sys device driver.

Problem accessing CSV from passive Cluster Node

I recently had a perplexing problem where Cluster Shared Volumes in a Hyper-V cluster were not working correctly.  The volumes were only accessible from the node currently owning the volume.  Attempts to access the volumes from any of the other nodes resulted Windows Explorer hanging indefinitely.  Enabling maintenance or redirected mode made no difference.

Event ID 5120 was logged:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer available on this node because of 'STATUS_BAD_NETWORK_PATH(c00000be)'. All I/O will temporarily be queued until a path to the volume is re-established.
Event ID 5142 also occurred:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.

I could ping all nodes over both the Production and Heartbeat network links, and I could access file shares from any node on any node.

The Problem
After much troubleshooting I realised I disabled both File and Print Sharing and Client for Microsoft Networks on the Heartbeat NIC on all nodes.  This is a best practice drummed into me since working on Microsoft Clustering when it was still code-named Wolfpack.

The Resolution
I enabled File and Print Sharing and Client for Microsoft Networks and immediately afterwards all my Cluster Shared Volumes started functioning as expected.

The Explanation
It’s documented in MS KB Article 2008795.  When accessing a CSV volume from a passive (non-coordinator) node, the disk I/O to the owning (coordinator) node is routed through a 'preferred' network adapter and requires SMB be enabled on that network adapter. For SMB connections to work on these network adapters, the aforementioned protocols must be enabled.  Ugh.

Creating a Hyper-V Cluster after the fact, or, how to preserve and add existing VMs to a Cluster.

I was faced with an interesting challenge recently. a Client was running two standalone Hyper-V hosts with about 4 VMs each running on a Storage Area Network(SAN).  I installed the SAN previously to provide increased IO performance for their SCADA (Citect, for those taking notes) system. 

This was essentially a very effective proof of concept as far as the client was concerned and they wished to take advantage of the more advanced features offered by Clustered Hyper-V (stuff like live migration etc.).

This posed a challenge, because we needed to convert the LUNs occupied by the VM’s to highly available Cluster Shared Volumes (CSV’s).  In Hyper-V, a VM needs to be hosted on a CSV in order to be made highly-available.  So off I went trying to figure out a non disruptive way to convert all my LUNs to CSV, without losing any data.  This is what I came up with.

1. Shut down your VM(s)
2. Open Disk Management on your Hyper-V host and remove drive letter from LUN hosting the VM
3. Open Failover Cluster Manager (FCM) –> Storage –> Add Disk –> Select Disk from Step 2 –> Click OK
4. Still in FCM - Go to Cluster Shared Volume - Add Storage – select the disk you added in Step 3
5. Open up Hyper-V Manager on the same host (notice the VM status is critical because you removed the drive letter).  Remove the VM
6. Create a new VM, opt to store it under the %systemdrive%\ClusterStorage folder which was created automatically when you performed Step 4.  VERY IMPORTANT – Do not add any disks to the VM!
7. Right click the VM you created in Step 6 and choose Edit Settings.  Add the original VM’s disks (boot drive to be added to IDE controller 0).  The existing VHD’s will be found in %systemdrive%\ClusterStorage
8. Open FCM – Go to Services and Applications – in the Action pane select Configure a Service or Application –> select Virtual Machine –> Check the VM created in Step 6 –> Complete Wizard
9. Ensure that the VM is connected to the correct network in Hyper-V Manager
10. Because we are connecting a new NIC to the VM you will have to re-specify the IP address inside the VM once the VM has started up

Rinse and repeat for all existing VMs you want to make highly available.  The Microsoft way would be to export the VM’s and import it again.  Nothing wrong with that, apart from the fact that it takes a lot of time and storage to do, depending on the size of the VM.  My way is quick and easy and it works!