Problem accessing CSV from passive Cluster Node

I recently had a perplexing problem where Cluster Shared Volumes in a Hyper-V cluster were not working correctly.  The volumes were only accessible from the node currently owning the volume.  Attempts to access the volumes from any of the other nodes resulted Windows Explorer hanging indefinitely.  Enabling maintenance or redirected mode made no difference.

Event ID 5120 was logged:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer available on this node because of 'STATUS_BAD_NETWORK_PATH(c00000be)'. All I/O will temporarily be queued until a path to the volume is re-established.
Event ID 5142 also occurred:  Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.

I could ping all nodes over both the Production and Heartbeat network links, and I could access file shares from any node on any node.

The Problem
After much troubleshooting I realised I disabled both File and Print Sharing and Client for Microsoft Networks on the Heartbeat NIC on all nodes.  This is a best practice drummed into me since working on Microsoft Clustering when it was still code-named Wolfpack.

The Resolution
I enabled File and Print Sharing and Client for Microsoft Networks and immediately afterwards all my Cluster Shared Volumes started functioning as expected.

The Explanation
It’s documented in MS KB Article 2008795.  When accessing a CSV volume from a passive (non-coordinator) node, the disk I/O to the owning (coordinator) node is routed through a 'preferred' network adapter and requires SMB be enabled on that network adapter. For SMB connections to work on these network adapters, the aforementioned protocols must be enabled.  Ugh.

Setting up a KMS server on Server 2008 R2

Today we’ll deal with setting up a Microsoft Key Management Server (KMS).  a KMS is used to activate Microsoft Volume Licensed products such as Windows 7, Office 2010 and Windows server 2008 R2, amongst others.

a KMS server activates a client for a period of 180 days.  The activated machine will communicate with the KMS every 7 days to renew it’s activation information.  It then resets the license counter back to 180 days if successful.  If not it attempts to background connect to the KMS every 2 hours.

If, after 180 days, the machine has not been able to contact the KMS it will go into the 30 day grace period and notify the user.  After that the machine will enter a reduced functionality mode until it can again connect to a KMS.

That was quite a mouthful – so let’s get down to setting up a KMS on a Windows 2008 R2 host.  In addition we’ll also set it up so that it can activate Office 2010 clients.

Setting up a KMS

1. Activate Windows with a KMS key.  This will automatically configure the server as a KMS
2. Download the Office 2010 KMS Host License Pack
3. Enter your KMS host key when prompted (you will get this key from your Microsoft Volume Licensing website)
4. Make sure to allow the Key Management Service through the Windows Firewall

Verify that KMS is published in DNS

nslookup -type=srv _vlmcs._tcp.<your DNS domain>

Checking the KMS status on your KMS

From an elevated command prompt, type SLMGR.vbs /dlv

Checking the license and activation status on a client

slmgr.vbs –dli

a KMS goes a significant way to easing administrative burden, so go ahead and set it up, it’s as easy-peasy!

Handy naviseccli Commands

I have been meaning to document this for ages.  I often find myself supporting clients who are located on the other side of a horribly slow WAN / VPN / 2 Cans and a piece of string link.  Slow as in even Navisphere Express times out in the web browser.  That’s when a ninja-admin such as myself whips out his command-line fu.  All commands below are to be entered on a single line and substitute %username% and %password%

Physical Container-Front End Ports Speeds

naviseccli –h <ip address> port –list -sfpstate
naviseccli –h <ip address> –set sp a –portid 0 2
naviseccli –h <ip address> backendbus –get –speeds 0

SP cache details
naviseccli -scope 0 -user %username% –password %password% -address <ip address> getcache

Get all the details of the LUN’s on the array
naviseccli -scope 0 -user %username% –password %password% -address <ip address> getlun

Review IO Ports on an array
naviseccli -h <ip address> -user %username% –password %password% -scope 0 ioportconfig -list |more

All details from the Array
naviseccli -scope 0 -user %username% -password %password%-address <ip address> getall

SP Reboot and Shutdown GUI
naviseccli –h <ip address> rebootsp
naviseccli –h <ip address> resetandhold

Apart from being faster than the GUI, knowing the naviseccli commands also allows you to incorporate them in scripts, pipe the output etc.  In other words it’s a very nice string to have in your bow!

Hyper-V 3.0 is full of win!

There has been a lot of movement in the virtualization space recently, what with the release of vSphere 5 and Microsoft giving us a sneak peek at the upcoming Hyper-V 3.0.  All in all it seems that Hyper-V is a very rapidly maturing product, and Microsoft is adding the features and scalability so craved by enterprises.  Indeed, it also appears that they are pulling ahead of vSphere in certain areas.

The table below highlights some key performance maximums

	vSphere 5	Hyper-V 3
Max Nodes per Cluster	32	63
Max VMs per Cluster	3000	4000
Max CPUs per VM	32	32
Max RAM per VM	1 TB	512 GB
Max VM Disk Size	2 TB	12 TB
Max Processor Cores per Host	160	160
Max RAM per Host	2 TB	2 TB

In addition to that Hyper-V 3.0 will also bring the following to the table:

• Live storage migration
• This allows you to move you VHDs to another volume whilst the VM is online.  The volume need not reside on shared storage
• Hyper-V Replica
• This allow replication via LAN, for incredibly easy and cost-efficient DR
• Native NIC teaming support in Windows Server 8
• Storage De-Duplication
• Offloaded Date Transfer (ODX), which basically offloads the grunt storage work to an ODX-enabled SAN
• Virtualization-Aware Domain Controllers
• You now can now make and revert to snapshots of a virtualized domain controller
• Domain controllers running as a VM can also be cloned

This is some of the more important features Hyper-V 3.0 has to offer.  It is quickly turning into a very viable and cost-effective alternative to vSphere.

Start-up Sequence for VM’s in a Hyper-V Environment

I’ve had customers phone me several times with regards to powering up their servers after an outage.  If not done properly you can run into a situation whereby you cannot log onto a host to start a VM, because the virtualised DC is still down.  Not good.

So, if me or my clients are recovering from a shutdown I usually do the following:

1. Power on the Hyper-V server hosting the VM that holds the PDC emulator FSMO role.  The VM should preferably also be a GC and DNS server and refer to itself for either primary or secondary DNS
2. Sign on to the Hyper-V host using a Local Administrative Account.
3. Start the VM.
4. Log out of the Hyper-V host once the VM has booted successfully.
5. Log into the Hyper-V host using a “proper” Domain Account.
6. Start any other virtualised DC’s / GC’s you might have.
7. Ensure you have at least one DC / GC running in each AD site.
8. Boot up the rest of your environment

Another option (pointed out to me in the comments section), is to set startup delays on your VM’s so that your DC’s starts up before your other servers.  This seems very basic, but it’s amazing how quickly common sense goes out of the window when the pressure is on to get an environment up and running. 

Another way of dealing with this (my preferred way) is to have the PDC FSMO role on a physical machine.  If you go this route also ensure that your physical machine is a GC.

Publishing Remote Desktop Gateway (RDG) with TMG 2010

I recently had the pleasure of creating an Remote Desktop Services (RDS), Remote Desktop Gateway (RDG) and RemoteApp environment for a client.  This was a bit more technical and involved than I originally envisioned, no thanks to the scant documentation that exists.  I will detail all that in a later blog post, for now I will focus on publishing your RDG, RDS and RA environment through a Microsoft TMG 2010 Firewall.

First we have to create an SSL Listener

1. Specify an IP address for the Listener
2. Enable both HTTP and SSL connections
3. For HTTP to HTTPS redirection select redirect all traffic from HTTP to HTTPS
4. On the Certificates tab select “Use a single certificate for this web listener” and select an appropriate certificate
5. Authentication should be set to “No Authentication”

Now we create the actual publishing rule:

1. Allow
2. From Anywhere
3. To – Your RDSG IP or Host – Forward original host header – request appears to come from TMG
4. Traffic HTTPS
5. Listener – Select the one we created earlier
6. Public name – This is the Public DNS name
7. Paths should be /rdsweb/* and /rpc/*
8. Authentication delegation – “No Delegation, client may authenticate directly”

These were the steps I had to take to successfully and securely publish the client's RDSG to the internet.  Once again I found the existing documentation to be lacking in the extreme.  Hope this helps someone out there.

How to Prepare an Offsite replica with DPM 2010

Sometimes the need will arise to backup your DPM replicas to removable storage for whatever reason.  It might be so that you can recover your DPM server in case of a disaster or you might even want to use the replicas to seed another DPM server in a DPM 2 DPM 4 DR scenario.  Here is an extremely simple and effective way to accomplish that:

1. On your DPM Server open an Admin Command Prompt
2. Navigate to the DPM bin folder (usually C:\
   Program Files\Microsoft DPM\DPM\bin\)
3. Execute dpmbackup -db
4. Execute dpmbackup -replicas
5. I prefer using robocopy to copy the data to USB (or any alternate) Storage like so: robocopy C:\Program Files\Microsoft DPM\DPM\Volumes\ShadowCopy\ %destination% /e /b

What the above does is creates snapshots of the replica volumes and then mounts those read-only snapshots under the \Program Files\Microsoft DPM\DPM\Volumes\ShadowCopy folder.  It is therefore a point in time replica which you can copy wherever you wish, be it disk or tape.  You can, of course, also use your favorite backup software to backup your replicas, just be sure to configure it to traverse mount points.