Partial/No Redundancy on iSCSI Datastores

Expensive fiber SANs are not price-compatible with a lot of my clients, therefore a lot of my time is spent in iSCSI environments.  I’ve noticed in all instances that the Multipathing Status for all my iSCSI datastores are Partial/No Redundancy when viewed on the Storage Views tabs in vCenter.  This bothers me because I always go to great lengths to ensure that I set up my iSCSI multipathing correctly.

I therefore breathed a big sigh of relief when I discovered that this behaviour is a bug as confirmed by VMware Technical Support. The rule for displaying the “Multipathing Status” is as follows:

Full Redundancy – If you have 2 separate adapters and 2 separate paths to the datastore
Partial/No Redundancy – If there is one path which is Up
Unknown – If there is at least one path with an “Unknown” status
All Paths Down – No way to reach the datastore

You will always only have one adapter when using a software iSCSI Initiator, this implies a single point of failure which gives us the dreaded “Partial/No Redundancy” status.  So as things stand now software iSCSI will always be displayed with a degraded status.  Methinks VMWare should develop separate rules / algorithms for fiber and iSCSI SANs…

HP Proliant NIC Teaming with Windows Server 2008 R2 Hyper-V

Installing a new HP Proliant Server typically involves booting the server from the HP SmartStart DVD and using that to do a guided OS installation.  This, quite handily, installs the HP Proliant Support Pack (PSP) which takes care of all drivers for you.
Turns out that if you follow this process and then install the Hyper-V role later on there is a possibility of running into problems.
In a nutshell, you need to install the OS, Hyper-V role and all the latest MS updates and Hotfixes.  Only then should you install the HP NIC teaming software
This HP Whitepaper (Using HP Proliant Network Teaming Software with Microsoft Windows Server 2008 Hyper-V or with Microsoft Windows Server 2008 R2 Hyper-V) explains in detail what the proper procedure is, and some of the issues you might run into.  As an extra it also delves into setting up VLAN’s.

Integrating Websense with ISA 2006

I recently had the dubious “pleasure” of having to install and integrate Websense with an ageing ISA 2006 Server.  After lots of trials and tribulations (then it works, then it doesn’t, or it appears to work but doesn’t filter etc. etc.)  I finally had to throw in the towel and call Websense support.  Here is what I had to do to get it working:

1. Install Websense (with Filtering module) on the ISA server, making sure to select "the “ISA Integration” option during installation
2. Installed the Websense ISAPI filter as per this Websense guide
3. Stop and disabled the Filtering Service
4. Edited the wsMSP.ini file (located in %windir%\system32) on the ISA 2006 server and change the EIMServerIP value to match the IP of the Websense server (in this case my ISA server)
5. Created a “ignore.txt” file as per the Configuring ISAPI Filter section in the Websense Installation Guide for Microsoft ISA Server
6. Websense support then had me disable ALL authentication methods in ISA Server.  I did this via Configuration – Networks – Internal – Properties – Web Proxy – Authentication
7. They then had me set up the following Access Rule in ISA Server: "All outbound traffic" TO AND FROM "Local Host" and "All Protected Networks"

Only after all this did it work as expected.  I have not had the opportunity to test this with ISA’s successor, Microsoft’s Threat Management Gateway, but I would imagine that the procedure should remain fairly similar.

Backing up Hyper-V VMs using DPM with software VSS Providers

DPM is a terrific backup solution for the all-round Microsoft shop.  In my humble opinion it’s also one of the best ways to back up Hyper-V Virtual Machines.  It integrates beautifully with your SAN vendors hardware VSS providers, but if you don’t have the budget to splash out on the requisite SAN licenses, there is a way to do it in software.

The Challenge

When backing up VM’s residing on a CSV, you might run into the following DPM error message:

Failed to prepare a Cluster Shared Volume (CSV) for backup as another backup using the same CSV is in progress. (ID 32612 Details: Unknown error (0x8007173d) (0x8007173D))

The Solution

1. Run the DSConfig.ps1 script (included below) on any node in your CSV containing cluster
2. This will create a file called DataSourceGroups.xml in the same folder that you executed the DSConfig.ps1 script
3. Copy DataSourceGroups.xml to the %programfiles%\Microsoft DPM\DPM\Config folder on your DPM Server
4. This file needs to be updated every time you create and protect a VM using DPM
5. If a protection group has already been created for the virtual machines, execute the Modify Protection Group Wizard. If a protection group has not been created, create a new protection group and the job serialization described above will take effect.

DSConfig.ps1 Script

The updated script can be downloaded from here

This will allow you to get rid of the pesky alerts and streamline your backup process.  The optimum solution would be to use a hardware VSS provider, but this is not always financially feasible, or if you have an iSCSI SAN.

Securing HP Procurve Switches (or how to placate Auditors)

I was recently approached by an organization who suffered through an IT Systems and Security Audit.  Needless to say the audited highlighted a couple of holes in their system, and I was tasked with implementing the audit recommendations.  Part of my scope of work included securing their HP Procurve switches, this post will outline a couple of best-practices when securing your network equipment.  It is by no means exhaustive, but it should provide you with a good baseline.  All instructions assume you're logged into the switch and at the (config)# prompt

Defaults Configurations and Secure Alternatives

Telnet vs SSH
Telnet is an antiquated and insecure protocol; it sends all traffic across the wire in clear text. This includes user
names and passwords. Any attacker may eavesdrop on a Telnet session and obtain credentials.  Below are the commands detailing how to disable Telnet and and implement SSH.

1. crypto key generate ssh
2. ip ssh
3. no telnet-server

HTTP vs HTTPS
ProCurve switches and routers can, by default, be configured through the web interface. The HTTP
interface has the same limitations as Telnet. It is therefore recommended that the HTTPS interface be configured and the HTTP interface be disabled, like so:

1. crypto key generate cert 1024
2. web-management ssl
3. no web-management plaintext

TFTP vs SFTP & SCP
TFTP client and server should be disabled as they do not require any authentication. Secure File
Transfer Protocol (SFTP) and Secure Copy Protocol (SCP) are part of the SSH protocol suite. As such provide an encrypted session using public/private keys between client and server just like
SSH.  You configure it like so:

1. crypto key generate ssh
2. ip ssh filetransfer 
3. no tftp server
4. no tftp client

SNMPv1/2c vs. SNMPv3
SNMP version 2 is enabled by default.  SNMPv2 community names are sent across the network as clear text (similiar to Telnet).  SNMP version 3 overcomes these weaknesses via encryption.  Here's how to configure it:

1. snmpv3 enable
2. snmpv3 only

If, for whatever reason, you cannot run SNMP v3, it is possible to run SNMP v2 in restricted mode, i.e. provide Read-Only access to your switch:

1. snmp-server community restricted 

It is also possible to completely disable SNMP altogether:

1. no snmp-server enable

Authorised IP Managers
It’s possible to restrict IP access to the switch to certain management IP's only.  This is configured via the following command:

1. ip authorized-manager <IP address> mask  

Whilst not the be-all and end-all of security, the above should go a long way to ensure a reasonably secure base line for your HP Procurve network equipment.

Exchange 2010 ActiveSync Issue (or the dreaded HTTP Error 500)

Exchange ActiveSync (EAS for us smart people) is a pretty rocking way to get Smart phones hooked up to your Exchange Server.  9 times out of ten it "just works" if you follow the docs, but sometimes it doesn't.  Otherwise I wouldn't writing about it now, would I?

I had a rather perplexing case where EAS only worked for some users.  On the non-working users the only error message displayed was the rather unhelpful HTTP Error 500.  The only suggestion the usually excellent www.testexchangeconnectivity.com could offer was an old article applicable to Exchange 2003.

Diagnosing

The next step was to scour the Application log on the Exchange Server, and lo and behold, the following error was logged:

Source: MSExchange ActiveSync
Event ID: 1053
Level: Error


Exchange ActiveSync doesn't have sufficient permissions to create the "CN=User Name,OU=Admins,OU=Employees,OU=People,DC=domain,DC=local" container under Active Directory user "Active Directory operation failed on server01.domain.local. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
".
Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type "msExchangeActiveSyncDevices" and doesn't have any deny permissions that block such operations.

Righto.  So it would appear that to actually use EAS, information has to be written to the user account and to do this the Exchange Servers group needs to have permissions to the user object.  Further investigation revealed that the only accounts giving the error was Admin accounts.

AdminSDHolder and SDPROP are the culprits

Because the failing accounts has domain admins rights their User Object permissions were being reset by
AdminSDHolder, via a process called SDPROP.  I quote "Each Active Directory domain has an object called AdminSDHolder, which resides in the System container of the domain. The AdminSDHolder object has a unique Access Control List (ACL), which is used to control the permissions of security principals that are members of built-in privileged Active Directory groups (what I like to call "protected" groups). Every hour, a background process runs on the domain controller that holds the PDC Emulator operations master role. It compares the ACL on all security principals (users, groups and computer accounts) that belong to protected groups against the ACL on the AdminSDHolder object. If the size or the binary string is different, the security descriptor on the object is overwritten by the security descriptor from the AdminSDHolder object.."

Resolution

Because SDPROP runs every hour, the fix needs to be applied and a EAS sync done before an hour has gone past.  Here is how:

1. Launch the EMC from your Exchange CAS Server
2. Browse to Server Configuration and select your server
3. Go to the System Settings tab to see which Domain Controller is in use
4. Launch ADUC from the DC located in step 3 and make sure Advanced View is enabled
5. Right-click the user account and go to Properties
6. Click the Security tab, then click Advanced
7. Check the Include inheritable permission.... checkbox
8. OK out of all the dialog boxes
9. Sync your device

The sync should be successful, providing not more than an hour has passed between the sync and step 1.  Whilst this has been a very difficult issue to nail down, there is a lesson in there somewhere...DON'T USE PRIVILEGED ACCOUNTS FOR DAY-TO-DAY USE!!!

ESX Hosts Disconnecting After Upgrade to vSphere 4.1

When you upgrade to vSphere 4.1 your hosts might start disconnecting from your vCenter Server with the following error message: A general system error occurred: internal error: vmodl.fault.HostCommunication.  Restarting the management agents does not resolve the error, nor does rebooting the host.  This VMware KB points to name resolution issues, but that is not at fault here.  The issue is vCenter Server cannot manage an ESX 4.1 host.

Workaround / Solution

Currently there are two solutions available:

1. Upgrade your vCenter Server to version 4.1.  (Once you've upgraded you'll have to remove the hosts from your inventory and re-add it - simply reconnecting didn't work in my case)
2. Downgrade your ESX hosts to version 4

Strangely enough I could not find this documented anywhere on the VMware Knowledge Base, even though it seems to be a pretty widely reported problem.