Enabling Jumbo Frames in vSphere

Hi Kids, todays post is brought to you by the letter J. I previously mentioned a vSphere deployment I had to do, connecting to a EMC AX4-5i iSCSI SAN. Once I got the storage hooked up to my ESX hosts I of course wanted to enable jumbo frames.

Jumbo Frames, for the uninitiated heathens out there, is basically a Ethernet frame with a payload of more than 1500 bytes, up to a typical maximum of 9000 bytes per frame. Why would I want to do this? Performance, in a nutshell. The only requirement would be that your switch supports Jumbo Frames, which I believe most, if not all, mid to high-end kit does. So let's get down to business, shall we?

Switch Configuration

I'm fairly proficient with Cisco and HP networking gear, so I'll give a quick rundown of the commands needed to enable jumbo frames on their kit:

• Cisco: Go into conf t mode and enter the following command "system mtu jumbo 9000". Once that is done issue a "reload" command to reboot your switch
• HP Procurve: On these babies Jumbo frames need to be enabled per VLAN. Execute the following command: "vlan # jumbo"

That takes care of the networking side of things, let's move on to the ESX...

Configure Jumbo frames in vSphere

1. Log on to your ESX host using your favorite SSH client
2. Change your chosen vSwitch MTU with the following command "esxcfg-vswitch -m 9000 vSwitch#" Replace vSwitch# with the name of the vSwitch you want to modify
3. Seeing as one cannot change an existing VMKernel port MTU, you will either have to remove and recreate your existing VMKernel port, or create a new one. To delete an existing VMKernel port, enter the following command: "esxcfg-vmknic -d -p VMKernelport"
4. Now let's add a VMKernel port called "iSCSI01" to our vSwitch, like so: "esxcfg-vswitch –A iSCSI01 vSwitch#"
5. Now we enable jumbo frames on our VMKernel port: "esxcfg-vmknic –a –i 192.168.x.x –n 255.255.255.0 –m 9000 iSCSI01". You do know you need to change the IP address and mask to reflect your environment, right?
   
6. Let's make sure we didn't screw up somewhere by running "esxcfg-vmknic –l". Verify that the MTU is set to 9000
7. All done! You can refer to my previous iSCSI Multipathing post to ensure Jumbo frame enabled iSCSI Multipathing goodness!

I have thought of merging this post with the iSCSI Multipathing post, but seeing as not all environments can/will support end to end Jumbo frames, I have decided to keep them seperate for the time being.

RPC Endpoint 6001, and why I couldn't ping it

Look, I love Exchange 2010. As a matter of fact I'm pretty sure Bono is busy installing it somewhere in a attempt to cure World Hunger. It's like caviar, dipped in celery salt and served from Uma Thurman's belly button...but geez Louise, it can be a frustrating beast at time as well, sometimes through no fault of it's own, as we'll see below.

There my Exchange 2010 was, humming along nicely, I already moved the mailboxes across, configured Autodiscover, Outlook Anywhere, ActiveSync, all those highly visible, nice to haves. I created a test account, tested all the remote services, and it worked beeeautifully! So with a flourish I sent a mail to the client, giving him the go-ahead to roll out the remote / mobile services to his users. Only to hear back from them 10 minutes later that bugger-all remote services works. Godammit. No OA, no AS, no OWA, nada, zip. After wiping the egg from my face, I started troubleshooting. First instinct was that I somehow fudged my ISA Publishing rules, but no, it was all good. I then headed off to the bestest Exchange Connectivity Troubleshooter in the history of the world. I ran a test or two and it told me in no uncertain terms that:
Attempting to ping RPC Endpoint 6001 (Exchange Information Store) on server exc.clientabc.local
Failed to ping Endpoint
Tell me more about this issue and how to resolve it
Additional Details
RPC_E_ACCESS_DENIED error (0x5) was thrown by the RPC Runtime

Now the interesting thing here was the fact that it was only doing this for existing users, newly created ones worked without a hitch. I had a antacid (for some reason troubleshooting permission issues on IIS gets to me) and got to work. Suffice to say that I compared IIS permissions, folder NTFS permissions, even user properties in ADSI Edit. I ensured the new users had exactly the same group membership as the existing ones, everything. Then my eye caught something in the security log, All existing users, when logging on via any remote services, threw a EventID 534 in the Security Log; The user has not been granted the requested logon type on this computer.

I was making progress! a Quick review of the existing users account properties revealed that users were only allowed to log onto their workstations. The new test accounts that I created had no such restriction of course. Once I removed these restrictions from the user accounts everything worked flawlessly!

P.S. I tried granulizing (is that even a word?) the logon restrictions by allowing the users to logon to the Exchange 2010 Server. This fixed the OWA, but OA and AS was still not working. The only way I could get OA / AS to work was to remove all logon restrictions.

Missing Intermediate Certificate in Certificate Chain

As promised, I will outline some of the issues I had with a recent Exchange 2010 migration. After Installing the required UCC and intermediary certificates on both the Exchange and ISA 2006 SP1 servers, following the GoDaddy instructions here, I ran into some issues. Namely ActiveSync refused to work. I headed of the best Exchange Connectivity Troubleshooting site EVAR! This gave me the following detailed info:

Validating certificate trust for Windows Mobile Devices
Certificate trust validation failed
Additional Details
Missing intermediate certificate in Certificate Chain. Subject =
SERIALNUMBER=07969287, CN=Go Daddy Secure Certification Authority,
OU=http://certificates.godaddy.com/repository, O="GoDaddy.com, Inc.",
L=Scottsdale, S=Arizona, C=US, See KB 927465 for more details.

After some serious troubleshooting, googling etc, I still didn't have the issue fixed. As a last resort I rebooted (thought those days were gone) the ISA server. Lo and behold it worked, but that's four hours of my life I'm not getting back anytime soon. What's weird about this is that it's the first time I've had to reboot a ISA server to get a certificate to "take". Oh well...

ESX 4 iSCSI Multipathing

There I was, setting up a small DR site for a client, hardware was a blinged up Proliant DL380, connecting to a sweet, sweet little EMC AX4-5i SAN. This being my first non-FC SAN, I obviously investigated the multipathing options. I was kind of (no, really!) taken aback when I discovered that it's not so straightforward to set up. What's a boy to do? I took a deep breath, bowed in the direction of the (not-so-sweet-anymore) AX4-5i SAN and unleashed the fury of my Google-Fu! Hee-Ya! About 3 hours later the fiendish ESX 4 and a AX4-5i had to yield to the fury of my touch typing - below I chronicle it for all eternity.

Setting up Multipathing
The DL380's I worked on here had an additional 4 port NIC installed, for a total of 8 1Gb ports. The 4 on-board ports I dedicated to the Service console, vmkernel et al, standard stuff. The 4 ports on the add-on NIC I used for iSCSI, perfect considering the AX4-5i also has 4 iSCSI ports. What I did then was set up a new vSwitch, added my 4 Nics to the vSwitch and then created 4 seperate VmKernel ports, so that we have a one-to-one mapping between the VMkernels and the NICs. Here's how to do that:

1. Create a new vSwitch that we're going to dedicate to iSCSI
2. Connect your NICs to this vSwitch by going vSwitch - Properties. Click the Network Adapters tab and click Add. Select your NIC's and click Next and Finish.
3. Now we create VMkernel ports for all our newly added adapters, like so: Go vSwitch - Properties and click the Ports tab and click Add. Select VMkernel, give it a nice label and a suitable IP address and click Finish. Create a VMkernel port for every adapter assigned to your iSCSI vSwitch.
4. Now the important bit, we're going to do the one-to-one VMkernel to NIC mapping: On the ports tab, select one of your new VMkernel port and click Edit.
5. Select the Nic Teaming Tab and check the Override vSwitch failover order
6. Make sure only one adapter is active, move all the other NIC's down to the Unused Adapters section
7. Rinse and repeat for every VMkernel port on your vSwitch

Alas, we're not done yet - now I'm going to introduce you to my little friend esxcli. This fun little guy will enable us to connect our newly created VMkernel ports to the ESX iSCSI initiator

Enable Software iSCSI Multipathing

1. From the CLI (console, ssh, pick your poison) run the following command esxcli swiscsi nic add -n vmk# -d vmhba33
2. Rinse and repeat for all your ports, verify your work by executing the following command: esxcli swiscsi nic list -d vmhba33
3. You can then rescan your iSCSI initiator via the vSphere GUI. You can also verify multpathing via the paths view for the vmhba33 adapter.

Happy Multipathing! It should be mentioned that iSCSI and Jumbo frames go hand-in-hand. I will do a post detailing that can of worms in the near future.

Exchange 2010 SAN / UCC Certificates Installation

So, I recently completed a Exchange 2003 to Exchange 2010 migration for a mid-sized client. There were a (of course) couple of hiccups, which I will detail as soon as I get my stuff in order. With that said, here's what I learned about UCC certificates (more specifically GoDaddy UCC Certs).

THE BASICS:

This client had 2 separate DNS namespaces, the AD DNS was clientabc.local, the external DNS was clientabc.com. Internally the Exchange server was called cabc-exc-001, externally it was mail.clientabc.com. So let's get down to the 3 commandments.

1. Any name by which your server will be accessed needs to be included on the certificate. In my case it was the following: mail.clientabc.com, cabc-exc-001.clientabc.local and finally clientabc.com
2. Make the common name the server's external DNS alias, eg. mail.clientabc.com
3. If you use the Autodiscover server (which you should, it RAWKS) you should add that to your UCC certificate. In my case: autodiscover.clientabc.com and autodiscover.clientabc.local

Generating the Certificate Request:

1. Fire up your EMC and click "Manage Databases" on the homepage
2. Click "Server Configuration", then click on "New Exchange Certificate" in the actions pane
3. You'll be prompted for a "Friendly Name". This is purely descriptive, so call it something descriptive.
4. On the "Domain Scope" dialog, do not select the "wildcard" option
5. Next up is the "Exchange Configuration" menu. Check the boxes for the services you plan to secure. The wizard will recommend names, ensure they're correct for your environment, keeping in mind our 3 commandments
6. On the next screen you'll be allowed to enter your Org info
7. Et viola! Click on the "Browse" button to save all hard work from above into a .req file
8. The contents of the .req file must now be submitted to your Certificate vendor of choice (I used Godaddy).
9. Once you've completed that you should be able to download your certificate. Once that is done it's on the next section.
10. It's of course also possible to do all of the above via the EMS. Using my example the command would be: "New-ExchangeCertificate -GenerateRequest -KeySize 2048 -SubjectName "c=NA, s=Erongo, l=Swakopmund, o=ClientABC, ou=Information Technology, cn=mail.clientabc.com" -DomainName cabc-exc-001.clientabc.local, autodiscover.clientabc.com, autodiscover.clientabc.local, clientabc.com -PrivateKeyExportable $True"
    

Installing Your UCC Certificate

1. Download and save the certificate from your provider
2. Now install any intermediary certificate, following instructions provided by your chosen CA. THIS IS CRUCIAL! Install this before you install your actual certificate.
3. Now start up the EMC again and click "Manage Databases" on the homepage. Click "Server Configuration", then select your certificate.
4. In the Action Pane, click on "Complete Pending Request"
5. Browse to your downloaded certificate, and click Open, Complete and Finish.
6. From the Action Pane, click "Assign Services to Certificate", select your server from the list and click Next
7. Select the necessary services, then click Next, Assign and Finish
8. Alternatively we can import our certificate with a EMS command: Import-ExchangeCertificate -path c:\certreq\mail.clientabc.com.crt -friendlyname "Your Friendly Name"
9. Then assign the services like so: Enable-exchangecertificate –services IIS –thumbprint

To make sure everything takes, you can restart the Exchange Transport Service. And that, as they say, is that!