Updating the firmware (DDOS) on a DataDomain

EMC has done a great job with DDOS 5.2, as such I’ve had to do a couple of upgrades recently.  The upgrade manual is quite a tome, so this is my notes, distilled.  I’ve added a couple of extra steps just to make sure there are no error conditions that can hamper the upgrade or the operation of the DataDomain.

Pre-Upgrade checks

Review the output of the below commands for any errors and warnings.

1. Reboot the DataDomain (I do it just because)
2. ssh into the DataDomain and issue the commands listed below
3. enclosure show powersupply 1
4. alert show current
5. reg show config.net (confirm the following settings)
1. noauto.enabled
2. noauto.speed
3. noauto.full_duplex
6. ifgroup
7. disk show state
8. disk show reliability-data
9. enclosure show all
10. enclosure show topology
11. system show hardware
12. system show ports
13. filesys status
14. log view debug/platform/kern.info

Upgrade Steps

Well after all that the steps to actually upgrade the DDOS is a bit of a let-down.

1. Copy your DDOS .rpm to /ddr/var/releases
1. You can do this via a NFS mount or you can upload via the GUI
2. ssh into the DataDomain and issue the commands listed below
3. system upgrade 5.2.2.4-370754.rpm
4. Monitor progress with “system upgrade status” or “log watch debug/platform/infra.log”

Post-Upgrade Steps

I guess strictly speaking these steps are not necessary, as with the pre-upgrade steps.  I do it just to ensure I have a nice neat base to work from

1. ssh into the DataDomain and issue the commands listed below
2. ost disable
3. ost enable
4. nfs disable
5. nfs enable

And that is all there is to it, mostly.

Configuring Backup Exec 2012 to backup to a DataDomain 670 using the OST plugin

DataDomain Configuration

First things first, let’s verify that we have the DD Boost is correctly configured on the DataDomain

1. Log into your DataDomain Appliance and navigate to System Settings – Licenses and verify the correct licenses, it should look like this:
   
   
2. Navigate to Data Management – DD Boost – Settings and verify that DD Boost is enabled, like so:
   
3. Now click on the + sign to add your BackupExec host.  Enter the hostname and click OK
   
4. Next we need to create a DD Boost Storage Unit.  Navigate to Data Management – DD Boost – Storage Units and click Create.  Enter a descriptive name and configure any quotas, if desired.  Click OK
   
5. Now we need to enable an interface for DD Boost.  Navigate to Data Management – DD Boost – IP Network.  Highlight and edit an interface group and tick the “Enabled” check box in the resultant dialogue box and click OK.
   

That takes care of the configuration on the DataDomain side of things.  Let’s move over to Backup Exec

Backup Exec 2012 Configuration

1. Download and install the latest version of the EMC DataDomain Boost for Symantec OpenStorage (Version 2.5.0.3-314845) at the time of writing.  This file is available from the DataDomain support site (Powerlink Login required)
2. Open up the Backup Exec console, select the Storage tab and click Configure Storage
3. The type of storage is OpenStorage
   
4. Enter a name and description for the DataDomain
   
5. Select the DataDomain provider (the DataDomain only shows up once you completed the OST plugin installation as per Step 1)
   
6. Enter the connection details for the your DataDomain device
   
7. Enter the Storage Location configured on the DataDomain (BackupExec in our case)
   
8. Select the amount of concurrent operations allowed to the DataDomain
   
9. Click Finish on the Summary screen and confirm that you would like to restart the Backup Exec services
   

With that completed you’ll be able to select the DataDomain as a deduplication target (as opposed to B2D device).

Directly Connecting a Brocade 815 HBA to a EMC VNX5300

I’m busy with a project which involves getting two ESXi hosts hooked up to a VNX5300 configured in block mode.  The order we placed with Dell specified Emulex 12000 HBA’s, but Dell got creative and shipped Brocade 815’s instead.  Only problem was that they didn’t work when directly connected to the front-end ports on the VNX.  I’m documenting the symptoms here as well, so that the next person does not have to battle for two days.

The Symptoms

When directly connecting the HBA’s to the VNX fiber ports the following events pop up in the SP event logs

• EV_VirtualArrayFeature::_mergeInternalObjects() - No parent for HBA,
• EV_TargetMapEntry::GetHostInitiatorPort() - NULL HBAPort pointer

Running NaviSECCli.exe -Address 172.20.10.27 port -list –sfpstate outputs the following:

SP Name:             SP A
SP Port ID:          1
SP UID:              50:06:01:60:BE:A0:72:F9:50:06:01:61:3E:A0:72:F9
Link Status:         Up
Port Status:         Online
Switch Present:      NO
SFP State:           Online

This tells us that things are fine on a physical layer, but not much else is happening higher up the stack.

The Fix

First we need to upgrade the HBA firmware to version 3.1.  There are various OS specific ways to do it, easiest is probably to download the livecd from Brocade.  Since this HBA is not on the ESXi 5.1 HCL we need to install the driver.  You need to install at least the v3.1   I include the steps for the sake of completeness

1. Enable SSH on your ESXi host
2. Use scp for Windows or the following command from a linux / max host:  scp brocade_driver_esx50_v3-1-0-0.tar root@<ip address>:/tmp
3. SSH into your ESXi host and navigate to the /tmp folder with cd /tmp
4. Execute tar xf brocadedriveresx50_v3-1-0-0.tar
5. Execute ./brocade_install_esxi.sh
6. Wait for the installation to finish (takes about 1 – 2 mins) and reboot host once done

Now we need to configure the HBA for direct connection, or more technically,  FC-AL mode

1. SSH into your ESXi host and navigate to /opt/brocade/bin/ by entering cd /opt/brocade/bin/
2. ./bcu port --topology 1/0 loop
3. ./bcu port —disable 1/0
4. ./bcu port —enable 1/0
5. ./bcu port --topology 2/0 loop
6. ./bcu port —disable 1/0
7. ./bcu port —enable 1/0

Your ESXi host should now show up as a host on the VNX where you can add it to a storage group and assign LUNs.

Handy naviseccli Commands

I have been meaning to document this for ages.  I often find myself supporting clients who are located on the other side of a horribly slow WAN / VPN / 2 Cans and a piece of string link.  Slow as in even Navisphere Express times out in the web browser.  That’s when a ninja-admin such as myself whips out his command-line fu.  All commands below are to be entered on a single line and substitute %username% and %password%

Physical Container-Front End Ports Speeds

naviseccli –h <ip address> port –list -sfpstate
naviseccli –h <ip address> –set sp a –portid 0 2
naviseccli –h <ip address> backendbus –get –speeds 0

SP cache details
naviseccli -scope 0 -user %username% –password %password% -address <ip address> getcache

Get all the details of the LUN’s on the array
naviseccli -scope 0 -user %username% –password %password% -address <ip address> getlun

Review IO Ports on an array
naviseccli -h <ip address> -user %username% –password %password% -scope 0 ioportconfig -list |more

All details from the Array
naviseccli -scope 0 -user %username% -password %password%-address <ip address> getall

SP Reboot and Shutdown GUI
naviseccli –h <ip address> rebootsp
naviseccli –h <ip address> resetandhold

Apart from being faster than the GUI, knowing the naviseccli commands also allows you to incorporate them in scripts, pipe the output etc.  In other words it’s a very nice string to have in your bow!

ESX 4 iSCSI Multipathing

There I was, setting up a small DR site for a client, hardware was a blinged up Proliant DL380, connecting to a sweet, sweet little EMC AX4-5i SAN. This being my first non-FC SAN, I obviously investigated the multipathing options. I was kind of (no, really!) taken aback when I discovered that it's not so straightforward to set up. What's a boy to do? I took a deep breath, bowed in the direction of the (not-so-sweet-anymore) AX4-5i SAN and unleashed the fury of my Google-Fu! Hee-Ya! About 3 hours later the fiendish ESX 4 and a AX4-5i had to yield to the fury of my touch typing - below I chronicle it for all eternity.

Setting up Multipathing
The DL380's I worked on here had an additional 4 port NIC installed, for a total of 8 1Gb ports. The 4 on-board ports I dedicated to the Service console, vmkernel et al, standard stuff. The 4 ports on the add-on NIC I used for iSCSI, perfect considering the AX4-5i also has 4 iSCSI ports. What I did then was set up a new vSwitch, added my 4 Nics to the vSwitch and then created 4 seperate VmKernel ports, so that we have a one-to-one mapping between the VMkernels and the NICs. Here's how to do that:

1. Create a new vSwitch that we're going to dedicate to iSCSI
2. Connect your NICs to this vSwitch by going vSwitch - Properties. Click the Network Adapters tab and click Add. Select your NIC's and click Next and Finish.
3. Now we create VMkernel ports for all our newly added adapters, like so: Go vSwitch - Properties and click the Ports tab and click Add. Select VMkernel, give it a nice label and a suitable IP address and click Finish. Create a VMkernel port for every adapter assigned to your iSCSI vSwitch.
4. Now the important bit, we're going to do the one-to-one VMkernel to NIC mapping: On the ports tab, select one of your new VMkernel port and click Edit.
5. Select the Nic Teaming Tab and check the Override vSwitch failover order
6. Make sure only one adapter is active, move all the other NIC's down to the Unused Adapters section
7. Rinse and repeat for every VMkernel port on your vSwitch

Alas, we're not done yet - now I'm going to introduce you to my little friend esxcli. This fun little guy will enable us to connect our newly created VMkernel ports to the ESX iSCSI initiator

Enable Software iSCSI Multipathing

1. From the CLI (console, ssh, pick your poison) run the following command esxcli swiscsi nic add -n vmk# -d vmhba33
2. Rinse and repeat for all your ports, verify your work by executing the following command: esxcli swiscsi nic list -d vmhba33
3. You can then rescan your iSCSI initiator via the vSphere GUI. You can also verify multpathing via the paths view for the vmhba33 adapter.

Happy Multipathing! It should be mentioned that iSCSI and Jumbo frames go hand-in-hand. I will do a post detailing that can of worms in the near future.