Rebuilding the Exchange 2010 Offline Address Book (OAB) from scratch

IT peeps supporting Microsoft Exchange can be divided into two groups – those that have experienced problems with the OAB and those that will.  If you’re in the first camp – I feel your pain.  Those of you lucky enough to be in the second camp – file this away, it might come in handy…no I’m not being facetious.
The process below will blow away your OAB and create a new one, so be mindful.  FWIW I’ve never had any issues with this process.  This is especially effective if you’ve screwed up on the public folder replication during Exchange Migrations (don’t ask).
We rebuild the Exchange 2010 OAB like so:

Create a new Offline Address Book object

1. Open the Exchange Management Console (EMC) and navigate to Organisation Configuration – Mailbox
2. Click the Offline Address Book tab.  Right click in the blank area and click New Offline Address Book
3. Give your OAB a different name than the existing one
4. Select your Exchange 2010 MBX server as the OAB generation server
5. Check Include the default Global Address Lists option
6. Check Enable Web-based Distribution as well as the Enable public folder distribution option
7. Finish the wizard

Restart Exchange Services

1. Restart the Microsoft Exchange System Attendant service
2. Restart the Microsoft Exchange File Distribution service

Update and set the OAB as default Offline Address Book

1. Right-click your newly created OAB and click Update.  This can take a couple of minutes, confirm successful completion via your Application log
2. Right-click the OAB from step 1 and and click Set as default

Assign the OAB to the affected users databases

1. Open the Exchange Management Shell (EMS)
2. Execute the following:
   Get-MailboxDatabase |  set-MailboxDatabase  -OfflineAddressBook "%your new OAB%"
3. Wait for Outlook to complete it’s OAB download cycle (can be as much as 24 hours)

You can safely delete the old OAB once you’ve verified that your clients are successfully downloading the newly created one.

NetApp Single Mailbox Restore (SMR) for Exchange for Virtualised Exchange Servers

NetApp Single Mailbox Restore for Exchange 2010 is, well, a snap to use when your Exchange server is running in the “NetApp way”.  What is the NetApp way you ask?  Well, in a nutshell, it is when you have your physical Exchange box hooked up to your SAN via iSCSI or FCP.  If you are virtualised then you’ll need to present your disks via RDM (vSphere) or pass-through if you live in MS land.

What I address here is the case where you have an Exchange server virtualised with Hyper-V, with your hard drives attached as VHD’s.  Even though this example uses Hyper-V, the principles are also applicable to a vSphere environment.

Mounting the NetApp Snapshot

1. Open NetApp SnapDrive on a host connected to your Filer via either FCP or iSCSI
2. Navigate to the Disks node and expand the LUN containing the VHD which in turn contains your Exchange DB’s.
   
3. Under Snapshot Copies, right-click point in time snapshot that you wish to restore and select Connect Disk
   
4. The Connect Disk Wizard will start.  Click Next
   
5. Select the appropriate snapshot and click next.
   
6. Click Next on the the “Important Properties…” screen (Don’t change anything here)
   
7. Set the LUN type as Dedicated and click Next
   
8. Assign a Drive Letter and click Next
   
9. Select your initiators and click Next
   
10. Select Manual on the Initiator Group Management Screen and click Next
    
11. Select the appropriate iGroup and click Next
    
12. Click Finish to complete the SnapDrive Connect Disk Wizard
    

Your NetApp snapshot should now be mounted as a drive accessible through Windows Explorer, If you browse to it it should contain the VHD hosting your Exchange DB.  The next step is to mount the VHD so that it is accessible to SMR.

Mounting the VHD

1. Open Server Manager.  Navigate to Storage – Disk Management.  Right-click Disk Management and click Attach VHD
   
2. Browse to the VHD from the previous section and click OK
   

Your VHD will now be mounted with the next available drive letter and accessible via Windows Explorer.  The next and final step will be to mount our mailbox with SMR and get restoring!

Restoring with SMR

1. Open Single Mailbox Recovery and click File – Open Source
   
2. Browse to your source EDB file, ignoring any warnings about missing log files (Hooray for application-aware snapshots!) and click OK
   
3. SMR will now process your database and allow you to restore a mailbox, folder or item to PST or an Exchange Server.
   

Awesome, but once done we have to clean up after ourselves by dismounting the VHD and disconnecting the temporary NetApp SnapShot LUN.

Cleanup

1. Open Server Manager. Navigate to Storage – Disk Management. Right-click Disk Management and click Detach VHD
   
2. Take care to *not* check the “Delete…” box and click OK
   
3. Open SnapDrive and go to the Disks node.  Right click your temporary SnapShot LUN and click Disconnect Disk.
   

Increasing the amount of concurrent mailbox moves in Exchange 2010

Exchange 2010, by default, allows you to do 2 concurrent mailbox moves.  This is a rather conservative approach, in this age of 10Gb Ethernet and super-fast SAN storage.  Fortunately Microsoft has provided a mechanism whereby we can increase the amount of concurrent moves allowed.

Said mechanism involves editing the MSExchangeMailboReplication.exe.config file, which resides under the C:\Program Files\Microsoft\Exchange Server\V14\Bin\ foler.

These are the changes you need to make if you’d want to allow 20 concurrent moves:

• MaxActiveMovesPerSourceMDB = “20"
• MaxActiveMovesPerTargetMDB = “20"
• MaxActiveMovesPerTargetServer = “20"

Once you’ve made the changes you will also need to restart the Microsoft Exchange Replication service.  Lastly – all the above needs to be done on the target server.

Quick and easy, just the way I like it!

AD and Exchange Forest Migration (Part I)

I’m currently in the middle of a big and relatively complex forest migration.  I’ve found that while there’s a ton of documentation on the subject, a lot of it is way too complex for 90% of engagements and the rest is very spotty.  Thus I’ve set out to document my processes in a simple and to the point way, keeping in mind that this is what works for me, in this specific client’s environment.  Caveat Emptor.

Current Environment

Source:

The source domain is a standalone forest, with a two-way forest trust to the target domain.

Source Domain Name:  olddomain.local
Domain Functional Level: Windows Server 2008 R2 domain level
Mode: Native
Forest Level: Windows Server 2008 R2 domain level
SMTP Address Space: company.com

Target:

The target domain is a child domain contained in a existing forest.

Target Domain Name: newdomain.local
Domain Functional Level: Windows Server 2008 R2 domain level
Mode: Native
Forest Level: Windows Server 2008 R2 domain level
SMTP Address Space: company.com

High-Level Overview

1. Clean up source domain by deleting unused accounts, mailboxes etc.
2. Setting up Name Resolution (DNS) to allow us to create a trust
3. Create a Two-Way Forest Trust between the source and target domains
4. Enable SID History and disable SID Filtering
5. Install the Active Directory Migration Tool (ADMT)
6. Install the ADMT Password Export Server (PES)
7. Use Prepare-MoveRequest.ps1 to create Mail Enabled Users (MEU’s) in the target domain
8. Configure Exchange servers in the source and target domains to operate within a shared address space
9. Use ADMT to migrate user accounts to the target domain
10. Use ADMT to re-ACL resources
11. Use ADMT to migrate computer accounts to the target domain
12. Move mailboxes to the Exchange server in the target domain
13. Decommission source Exchange server
14. Use ADMT to remove old ACL’s from resources
15. Use ADMT to migrate servers to the target domain
16. Decommission old servers, domain and forest

I will use the next series of blog posts to document all the above steps in detail.  As I said, I have been unable to find a single authoritative source for the process, so I aim to make my life easier the next time I’m faced with this challenge.  Hopefully I also save someone else some time and effort.

I want to conclude by saying that even though my documentation might suit your environment to a T, it is imperative that you lab the living daylights out of your processes.  Also, make sure you understand what each step does, and have a rollback procedure in place.

Exchange 2010 Pre-Install Requirements

Lately I have found myself doing quite a few Exchange 2010 installations, and every single time the pre-install requirements trips me up.  Not really a show-stopper but it gets old quickly having to exit out of the installer, do something, start the installer and then have it error out on the very next step.  So without further ado – here is a list of things that needs doing to ensure Exchange 2010 installs smoothly

1. Target server needs to be running either
   • Windows Server 2008 R2 64-bit with SP2
   • Windows Server 2008 64-bit R2
2. Install the Microsoft Filter Pack (only if you’re going to host the Hub Transport or Mailbox Server roles).  The filter pack is available here
3. Run the following from an elevated PowerShell console: Import-Module ServerManager
4. Run the following from an elevated PowerShell console: Add-WindowsFeature NET-Framework,RSAT-ADDS,Web-Server,Web-Basic-Auth,Web-Windows-Auth,Web-Metabase,Web-Net-Ext,Web-Lgcy-Mgmt-Console,WAS-Process-Model,RSAT-Web-Server,Web-ISAPI-Ext,Web-Digest-Auth,Web-Dyn-Compression,NET-HTTP-Activation,RPC-Over-HTTP-Proxy –Restart
5. Run the following from an elevated PowerShell console: Set-Service NetTcpPortSharing -StartupType Automatic
6. Install latest patches and service packs via Windows Update
7. Proceed with the Exchange 2010 installation

If you want to go straight to the source, here is the Microsoft Technet article

Exchange 2010 ActiveSync Issue (or the dreaded HTTP Error 500)

Exchange ActiveSync (EAS for us smart people) is a pretty rocking way to get Smart phones hooked up to your Exchange Server.  9 times out of ten it "just works" if you follow the docs, but sometimes it doesn't.  Otherwise I wouldn't writing about it now, would I?

I had a rather perplexing case where EAS only worked for some users.  On the non-working users the only error message displayed was the rather unhelpful HTTP Error 500.  The only suggestion the usually excellent www.testexchangeconnectivity.com could offer was an old article applicable to Exchange 2003.

Diagnosing

The next step was to scour the Application log on the Exchange Server, and lo and behold, the following error was logged:

Source: MSExchange ActiveSync
Event ID: 1053
Level: Error


Exchange ActiveSync doesn't have sufficient permissions to create the "CN=User Name,OU=Admins,OU=Employees,OU=People,DC=domain,DC=local" container under Active Directory user "Active Directory operation failed on server01.domain.local. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
".
Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type "msExchangeActiveSyncDevices" and doesn't have any deny permissions that block such operations.

Righto.  So it would appear that to actually use EAS, information has to be written to the user account and to do this the Exchange Servers group needs to have permissions to the user object.  Further investigation revealed that the only accounts giving the error was Admin accounts.

AdminSDHolder and SDPROP are the culprits

Because the failing accounts has domain admins rights their User Object permissions were being reset by
AdminSDHolder, via a process called SDPROP.  I quote "Each Active Directory domain has an object called AdminSDHolder, which resides in the System container of the domain. The AdminSDHolder object has a unique Access Control List (ACL), which is used to control the permissions of security principals that are members of built-in privileged Active Directory groups (what I like to call "protected" groups). Every hour, a background process runs on the domain controller that holds the PDC Emulator operations master role. It compares the ACL on all security principals (users, groups and computer accounts) that belong to protected groups against the ACL on the AdminSDHolder object. If the size or the binary string is different, the security descriptor on the object is overwritten by the security descriptor from the AdminSDHolder object.."

Resolution

Because SDPROP runs every hour, the fix needs to be applied and a EAS sync done before an hour has gone past.  Here is how:

1. Launch the EMC from your Exchange CAS Server
2. Browse to Server Configuration and select your server
3. Go to the System Settings tab to see which Domain Controller is in use
4. Launch ADUC from the DC located in step 3 and make sure Advanced View is enabled
5. Right-click the user account and go to Properties
6. Click the Security tab, then click Advanced
7. Check the Include inheritable permission.... checkbox
8. OK out of all the dialog boxes
9. Sync your device

The sync should be successful, providing not more than an hour has passed between the sync and step 1.  Whilst this has been a very difficult issue to nail down, there is a lesson in there somewhere...DON'T USE PRIVILEGED ACCOUNTS FOR DAY-TO-DAY USE!!!

RPC Endpoint 6001, and why I couldn't ping it

Look, I love Exchange 2010. As a matter of fact I'm pretty sure Bono is busy installing it somewhere in a attempt to cure World Hunger. It's like caviar, dipped in celery salt and served from Uma Thurman's belly button...but geez Louise, it can be a frustrating beast at time as well, sometimes through no fault of it's own, as we'll see below.

There my Exchange 2010 was, humming along nicely, I already moved the mailboxes across, configured Autodiscover, Outlook Anywhere, ActiveSync, all those highly visible, nice to haves. I created a test account, tested all the remote services, and it worked beeeautifully! So with a flourish I sent a mail to the client, giving him the go-ahead to roll out the remote / mobile services to his users. Only to hear back from them 10 minutes later that bugger-all remote services works. Godammit. No OA, no AS, no OWA, nada, zip. After wiping the egg from my face, I started troubleshooting. First instinct was that I somehow fudged my ISA Publishing rules, but no, it was all good. I then headed off to the bestest Exchange Connectivity Troubleshooter in the history of the world. I ran a test or two and it told me in no uncertain terms that:
Attempting to ping RPC Endpoint 6001 (Exchange Information Store) on server exc.clientabc.local
Failed to ping Endpoint
Tell me more about this issue and how to resolve it
Additional Details
RPC_E_ACCESS_DENIED error (0x5) was thrown by the RPC Runtime

Now the interesting thing here was the fact that it was only doing this for existing users, newly created ones worked without a hitch. I had a antacid (for some reason troubleshooting permission issues on IIS gets to me) and got to work. Suffice to say that I compared IIS permissions, folder NTFS permissions, even user properties in ADSI Edit. I ensured the new users had exactly the same group membership as the existing ones, everything. Then my eye caught something in the security log, All existing users, when logging on via any remote services, threw a EventID 534 in the Security Log; The user has not been granted the requested logon type on this computer.

I was making progress! a Quick review of the existing users account properties revealed that users were only allowed to log onto their workstations. The new test accounts that I created had no such restriction of course. Once I removed these restrictions from the user accounts everything worked flawlessly!

P.S. I tried granulizing (is that even a word?) the logon restrictions by allowing the users to logon to the Exchange 2010 Server. This fixed the OWA, but OA and AS was still not working. The only way I could get OA / AS to work was to remove all logon restrictions.

Missing Intermediate Certificate in Certificate Chain

As promised, I will outline some of the issues I had with a recent Exchange 2010 migration. After Installing the required UCC and intermediary certificates on both the Exchange and ISA 2006 SP1 servers, following the GoDaddy instructions here, I ran into some issues. Namely ActiveSync refused to work. I headed of the best Exchange Connectivity Troubleshooting site EVAR! This gave me the following detailed info:

Validating certificate trust for Windows Mobile Devices
Certificate trust validation failed
Additional Details
Missing intermediate certificate in Certificate Chain. Subject =
SERIALNUMBER=07969287, CN=Go Daddy Secure Certification Authority,
OU=http://certificates.godaddy.com/repository, O="GoDaddy.com, Inc.",
L=Scottsdale, S=Arizona, C=US, See KB 927465 for more details.

After some serious troubleshooting, googling etc, I still didn't have the issue fixed. As a last resort I rebooted (thought those days were gone) the ISA server. Lo and behold it worked, but that's four hours of my life I'm not getting back anytime soon. What's weird about this is that it's the first time I've had to reboot a ISA server to get a certificate to "take". Oh well...

Exchange 2010 SAN / UCC Certificates Installation

So, I recently completed a Exchange 2003 to Exchange 2010 migration for a mid-sized client. There were a (of course) couple of hiccups, which I will detail as soon as I get my stuff in order. With that said, here's what I learned about UCC certificates (more specifically GoDaddy UCC Certs).

THE BASICS:

This client had 2 separate DNS namespaces, the AD DNS was clientabc.local, the external DNS was clientabc.com. Internally the Exchange server was called cabc-exc-001, externally it was mail.clientabc.com. So let's get down to the 3 commandments.

1. Any name by which your server will be accessed needs to be included on the certificate. In my case it was the following: mail.clientabc.com, cabc-exc-001.clientabc.local and finally clientabc.com
2. Make the common name the server's external DNS alias, eg. mail.clientabc.com
3. If you use the Autodiscover server (which you should, it RAWKS) you should add that to your UCC certificate. In my case: autodiscover.clientabc.com and autodiscover.clientabc.local

Generating the Certificate Request:

1. Fire up your EMC and click "Manage Databases" on the homepage
2. Click "Server Configuration", then click on "New Exchange Certificate" in the actions pane
3. You'll be prompted for a "Friendly Name". This is purely descriptive, so call it something descriptive.
4. On the "Domain Scope" dialog, do not select the "wildcard" option
5. Next up is the "Exchange Configuration" menu. Check the boxes for the services you plan to secure. The wizard will recommend names, ensure they're correct for your environment, keeping in mind our 3 commandments
6. On the next screen you'll be allowed to enter your Org info
7. Et viola! Click on the "Browse" button to save all hard work from above into a .req file
8. The contents of the .req file must now be submitted to your Certificate vendor of choice (I used Godaddy).
9. Once you've completed that you should be able to download your certificate. Once that is done it's on the next section.
10. It's of course also possible to do all of the above via the EMS. Using my example the command would be: "New-ExchangeCertificate -GenerateRequest -KeySize 2048 -SubjectName "c=NA, s=Erongo, l=Swakopmund, o=ClientABC, ou=Information Technology, cn=mail.clientabc.com" -DomainName cabc-exc-001.clientabc.local, autodiscover.clientabc.com, autodiscover.clientabc.local, clientabc.com -PrivateKeyExportable $True"
    

Installing Your UCC Certificate

1. Download and save the certificate from your provider
2. Now install any intermediary certificate, following instructions provided by your chosen CA. THIS IS CRUCIAL! Install this before you install your actual certificate.
3. Now start up the EMC again and click "Manage Databases" on the homepage. Click "Server Configuration", then select your certificate.
4. In the Action Pane, click on "Complete Pending Request"
5. Browse to your downloaded certificate, and click Open, Complete and Finish.
6. From the Action Pane, click "Assign Services to Certificate", select your server from the list and click Next
7. Select the necessary services, then click Next, Assign and Finish
8. Alternatively we can import our certificate with a EMS command: Import-ExchangeCertificate -path c:\certreq\mail.clientabc.com.crt -friendlyname "Your Friendly Name"
9. Then assign the services like so: Enable-exchangecertificate –services IIS –thumbprint

To make sure everything takes, you can restart the Exchange Transport Service. And that, as they say, is that!