Exchange 2010 ActiveSync Issue (or the dreaded HTTP Error 500)

Exchange ActiveSync (EAS for us smart people) is a pretty rocking way to get Smart phones hooked up to your Exchange Server.  9 times out of ten it "just works" if you follow the docs, but sometimes it doesn't.  Otherwise I wouldn't writing about it now, would I?

I had a rather perplexing case where EAS only worked for some users.  On the non-working users the only error message displayed was the rather unhelpful HTTP Error 500.  The only suggestion the usually excellent www.testexchangeconnectivity.com could offer was an old article applicable to Exchange 2003.

Diagnosing

The next step was to scour the Application log on the Exchange Server, and lo and behold, the following error was logged:

Source: MSExchange ActiveSync
Event ID: 1053
Level: Error


Exchange ActiveSync doesn't have sufficient permissions to create the "CN=User Name,OU=Admins,OU=Employees,OU=People,DC=domain,DC=local" container under Active Directory user "Active Directory operation failed on server01.domain.local. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
".
Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type "msExchangeActiveSyncDevices" and doesn't have any deny permissions that block such operations.

Righto.  So it would appear that to actually use EAS, information has to be written to the user account and to do this the Exchange Servers group needs to have permissions to the user object.  Further investigation revealed that the only accounts giving the error was Admin accounts.

AdminSDHolder and SDPROP are the culprits

Because the failing accounts has domain admins rights their User Object permissions were being reset by
AdminSDHolder, via a process called SDPROP.  I quote "Each Active Directory domain has an object called AdminSDHolder, which resides in the System container of the domain. The AdminSDHolder object has a unique Access Control List (ACL), which is used to control the permissions of security principals that are members of built-in privileged Active Directory groups (what I like to call "protected" groups). Every hour, a background process runs on the domain controller that holds the PDC Emulator operations master role. It compares the ACL on all security principals (users, groups and computer accounts) that belong to protected groups against the ACL on the AdminSDHolder object. If the size or the binary string is different, the security descriptor on the object is overwritten by the security descriptor from the AdminSDHolder object.."

Resolution

Because SDPROP runs every hour, the fix needs to be applied and a EAS sync done before an hour has gone past.  Here is how:

1. Launch the EMC from your Exchange CAS Server
2. Browse to Server Configuration and select your server
3. Go to the System Settings tab to see which Domain Controller is in use
4. Launch ADUC from the DC located in step 3 and make sure Advanced View is enabled
5. Right-click the user account and go to Properties
6. Click the Security tab, then click Advanced
7. Check the Include inheritable permission.... checkbox
8. OK out of all the dialog boxes
9. Sync your device

The sync should be successful, providing not more than an hour has passed between the sync and step 1.  Whilst this has been a very difficult issue to nail down, there is a lesson in there somewhere...DON'T USE PRIVILEGED ACCOUNTS FOR DAY-TO-DAY USE!!!

RPC Endpoint 6001, and why I couldn't ping it

Look, I love Exchange 2010. As a matter of fact I'm pretty sure Bono is busy installing it somewhere in a attempt to cure World Hunger. It's like caviar, dipped in celery salt and served from Uma Thurman's belly button...but geez Louise, it can be a frustrating beast at time as well, sometimes through no fault of it's own, as we'll see below.

There my Exchange 2010 was, humming along nicely, I already moved the mailboxes across, configured Autodiscover, Outlook Anywhere, ActiveSync, all those highly visible, nice to haves. I created a test account, tested all the remote services, and it worked beeeautifully! So with a flourish I sent a mail to the client, giving him the go-ahead to roll out the remote / mobile services to his users. Only to hear back from them 10 minutes later that bugger-all remote services works. Godammit. No OA, no AS, no OWA, nada, zip. After wiping the egg from my face, I started troubleshooting. First instinct was that I somehow fudged my ISA Publishing rules, but no, it was all good. I then headed off to the bestest Exchange Connectivity Troubleshooter in the history of the world. I ran a test or two and it told me in no uncertain terms that:
Attempting to ping RPC Endpoint 6001 (Exchange Information Store) on server exc.clientabc.local
Failed to ping Endpoint
Tell me more about this issue and how to resolve it
Additional Details
RPC_E_ACCESS_DENIED error (0x5) was thrown by the RPC Runtime

Now the interesting thing here was the fact that it was only doing this for existing users, newly created ones worked without a hitch. I had a antacid (for some reason troubleshooting permission issues on IIS gets to me) and got to work. Suffice to say that I compared IIS permissions, folder NTFS permissions, even user properties in ADSI Edit. I ensured the new users had exactly the same group membership as the existing ones, everything. Then my eye caught something in the security log, All existing users, when logging on via any remote services, threw a EventID 534 in the Security Log; The user has not been granted the requested logon type on this computer.

I was making progress! a Quick review of the existing users account properties revealed that users were only allowed to log onto their workstations. The new test accounts that I created had no such restriction of course. Once I removed these restrictions from the user accounts everything worked flawlessly!

P.S. I tried granulizing (is that even a word?) the logon restrictions by allowing the users to logon to the Exchange 2010 Server. This fixed the OWA, but OA and AS was still not working. The only way I could get OA / AS to work was to remove all logon restrictions.