Last two months I was engaged on decent project for Microsoft important account in Netherlands. This worldwide consulting company was deploying 4000 Windows Mobile devices to it’s employees. They decided to go with System Center Mobile Device Manager 2008, the new Microsoft product for treating mobile devices as corporate computers. You might be interested why software development was involved in “mostly” infrastructure project. Well, you would be surprised but infrastructure part was quite piece of cake. The real challenges were waiting for developer (me). In this blog post I would like to share some experience from such a large deployment of Windows Mobile and System Center Mobile Device Manager 2008. Please not that for some business reasons I can’t mention real name of the customer.

Infrastructure setup

Whole infrastructure was set up by the customer’s internal IT and Microsoft field engineers. In terms of infrastructure there were no special extraordinaries. Gateway server, management server and IIS farm for self service portal. The only thing to consider beside the live environment of respectable consulting company were load balancers. Of course the testing environment was set as well and was available for the whole deployment time. It was handy for testing changes we were doing when system was already “live and serving”.

Sony Ericsson XPERIA X1 400x239 Sony Ericsson XPERIA X1

How to get devices to users?

How would you distribute 4000 mobile devices to non-technical people set all over the country and give them smooth user experience? That was challenging process experience. Before I get to more details I should say that there were “shoot out” from three different device models: HTC Touch Diamond, HTC Touch Pro and Sony Ericsson Xperia X1. Final decision set Xperia X1 as the one and only. It was good for quality of whole deployment process. Same model for everyone makes “learning process” easier and makes life easier for service desk operators.

Devices were distributed by post to employee’s home address. On every Xperia X1 box were sticker telling user to open mymobile web site on the corporate network. It means that every user gets “step by step” guide on his/her computer screen even before opening box with his/her new mobile device. The mymobile web site leads users through the first start of device, clock settings, domain enrollment and post enrollment settings. Even if I grind my teeth every time I was changing content of the site upon latest customer requests - I can say that result was really nice and functional! There were no calls for IT service desk complaining, that someone is lost in the process steps. It was amazing success because as I said earlier, most users are non-techies.

Enrollment portal 1 320x234 Figure 1.

Enrollment portal 2 320x234 Figure 2.

Domain enrollment process

If you are familiar with SCMDM 2008 then you know that every domain enrollment needs enrollment request with one time password (PIN). I think everybody understands that creating 4000 enrollment request manually by the admin is impossible. Especially because enrollment requests has quite short expiration time and every enrollment request contains name of the device which needs to be unique and set by some company ruled named convention. It means that this part of the process was integrated into mymobile website. As user was click through step-by-step guide, his/her enrollment request was created and important “personalized” page with PIN and expiration date was shown as one of the steps. Also reentering mymobile step-by-step guide was covered. If there was a pending request for the user, nothing was created and only information (PIN, expiration) about pending request were shown.

The naming convention for devices was derived from mobile phone number of each user. I can’t say that this is the best practice but it was customers wish. Login-name would serve better but device name can be only 15 characters long. Do you think it’s enough? No, it is not for our customer. Believe it or not they have some users with login-name over 15 characters. Employee number was option two. However, there are some “contractors” employees who have all same employee number. That’s why they choose mobile phone number. Final name was set as two chars country prefix then unique part of the mobile phone number and device index after dash (something like US12345678-01). There were few errors because some users has not mobile number filled in the AD so their enrollment request creation failed. Luckily it was really really few users.

In the future I will probably blog a bit more about possibilities to control SCMDM 2008 from your own application.

Enrollment portal 3 320x234 Figure 3.

Enrollment portal 4 320x234 Figure 4.

Post enrollment configuration

System Center Mobile Device Manager 2008 is a mighty tool but current version has no support for personalized settings. It means you can’t setup ActiveSync with username or deploy user specific certificate. However, we were in need to provide smooth user experience even for device settings that can not be done using SCMDM 208. Here is the list of some settings that was necessary to perform on the device:

  1. Set Ducth locales
  2. Uninstall some apps. that comes with Xperia X1
  3. Set Internet Explorer favorites
  4. Enroll user certificate
  5. Set-up ActiveSync

Some of these can be done from SCMDM but you can’t guarantee the time when it’s happen and we needed all of them “instantly”. That’s why I wrote quite sophisticated tool called Configuration Wizard. It’s fully configurable by XML files (even user interface), so customers can use it on their own without my assistance. With Configuration Wizard it’s possible to perform several tasks, which will be processed on the device. And what is most important - parametrize those tasks! It means that once user enrolled his/her device, he/she pointed pocket internet explorer to specific website and downloaded self-executable .CAB file with Configuration Wizard.

Configuration Wizard looks like standard step-by-step wizard, which collects user information (username, password) and process several steps. Especially enrollment of user certificate and configuring ActiveSync were esential. Immodestly I can say, that I’ve done good job on Configuration Wizard especially because it’s very easy reusable on other projects without touching the source code.

Enrollment portal 3 320x234 Figure 5.

3-2-1 Roll-out!

It was a cold and wet December day when we touched the ground at J.F.K. Schipol (ref: U2). Even if the most work was done “off-site”, on-site presence during roll-out was necessary. W-week was set to be right before Christmas. Why not? I spent nice weekend in Amsterdam. Distribution of devices was done in four waves. First wave was on Monday, including only people from internal IT, service desk and other technical personal (about 80 devices). This was very clever because it gave us certain “sentinel” before second “the large” wave. First wave proceeded fine, same as previous intensive testing with 40 devices.

Second wave took place on Thursday with 1.500 devices (people received them on Wednesday). Everything went right, there were minimum calls on the services desk and most of them were a bit “irrelevant”. There were some failed creations of enrollment requests within 10 AM and 11 AM. After investigation we found out some time-outs in communication with AD. Causes of problem were not fully recognized but most probable was the AD replication that was done at the same time. In the large corporation with subsidiaries all over the world is really hard to orchestrate all major intervention in IT infrastructure. I’ve quickly implemented configurable “maintenance hours” into mymobile web site. Just in case that we will need to take system “out of order” in the future, without Server unreachable error. After first day there were about 500 devices/users enrolled.

On Friday we were expecting another 1.500 people to enroll their devices. Unfortunately, Friday or maybe Thursday-night brought us one a bit more serious problem. There was a user who received enrollment request but was not able to enroll his device. Obviously his request was not created in the system. After investigations and testing it turns out that SCMDM 2008 has a bit unexpected behavior when certain conditions are met. The core of the problem was that our customer is using commas (,) in their AD common names. Imagine LDAP path CN=Bansky\, Pavel,OU=SCMDM2008 TestUsers,DC=intra,DC=contoso,DC=com - the comma between my last and first name has to be escaped. However, enrollment request creation method don’t deal with this in a best way. Long story short, when several others conditions were met some people did not received enrollment request information from their freshly created request but from users with the same last name and already pending request. I’ve changed application logic in mymobile website a bit and problem was solved once for all. Again, this behavior requires many conditions to met including two people with same name creating request at the same time. This unexpected behavior with repro steps were submitted into SCMDM product group and it will be fixed soon.

Weekend I spent with shopping Christmas presents and hanging out in Amsterdam center. It’s really nice country this Netherlands, everyone (including beggars) is speaking perfect English. It’s really easy to survive in Amsterdam - they all speak English, drinking Heineken and they are proud of their red light district and coffee shops.

Whole weekend, Monday and Tuesday were basically boring days in terms of SCMDM. Everything went really smoothly and we were seeing more and more devices getting in. It was really frustrating that nothing were happening, no errors, no calls. You didn’t knew whether people already smashed their devices against wall or whether they are using it. We were proactively browsing logs and checking everything, but really everything went fine. At this point I can point to very interesting command-let Get-ActivesyncDeviceStatistics in Exchange 2007. Using this you can check on last successful synchronization with mobile device for given user. So it’s very easy to verify whether enrolled users get’s their personal certificates, configured ActiveSync properly and receiving their e-mails. More examples on this command-let is described in this article on Exchangepedia.

I was leaving Amsterdam on Tuesday 23 December evening and there were 3000 of total 4000 devices successfully enrolled. Until now 1st January 2009 nobody contacted me with some critical situation - so, mission accomplished.

Amsterdam Czech Flag 300x400 Amsterdam, feels like home (Czech flag)

Conclusion

It was really nice lesson learned from this large SCMDM deployment and I was happy to play one of the major roles in this project. There are still some work in progress, I’am developing application that should make life with Windows Mobile easier to our customer. I have to say that they choose very good strategy: no restrictions now, tighten bolts later. It gives users better experience with their new device and also warm welcome to their hardware. I have to say that my next deployment will be done in exactly same way :)

If you have more question or you want to consult your future deployment of Windows Mobile devices, feel free to use comments bellow or contact me directly at pavel.bansky [at] microsoft.com