Identify the mailboxes to be moved

Once you figure out the syntax for the “-Filter” flag to get-mailbox, this is easy

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*conference*") }

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Conference Room2          ConferenceRoom2      exch2010be1      unlimited
Production Conference ... productionconf       exch2010be1      unlimited
Conference Room 1         conference1          exch2010be1      unlimited
L&D Conference Room       ldconference         exch2010be1      unlimited
Tech Conference Room      techconference       exch2010be1      unlimited
Client Services Confer... csconference         exch2010be1      unlimited
Suite 202 Conference Room 202conf              exch2010be1      unlimited

Convert them to rooms

As Microsoft says in this story about converting mailboxes to rooms, this can only be done via Exchange Management Shell (not EMC), so just pipe the output from the previous command to Set-Mailbox -Type Room:

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*confe
rence*") } | set-mailbox -type room
[PS] C:\Windows\system32>

Done! Now when you create an appointment in Outlook 2007, in Scheduling Assistant, you can click the “Add Room” button to add a room. Hooray.

[evan@ehoffman 10:35:50 ~]$ ldapsearch -x -LLL -D "ldapuser@example.com" -w password -b "OU=Users,DC=example,DC=com" -s sub -H ldaps://activedirectory.example.com "(sn=hoffman)" cn mail displayName samaccountname
dn: CN=Evan Hoffman,OU=Tech,OU=Users,DC=example,DC=com
cn: Evan Hoffman
displayName: Evan D. Hoffman
sAMAccountName: ehoffman
mail: Evan.Hoffman@example.com

I went in with Powershell and checked their autoreply status via Get-MailboxAutoReplyConfiguration and it appears that it is, in fact, disabled:

[PS] C:\Windows\system32>Get-mailbox -identity username | Get-MailboxAutoReplyConfiguration
RunspaceId : 7ad7e9af-cd57-4572-a4fd-c1e999e4b9a5
AutoReplyState : Disabled
EndTime : 8/12/2010 12:00:00 PM
ExternalAudience : All
ExternalMessage :
InternalMessage :
StartTime : 8/11/2010 12:00:00 PM
MailboxOwnerId : [removed]
Identity : [removed]
IsValid : True


I used Set-MailboxAutoReplyConfiguration to set the messages to “” (empty string) and it’s still sending the user’s autoresponse, from before I blanked it out. My working theory right now is that the out-of-office message was set on both the Exchange 2010 server and the Exchange 2003 server (where the mailboxes were before I migrated them to 2010).

What a fun problem! It’s hard to test whether I’ve fixed it, since each sender only receives the message once, so I have to keep creating new test email addresses to send test messages.

Also, as an aside, why is “out-of-office” abbreviated “OOF” in Microsoft’s docs?

Edit 1: I had one user verify the message was off in OWA and then start Outlook via Start -> Run… “outlook /cleanrules” and this seemed to resolve the issue. Hopefully this isn’t required every time…

The new OWA has a zillion other awesome features, my favorite being that Firefox and Chrome are no longer second-class-citizens and can use the “full version” now, even on Linux. So anyway, I guess all my work was for nothing. Not the first time (or the last).

I found a possible solution in KB 940726. Basically you use this cmdlet to change the Autodiscover URI for internal clients:

Set-ClientAccessServer –AutodiscoverServiceInternalUri https://mail.contoso.com/autodiscover/autodiscover.xml

You’d replace mail.contoso.com with the external URL of your OWA server (in my case, mail.domain.com). I’ve made the changes but I think I need to wait for AD propagation. Hopefully this will resolve it, because I don’t want to move everyone’s mailboxes over until this thing is “perfect,” whatever that means.

Edit: I also needed to add a SRV record so Outlook would know what host to check for autodiscovery when outside the domain.

Edit 2:: Also need to install a hotfix or be running Outlook 2007 SP1 or later for the SRV functionality.

Edit 3: It occurs to me that a simpler fix for this issue may be simply to delete the DNS record for autodiscover entirely. That way, when Outlook attempts to open the SSL connection to autodiscover.domain.com, it gets a NXDOMAIN error (should) silently skip it. Unfortunately we have wildcard DNS active for our domain.

Other useful resources:

• MS Exchange Team blog post comparing the various autodiscover schemes.
• Set-ClientAccessServer.
• Test Exchange Connectivity
• Setting Autodiscover URL via DNS SRV record
• Autodiscover whitepaper.
• Example Autodiscover BIND record – _autodiscover._tcp.domain.com. SRV 0 0 443 webmail.domain.com.
• Debug Autodiscover by right-clicking the Outlook icon in the system tray while holding down Ctrl
• Verifying SRV records exist with nslookup
• What version of Outlook am I running? You need SP1 or later for the SRV hack.
• Hotfix for Outlook 2007 (pre-SP1) to use SRV records for autodiscovery

Back in January or February, I got fed up with the Exchange setup I inherited: our Exchange 2003 server was running on a server in the basement of our office, on non-UPS power, with a power company that likes to pull shenanigans (like 3-4 hour outages every few months). In addition, the physical machine itself has some weird bug where it would hang at the POST screen complaining about some USB device, even though there are no USB devices plugged in, and USB is disabled in the BIOS. Meanwhile, in the datacenter, I had recently finished migrating most of our ancient physical servers to virtual machines on beautiful new hardware. It didn’t take long to see the solution that seemed to be obvious: move Exchange to the datacenter, in a VM.

There was a major wrinkle in this plan, however: there were no quota limits enforced in Exchange, and the average mailbox was 6-7 gigabytes, with 4 users over 10 gigs. At the time, we only had a 5 mbit upload connection to the datacenter, and the total size of the mailboxes was around 400 gigs. I didn’t want to spend weeks and weeks moving tons of mail over a slow pipe – and with mailboxes being so big, I wasn’t sure I could even complete some of them overnight.

At this point I brought up the idea of migrating the company to Google Apps. I’m a big fan of Gmail and moving off of Exchange would have certainly simplified some aspects of my job, and nobody would need Outlook (especially not me). I knew it would be a tough sell internally, but the pricing certainly didn’t help; it came out to $83/user/year for Google Apps + document retention. The price came out to about the same as upgrading to Exchange 2010. If it had been half or a third the cost I may have pushed harder, but to make the story (a little) shorter, we ended up sticking with Exchange, and instituting quotas.

We phased in the quotas over the course of a month to give users time to archive and clean up their mailboxes. Once that was done, I setup a new Exchange 2003 frontend server (in a VM) in the datacenter and pointed our webmail (OWA & ActiveSync) there. So we had the frontend in the datacenter and the backend “mailbox” server still in the office. I then setup another VM running Exchange 2003 in the datacenter. This enabled me to move mailboxes over one at a time with almost no interruption in service, except for the user whose mail was in transit. Since we instituted quotas, the mailboxes were all under 2 GB, and I was able to do 6-10 mailboxes each night.

I can’t tell you how happy I was when we lost power yet everyone retained full connectivity to email via their phones (except BlackBerry users, since BES was still in the basement — note to RIM: ActiveSync!).

So phase 1 & 2 (instituting quotas and moving email out of the basement) were complete. Phase 3 was the bigger unknown – moving to Exchange 2010. After lots of reading and planning, installing, configuring and testing, about two weeks ago I setup a Client Access Server to serve as the new webmail “frontend.” Microsoft has some pretty great instructions for setting up 2003 and 2010 in coexistence, but basically you point your “real” webmail URL to the 2010 CAS and move your “old” Exchange 2003 webmail to another url (they suggest legacy.company.com). Then people log in to the 2010 interface, and if their mailbox is housed on the 2003 server, it seamlessly redirects them to https://legacy.company.com/, and they don’t have to log in again. Pretty slick, and I didn’t believe it would work until I saw it for myself (which, btw, it does). So ActiveSync and Outlook Anywhere were working through the 2010 CAS even for the users housed on the 2003 server (which was all of them).

This week I started moving users over to Exchange 2010. So far it’s been mostly positive. We have several Mac users, so the ability for them to have native mail & calendaring is pretty epic. The Outlook Web App in Exchange 2010 is phenomenal. I mean, it almost brings a tear to my eye, it’s so beautiful – especially when compared with 2003. And being able to do server-side searching in OWA & on my iPhone is fabulous.

All is not perfect, though. I keep getting stupid certificate errors for Autodiscover when I open Outlook 2007. I guess I’ll need to buy another SSL certificate and dedicate another IP to this service… ugh. And now that I moved my mailbox to Exchange 2010, Outlook Anywhere appears not to work. Oh well, almost there…

References:

• http://communities.vmware.com/message/1553296#1553296

• http://www.experts-exchange.com/Software/VMWare/Q_26251559.html

The first step was to setup a VPN between the office and the datacenter so that users in the office would be able to connect seamlessly to the Exchange server once it was moved. This turned out to be relatively easy. The next step was basically to move the Exchange server. This originally seemed like it would be an easy thing to do — having a long history with PostgreSQL I figured I could do essentially a “dump and restore” – run some command that would backup contents of the mail database to a file and then restore it to a new machine. Well, I quickly learned that wasn’t possible, at least not given the factors involved.

Microsoft suggests two ways of moving an Exchange server to new hardware: 1) replacing a machine in-place with another one that takes its name and doing a restore, and 2) setting up the new server “next to” the old one and moving mailboxes over one at a time. I ruled out the first method because it seemed like a total crapshoot with no easy “rollback” mechanism. Plus I had no idea how long it would take to do a restore of our Exchange server – total mailbox size at the time was over 300 GB, and it took about 28 hours just to do the backup, so it seemed like it could easily take over 72 hours, meaning even if we started it Friday at 6 PM, it wouldn’t complete by Monday morning, and people would come in to work to find they had no email. No good.

This left the second option – setting up another server and moving mailboxes one at a time. This seemed pretty simple, except for the fact that people frequently use Outlook Web Access (webmail) to check their mail when out of the office, and ActiveSync to get mail on their phones. We tested the 2-server setup a while back and while mail gets routed properly, and users in the office are able to connect to both Exchange servers without problems, when they try accessing their mail from outside the office it fails. This is because if A is the old server (which people use for webmail) and B is the new server, if you log in to webmail (server A) but your mailbox is homed on server B, webmail will issue you a 302 redirect to http://B . If that’s not a valid URL outside your office (as is the case with us) it won’t work. If we could move everybody’s mailboxes from A to B overnight, and then make webmail point to B rather than A, that would solve the problem, but again, we had no way to know how long that would take, and I didn’t want to risk making anyone’s mail unavailable.

The plan I then came up with was to set up an Exchange frontend server in our office in front of our existing Exchange server. The frontend server would handle all the OWA/ActiveSync stuff and abstract that away from the backend server (where the mailboxes live). I could then set up an Exchange server in our office in a VM, migrate mailboxes over to them one at a time, and when it was done, copy the VM Exchange server to an external USB drive and drive it to the datacenter (about 25 miles away) and import the VM to our VMware production cluster, fiddle with its IP address and voila – the Exchange server would be moved.

But then I had a better idea: set up the frontend server and the new backend server in the datacenter in the VMware cluster from the get-go. Then when people accessed webmail they’d be hitting a server in the datacenter, which would connect to the Exchange server in the office transparently and relay them their mail. I could then move each mailbox from A to B with B being in the datacenter and the move taking place over the VPN.

Well, this is what I ended up doing, and there have been some wrinkles in the process, but so far it’s generally been working as I expected. I moved my mailbox to the new server today, and the move itself went fine – took about 90 minutes to move my 1.5 GB mailbox. It wasn’t quite a seamless process – the mailbox was moved but I couldn’t send or receive mail from the other server or the Internet in general. I managed to fix outbound SMTP pretty quickly (we relay mail through a smarthost in the datacenter) but inbound wasn’t working because the old server and new one couldn’t communicate for some reason, and all mail was being delivered to the old server. Among the things I did in attempting to solve this problem were create a new routing group for the servers in the datacenter (since we only had one Exchange server before, we only had one routing group), and then setup a Routing Group Connector between the two. This seemed like it should have resolved it but it didn’t. From server A, I could “telnet B 25″ and the connection would succeed, but if I issued a HELO I got 500 5.3.3 Unrecognized command. Same thing happened if I tried B -> A. After hours of checking settings, I came across a post on Experts Exchange that suggested the problem may be with the firewall (Cisco ASA) inspecting SMTP traffic. This was something that had flitted around in my head for a couple of seconds but I didn’t actually check it. In the end though, that’s what it was – the ASA in the datacenter was mangling the SMTP packets somehow and preventing the two from communicating. Once I issued the “no inspect esmtp” line, the whole day’s worth of mail came flooding through to my inbox (now on server B).

For some reason, however, mail was still not going B->A. I spent a while trying to figure out why – looking in logs, doing “telnet A 25″ and everything seemed fine. The mail queue kept showing queued messages though and an error like “remote server didn’t respond to the connection.” What ended up solving it, though, was deleting the Routing Group Connector associated with the datacenter routing group and re-adding it. For whatever reason, that cleared it right up.

So as of right now, we have Office and Datacenter, with Office having Exchange server A, and Datacenter having Exchange servers B and C – B being the new backend and C being the new frontend. DNS has been updated so webmail points to C, and C connects to A or B to get the user’s mail for OWA/ActiveSync. It works, it’s fast, I’m mostly happy.

I should probably mention that we discussed moving to Google Apps in the midst of this project. I was about 70% in favor of it, but in the end it seemed too expensive. We’ve already paid for our Exchange licenses and a hardware message archiver. Google’s price for Google Apps is $83/person per year if you include their 10-year archival option. If you don’t already have infrastructure in place, that might be cheap, but when you’re comparing it to “$0″ (and yes, I realize projects like the one I mentioned above aren’t free), it is a lot when you have ~100 users. In addition, most people at my company weren’t comfortable with the privacy/legal implications of having Google host our mail in the cloud – not to mention lots of people are Outlook addicts. They offered 25 GB storage per user, which was pretty compelling, and I personally love the Gmail interface, but in the end we opted to stick with Exchange for the time being.

(Edited 5/14/2010 to replace the inlined Perl script with a link to the script as a text file.)

changeadpasswd.pl