This just started happening again, with these errors appearing in the event viewer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.
 
Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/17/2011 6:47:07 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' terminated
unexpectedly. The process id was '3108'. The process exit code was
'0x800703e9'.
 
Log Name: Application
Source: Application Error
Date: 9/17/2011 6:46:30 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time
stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time
stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

After reviewing the IIS logs and the event logs, I think it has to do with the WebReady document viewer – the thing in OWA that renders and lets you view .doc attachments within the browser rather than forcing you to open Word or Excel. I think users were attempting to open corrupted files and that was causing it to crash. I’ve disabled Webready in EMC (Server Config -> CAS) and I’ll see what happens.

Long ago — eons, perhaps — before I had anything to do with the Windows environment here, someone created the AD domain in my company as a single-label domain (e.g. instead of “example.com” our domain is just “example”). Over the years this has led to lots of “fun” on the part of Windows admins who’ve worked here as the implications of this choice became more apparent.

Since I inherited this system about a year ago, I haven’t really bumped up against any problems stemming from the single-label domain issue… until now. I recently attempted to add a new Windows 2008r2 file server to our DFS replication group/namespace. This totally failed for some mysterious reason. Well, I shouldn’t say “totally” failed, as I was able to add it to the DFS replication group, but unable to add it to the DFS namespace. In my attempt to debug the namespace issue, I deleted the namespace and attempted to recreate it, but just kept getting this error: The namespace cannot be queried. The specified domain either does not exist or could not be contacted.. I couldn’t do anything with the namespace – even clicking on it in the DFS Management console brought up an error. After some searching I found that this was likely due to having a single-label domain. I wasn’t sure why the error was happening even on Windows 2003 machines though, maybe joining a 2008r2 box to the domain made some schema changes? I tried a few suggestions like editing the hosts file but nothing seemed to resolve this.

Fortunately, we didn’t really need DFS namespaces and were able to just direct everybody to the fileserver via its DNS name, though as you can imagine this was clumsy. However, since this has been a problem since time immemorial, I figured it was time to see if it was fixable. After some quick searching, I found RENDOM. However, after even more searching I discovered this TechNet article which says:

The domain rename operation is not supported in Microsoft Exchange Server 2007 or Exchange Server 2010. DNS domain rename is supported in Exchange Server 2003. However, renaming of the NetBIOS domain name is not supported in any version of Exchange Server. Other non-Microsoft applications might also not support domain rename.

Well. We’re running Exchange 2010. So now what? I guess we’re going to have to create a second domain and migrate over to it. We’d already discussed this as a likely way of implementing the rename anyway, since it didn’t seem like “RENDOM” had any rollback procedure – it either just works (hahaha) or semi-works and semi-fails, leaving a wake of destruction throughout AD. Building a second domain seems like a lot of work, but at least we can move users over one at a time, and we get the side benefit of starting fresh, outgrowing the 5+ years of crud that’s accumulated in our AD.

Guess we’ll see what happens. Neither option seems like much fun. I guess the alternative is do nothing, but Microsoft clearly doesn’t think very highly of single-label domains, and anyone who asks about them gets looked at funny. At least it gives us something to do!

I had Autodiscover working for months but recently it just stopped. I’m not sure why, but it may be related to removing the last Exchange 2003 servers from service recently. Maybe some setting got wiped from AD when I uninstalled Exchange 2003 (as per the procedure Microsoft gives). Basically what was happening was that the email address field was being autopopulated by the user’s UPN rather than their email address. Since we have a single label domain, the UPN isn’t a valid email address, and autodiscovery fails.

Anyway, I ran Get-AutodiscoverVirtualDirectory and it looks like the autodiscover URL isn’t set. Pretty sure I set this at some point.

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl
 
InternalUrl :
ExternalUrl :
 
[PS] C:\Windows\system32>

I just piped this to Set-AutodiscoverVirtualDirectory to correct the problem:

[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | Set-AutodiscoverVirtualDirectory -ExternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml' -InternalUrl 'https://webmail.example.com/Autodiscover/Autodiscover.xml'
[PS] C:\Windows\system32>Get-AutodiscoverVirtualDirectory -server exch2010fe1  | fl InternalUrl,ExternalUrl
 
 
InternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml
ExternalUrl : https://webmail.example.com/Autodiscover/Autodiscover.xml
 
 
[PS] C:\Windows\system32>

After resetting the InternalURL and ExternalURL, autodiscover works again (we have SRV records that tell Outlook to look at webmail.example.com for the Autodiscover service).

Hooray!

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Some of the errors SquirrelMail was reporting were “Error connecting to IMAP server xxxx Server error: (0)” and “Error connecting to IMAP server: tls://xxxx:993. 0: ”

Nothing would actually work on port 993. Telnet to 993 got this:

$ telnet 10.0.20.18 993
Trying 10.0.20.18...
Connected to 10.0.20.18.
Escape character is '^]'.
* BYE Connection is closed. 14
Connection closed by foreign host.

After too much poking, I decided to go down to a lower level and do a simple openssl certificate retrieval and see what came back:

$ openssl s_client -connect 10.0.20.18:993
CONNECTED(00000003)
140281653434184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 113 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

That didn’t look right, so I ran it against the same server on port 443 and got back a real certificate. Same for port 995 (pop3s):

$ openssl s_client -connect 10.0.20.18:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = info@valicert.com
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1

(snip)

So there’s just something wrong with SSL on port 993. To make a long story short, I had to use the Enable-ExchangeCertificate to apply the SSL certificate to port 993. First, run “Get-ExchangeCertificate” to list the available certificates and retrieve the Thumbprint.

[PS] C:\Windows\system32>Get-ExchangeCertificate
 
Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  .P....     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  I..W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

Copy & paste the thumbprint for whichever cert you want to use into Enable-ExchangeCertificate:

[PS] C:\Windows\system32>Enable-ExchangeCertificate -ThumbPrint xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -Services IIS,P
OP,IMAP -DoNotRequireSSL
[PS] C:\Windows\system32>Get-ExchangeCertificate
 
Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  ......     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  IP.W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

After running that, imaps on port 993 worked perfectly. I can connect to it with both SquirrelMail and Thunderbird.

The SquirrelMail config looks like this:

IMAP Settings
--------------
4.  IMAP Server            : webmail.example.com
5.  IMAP Port              : 993
6.  Authentication type    : login
7.  Secure IMAP (TLS)      : true
8.  Server software        : exchange
9.  Delimiter              : detect

Edit Feb 15, 2011: I just renewed the SSL cert and ran into a problem with a Ruby script that was suddenly unable to check a mailbox over IMAPS. The error received was:

/usr/lib/ruby/1.8/net/imap.rb:898:in `connect': unknown protocol (OpenSSL::SSL::SSLError)
        from /usr/lib/ruby/1.8/net/imap.rb:898:in `initialize'

After a few minutes, I remembered this blog post and ran Enable-ExchangeCertificate and it worked again. Glad I wrote it down.

CONNECTED(00000003) 26831:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

A few years ago we moved from Nagios to Zabbix for our server monitoring needs. I wasn’t a big fan of Nagios, finding it a pain to manage with its myriad configuration files. It’s probably gotten better since I last toyed with it but since we moved to Zabbix I haven’t had much reason to look at Nagios again.

I also try to use SNMP monitoring for everything. SNMP is widely supported – all sorts of hardware has SNMP support, and with the net-snmp package you can pretty easily create your own SNMP-monitorable stuff on Linux. Since almost all of our stuff runs on Linux this has worked out pretty well, but our Exchange server is probably going to be running on Windows for the foreseeable future. Windows has SNMP support, it’s just not on by default. However, even when it’s enabled it doesn’t have the simple “dskPercent” monitoring I’ve come to know and love with net-snmp on Linux, which simply tells you how full a given disk is as a percent. This makes it easy to set alerts when a disk reaches 80% full.

On Windows I found these objects that can be used to get something similar:

[evan@monitoring02 14:41:24 ~]$ snmpwalk -v 2c -c community 192.168.1.20 | grep -i storage
HOST-RESOURCES-MIB::hrStorageIndex.1 = INTEGER: 1
HOST-RESOURCES-MIB::hrStorageIndex.2 = INTEGER: 2
HOST-RESOURCES-MIB::hrStorageIndex.3 = INTEGER: 3
HOST-RESOURCES-MIB::hrStorageIndex.4 = INTEGER: 4
HOST-RESOURCES-MIB::hrStorageIndex.5 = INTEGER: 5
HOST-RESOURCES-MIB::hrStorageIndex.6 = INTEGER: 6
HOST-RESOURCES-MIB::hrStorageType.1 = OID: HOST-RESOURCES-TYPES::hrStorageRemovableDisk
HOST-RESOURCES-MIB::hrStorageType.2 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.3 = OID: HOST-RESOURCES-TYPES::hrStorageCompactDisc
HOST-RESOURCES-MIB::hrStorageType.4 = OID: HOST-RESOURCES-TYPES::hrStorageFixedDisk
HOST-RESOURCES-MIB::hrStorageType.5 = OID: HOST-RESOURCES-TYPES::hrStorageVirtualMemory
HOST-RESOURCES-MIB::hrStorageType.6 = OID: HOST-RESOURCES-TYPES::hrStorageRam
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: A:\
HOST-RESOURCES-MIB::hrStorageDescr.2 = STRING: C:\ Label:  Serial Number b78d19
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: D:\ Label:EXCH201064  Serial Number xxxxxxxx
HOST-RESOURCES-MIB::hrStorageDescr.4 = STRING: E:\ Label:Exchange2010  Serial Number xxxxxxxx
HOST-RESOURCES-MIB::hrStorageDescr.5 = STRING: Virtual Memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Physical Memory
HOST-RESOURCES-MIB::hrStorageAllocationUnits.1 = INTEGER: 0 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.2 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.3 = INTEGER: 2048 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.4 = INTEGER: 4096 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.5 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageAllocationUnits.6 = INTEGER: 65536 Bytes
HOST-RESOURCES-MIB::hrStorageSize.1 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageSize.2 = INTEGER: 10459647
HOST-RESOURCES-MIB::hrStorageSize.3 = INTEGER: 546570
HOST-RESOURCES-MIB::hrStorageSize.4 = INTEGER: 104824319
HOST-RESOURCES-MIB::hrStorageSize.5 = INTEGER: 393172
HOST-RESOURCES-MIB::hrStorageSize.6 = INTEGER: 196600
HOST-RESOURCES-MIB::hrStorageUsed.1 = INTEGER: 0
HOST-RESOURCES-MIB::hrStorageUsed.2 = INTEGER: 5885720
HOST-RESOURCES-MIB::hrStorageUsed.3 = INTEGER: 546570
HOST-RESOURCES-MIB::hrStorageUsed.4 = INTEGER: 44650892
HOST-RESOURCES-MIB::hrStorageUsed.5 = INTEGER: 166057
HOST-RESOURCES-MIB::hrStorageUsed.6 = INTEGER: 152902

I thought initially that the hrStorageUsed and hrStorageSize values were being reported in bytes, but according to this MSDN article, the units are reported in “allocation units,” which I assume are being reported under hrStorageAllocationUnits, so you just need to multiply the values by the allocation units.

In Zabbix, I check hrStorageUsed every 15 minutes as “disk_1_used”. I check hrStorageSize every 2 hours (since the actual size of the disk/partition isn’t likely to change that often) as “disk_1_size”. To calculate the percentage, I created a “Calculated” item with this formula:

100*(last("disk_1_used") / last("disk_1_size"))

Windows disk usage percent

The values for disk_1_used and disk_1_size are in Storage Allocation Units, but since this is a percentage that doesn’t matter. However, I do also like to get an idea of the actual disk space being consumed; luckily this is also relatively easy to obtain in Zabbix using Calculated items. I monitor hrStorageAllocationUnits as “disk_1_allocunit” (every 7200 seconds since this too is unlikely to change much) and then the formula for the calculated used disk space is simply:

last("disk_1_used") * last("disk_1_allocunit")

Windows disk used (bytes)

Once all the work is done, here’s what the result looks like:

Zabbix SNMP monitored Windows disk items

When I log in to the actual machine (my vCenter VM in this case) and check disk usage, the numbers match what Zabbix’s calculated values show, though Zabbix seems to be reporting values in “mebibytes” rather than “megabytes”:

Actual Windows disk usage

I created a template in Zabbix which monitors these data for disks 1-5 and then applied it to all Windows servers; now I just need to apply some alert triggers and mission accomplished.

Edit 2011-01-14: here’s the template exported from Zabbix: Zabbix Windows disk usage template

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

My first guess was that I was being punished for having forgotten to upgrade the LDAP address lists to OPATH. I don’t really know what that even means, but when I attempted to edit the address lists in EMC I’d get an error that they needed to be upgraded. Fortunately, this Technet article lists the commands needed to upgrade the lists. I did it but this didn’t appear to resolve all the issues.

At this point, after some Googling, I came across this tidbit:

If you’re moving from Exchange 2000/2003 to Exchange 2007 or Exchange 2010, you’re going to want to convert all your domain local and global distribution and mail-enabled security groups to universal groups so they can be managed using the Exchange management tools.

This explains a few things we’ve noticed – inability to add Global (Non-Universal) groups to a newly created (Universal) group, for one. So it appears what we should do is upgrade all the Global groups to Universal. First, how do we get a list of all the Global groups? EMS/PowerShell to the rescue:

[PS] C:\Windows\system32>Get-Group  | Where {$_.GroupType -Like "Global*"  -AND $_.RecipientType -eq "MailNonUniversalGroup"} |
Export-Csv -encoding "utf8" -Path \\fileserver\Tech\groups1.csv

You can refine the filter further, and when it looks correct you can just pipe the output to Set-Group:

Get-Group  | Where {$_.GroupType -Like "Global*"  -AND $_.RecipientType -eq "MailNonUniversalGroup"} | Set-Group -Universal

But now for the most important question: will this break anything? I have no idea. We only have a single domain in our AD forest so we’ve never had need to use Universal, and I don’t think there should be a problem, but I don’t really have any idea.

I ran the Get-Group/Set-Group commands and they seemed to work as intended for all but about 60 of the target groups. The groups that didn’t get converted all had weird issues – aliases that contained illegal characters (which I fixed), or some of them complained that a particular user (I think the Owner of the group in AD) was not found (even though it was in the exact location it was saying it wasn’t, though the user was disabled). I “manually” converted these groups to Universal via the radio button in the properties dialog in Active Dir Users & Groups. Not the most elegant solution but it worked. So all the groups in question are now Universal Security groups. Will this solve the problem? Well, I’ll have to wait until tomorrow to find out.

Reference links:

• Need to convert Global groups to Universal groups? Do you have messages to global groups disappearing?
• Changing Group Type via PowerShell
• How to Convert Local and Global Groups to Universal Groups
• Forum post about the “BypassSecurityGroupManagerCheck” security error
• Exchange Server 2007 and Universal Groups
• KB 231273: Group Type and Scope Usage in Windows
• Mail Universal Groups vs Mail Non-Universal Groups on Experts Exchange
• Technet blog: Recipients List
• Technet forum: Mail enabled distribution groups in AD

I’ve been working on migrating our Exchange environment from 2003 to 2010 for several months. My first post about this is from April 14th, when I was just trying to virtualize our existing Exchange 2003 system. Once that was complete, I started playing around with Exchange 2010 around June or July, and had most of the users moved over to 2010 by the end of August. The last holdouts were Blackberry users. I couldn’t move their mailboxes because our BES was hosted on our original Exchange 2003 server.

BES is another product that I inherited that I had no experience with. It’s BES 4.1.x and while I wasn’t a fan of the UI it seemed to do its job. However, when I started moving people to Exchange 2010 I learned that BES 4.1 doesn’t support Exchange 2010. So, to cut the (absurdly long) story short, I setup BES Express on a new VM, pointed it at our Exchange 2010 server, tested it out (and it worked), and just last week was able (finally) to move the last few users over to Exchange 2010. BES users had to have their phones wiped to join them to the BES Express server, which was the major sticking point.

I can’t believe it actually took that long to complete, but we managed to move all user mailboxes twice (Ex2003 physical -> Ex2003 VM, then Ex2003 VM -> Ex2010 VM) with no noticeable interruption to users (we did the moves at night). OWA 2010 alone would make it worth the upgrade, but I’m actually loving the Exchange Management Shell too.

Anyway… nice to have it completed.

I’ve always hated iTunes. It’s a huge pile of bloatware and it’s slow as poo. It’s like 100 mb or more for an mp3 player. I remember winamp playing mp3s when it was a 500k download. Anyway.

I keep all my music on a Linux machine running samba. This way it’s available to every machine in the house. When I had Winamp on all my machines this was wonderful. But now that I’m forced into iTunes (thanks to having an iPhone), it turns out to be a major pain. In iTunes I unchecked the box for “let iTunes keep my libary organized” to prevent it from copying the entire library to each computer’s local disk. Initially adding my library of ~4000 tracks to iTunes takes over an hour (100 mbit wire) – it would take about 5 minutes in Winamp, even reading the ID3 tags for each track as it was added (rather than lazily as the song was played).

But the thing that iTunes does that is so annoying it prompted me to write this whiny rant is:

iTunes 'Song Not Found'

If, for some reason, my M: drive (where the Samba share is mapped) is not connected when iTunes starts, every song in the library gets this “!” exclamation point of doom. If I attempt to play any of these tracks, I am given the option to locate the file. Nice in theory, but locating all 4000 tracks isn’t realistic. If I quit iTunes, reconnect the M: drive, and reopen iTunes, the ! persists. The only solution I’ve found to this is deleting the entire library from iTunes and re-adding it, which as I said, takes an extremely long time.

I have other reasons for hating iTunes, this is a blog, not a book.

I’m wrapping up moving mailboxes to Exchange 2010. The last ones to be moved (except for BlackBerry users… thanks BES) are the conference rooms. So the first step was to move them using the Local Move tool, which was pretty simple. But I don’t want them in 2010 as user mailboxes if they can be designated as “rooms,” which they can. So here’s how I’m doing it:

Identify the mailboxes to be moved

Once you figure out the syntax for the “-Filter” flag to get-mailbox, this is easy

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*conference*") }
 
Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
Conference Room2          ConferenceRoom2      exch2010be1      unlimited
Production Conference ... productionconf       exch2010be1      unlimited
Conference Room 1         conference1          exch2010be1      unlimited
L&D Conference Room       ldconference         exch2010be1      unlimited
Tech Conference Room      techconference       exch2010be1      unlimited
Client Services Confer... csconference         exch2010be1      unlimited
Suite 202 Conference Room 202conf              exch2010be1      unlimited

Convert them to rooms

As Microsoft says in this story about converting mailboxes to rooms, this can only be done via Exchange Management Shell (not EMC), so just pipe the output from the previous command to Set-Mailbox -Type Room:

[PS] C:\Windows\system32>get-mailbox -filter { (RecipientTypeDetails -eq "UserMailbox") -and ( DisplayName -like "*conference*") } | set-mailbox -type room
[PS] C:\Windows\system32>

Done! Now when you create an appointment in Outlook 2007, in Scheduling Assistant, you can click the “Add Room” button to add a room. Hooray.

Just putting this here for safekeeping since I couldn’t remember the exact syntax.

[evan@ehoffman 10:35:50 ~]$ ldapsearch -x -LLL -D "ldapuser@example.com" -w password -b "OU=Users,DC=example,DC=com" -s sub -H ldaps://activedirectory.example.com "(sn=hoffman)" cn mail displayName samaccountname
dn: CN=Evan Hoffman,OU=Tech,OU=Users,DC=example,DC=com
cn: Evan Hoffman
displayName: Evan D. Hoffman
sAMAccountName: ehoffman
mail: Evan.Hoffman@example.com

Explanation: Connect to activedirectory.example.com using ldaps (SSL) with simple authentication, binding as ldapuser@example.com with password password; search for (sn=hoffman) within the OU=Users,DC=example,DC=com search base (branch), and search the subtree. Return the cn, displayName, and samaccountname fields.

Refer to the ldapsearch man page for more options.

Two users reported the same problem this week: they turned on their out-of-office reply while they were out, then came back and turned it off. Except even after they turned it off, the autoreply was still being sent out. I had them log in to OWA and make sure it was off (maybe some weird bug with Outlook not registering the change in the server), which it was in both cases. I Googled hard and fast and couldn’t find anyone with this same problem.

I went in with Powershell and checked their autoreply status via Get-MailboxAutoReplyConfiguration and it appears that it is, in fact, disabled:

[PS] C:\Windows\system32>Get-mailbox -identity username | Get-MailboxAutoReplyConfiguration
RunspaceId       : 7ad7e9af-cd57-4572-a4fd-c1e999e4b9a5
AutoReplyState   : Disabled
EndTime          : 8/12/2010 12:00:00 PM
ExternalAudience : All
ExternalMessage  :
InternalMessage  :
StartTime        : 8/11/2010 12:00:00 PM
MailboxOwnerId   :  [removed]
Identity         :  [removed]
IsValid          : True

I used Set-MailboxAutoReplyConfiguration to set the messages to “” (empty string) and it’s still sending the user’s autoresponse, from before I blanked it out. My working theory right now is that the out-of-office message was set on both the Exchange 2010 server and the Exchange 2003 server (where the mailboxes were before I migrated them to 2010).

What a fun problem! It’s hard to test whether I’ve fixed it, since each sender only receives the message once, so I have to keep creating new test email addresses to send test messages.

Also, as an aside, why is “out-of-office” abbreviated “OOF” in Microsoft’s docs?

Edit 1: I had one user verify the message was off in OWA and then start Outlook via Start -> Run… “outlook /cleanrules” and this seemed to resolve the issue. Hopefully this isn’t required every time…

A few months ago I was on a quest to figure out how to change my Active Directory password via a browser (for Linux/Mac users). I finally figured it out, but since I’ve been working on this Exchange 2010 migration I noticed one of the features of OWA (Outlook Web App) in Exchange 2010 is that you can change the AD password right in the browser from within the app:

The new OWA has a zillion other awesome features, my favorite being that Firefox and Chrome are no longer second-class-citizens and can use the “full version” now, even on Linux. So anyway, I guess all my work was for nothing. Not the first time (or the last).

One of the more annoying side effects of migrating my mailbox to Exchange 2010 has been the nagging of Outlook 2007′s Autodiscovery feature. Now, every time I start Outlook I get hit with a certificate error for autodiscover.domain.com. Now, autodiscover.domain.com is a CNAME to mail.domain.com, which is the OWA URL for the CAS. The SSL certificate is valid – but it’s valid for mail.domain.com. I could buy a SSL certificate from GoDaddy for $12.99 (an insanely great price, btw) for “autodiscover” but that would also require using another IP address on the CAS (since you can can only bind one SSL certificate to an IP:port pair), and that seems like a waste of an IP address.

I found a possible solution in KB 940726. Basically you use this cmdlet to change the Autodiscover URI for internal clients:

Set-ClientAccessServer –AutodiscoverServiceInternalUri https://mail.contoso.com/autodiscover/autodiscover.xml

You’d replace mail.contoso.com with the external URL of your OWA server (in my case, mail.domain.com). I’ve made the changes but I think I need to wait for AD propagation. Hopefully this will resolve it, because I don’t want to move everyone’s mailboxes over until this thing is “perfect,” whatever that means.

Edit: I also needed to add a SRV record so Outlook would know what host to check for autodiscovery when outside the domain.

Edit 2:: Also need to install a hotfix or be running Outlook 2007 SP1 or later for the SRV functionality.

Edit 3: It occurs to me that a simpler fix for this issue may be simply to delete the DNS record for autodiscover entirely. That way, when Outlook attempts to open the SSL connection to autodiscover.domain.com, it gets a NXDOMAIN error (should) silently skip it. Unfortunately we have wildcard DNS active for our domain.

Other useful resources:

• MS Exchange Team blog post comparing the various autodiscover schemes.
• Set-ClientAccessServer.
• Test Exchange Connectivity
• Setting Autodiscover URL via DNS SRV record
• Autodiscover whitepaper.
• Example Autodiscover BIND record – _autodiscover._tcp.domain.com. SRV 0 0 443 webmail.domain.com.
• Debug Autodiscover by right-clicking the Outlook icon in the system tray while holding down Ctrl
• Verifying SRV records exist with nslookup
• What version of Outlook am I running? You need SP1 or later for the SRV hack.
• Hotfix for Outlook 2007 (pre-SP1) to use SRV records for autodiscovery

I’ve been working on migrating from Exchange 2003 to Exchange 2010 for several weeks. Actually, at this point it feels like several months. Now that I think about it, I guess that’s because it’s actually been several months.

Back in January or February, I got fed up with the Exchange setup I inherited: our Exchange 2003 server was running on a server in the basement of our office, on non-UPS power, with a power company that likes to pull shenanigans (like 3-4 hour outages every few months). In addition, the physical machine itself has some weird bug where it would hang at the POST screen complaining about some USB device, even though there are no USB devices plugged in, and USB is disabled in the BIOS. Meanwhile, in the datacenter, I had recently finished migrating most of our ancient physical servers to virtual machines on beautiful new hardware. It didn’t take long to see the solution that seemed to be obvious: move Exchange to the datacenter, in a VM.

There was a major wrinkle in this plan, however: there were no quota limits enforced in Exchange, and the average mailbox was 6-7 gigabytes, with 4 users over 10 gigs. At the time, we only had a 5 mbit upload connection to the datacenter, and the total size of the mailboxes was around 400 gigs. I didn’t want to spend weeks and weeks moving tons of mail over a slow pipe – and with mailboxes being so big, I wasn’t sure I could even complete some of them overnight.

At this point I brought up the idea of migrating the company to Google Apps. I’m a big fan of Gmail and moving off of Exchange would have certainly simplified some aspects of my job, and nobody would need Outlook (especially not me). I knew it would be a tough sell internally, but the pricing certainly didn’t help; it came out to $83/user/year for Google Apps + document retention. The price came out to about the same as upgrading to Exchange 2010. If it had been half or a third the cost I may have pushed harder, but to make the story (a little) shorter, we ended up sticking with Exchange, and instituting quotas.

We phased in the quotas over the course of a month to give users time to archive and clean up their mailboxes. Once that was done, I setup a new Exchange 2003 frontend server (in a VM) in the datacenter and pointed our webmail (OWA & ActiveSync) there. So we had the frontend in the datacenter and the backend “mailbox” server still in the office. I then setup another VM running Exchange 2003 in the datacenter. This enabled me to move mailboxes over one at a time with almost no interruption in service, except for the user whose mail was in transit. Since we instituted quotas, the mailboxes were all under 2 GB, and I was able to do 6-10 mailboxes each night.

I can’t tell you how happy I was when we lost power yet everyone retained full connectivity to email via their phones (except BlackBerry users, since BES was still in the basement — note to RIM: ActiveSync!).

So phase 1 & 2 (instituting quotas and moving email out of the basement) were complete. Phase 3 was the bigger unknown – moving to Exchange 2010. After lots of reading and planning, installing, configuring and testing, about two weeks ago I setup a Client Access Server to serve as the new webmail “frontend.” Microsoft has some pretty great instructions for setting up 2003 and 2010 in coexistence, but basically you point your “real” webmail URL to the 2010 CAS and move your “old” Exchange 2003 webmail to another url (they suggest legacy.company.com). Then people log in to the 2010 interface, and if their mailbox is housed on the 2003 server, it seamlessly redirects them to https://legacy.company.com/, and they don’t have to log in again. Pretty slick, and I didn’t believe it would work until I saw it for myself (which, btw, it does). So ActiveSync and Outlook Anywhere were working through the 2010 CAS even for the users housed on the 2003 server (which was all of them).

This week I started moving users over to Exchange 2010. So far it’s been mostly positive. We have several Mac users, so the ability for them to have native mail & calendaring is pretty epic. The Outlook Web App in Exchange 2010 is phenomenal. I mean, it almost brings a tear to my eye, it’s so beautiful – especially when compared with 2003. And being able to do server-side searching in OWA & on my iPhone is fabulous.

All is not perfect, though. I keep getting stupid certificate errors for Autodiscover when I open Outlook 2007. I guess I’ll need to buy another SSL certificate and dedicate another IP to this service… ugh. And now that I moved my mailbox to Exchange 2010, Outlook Anywhere appears not to work. Oh well, almost there…

I got the above error today after running Windows Update on my XP VM a few days ago. A quick search showed that the error is caused by a Microsoft update to the .NET framework. To resolve it, remove update KB980773 (Add/Remove programs, make sure “Show Updates” is checked; KB980773 is under “Microsoft .NET Framework 2.0 Service Pack 2″). I removed it and was able to log in without problems.

References:

• http://communities.vmware.com/message/1553296#1553296

• http://www.experts-exchange.com/Software/VMWare/Q_26251559.html

Edit 10/22/2010: You can also resolve this by upgrading your vCenter client to 4.1, which I recently did. 4.1 is available on vmware.com.

I’m installing Exchange 2010 and in the docs it shows a bunch of groups that get created in the AD Schema during the domain prep part. After running prep, I looked to see if the groups were there, and sure enough they were (yay). What caught my eye was that one of the groups is called Hygiene Management. I thought maybe this was an Easter Egg from MS, but apparently it’s just the name of the group of people who can manage the Exchange antivirus/antispam features. Still funny though.

So our Exchange server is located in our office building. This made sense at the time because that’s where the users are. Over time though, this has proved problematic for a few reasons. Primarily, our office is certainly not a datacenter and doesn’t offer the amenities of one – clean, reliable power, and redundant cooling. In an average year we lose power probably 10-15 times, often for an hour or more. The rest of our production environment is hosted in a top-tier datacenter, so after a while I started to wonder why our Exchange server wasn’t there, and making plans to move it there. Oh, and did I mention I’m not an Exchange admin in any sense of the term? I just inherited the Exchange server about 2 months ago.

The first step was to setup a VPN between the office and the datacenter so that users in the office would be able to connect seamlessly to the Exchange server once it was moved. This turned out to be relatively easy. The next step was basically to move the Exchange server. This originally seemed like it would be an easy thing to do — having a long history with PostgreSQL I figured I could do essentially a “dump and restore” – run some command that would backup contents of the mail database to a file and then restore it to a new machine. Well, I quickly learned that wasn’t possible, at least not given the factors involved.

Microsoft suggests two ways of moving an Exchange server to new hardware: 1) replacing a machine in-place with another one that takes its name and doing a restore, and 2) setting up the new server “next to” the old one and moving mailboxes over one at a time. I ruled out the first method because it seemed like a total crapshoot with no easy “rollback” mechanism. Plus I had no idea how long it would take to do a restore of our Exchange server – total mailbox size at the time was over 300 GB, and it took about 28 hours just to do the backup, so it seemed like it could easily take over 72 hours, meaning even if we started it Friday at 6 PM, it wouldn’t complete by Monday morning, and people would come in to work to find they had no email. No good.

This left the second option – setting up another server and moving mailboxes one at a time. This seemed pretty simple, except for the fact that people frequently use Outlook Web Access (webmail) to check their mail when out of the office, and ActiveSync to get mail on their phones. We tested the 2-server setup a while back and while mail gets routed properly, and users in the office are able to connect to both Exchange servers without problems, when they try accessing their mail from outside the office it fails. This is because if A is the old server (which people use for webmail) and B is the new server, if you log in to webmail (server A) but your mailbox is homed on server B, webmail will issue you a 302 redirect to http://B . If that’s not a valid URL outside your office (as is the case with us) it won’t work. If we could move everybody’s mailboxes from A to B overnight, and then make webmail point to B rather than A, that would solve the problem, but again, we had no way to know how long that would take, and I didn’t want to risk making anyone’s mail unavailable.

The plan I then came up with was to set up an Exchange frontend server in our office in front of our existing Exchange server. The frontend server would handle all the OWA/ActiveSync stuff and abstract that away from the backend server (where the mailboxes live). I could then set up an Exchange server in our office in a VM, migrate mailboxes over to them one at a time, and when it was done, copy the VM Exchange server to an external USB drive and drive it to the datacenter (about 25 miles away) and import the VM to our VMware production cluster, fiddle with its IP address and voila – the Exchange server would be moved.

But then I had a better idea: set up the frontend server and the new backend server in the datacenter in the VMware cluster from the get-go. Then when people accessed webmail they’d be hitting a server in the datacenter, which would connect to the Exchange server in the office transparently and relay them their mail. I could then move each mailbox from A to B with B being in the datacenter and the move taking place over the VPN.

Well, this is what I ended up doing, and there have been some wrinkles in the process, but so far it’s generally been working as I expected. I moved my mailbox to the new server today, and the move itself went fine – took about 90 minutes to move my 1.5 GB mailbox. It wasn’t quite a seamless process – the mailbox was moved but I couldn’t send or receive mail from the other server or the Internet in general. I managed to fix outbound SMTP pretty quickly (we relay mail through a smarthost in the datacenter) but inbound wasn’t working because the old server and new one couldn’t communicate for some reason, and all mail was being delivered to the old server. Among the things I did in attempting to solve this problem were create a new routing group for the servers in the datacenter (since we only had one Exchange server before, we only had one routing group), and then setup a Routing Group Connector between the two. This seemed like it should have resolved it but it didn’t. From server A, I could “telnet B 25″ and the connection would succeed, but if I issued a HELO I got 500 5.3.3 Unrecognized command. Same thing happened if I tried B -> A. After hours of checking settings, I came across a post on Experts Exchange that suggested the problem may be with the firewall (Cisco ASA) inspecting SMTP traffic. This was something that had flitted around in my head for a couple of seconds but I didn’t actually check it. In the end though, that’s what it was – the ASA in the datacenter was mangling the SMTP packets somehow and preventing the two from communicating. Once I issued the “no inspect esmtp” line, the whole day’s worth of mail came flooding through to my inbox (now on server B).

For some reason, however, mail was still not going B->A. I spent a while trying to figure out why – looking in logs, doing “telnet A 25″ and everything seemed fine. The mail queue kept showing queued messages though and an error like “remote server didn’t respond to the connection.” What ended up solving it, though, was deleting the Routing Group Connector associated with the datacenter routing group and re-adding it. For whatever reason, that cleared it right up.

So as of right now, we have Office and Datacenter, with Office having Exchange server A, and Datacenter having Exchange servers B and C – B being the new backend and C being the new frontend. DNS has been updated so webmail points to C, and C connects to A or B to get the user’s mail for OWA/ActiveSync. It works, it’s fast, I’m mostly happy.

I should probably mention that we discussed moving to Google Apps in the midst of this project. I was about 70% in favor of it, but in the end it seemed too expensive. We’ve already paid for our Exchange licenses and a hardware message archiver. Google’s price for Google Apps is $83/person per year if you include their 10-year archival option. If you don’t already have infrastructure in place, that might be cheap, but when you’re comparing it to “$0″ (and yes, I realize projects like the one I mentioned above aren’t free), it is a lot when you have ~100 users. In addition, most people at my company weren’t comfortable with the privacy/legal implications of having Google host our mail in the cloud – not to mention lots of people are Outlook addicts. They offered 25 GB storage per user, which was pretty compelling, and I personally love the Gmail interface, but in the end we opted to stick with Exchange for the time being.

I had to give up on PHP and go to Perl, but it turned out not to be so bad. Users can now change their Active Directory passwords via a self-service web page that doesn’t require admin credentials. The Perl code is below.  Authentication to the script is done via .htaccess LDAP authentication, so the REMOTE_USER env variable is assumed to contain the user’s username (sAMAccountName) by the time this script is called.  There is a simple check for $ENV{HTTPS} to ensure the script is called via SSL, and AD requires password changes to be done via ldaps, so the whole thing should be encrypted end to end.

(Edited 5/14/2010 to replace the inlined Perl script with a link to the script as a text file.)

(Edited 10/6/2011 to replace link to script with link to gist)