We’ve been trying now literally for weeks to configure our Juniper SRX VPN using RADIUS authentication. We previously had a Cisco ASA 5510 and it worked fine but we moved to an SRX because it was much easier to administer and was about 1/3 the cost. For the most part we’ve been pretty happy with the SRX vs the ASA – common-sense features like saving previous configs for trivial rollback, a web gui that’s way easier to use than ASDM, etc. But this VPN thing is a real fiasco. Basically what we want to do is use the SRX to authenticate against our Active Directory using IAS.

The closest we’ve gotten so far is having the SRX authenticate via RADIUS but apparently we still need to maintain a local user list on the SRX itself, defeating the purpose of central authentication. We were sold 10 VPN licenses with the understanding that this meant we could have up to 10 simultaneous VPN connections open. Apparently that’s true, but we’d have to specify which 10 users can access the VPN, which won’t work for us.

This doesn’t seem like we’re trying to do anything exotic here but according to the vendor and JTAC support, they’re not aware of anyone doing this, and have no idea how to do it. The ticket’s been open with Juniper for over 1 month now. We’re at the point where we just want a refund for the VPN licenses rather than continuing to bang our heads against the wall with this. The vendor originally wanted us to buy an SA VPN appliance in addition to the SRX, but assured us the SRX would do what we needed with the caveat that the JunOS Pulse client for the SRX VPN was kind of crappy.

So after opening tickets in JTAC and the vendor’s support system, the net result apparently is that this is actually not possible. Not only is it not possible but everybody acts like we’re trying to do some strange voodoo. I mean, we have other stuff hooked into AD IAS already – wifi access points for one, and the old ASA AnyConnect VPN for another. We even went so far as to create a FreeRadius server to serve as an intermediary between the SRX and AD so we could get better log messages, and that didn’t work either.

So hopefully we can get a refund on the 10 useless VPN licenses and use our old ASA for AnyConnect VPN. I’m not holding my breath though. Fortunately we’ve eliminated the need for VPN for almost everything, the local fileserver is really the only thing left that requires VPN access.

Long ago — eons, perhaps — before I had anything to do with the Windows environment here, someone created the AD domain in my company as a single-label domain (e.g. instead of “example.com” our domain is just “example”). Over the years this has led to lots of “fun” on the part of Windows admins who’ve worked here as the implications of this choice became more apparent.

Since I inherited this system about a year ago, I haven’t really bumped up against any problems stemming from the single-label domain issue… until now. I recently attempted to add a new Windows 2008r2 file server to our DFS replication group/namespace. This totally failed for some mysterious reason. Well, I shouldn’t say “totally” failed, as I was able to add it to the DFS replication group, but unable to add it to the DFS namespace. In my attempt to debug the namespace issue, I deleted the namespace and attempted to recreate it, but just kept getting this error: The namespace cannot be queried. The specified domain either does not exist or could not be contacted.. I couldn’t do anything with the namespace – even clicking on it in the DFS Management console brought up an error. After some searching I found that this was likely due to having a single-label domain. I wasn’t sure why the error was happening even on Windows 2003 machines though, maybe joining a 2008r2 box to the domain made some schema changes? I tried a few suggestions like editing the hosts file but nothing seemed to resolve this.

Fortunately, we didn’t really need DFS namespaces and were able to just direct everybody to the fileserver via its DNS name, though as you can imagine this was clumsy. However, since this has been a problem since time immemorial, I figured it was time to see if it was fixable. After some quick searching, I found RENDOM. However, after even more searching I discovered this TechNet article which says:

The domain rename operation is not supported in Microsoft Exchange Server 2007 or Exchange Server 2010. DNS domain rename is supported in Exchange Server 2003. However, renaming of the NetBIOS domain name is not supported in any version of Exchange Server. Other non-Microsoft applications might also not support domain rename.

Well. We’re running Exchange 2010. So now what? I guess we’re going to have to create a second domain and migrate over to it. We’d already discussed this as a likely way of implementing the rename anyway, since it didn’t seem like “RENDOM” had any rollback procedure – it either just works (hahaha) or semi-works and semi-fails, leaving a wake of destruction throughout AD. Building a second domain seems like a lot of work, but at least we can move users over one at a time, and we get the side benefit of starting fresh, outgrowing the 5+ years of crud that’s accumulated in our AD.

Guess we’ll see what happens. Neither option seems like much fun. I guess the alternative is do nothing, but Microsoft clearly doesn’t think very highly of single-label domains, and anyone who asks about them gets looked at funny. At least it gives us something to do!

Now that everyone’s been moved to Exchange 2010 we’ve started using the 2010 Exchange Managment Console/Shell exclusively which has revealed some weirdness. First, we created a new group in AD using an old script (which used LDAP) and created a Mail-enabled Global Security group. We put people in the group, and everything seemed to be working fine until it was discovered that users in the group couldn’t see the group in the Global Address List. Users not in the group had no problem seeing the group. Additionally, users in the group couldn’t see users added directly in 2010. This only appeared to affect the GAL; the users were able to send/receive email fine with the full SMTP addresses.

My first guess was that I was being punished for having forgotten to upgrade the LDAP address lists to OPATH. I don’t really know what that even means, but when I attempted to edit the address lists in EMC I’d get an error that they needed to be upgraded. Fortunately, this Technet article lists the commands needed to upgrade the lists. I did it but this didn’t appear to resolve all the issues.

At this point, after some Googling, I came across this tidbit:

If you’re moving from Exchange 2000/2003 to Exchange 2007 or Exchange 2010, you’re going to want to convert all your domain local and global distribution and mail-enabled security groups to universal groups so they can be managed using the Exchange management tools.

This explains a few things we’ve noticed – inability to add Global (Non-Universal) groups to a newly created (Universal) group, for one. So it appears what we should do is upgrade all the Global groups to Universal. First, how do we get a list of all the Global groups? EMS/PowerShell to the rescue:

[PS] C:\Windows\system32>Get-Group  | Where {$_.GroupType -Like "Global*"  -AND $_.RecipientType -eq "MailNonUniversalGroup"} |
Export-Csv -encoding "utf8" -Path \\fileserver\Tech\groups1.csv

You can refine the filter further, and when it looks correct you can just pipe the output to Set-Group:

Get-Group  | Where {$_.GroupType -Like "Global*"  -AND $_.RecipientType -eq "MailNonUniversalGroup"} | Set-Group -Universal

But now for the most important question: will this break anything? I have no idea. We only have a single domain in our AD forest so we’ve never had need to use Universal, and I don’t think there should be a problem, but I don’t really have any idea.

I ran the Get-Group/Set-Group commands and they seemed to work as intended for all but about 60 of the target groups. The groups that didn’t get converted all had weird issues – aliases that contained illegal characters (which I fixed), or some of them complained that a particular user (I think the Owner of the group in AD) was not found (even though it was in the exact location it was saying it wasn’t, though the user was disabled). I “manually” converted these groups to Universal via the radio button in the properties dialog in Active Dir Users & Groups. Not the most elegant solution but it worked. So all the groups in question are now Universal Security groups. Will this solve the problem? Well, I’ll have to wait until tomorrow to find out.

Reference links:

• Need to convert Global groups to Universal groups? Do you have messages to global groups disappearing?
• Changing Group Type via PowerShell
• How to Convert Local and Global Groups to Universal Groups
• Forum post about the “BypassSecurityGroupManagerCheck” security error
• Exchange Server 2007 and Universal Groups
• KB 231273: Group Type and Scope Usage in Windows
• Mail Universal Groups vs Mail Non-Universal Groups on Experts Exchange
• Technet blog: Recipients List
• Technet forum: Mail enabled distribution groups in AD

Just putting this here for safekeeping since I couldn’t remember the exact syntax.

[evan@ehoffman 10:35:50 ~]$ ldapsearch -x -LLL -D "ldapuser@example.com" -w password -b "OU=Users,DC=example,DC=com" -s sub -H ldaps://activedirectory.example.com "(sn=hoffman)" cn mail displayName samaccountname
dn: CN=Evan Hoffman,OU=Tech,OU=Users,DC=example,DC=com
cn: Evan Hoffman
displayName: Evan D. Hoffman
sAMAccountName: ehoffman
mail: Evan.Hoffman@example.com

Explanation: Connect to activedirectory.example.com using ldaps (SSL) with simple authentication, binding as ldapuser@example.com with password password; search for (sn=hoffman) within the OU=Users,DC=example,DC=com search base (branch), and search the subtree. Return the cn, displayName, and samaccountname fields.

Refer to the ldapsearch man page for more options.

A few months ago I was on a quest to figure out how to change my Active Directory password via a browser (for Linux/Mac users). I finally figured it out, but since I’ve been working on this Exchange 2010 migration I noticed one of the features of OWA (Outlook Web App) in Exchange 2010 is that you can change the AD password right in the browser from within the app:

The new OWA has a zillion other awesome features, my favorite being that Firefox and Chrome are no longer second-class-citizens and can use the “full version” now, even on Linux. So anyway, I guess all my work was for nothing. Not the first time (or the last).

I found a script a few months ago that generated a CSV report of mailbox size, which included the Mailbox Name (usually the user’s name), size in Kbytes, number of items, which server it’s on, etc. This was very helpful, but I wanted to see which department within the company used the most space on the mail server, and the department wasn’t one of the pieces of data included in the report. It took a while but I figured out how to do LDAP lookups in vbscript and was able to add that info, so the report now has the user’s department, office location, and quota limit in it as well as the other fields. This makes it very easy to do a PivotChart in Excel to generate a pie chart of the size by department. The script is attached – change the extension to .vbs to run it. You’ll need to plug in your Exchange server and domain controller where the placeholders currently are.

EmailSizeByDepartment.vbs

I had to give up on PHP and go to Perl, but it turned out not to be so bad. Users can now change their Active Directory passwords via a self-service web page that doesn’t require admin credentials. The Perl code is below.  Authentication to the script is done via .htaccess LDAP authentication, so the REMOTE_USER env variable is assumed to contain the user’s username (sAMAccountName) by the time this script is called.  There is a simple check for $ENV{HTTPS} to ensure the script is called via SSL, and AD requires password changes to be done via ldaps, so the whole thing should be encrypted end to end.

(Edited 5/14/2010 to replace the inlined Perl script with a link to the script as a text file.)

(Edited 10/6/2011 to replace link to script with link to gist)

So I got everything working with .htaccess and AD/LDAP authentication. Just add LDAPVerifyServerCert Off to the httpd config to let Apache authenticate against an AD server with a self-signed certificate (without dealing with the annoyance of putting the cert on each Apache server).

With that piece of the puzzle largely solved, I moved on to another: how will users change their passwords (which are all stored in Active Directory)? For users running Windows this is pretty trivial — they can do it right in Windows when they’re logged into the domain. But what about Linux users? I figured the easiest thing to do would be to make a web form to do this. The user would login (with the http/LDAP auth I previously setup) and the form would ask for their password (twice) and update it in Active Directory. Sounds pretty simple to me. I think if this were OpenLDAP it probably would be, but being AD, it’s not.

I’d already spent an hour or two writing the script (in PHP) when I was able to test its basic functionality. What I got was this:

Warning: ldap_mod_replace() [function.ldap-mod-replace]: Modify: Insufficient access in /home/evan/public_html/authtest/index.php

After some hair pulling I realized that I was binding with my “dummy” user when attempting to change the password. I figured the solution would be to re-bind as the user whose password I was attempting to change, which is what I did. I verified that the new bind worked, but I still couldn’t change the password. I decided to fall back to command line and started issuing some ldapmodify commands to see what I could and couldn’t do as the user. For whatever reason, it appears that even though the user CAN change his password in AD (the “user cannot change password” setting is NOT selected — I checked), the user cannot change his password through LDAP.

$ ldapmodify -vv -x -d8 -D "CN=Boba Fett,OU=Utility,OU=Users,DC=example,DC=com" -w secret -H ldaps://activedirectory.example.com -f bfett.ldif
ldap_initialize( ldaps://activedirectory.example.com )
TLS certificate verification: Error, unable to get local issuer certificate
request done: ld 0x9dd86e8 msgid 1
replace unicodePwd:
        "
modifying entry "CN=Boba Fett,OU=Utility,OU=Users,DC=example,DC=com"
modify complete
request done: ld 0x9dd86e8 msgid 2
ldapmodify: Insufficient access (50)
        additional info: 00000005: SecErr: DSID-031A0F44, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

I Googled for the error code – DSID-031A0F44 – and found a couple of threads about people with the same problem, but no solutions.

The LDIF file I fed in looks like this:

dn: CN=Boba Fett,OU=Utility,OU=Users,DC=example,DC=com
changetype: modify
replace: unicodePwd
unicodePwd::IgBhAGIAYwBkAGUAZgBnAGgAIgA=
-

When I run the same command with my own DN/password for binding it works fine (though I did discover that you can assign multiple values to unicodePwd, meaning that, until I fixed it, the test user had two valid passwords, which seems like a bug on Microsoft’s part), I assume because I’m a domain admin. An “easy” out would be simply to have the script bind as a domain admin, but that means I’d be hardcoding a Domain Admin password in the script, which I’m against. I’ll have to put some more thought into this. Maybe Linux users will just have to log in to a Windows workstation to change their passwords. That’s not ideal, but it does already work.

Grr…

Update – Click the “ldap” tag below to see all the LDAP-related posts, including the solution to changing AD passwords via a browser.

Following up on my previous post, it turned out not to be as big of a deal as I’d originally expected to have Apache authenticate against AD and only allow users whose accounts weren’t disabled. In a nutshell, here’s what I did:

In your .htaccess file:

AuthBasicProvider ldap
AuthType basic
AuthName "AD LDAP Test"
AuthLDAPURL     "ldap://activedirectory.example.com/OU=Users,DC=example,DC=com?sAMAccountName?sub?(!(userAccountControl:1.2.840.113556.1.4.803:=2))"
AuthzLDAPAuthoritative On
AuthLDAPGroupAttribute member
AuthLDAPBindDN ldapuser@example.com
AuthLDAPBindPassword password
Require ldap-group CN=Sysadmins,OU=Internal Groups,OU=Groups,DC=example,DC=com

The key here is this LDAP filter: (!(userAccountControl:1.2.840.113556.1.4.803:=2)). This is the bitwise “AND” of the userAccountControl field and the decimal number 2, which is Microsoft’s value for “account is disabled.” The codes are listed here: http://support.microsoft.com/kb/305144

In httpd.conf (or some other server config file – I did it in /etc/httpd/conf.d/mod_authz_ldap.conf inside the section), add this directive:

LDAPOpCacheEntries 0

This tells Apache not to cache the results of the LDAP op. If you don’t put this in there, the server will cache the result of the user’s login for whatever the TTL is, and the user will be able to login even after you disable the account (until the cache expires). There may be other ways around this issue, but this works for me.

This works pretty well so far. Now I can create a “SVN Users” group in Active Directory, put the people I want in that group, use the above method for authentication and everyone’s SVN login will be the same as their domain login. Single sign-on one step closer. Yay!

One project we’ve been working on for a while is single sign-on across all our servers and other services (e.g. SVN repository, a few other things). One thing I wanted to avoid, I guess for mostly religious reasons, was reliance on a Windows instance for any of our production environment. The logical part of my brain knows that people build huge websites with Windows farms and AD, but my gut still doesn’t trust it. So what I wanted to do was setup OpenLDAP as a “slave” to an Active Directory “master” and have all the LDAP info propagate over the slave whenever any changes were made in the master. I’ve done this with DNS – setup Bind as a slave to an AD server and everything basically works as I expect in a Bind-Bind master/slave scenario. Well, it turns out that it doesn’t work like that when it comes to LDAP. Apparently AD doesn’t follow the RFC for LDAP (surprise!) so many things that would be expected to work with OpenLDAP won’t.

I thought to myself, well, I can just go ahead and write something that will sync AD -> OpenLDAP every 5 minutes. This would mostly work, except there’s apparently no way to retrieve the user’s password via LDAP – the field is write-only. This makes it so I could essentially clone the whole LDAP tree… except for the piece of info that would let people login, which is the only thing I want it for. This was frustrating.

I decided to explore a caching proxy. I was thinking if I could setup OpenLDAP as a caching proxy to Active Directory, and set a cache time of ~15 minutes, it would at least let me withstand a Windows reboot (it’s 2009 and I still feel like any Windows fix begins and ends with a reboot). I figured I’d start with a regular proxy first. This turned out to be relatively simple: in slapd.conf, at the bottom, add:

database ldap
suffix          "dc=example,dc=com"
uri "ldap://activedirectory.example.com"
acl-bind bindmethod=simple binddn="some user's DN" credentials=password

Restart slapd and do an ldapsearch against the new ldap server and it will relay the request to the AD server and relay the response to you.

Caching is another story. It seemed like it should be straightforward – yum install openldap-servers-overlays – but on my FC11 box, the package didn’t exist, and on my CentOS 5.4 box, openldap-servers-overlays didn’t appear to contain the pcache overlay used for caching. So I gave up on that.

I ended up shelving the whole idea for a while and just created 2 Windows AD VMs that will serve as Production AD auth boxes. I got everything configured – here’s a sample .htaccess file that queries our AD server for a user (by sAMAccountName, e.g. the “user” part of a username in the user@domain login name) and checks that the user is in the Sysadmins group:

AuthBasicProvider ldap
AuthType basic
AuthName "AD LDAP Test"
AuthLDAPURL     "ldap://activedirectory.example.com/OU=Users,DC=example,DC=com?sAMAccountName?sub"
AuthzLDAPAuthoritative On
AuthLDAPGroupAttribute member
AuthLDAPBindDN ldapuser@example.com
AuthLDAPBindPassword password
Require ldap-group CN=Sysadmins,OU=Internal Groups,OU=Groups,DC=example,DC=com

This works (I’ve modified it so it doesn’t include our real info, though). It will let the user in if their credentials are OK and if they are in the required group. What I discovered, however, is that if you go into AD and disable the user’s account, they are still allowed to log in (assuming the user is still in the account). This seemed stupid. I did some more research and discovered userAccountControl. Apparently you need to use bit masking against this attribute to determine if an account is disabled. This is as far as I’d gotten before vacation, but let me say this:

BRILLIANT, MICROSOFT!