Edit Jan 24, 2012: Deleted all the crap from this story and just left the recommended Apache and Nginx SSL cipher suites for maximum security without SSLv2 and without BEAST vulnerability (at least according to Qualys).

Apache httpd

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
SSLHonorCipherOrder on

nginx

        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;

Source:

• Qualys
• SSL checker

RT has its own internal accounting & tracking system for logging activity, but I was interested in even more granular stuff, like seeing who looked at which tickets. I figured it wouldn’t be that hard to log this in Apache. Well, I was kind of right, in that it wasn’t “hard,” but it took me a long time to find the right place to do it. I did finally get it though.

httpd.conf

In httpd.conf I created a new LogFormat:

LogFormat "%h %l %{RTUSER}e %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined-rt

So instead of the HTTP-auth user, it puts the RT user in this field. Make sure to update your VirtualHost config to use the combined-rt LogFormat.

/usr/share/rt3/html/autohandler

In /usr/share/rt3/html/autohandler, right under this section:

# If we've got credentials, let's serve the file up.
if (    ( defined $session{'CurrentUser'} )
    and ( $session{'CurrentUser'}->Id ) )
{

Add this:

        $ENV{'RTUSER'} = $session{'CurrentUser'}->UserObj->EmailAddress;

This populates the RTUSER environment variable with the currently-logged-in user’s email address.

Restart httpd and the RT user’s email address should now appear in access_log. Note that it will only appear for Perl pages (not gifs/jpgs or other static content, since Perl doesn’t process those):

10.0.0.10 - evan@example.com [08/Aug/2011:17:30:34 -0400] "GET /rt3/index.html HTTP/1.1" 200 46809 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"
10.0.0.10 - - [08/Aug/2011:17:30:34 -0400] "GET /rt3/NoAuth/images//css/rolldown-arrow.gif HTTP/1.1" 200 83 "https://help.example.com/rt3/NoAuth/css/3.5-default/main-squished.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"

Putting this here for safekeeping so my future self can find it. Mod_rewrite is one of my favorite tools, but it’s easy to spend 30 minutes crafting a 2-line directive that actually does what you want. I put this in a .htaccess file in the DocumentRoot of the server, put a “We’re down” message in maintenance.html (or whatever), and all requests will get a 302 redirect to /maintenance.html, except requests for /maintenance.html (for obvious reasons). It appends the original request in case you want to do something with it but that’s not really important. It also doesn’t do the redirect for images/js/css so those can actually be used in the maintenance message.

RewriteEngine on
RewriteRule ^maintenance.html	-	[PT,L]
 
RewriteCond %{REQUEST_URI}	!(gif|jpg|css|js)$
RewriteRule (.*) /maintenance.html?redir=true&originalRequest=%{REQUEST_URI} [R=302]

I never like typing a password into a non-SSL site, no matter how trivial it is. In order to give my own site this ability I simply used mod_rewrite to force requests to WordPress’s admin pages to go over SSL.

The .htaccess file for the site looks like this:

# BEGIN WordPress
<ifModule mod_rewrite.c>
RewriteEngine On
RewriteBase /evan/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /evan/index.php [L]
</ifModule>
 
# END WordPress

To force the admin pages to SSL, just add these lines under RewriteEngine On:

RewriteCond %{HTTPS} !=on
RewriteRule ^wp-(.*)$ https://%{HTTP_HOST}%{REQUEST_URI}      [R,L]

Edit – The above code screws up uploads (which go into the /wp-content directory). I replaced that with the following and it Worked As Intended.

RewriteCond %{HTTPS} !=on
RewriteRule ^wp-login(.*)$ https://%{HTTP_HOST}%{REQUEST_URI}      [R,L]
RewriteCond %{HTTPS} !=on
RewriteRule ^wp-admin(.*)$ https://%{HTTP_HOST}%{REQUEST_URI}      [R,L]

That’s pretty much it. If your request starts with “wp-” it’ll redirect you to the same URL, but starting with https://. Problem solved. You do need to make sure you have an SSL VirtualHost pointing to your WordPress DocumentRoot so that https://yoursite.com goes to the same place as http://yoursite.com.

I use Akismet to block comment spam, but it still annoys me that it even exists. Last night I put a simple IP ban into my httpd config. But who to block?

I used a grep & Perl to get a rough guess of which IPs were submitting the most comments (working on the assumption that one IP address submits many spam comments) It took me about 20 minutes to write this mess but it does what I wanted to do:

[root@lunix ~]# zgrep POST /var/log/httpd/evanhoffman-access_log-201008??.gz | grep comment | perl -ne 'chomp; $_ =~ m/(?:\d{1,3}\.){3}\d{1,3}/; print "$&\n";' | perl -e '%a = (); while (<>) { chomp; $a{$_} += 1; } while (my ($key, $value) = each (%a)) { if ($value > 1) { print "$value\t=>\t$key\n";}}'
2 => 218.6.9.140
180 => 91.201.66.34
2 => 213.5.67.41
2 => 188.187.102.74
[root@lunix ~]#


That’s pretty hard to read. Here’s a quick explanation of each piece:

zgrep POST /var/log/httpd/evanhoffman-access_log-201008??.gz

Use zgrep to search for the string “POST” in all of the gzipped Apache logs for August. Pipe the results (the matching lines) to the next part:

grep comment

grep for the string “comment”. This isn’t really scientific, but I feel safe in assuming that if “POST” and “comment” both appear in the HTTP request, it’s probably someone posting a comment. Pipe the matches to…

perl -ne ‘chomp; $_ =~ m/(?:\d{1,3}\.){3}\d{1,3}/; print “$&\n”;’

This is a perl one-liner that uses a regular expression to match an IP address in a given line and print it out. The original regex I used was \d+\.\d+\.\d+\.\d+, this one was slightly fancier but did the same work in this case. It’s worth noting that this will only print out the first match in the given line, but since the requester’s IP (REMOTE_ADDR) is the first field in Combined Log Format, that’s fine this case.

The output (the IPs from which comment posts have been made) is piped to…

perl -e ‘%a = (); while (<>) { chomp; $a{$_} += 1; } while (my ($key, $value) = each (%a)) { if ($value > 1) { print “$value\t=>\t$key\n”;}}’

This is another perl one-liner. Basically, it maintains a hash of String=>count pairs, so each time it sees a string it increments a “counter” for that line. Then when it’s done receiving input (i.e. all the data has been processed) it prints out the contents of the hash for keys that have a value > 1 (i.e. IPs that have POSTed more than 1 comment).

The output shows pretty clearly where the spam is coming from:

2 => 218.6.9.140
180 => 91.201.66.34
2 => 213.5.67.41
2 => 188.187.102.74

180 submits from 91.201.66.34. Out of curiosity I looked up that IP in whois:

[root@lunix ~]# whois 91.201.66.34
[Querying whois.ripe.net]
[whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '91.201.64.0 - 91.201.67.255'

inetnum:        91.201.64.0 - 91.201.67.255
netname:        Donekoserv
descr:          DonEkoService Ltd
country:        RU
org:            ORG-DS41-RIPE
admin-c:        MNV32-RIPE
tech-c:         MNV32-RIPE
status:         ASSIGNED PI
mnt-by:         RIPE-NCC-END-MNT
mnt-by:         MNT-DONECO
mnt-by:         MNT-DONECO
mnt-lower:      RIPE-NCC-END-MNT
mnt-routes:     MHOST-MNT
mnt-routes:     MNT-PIN
mnt-domains:    MHOST-MNT
source:         RIPE # Filtered

organisation:   ORG-DS41-RIPE
org-name:       DonEko Service
org-type:       OTHER
address:        novocherkassk, ul stremyannaya d.6
e-mail:         admin@pinspb.ru
mnt-ref:        MNT-PIN
mnt-by:         MNT-PIN
source:         RIPE # Filtered

person:         Metluk Nikolay Valeryevich
address:        korp. 1a 40 Slavy ave.,
address:        St.-Petersburg, Russia
e-mail:         nm@internet-spb.ru
phone:          +7 812 4483863
fax-no:         +7 901 3149449
nic-hdl:        MNV32-RIPE
mnt-by:         MNT-PIN
source:         RIPE # Filtered

% Information related to '91.201.66.0/23AS21098'

route:          91.201.66.0/23
descr:          Route MHOST IDC
origin:         AS21098
mnt-by:         MHOST-MNT
source:         RIPE # Filtered

[root@lunix ~]#

Not much info other than the IP is based in Russia. Well, anyway, I IP blocked 91.0.0.0/8 (sorry, Russia), so if you’re in that subnet you’re probably seeing a 403 now.

Edit: It occurred to me that I can accomplish the same thing while being less draconian if I wrap the Deny in a <Limit></Limit> clause. This way everyone can still see the site but certain IP ranges won’t be able to POST anything:

<Limit POST PUT DELETE>
Order Allow,Deny
Allow from all
Deny from 218.6.9.
Deny from 173.203.101.
Deny from 122.162.28.
Deny from 91.
Deny from 213.5
</Limit>

So I got everything working with .htaccess and AD/LDAP authentication. Just add LDAPVerifyServerCert Off to the httpd config to let Apache authenticate against an AD server with a self-signed certificate (without dealing with the annoyance of putting the cert on each Apache server).

With that piece of the puzzle largely solved, I moved on to another: how will users change their passwords (which are all stored in Active Directory)? For users running Windows this is pretty trivial — they can do it right in Windows when they’re logged into the domain. But what about Linux users? I figured the easiest thing to do would be to make a web form to do this. The user would login (with the http/LDAP auth I previously setup) and the form would ask for their password (twice) and update it in Active Directory. Sounds pretty simple to me. I think if this were OpenLDAP it probably would be, but being AD, it’s not.

I’d already spent an hour or two writing the script (in PHP) when I was able to test its basic functionality. What I got was this:

Warning: ldap_mod_replace() [function.ldap-mod-replace]: Modify: Insufficient access in /home/evan/public_html/authtest/index.php

After some hair pulling I realized that I was binding with my “dummy” user when attempting to change the password. I figured the solution would be to re-bind as the user whose password I was attempting to change, which is what I did. I verified that the new bind worked, but I still couldn’t change the password. I decided to fall back to command line and started issuing some ldapmodify commands to see what I could and couldn’t do as the user. For whatever reason, it appears that even though the user CAN change his password in AD (the “user cannot change password” setting is NOT selected — I checked), the user cannot change his password through LDAP.

$ ldapmodify -vv -x -d8 -D "CN=Boba Fett,OU=Utility,OU=Users,DC=example,DC=com" -w secret -H ldaps://activedirectory.example.com -f bfett.ldif
ldap_initialize( ldaps://activedirectory.example.com )
TLS certificate verification: Error, unable to get local issuer certificate
request done: ld 0x9dd86e8 msgid 1
replace unicodePwd:
        "
modifying entry "CN=Boba Fett,OU=Utility,OU=Users,DC=example,DC=com"
modify complete
request done: ld 0x9dd86e8 msgid 2
ldapmodify: Insufficient access (50)
        additional info: 00000005: SecErr: DSID-031A0F44, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0

I Googled for the error code – DSID-031A0F44 – and found a couple of threads about people with the same problem, but no solutions.

The LDIF file I fed in looks like this:

dn: CN=Boba Fett,OU=Utility,OU=Users,DC=example,DC=com
changetype: modify
replace: unicodePwd
unicodePwd::IgBhAGIAYwBkAGUAZgBnAGgAIgA=
-

When I run the same command with my own DN/password for binding it works fine (though I did discover that you can assign multiple values to unicodePwd, meaning that, until I fixed it, the test user had two valid passwords, which seems like a bug on Microsoft’s part), I assume because I’m a domain admin. An “easy” out would be simply to have the script bind as a domain admin, but that means I’d be hardcoding a Domain Admin password in the script, which I’m against. I’ll have to put some more thought into this. Maybe Linux users will just have to log in to a Windows workstation to change their passwords. That’s not ideal, but it does already work.

Grr…

Update – Click the “ldap” tag below to see all the LDAP-related posts, including the solution to changing AD passwords via a browser.

I was about to write a script using OpenSSL to monitor the SSL certificate expiration dates for a few servers when it dawned on me that someone had probably done this already.

The ssl-cert-check shell script takes a list of hosts/ports and prints out the expiration date (and how many days away that date is). It can also be configured to email a message for any certificates expiring in less than N days for easy cronibility (?). Seems lovely!