We wanted to setup a loadbalanced web cluster in AWS for expansion. My first inclination was to use ELB for this, but I soon learned that ELB doesn’t let you allocate a static IP, requiring you to refer to it only by DNS name. This would be OK except for the fact that our current DNS provider, Dyn, requires IP addresses when using their GSLB (geo-based load balancer) service.

Rather than let this derail the whole project, I decided to look into the software options available for loadbalancing in EC2. I’ve been a fan of hardware load balancers for a while, sort of looking down at software-based solutions without any real rationale, but in this case I really had no choice so I figured I’d give it a try.

My first stop was Nginx. I’ve used it before in a reverse-proxy scenario and like it. The problem I had with it was that it doesn’t support active polling of nodes – the ability to send requests to the webserver and mark the node as up or down based on the response. As far as I can tell, using multiple upstream servers in Nginx allows you to specify max_fails and fail_timeout, however a “fail” is determined when a real request comes in. I don’t want to risk losing a real request – I like active polling.

This led me to HAProxy. I’d never used HAProxy before but it seemed to be ideally suited to this (since it’s exclusively a load balancer). The option httpchk even allows for active polling of nodes – yay!

Unfortunately, HAProxy doesn’t support SSL. From the HAProxy site:

People often ask for SSL and Keep-Alive support. Both features will complicate the code and render it fragile for several releases. By the way, both features have a negative impact on performance :

Having SSL in the load balancer itself means that it becomes the bottleneck. When the load balancer’s CPU is saturated, the overall response times will increase and the only solution will be to multiply the load balancer with another load balancer in front of them. the only scalable solution is to have an SSL/Cache layer between the clients and the load balancer. Anyway for small sites it still makes sense to embed SSL, and it’s currently being studied. There has been some work on the CyaSSL library to ease integration with HAProxy, as it appears to be the only one out there to let you manage your memory yourself.

Poop! I figured out a workaround however, by using both Nginx and HAProxy on the same instance. HAProxy listens on port 80 and 8443 (so that it can relay decrypted SSL traffic to the nodes on a separate port, so that the nodes are aware that it was originally SSL traffic). Nginx is configured as a reverse proxy, listens on port 443 only, and has the SSL cert & key. The upstream for the Nginx is just localhost:8443 – HAProxy.

This was pretty easy to setup and works very well. I benchmarked HAProxy on an EC2 t1.micro instance (in front of two m1.large instances running our webapp) using ab -n 5000 -c 50 -t 60 and found it actually performed better than one of our hardware load balancers. That was pretty eye-opening (and sad).

The HAProxy and Nginx configs are below, in the hopes that it helps someone. The main warning I’d give is that using this will cause the logs on your nodes to interpret all requests as coming from the IP of the load balancer. I had to rewrite some code to have the app use the X-Forwarded-For address rather than the REMOTE_ADDR, but other than that this has been working out pretty well.

/etc/nginx/nginx.conf
Main thing is to make sure the server isn’t listening on port 80 (since HAProxy needs to).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

user              nginx;
worker_processes  1;
 
error_log  /var/log/nginx/error.log;
 
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    keepalive_timeout  65;
 
    #
    # The default server
    #
    server {
        listen       81;
        server_name  _;
 
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
 
        error_page  404              /404.html;
        location = /404.html {
            root   /usr/share/nginx/html;
        }
 
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
 
    }
 
    # Load config files from the /etc/nginx/conf.d directory
    include /etc/nginx/conf.d/*.conf;
 
}

/etc/nginx/conf.d/ssl-offloader.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

upstream haproxy {
        server localhost:8443 ;
}
 
server {
        listen       443;
        server_name f.q.d.n 1.2.3.4 ; # I put the FQDN and IP here, but maybe "_" will work too
#  server_name  _;
 
        ssl                  on;
        ssl_certificate      /etc/nginx/ssl-cert/cert.pem;
        ssl_certificate_key  /etc/nginx/ssl-cert/cert.key;
 
        ssl_session_timeout  5m;
 
        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;
 
        location / {
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
 
                proxy_pass http://haproxy/;
                proxy_redirect default;
                proxy_redirect http://$host/ https://$host/;
                proxy_redirect http://hostname/ https://$host/;
 
                proxy_read_timeout 15s;
                proxy_connect_timeout 15s;
        }
 
}

/etc/haproxy/haproxy.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2
 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
 
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
 
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    3s
    timeout queue           1m
    timeout connect         2s
    timeout client          5s
    timeout server          5s
    timeout http-keep-alive 1s
    timeout check           10s
    maxconn                 3000
 
       stats enable
       stats auth evan:change_me_brother
 
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main_http *:80
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_http
 
frontend main_https *:8443
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_https
 
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend web_http
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:80 check port 80
    server  node2 192.168.1.30:80 check port 80
    server  node3 192.168.1.40:80 check port 80
 
 
backend web_https
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:8443 check port 8443
    server  node2 192.168.1.30:8443 check port 8443
    server  node3 192.168.1.40:8443 check port 8443

Edit Jan 24, 2012: Deleted all the crap from this story and just left the recommended Apache and Nginx SSL cipher suites for maximum security without SSLv2 and without BEAST vulnerability (at least according to Qualys).

Apache httpd

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
SSLHonorCipherOrder on

nginx

        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;

Source:

• Qualys
• SSL checker

I’m setting up SquirrelMail to point to my Exchange 2010 server via IMAP (don’t ask) and couldn’t get SM to talk to Exchange on port 993 (imaps). Even though the servers on the same subnet, any time passwords are being sent over the network I like to opt for SSL. I found a couple of sites suggesting that the problem was that there was no SSL certificate installed, but I knew for a fact there was a valid certificate because I could get to https://webmail.example.com/ for OWA.

Some of the errors SquirrelMail was reporting were “Error connecting to IMAP server xxxx Server error: (0)” and “Error connecting to IMAP server: tls://xxxx:993. 0: ”

Nothing would actually work on port 993. Telnet to 993 got this:

$ telnet 10.0.20.18 993
Trying 10.0.20.18...
Connected to 10.0.20.18.
Escape character is '^]'.
* BYE Connection is closed. 14
Connection closed by foreign host.

After too much poking, I decided to go down to a lower level and do a simple openssl certificate retrieval and see what came back:

$ openssl s_client -connect 10.0.20.18:993
CONNECTED(00000003)
140281653434184:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:699:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 113 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

That didn’t look right, so I ran it against the same server on port 443 and got back a real certificate. Same for port 995 (pop3s):

$ openssl s_client -connect 10.0.20.18:443
CONNECTED(00000003)
depth=3 L = ValiCert Validation Network, O = "ValiCert, Inc.", OU = ValiCert Class 2 Policy Validation Authority, CN = http://www.valicert.com/, emailAddress = info@valicert.com
verify return:1
depth=2 C = US, O = "The Go Daddy Group, Inc.", OU = Go Daddy Class 2 Certification Authority
verify return:1

(snip)

So there’s just something wrong with SSL on port 993. To make a long story short, I had to use the Enable-ExchangeCertificate to apply the SSL certificate to port 993. First, run “Get-ExchangeCertificate” to list the available certificates and retrieve the Thumbprint.

[PS] C:\Windows\system32>Get-ExchangeCertificate
 
Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  .P....     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  I..W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

Copy & paste the thumbprint for whichever cert you want to use into Enable-ExchangeCertificate:

[PS] C:\Windows\system32>Enable-ExchangeCertificate -ThumbPrint xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx -Services IIS,P
OP,IMAP -DoNotRequireSSL
[PS] C:\Windows\system32>Get-ExchangeCertificate
 
Thumbprint                                Services   Subject
----------                                --------   -------
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy  ......     CN=exch2010fe1
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  IP.W..     CN=webmail.example.com, OU=Domain Control Validated, O=webmail.ex...

After running that, imaps on port 993 worked perfectly. I can connect to it with both SquirrelMail and Thunderbird.

The SquirrelMail config looks like this:

IMAP Settings
--------------
4.  IMAP Server            : webmail.example.com
5.  IMAP Port              : 993
6.  Authentication type    : login
7.  Secure IMAP (TLS)      : true
8.  Server software        : exchange
9.  Delimiter              : detect

Edit Feb 15, 2011: I just renewed the SSL cert and ran into a problem with a Ruby script that was suddenly unable to check a mailbox over IMAPS. The error received was:

/usr/lib/ruby/1.8/net/imap.rb:898:in `connect': unknown protocol (OpenSSL::SSL::SSLError)
        from /usr/lib/ruby/1.8/net/imap.rb:898:in `initialize'

After a few minutes, I remembered this blog post and ran Enable-ExchangeCertificate and it worked again. Glad I wrote it down.

CONNECTED(00000003) 26831:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:188:

I got this idea (from this blog post) that it would be cool and great if we had an IRC server to facilitate communication within our team.  I started checking out some different IRC servers to see which ones supported the main features I wanted (LDAP auth & SSL).  I started out with ratbox but I didn’t really like it.  I found InspIRCd and after some config tweaking it’s working well.  Here’s the info from modules.conf for authentication against Active Directory (so people login with their AD usernames):

1
2
3
4
5
6
7
8
9
10

<module name="m_ldapauth.so">
   <ldapauth baserdn="OU=Users,DC=example,DC=com"
   attribute="sAMAccountName"
   server="ldap://activedirectory.example.com"
   killreason="LDAP auth failed"
   searchscope="subtree"
   binddn="CN=binduser,OU=Users,DC=example,DC=com"
   bindauth="password"
   verbose="yes"
   userfield="yes">

Here’s the configure line (since I had to build from source):

./configure --prefix=/usr/local/inspircd --enable-gnutls --uid 101 --enable-extras=m_ldapauth.cpp --enable-extras=m_ldapoper.cpp

I still have some more customization to do to make ChanServ work, but since this is internal I don’t know if that even matters.  Also I can’t seem to get SSL working – I tried compiling with –enable-openssl but that failed, so I went with –enable-gnutls, which worked, but I can’t connect via ircs:// in my client.  If it looks like people are interested in this I may fix it up, but I have a feeling most people won’t be as enthused about it as I am.

I never like typing a password into a non-SSL site, no matter how trivial it is. In order to give my own site this ability I simply used mod_rewrite to force requests to WordPress’s admin pages to go over SSL.

The .htaccess file for the site looks like this:

# BEGIN WordPress
<ifModule mod_rewrite.c>
RewriteEngine On
RewriteBase /evan/
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /evan/index.php [L]
</ifModule>
 
# END WordPress

To force the admin pages to SSL, just add these lines under RewriteEngine On:

RewriteCond %{HTTPS} !=on
RewriteRule ^wp-(.*)$ https://%{HTTP_HOST}%{REQUEST_URI}      [R,L]

Edit – The above code screws up uploads (which go into the /wp-content directory). I replaced that with the following and it Worked As Intended.

RewriteCond %{HTTPS} !=on
RewriteRule ^wp-login(.*)$ https://%{HTTP_HOST}%{REQUEST_URI}      [R,L]
RewriteCond %{HTTPS} !=on
RewriteRule ^wp-admin(.*)$ https://%{HTTP_HOST}%{REQUEST_URI}      [R,L]

That’s pretty much it. If your request starts with “wp-” it’ll redirect you to the same URL, but starting with https://. Problem solved. You do need to make sure you have an SSL VirtualHost pointing to your WordPress DocumentRoot so that https://yoursite.com goes to the same place as http://yoursite.com.

One of the more annoying side effects of migrating my mailbox to Exchange 2010 has been the nagging of Outlook 2007′s Autodiscovery feature. Now, every time I start Outlook I get hit with a certificate error for autodiscover.domain.com. Now, autodiscover.domain.com is a CNAME to mail.domain.com, which is the OWA URL for the CAS. The SSL certificate is valid – but it’s valid for mail.domain.com. I could buy a SSL certificate from GoDaddy for $12.99 (an insanely great price, btw) for “autodiscover” but that would also require using another IP address on the CAS (since you can can only bind one SSL certificate to an IP:port pair), and that seems like a waste of an IP address.

I found a possible solution in KB 940726. Basically you use this cmdlet to change the Autodiscover URI for internal clients:

Set-ClientAccessServer –AutodiscoverServiceInternalUri https://mail.contoso.com/autodiscover/autodiscover.xml

You’d replace mail.contoso.com with the external URL of your OWA server (in my case, mail.domain.com). I’ve made the changes but I think I need to wait for AD propagation. Hopefully this will resolve it, because I don’t want to move everyone’s mailboxes over until this thing is “perfect,” whatever that means.

Edit: I also needed to add a SRV record so Outlook would know what host to check for autodiscovery when outside the domain.

Edit 2:: Also need to install a hotfix or be running Outlook 2007 SP1 or later for the SRV functionality.

Edit 3: It occurs to me that a simpler fix for this issue may be simply to delete the DNS record for autodiscover entirely. That way, when Outlook attempts to open the SSL connection to autodiscover.domain.com, it gets a NXDOMAIN error (should) silently skip it. Unfortunately we have wildcard DNS active for our domain.

Other useful resources:

• MS Exchange Team blog post comparing the various autodiscover schemes.
• Set-ClientAccessServer.
• Test Exchange Connectivity
• Setting Autodiscover URL via DNS SRV record
• Autodiscover whitepaper.
• Example Autodiscover BIND record – _autodiscover._tcp.domain.com. SRV 0 0 443 webmail.domain.com.
• Debug Autodiscover by right-clicking the Outlook icon in the system tray while holding down Ctrl
• Verifying SRV records exist with nslookup
• What version of Outlook am I running? You need SP1 or later for the SRV hack.
• Hotfix for Outlook 2007 (pre-SP1) to use SRV records for autodiscovery

I was about to write a script using OpenSSL to monitor the SSL certificate expiration dates for a few servers when it dawned on me that someone had probably done this already.

The ssl-cert-check shell script takes a list of hosts/ports and prints out the expiration date (and how many days away that date is). It can also be configured to email a message for any certificates expiring in less than N days for easy cronibility (?). Seems lovely!