Looking for the paradigm » work

Load balancing in EC2 with Nginx and HAProxy

Evan Hoffman — Mon, 09 Jan 2012 22:00:22 +0000

We wanted to setup a loadbalanced web cluster in AWS for expansion. My first inclination was to use ELB for this, but I soon learned that ELB doesn’t let you allocate a static IP, requiring you to refer to it only by DNS name. This would be OK except for the fact that our current DNS provider, Dyn, requires IP addresses when using their GSLB (geo-based load balancer) service.

Rather than let this derail the whole project, I decided to look into the software options available for loadbalancing in EC2. I’ve been a fan of hardware load balancers for a while, sort of looking down at software-based solutions without any real rationale, but in this case I really had no choice so I figured I’d give it a try.

My first stop was Nginx. I’ve used it before in a reverse-proxy scenario and like it. The problem I had with it was that it doesn’t support active polling of nodes – the ability to send requests to the webserver and mark the node as up or down based on the response. As far as I can tell, using multiple upstream servers in Nginx allows you to specify max_fails and fail_timeout, however a “fail” is determined when a real request comes in. I don’t want to risk losing a real request – I like active polling.

This led me to HAProxy. I’d never used HAProxy before but it seemed to be ideally suited to this (since it’s exclusively a load balancer). The option httpchk even allows for active polling of nodes – yay!

Unfortunately, HAProxy doesn’t support SSL. From the HAProxy site:

People often ask for SSL and Keep-Alive support. Both features will complicate the code and render it fragile for several releases. By the way, both features have a negative impact on performance :

Having SSL in the load balancer itself means that it becomes the bottleneck. When the load balancer’s CPU is saturated, the overall response times will increase and the only solution will be to multiply the load balancer with another load balancer in front of them. the only scalable solution is to have an SSL/Cache layer between the clients and the load balancer. Anyway for small sites it still makes sense to embed SSL, and it’s currently being studied. There has been some work on the CyaSSL library to ease integration with HAProxy, as it appears to be the only one out there to let you manage your memory yourself.

Poop! I figured out a workaround however, by using both Nginx and HAProxy on the same instance. HAProxy listens on port 80 and 8443 (so that it can relay decrypted SSL traffic to the nodes on a separate port, so that the nodes are aware that it was originally SSL traffic). Nginx is configured as a reverse proxy, listens on port 443 only, and has the SSL cert & key. The upstream for the Nginx is just localhost:8443 – HAProxy.

This was pretty easy to setup and works very well. I benchmarked HAProxy on an EC2 t1.micro instance (in front of two m1.large instances running our webapp) using ab -n 5000 -c 50 -t 60 and found it actually performed better than one of our hardware load balancers. That was pretty eye-opening (and sad).

The HAProxy and Nginx configs are below, in the hopes that it helps someone. The main warning I’d give is that using this will cause the logs on your nodes to interpret all requests as coming from the IP of the load balancer. I had to rewrite some code to have the app use the X-Forwarded-For address rather than the REMOTE_ADDR, but other than that this has been working out pretty well.

/etc/nginx/nginx.conf
Main thing is to make sure the server isn’t listening on port 80 (since HAProxy needs to).

user              nginx;
worker_processes  1;
 
error_log  /var/log/nginx/error.log;
 
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    keepalive_timeout  65;
 
    #
    # The default server
    #
    server {
        listen       81;
        server_name  _;
 
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
 
        error_page  404              /404.html;
        location = /404.html {
            root   /usr/share/nginx/html;
        }
 
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
 
    }
 
    # Load config files from the /etc/nginx/conf.d directory
    include /etc/nginx/conf.d/*.conf;
 
}

/etc/nginx/conf.d/ssl-offloader.conf

upstream haproxy {
        server localhost:8443 ;
}
 
server {
        listen       443;
        server_name f.q.d.n 1.2.3.4 ; # I put the FQDN and IP here, but maybe "_" will work too
#  server_name  _;
 
        ssl                  on;
        ssl_certificate      /etc/nginx/ssl-cert/cert.pem;
        ssl_certificate_key  /etc/nginx/ssl-cert/cert.key;
 
        ssl_session_timeout  5m;
 
        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;
 
        location / {
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
 
                proxy_pass http://haproxy/;
                proxy_redirect default;
                proxy_redirect http://$host/ https://$host/;
                proxy_redirect http://hostname/ https://$host/;
 
                proxy_read_timeout 15s;
                proxy_connect_timeout 15s;
        }
 
}

/etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2
 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
 
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
 
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    3s
    timeout queue           1m
    timeout connect         2s
    timeout client          5s
    timeout server          5s
    timeout http-keep-alive 1s
    timeout check           10s
    maxconn                 3000
 
       stats enable
       stats auth evan:change_me_brother
 
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main_http *:80
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_http
 
frontend main_https *:8443
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_https
 
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend web_http
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:80 check port 80
    server  node2 192.168.1.30:80 check port 80
    server  node3 192.168.1.40:80 check port 80
 
 
backend web_https
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:8443 check port 8443
    server  node2 192.168.1.30:8443 check port 8443
    server  node3 192.168.1.40:8443 check port 8443

Installing Sun (Oracle) JDK 1.5 on an EC2 instance

Evan Hoffman — Wed, 16 Nov 2011 16:52:55 +0000

I’m currently working on moving a Tomcat-based application into EC2. The code was written for Java 5.0. While Java 6 would probably work, I’d like to keep everything as “same” as possible, since EC2 presents its own challenges. I spun up a couple of t1.micro instances and copied everything over, including the Java 5 JDK, jdk-1_5_0_22-linux-amd64.rpm. Installing from RPM was easy, but the EC2 instance defaults to using OpenJDK 1.6:

[root@ec2 ~]# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.10) (amazon-52.1.9.10.40.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

There were a couple of things I had to do to get the system to accept the Sun JDK as its “real” java.

Alternatives

Red Hat’s “alternatives” system is designed to allow a system to have multiple versions of a program installed and make it easy to choose which one you want to run. Unfortunately I’ve found the syntax a bit strange and always have to Google it, so I figured I’d document it here for posterity.

So here’s the default:

[root@ec2 ~]# alternatives --config java
 
There is 1 program that provides 'java'.
 
  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
 
Enter to keep the current selection[+], or type selection number:

Here’s how to add Sun java, assuming the java binary is in /usr/java/jdk1.5.0_22/jre/bin/java (where the RPM puts it).

[root@ec2 ~]# alternatives --install /usr/bin/java java /usr/java/jdk1.5.0_22/jre/bin/java 1
[root@ec2 ~]# alternatives --config java
There are 2 programs which provide 'java'.
 
  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
   2           /usr/java/jdk1.5.0_22/jre/bin/java
 
Enter to keep the current selection[+], or type selection number: 2
[root@ec2 ~]# java -version
java version "1.5.0_22"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_22-b03, mixed mode)

Yay! Unfortunately this doesn’t help with the other problem I had with Tomcat, which was that EC2 instances set the JAVA_HOME var to OpenJDK as well (/usr/lib/jvm/jre). Fortunately this is an easy fix as well.

Setting JAVA_HOME

The JAVA_HOME var is set in /etc/profile.d/aws-apitools-common.sh. Comment out this line:

export JAVA_HOME=/usr/lib/jvm/jre

Create a new file, /etc/profile.d/sun-java.sh, and put this in it:

export JAVA_HOME=/usr/java/jdk1.5.0_22/jre

Also in that file I added the following to instruct the JVM to process all dates in America/New_York, since that’s the timezone all of our other servers use, and it makes reading log files easier when all dates are in the same tz:

export TZ=America/New_York

(I found I had to do this even after pointing /etc/localtime to the correct zoneinfo – Java was stuck on UTC even after the rest of the system was using America/New_York.)

Rescan SATA bus (aka hot-adding a SATA disk on a Linux guest in VMware without rebooting)

Evan Hoffman — Mon, 24 Oct 2011 19:20:51 +0000

Linux supports hot-adding disks but whenever I add a new vdisk in VMware the new disk doesn’t show up unless I reboot, which defeats the purpose of hot-add. This command forces a rescan of the bus:

echo "- - -" > /sys/class/scsi_host/host0/scan

dmesg shows the new disk has been found:

  Vendor: VMware    Model: Virtual disk      Rev: 1.0 
  Type:   Direct-Access                      ANSI SCSI revision: 02
 target0:0:2: Beginning Domain Validation
 target0:0:2: Domain Validation skipping write tests
 target0:0:2: Ending Domain Validation
 target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
 sdd: unknown partition table
sd 0:0:2:0: Attached scsi disk sdd
sd 0:0:2:0: Attached scsi generic sg3 type 0

Now, why there’s no “rescan_sata” command is something I can’t fathom, but that’s Linux for you.

Making sure SSLv2 is disabled in Apache (and Nginx)

Evan Hoffman — Tue, 20 Sep 2011 18:06:38 +0000

Edit Jan 24, 2012: Deleted all the crap from this story and just left the recommended Apache and Nginx SSL cipher suites for maximum security without SSLv2 and without BEAST vulnerability (at least according to Qualys).

Apache httpd

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
SSLHonorCipherOrder on

nginx

        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;

Source:

Exchange (OWA) CAS crashes with 503 error – again

Evan Hoffman — Sun, 18 Sep 2011 16:22:46 +0000

This just started happening again, with these errors appearing in the event viewer:

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.
 
Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/17/2011 6:47:07 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' terminated
unexpectedly. The process id was '3108'. The process exit code was
'0x800703e9'.
 
Log Name: Application
Source: Application Error
Date: 9/17/2011 6:46:30 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time
stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time
stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

After reviewing the IIS logs and the event logs, I think it has to do with the WebReady document viewer – the thing in OWA that renders and lets you view .doc attachments within the browser rather than forcing you to open Word or Excel. I think users were attempting to open corrupted files and that was causing it to crash. I’ve disabled Webready in EMC (Server Config -> CAS) and I’ll see what happens.

64-bit Firefox 6 on CentOS 6 x86_64 (or Fedora) with Flash 11

Evan Hoffman — Wed, 31 Aug 2011 20:15:14 +0000

So I’ve gone back to Linux from Mac, due to the SSD issues I had with my Macbook Pro basically making VMware unusable. A Win7 VM would grind the guest and host to a halt on the 7200 RPM SATA OEM drive, and the SSD wouldn’t work, so I put the SSD back in my HP and installed CentOS 6 x86_64. Not really ideal, but at least it works.

CentOS 6 ships with Firefox 3.6.9, which is really old by now. Fine for a server, but I wanted FF6. I grabbed the Firefox 6.0.1 bz2 from VoxCast, unzipped it and copied it to /usr/lib64/firefox-6. Then I ran yum remove firefox to remove 3.6.9 and avoid any issues. I tried the instructions from my older post on this subject but for whatever reason it didn’t work – I’m guessing because I’m not using the distro’s Firefox RPM.

I followed these instructions to get the Flash 11 64-bit plugin installed. It still wasn’t working though.

I ended up creating a symlink:

ln -s /usr/lib64/mozilla/plugins/libflashplayer.so /home/evan/.mozilla/plugins/libflashplayer.so

When I started up Firefox after creating the symlink, Flash worked.

Additionally, to set Firefox 6 as the default browser, run gnome-default-applications-properties, select Custom, and paste /usr/lib64/firefox-6/firefox %s in the Command: field.

Update: I upgraded my home desktop PC from FC12 to FC14 last night and used the above procedure to install Firefox 6.0.2 on it with Flash 11. So, in case anyone was wondering, the above works for Fedora as well.

Macbook Pro locks up with SSD installed.

Evan Hoffman — Tue, 23 Aug 2011 16:04:54 +0000

A few weeks ago I switched from my trusty old HP nc8430 to a Macbook Pro (MC118LL/A) that was left spare when another employee left. I mostly enjoyed using Linux but I was tired of dealing with weird quirks like having X lock up, essentially forcing me to do a hard reboot.

To transition, I copied my documents from Linux to Mac, then turned off the Linux laptop. Surprisingly I found I didn’t need to turn Linux back on at all.

Last week, I decided to put the final nail in Linux’s coffin by taking the SSD (Corsair CMFSSD-128GBG2D)out of it and putting it in my Macbook. The Macbook was pretty fast (Core 2 Duo @ 2.5 GHz) but some things were noticeably slower on its 7200RPM disk than on Linux with an SSD, especially running Windows VMs.

I booted Linux to Knoppix and zeroed out the disk, then removed it. I backed my Mac up with Time Machine, shut it down, then undid the ~12 tiny screws, removed the bottom plate of the Macbook and popped the SSD in. I booted from the Mac DVD, restored from Time Machine and went home (it took ~4 hours to restore).

When I got in the next day, the restore was complete, though I had to click “Restart” to finish, which was annoying. Everything worked fine, and I was pretty impressed. The machine was kind of sluggish due to Spotlight indexing but once that was done I was pretty amazed at the transformation. Every app opened in under 1 second. Windows VMs were super snappy. Things were going well.

But then I started noticing periods of extended hanging. In the middle of some task, I’d get the beachball and the whole computer would become unresponsive (cursor would spin & move around but I couldn’t click anything). This would last about 30-60 seconds. I assumed it was some behind-the-scenes optimization, or some residual spotlight indexing.

Unfortunately, it hasn’t gone away. Earlier today I copied a 3 GB zip file from our file server to my laptop and it beachballed me on and off (about 60-70% of the time) for about 15 minutes as it copied. What’s odd is that the transfer speeds were pretty good, it appeared to be my computer itself that was bottlenecking it. After the download completed, I attempted to unzip it and was beachballed again. I checked Activity Monitor and it was peaking at 30 MB/s, but had extended periods of zeroes. I ran iostat and got basically the same information:

EvanMBP:~ root# iostat -Kw 3
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    9.33   3  0.03     0.00   0  0.00   9  4 87  0.25 0.28 0.32
    0.00   0  0.00     0.00   0  0.00  13  5 82  0.23 0.27 0.32
   20.00   0  0.01     0.00   0  0.00  15  5 80  0.23 0.27 0.32
   36.00   1  0.02     0.00   0  0.00  16  6 77  0.21 0.27 0.32
    0.00   0  0.00     0.00   0  0.00  12  5 83  0.21 0.27 0.32
   20.02  60  1.18     0.00   0  0.00  17 10 74  0.19 0.26 0.32
   24.60 363  8.72     0.00   0  0.00  18 13 69  0.17 0.26 0.31
   25.40 307  7.60     0.00   0  0.00  14 11 75  0.17 0.26 0.31
   21.95 426  9.14     0.00   0  0.00  15 12 73  0.32 0.29 0.32
   82.50 352 28.35     0.00   0  0.00  17 11 73  0.32 0.29 0.32
  809.70  84 66.41     0.00   0  0.00  22  9 69  0.29 0.28 0.32
    0.00   0  0.00     0.00   0  0.00   9  5 86  0.27 0.28 0.32
    0.00   0  0.00     8.89  11  0.09   9  5 86  0.27 0.28 0.32
    0.00   0  0.00     0.00   0  0.00   9  5 87  0.33 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  12  7 81  0.33 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  14  8 78  0.38 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.35 0.29 0.33
    0.00   0  0.00     0.00   0  0.00  10  6 84  0.35 0.29 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 84  0.32 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  11  6 84  0.32 0.29 0.32
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  10  5 84  0.30 0.28 0.32
    0.00   0  0.00     0.00   0  0.00  11  6 84  0.27 0.28 0.32
    0.00   0  0.00     0.00   0  0.00  11  5 84  0.27 0.28 0.32
    0.00   0  0.00     0.00   0  0.00   9  5 85  0.25 0.27 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.25 0.27 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.31 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.45 0.31 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.45 0.31 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.41 0.31 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.41 0.31 0.33
  384.49  10  3.85    10.00   0  0.00  12  6 82  0.32 0.29 0.32
  291.73 126 35.78     0.00   0  0.00  23 13 64  0.38 0.31 0.33
  236.65 338 78.20     0.00   0  0.00  30 17 53  0.34 0.30 0.33
  397.61  21  8.02     0.00   0  0.00  15  8 77  0.34 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.32 0.30 0.32
    0.00   0  0.00     0.00   0  0.00  12  7 81  0.32 0.30 0.32
    0.00   0  0.00     0.00   0  0.00  12  6 82  0.29 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.35 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  13  6 80  0.35 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.32 0.30 0.32
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  13  8 78  0.32 0.30 0.32
    0.00   0  0.00     0.00   0  0.00  11  7 82  0.29 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 86  0.35 0.30 0.33
  148.94 124 18.03     0.00   0  0.00  18  8 73  0.35 0.30 0.33
  267.73 121 31.72     0.00   0  0.00  17  8 76  0.32 0.30 0.32
  355.54 162 56.35     0.00   0  0.00  22  8 69  0.32 0.30 0.32
  738.07  38 27.38     0.00   0  0.00  16  6 78  0.30 0.29 0.32
  512.42  67 33.52     0.00   0  0.00  20  7 73  0.27 0.29 0.32
  835.74  61 50.05     0.00   0  0.00  19  7 73  0.27 0.29 0.32
  536.83  69 36.17     0.00   0  0.00  17  6 76  0.49 0.33 0.34
  543.89  83 43.90     0.00   0  0.00  20  8 72  0.49 0.33 0.34
  720.70  59 41.74     0.00   0  0.00  18  7 76  0.45 0.33 0.33
  541.23 124 65.70     0.00   0  0.00  22  9 70  0.41 0.32 0.33
  260.54 210 53.37     0.00   0  0.00  22  9 70  0.41 0.32 0.33
  806.93  73 57.78     0.00   0  0.00  20  8 72  0.46 0.33 0.34
  874.98  43 37.02     0.00   0  0.00  13  7 80  0.46 0.33 0.34
    0.00   0  0.00     0.00   0  0.00  11  4 85  0.42 0.33 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.39 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   9  4 87  0.39 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.44 0.33 0.33
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  11  5 84  0.44 0.33 0.33
    0.00   0  0.00     0.00   0  0.00  13  6 81  0.48 0.34 0.34
    0.00   0  0.00     0.00   0  0.00  10  4 86  0.44 0.34 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.44 0.34 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.41 0.33 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.41 0.33 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 87  0.38 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   7  4 89  0.35 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 87  0.35 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   7  3 89  0.32 0.31 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.32 0.31 0.33
    0.00   0  0.00     0.00   0  0.00   7  3 90  0.29 0.31 0.33
  347.11  92 31.29     0.00   0  0.00  20  8 72  0.43 0.34 0.33
   49.98 656 32.03     0.00   0  0.00  32 10 59  0.43 0.34 0.33
  113.45 351 38.90     0.00   0  0.00  40 15 45  0.47 0.35 0.34
  819.41  34 27.20     0.00   0  0.00  26 12 63  0.47 0.35 0.34
  686.99  50 33.76     0.00   0  0.00  18  7 76  0.52 0.36 0.34
  878.17  23 20.01     0.00   0  0.00  20  8 72  0.47 0.35 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 87  0.47 0.35 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.44 0.34 0.34

You can see in the MB/s column, lots of “0.00″ followed by some bursts of ~30 MB/s. The zeros didn’t actually print at the time, but flooded the screen in bursts when the bottleneck cleared up. It seems to me like it might be some problem with queueing or caching, or maybe the SATA controller on this Mac just isn’t up to the task of SSDs. I’m not sure, but at this point I’m afraid I might have to go back to the 7200 RPM Seagate that came with the Mac. 30-second hangups are far more annoying than having lots of things be slower. Kind of a strange amortization, if you think about it. Anyway, I’ll keep looking into it, now that I know how to reproduce the problem (unzip a huge file).

Updated: A quick Google search for “beach ball mac SSD” turned up this thread which seems to be about this same problem, with a different model SSD. Also referenced in this thread on Apple.com. It feels like the problem may be due to an “old” SSD.

Updated again: Here’s someone having the problem with a Corsair 128 GB SSD: http://forum.corsair.com/v3/showthread.php?t=91061.

Updated again: According to this post, this appears to be a problem with the SATA controller in the 2009 Macbooks. Bah.

Logging RT username in Apache access_log

Evan Hoffman — Mon, 08 Aug 2011 21:34:18 +0000

RT has its own internal accounting & tracking system for logging activity, but I was interested in even more granular stuff, like seeing who looked at which tickets. I figured it wouldn’t be that hard to log this in Apache. Well, I was kind of right, in that it wasn’t “hard,” but it took me a long time to find the right place to do it. I did finally get it though.

httpd.conf

In httpd.conf I created a new LogFormat:

LogFormat "%h %l %{RTUSER}e %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined-rt

So instead of the HTTP-auth user, it puts the RT user in this field. Make sure to update your VirtualHost config to use the combined-rt LogFormat.

/usr/share/rt3/html/autohandler

In /usr/share/rt3/html/autohandler, right under this section:

# If we've got credentials, let's serve the file up.
if (    ( defined $session{'CurrentUser'} )
    and ( $session{'CurrentUser'}->Id ) )
{

Add this:

        $ENV{'RTUSER'} = $session{'CurrentUser'}->UserObj->EmailAddress;

This populates the RTUSER environment variable with the currently-logged-in user’s email address.

Restart httpd and the RT user’s email address should now appear in access_log. Note that it will only appear for Perl pages (not gifs/jpgs or other static content, since Perl doesn’t process those):

10.0.0.10 - evan@example.com [08/Aug/2011:17:30:34 -0400] "GET /rt3/index.html HTTP/1.1" 200 46809 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"
10.0.0.10 - - [08/Aug/2011:17:30:34 -0400] "GET /rt3/NoAuth/images//css/rolldown-arrow.gif HTTP/1.1" 200 83 "https://help.example.com/rt3/NoAuth/css/3.5-default/main-squished.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"

Amazon SES: “illegal headers” with ses-send-email.pl (followup)

Evan Hoffman — Tue, 02 Aug 2011 18:39:45 +0000

A few people have emailed me asking me to integrate the perl code snippet into I wrote to strip illegal headers when sending email via Amazon SES into something actually usable. I’ve done so! I haven’t really tested this beyond sending some test emails, but here it is. Use this at your own risk, I make no warranty, blah blah blah.

This fix requires editing ses-send-email.pl, so I’d advise making a backup copy, though I imagine you can always get a fresh version from Amazon if necessary.

Open ses-send-email.pl in a text editor and find the read_message method. It should look like this:

Delete that and paste this in its place:

I tried it on the command line and it gave the output I expected – I haven’t tried integrating it with sendmail/postfix since I haven’t encountered this problem. Here’s the test message I attempted to send:

From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!
 
Now we're out of the header, into the body.  Grand!

From, To, and Subject are required headers. Chicken & Cats are illegal, X-Zombies should be ok since it’s X-ified. Here’s what happened when I tried to send with the unmodified ses-send-email.pl:

[Tue Aug 02 14:14:51 evan@EvanMBP 62 amazon-email]$ ./ses-send-email.pl --verbose -k aws-credentials -r
From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!

Now we're out of the header, into the body.  Grand!

    Sender
    InvalidParameterValue
    Illegal header 'Chicken'.

  5ffad294-bd33-11e0-b6e3-affca9ad1eb5

Illegal header 'Chicken'.

Illegal header error. Now with the modified version:

[Tue Aug 02 14:15:17 evan@EvanMBP 63 amazon-email]$ ./ses-send-email-x-headers.pl --verbose -k aws-credentials -r
From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!

Now we're out of the header, into the body.  Grand!

    000001318bb51442-c2cfb780-0604-4363-925a-54a57015e567-000000

    64e75a47-bd33-11e0-9d09-8f08f31615ad

No errors, and I received the email below (extraneous SMTP headers added by Barracuda/Exchange removed):

From: <evan@example.com>
To: <evan@example.com>
Subject: Email
X-Chicken: yummy
X-ASG-Orig-Subj: Email
X-Cats: yucky
X-Zombies: kill them!
Date: Tue, 2 Aug 2011 18:15:25 +0000
Message-ID: <000001318bb51442-c2cfb780-0604-4363-925a-54a57015e567-000000@email.amazonses.com>
Content-Type: text/plain
MIME-Version: 1.0

Now we're out of the header, into the body.  Grand!

Illegal headers have been X-ified. Hope this helps someone.

33 hours to restore a Postgres DB

Evan Hoffman — Tue, 26 Jul 2011 22:10:01 +0000

Having migrated our DB to a new machine, I was left with a pretty good machine unused. I decided to rebuild it and try out Postgres 9.0.

I downloaded and installed CentOS 6.0 x86_64, built Postgres 9.0.4 from SRPMS (makes me feel better building it myself), installed it and did initdb, dumped the primary DB to our NAS using psql 9.0 (pg_dump [db] | gzip > /path/to/file.gz) and began the restore.

Not much of a story but it took 33 hours to restore the DB, with the bulk of the time spent building indices. When the restore was complete I did an analyze verbose which took another 2 hours, for a total of 35 hours, not counting the time it took to create the dump file – about 14 hours, but this was with the DB under heavy load, made heavier by the dump, so I don’t know how long it would really take. I’m glad I didn’t have to do this to migrate, but I’m not relishing the idea of doing a real upgrade to 9.0 if it’s going to take this long. For the restore I disabled autovacuum and set fsync = off. It’s an ext4 filesystem on a 24-disk RAID 10 of 10krpm SAS drives.

DB size on disk after a clean restore was 1156 GB, as per pg_database_size(), versus 1237 GB on disk for the one that was migrated via bit-copy, so there’s a sizable chunk of crud accumulated in there.

Also, in doing this I discovered the logger command. One of the things that I’ve always found annoying about doing a pg restore is that the output is just a bunch of statements like this:

SET
SET
SET
CREATE SCHEMA
ALTER SCHEMA
CREATE SCHEMA
ALTER SCHEMA
SET
CREATE TABLE
ALTER TABLE
CREATE FUNCTION
ALTER FUNCTION

This is just what’s returned from the SQL commands, but it’s not very informative, and most importantly it doesn’t show you the timestamp so you have no idea how long anything’s taking. With logger, you can just pipe the output to it and it’ll be logged with syslog. So my restore command looked like this:

zcat /nfs/db/pgdump-90.sql.gz | psql -Upguser db 2> ~/restore.20110723-1916.log | logger -t PGRESTORE

This made it trivial to get the messages out of /var/log/messages with grep:

[root@link log]# grep PGRESTORE messages | head
Jul 24 03:36:43 link PGRESTORE: ALTER TABLE
Jul 24 03:36:43 link PGRESTORE: SET
Jul 24 03:36:44 link PGRESTORE: ALTER TABLE
Jul 24 03:36:44 link PGRESTORE: ALTER TABLE
Jul 24 03:38:45 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: SET
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE

In addition to having the timestamps it eliminates the accumulation of the random log files that accumulate like droppings from myriad one-off scripts. I modified about 20 cron jobs to pipe their output to logger rather than junk files in /tmp or my homedir. I can’t believe I never heard of logger before! Hooray for logger!

Benchmarking disk IO on ext3 vs ext4 vs xfs with fio

Evan Hoffman — Fri, 22 Jul 2011 11:15:05 +0000

With the old database phased out, I figured this was a good time to benchmark IO on it before either repurposing it or mothballing it. In the past I’ve used dd for elementary sequential read/write testing, but I recently found fio which is much more versatile. Apparently developed by FusionIO, fio lets you control concurrency, blocksize and many other parameters and reports lots more data, most notably IOPS.

I created 4 fio config files, basically the same as the one below, for direct / buffered disk access and sequential / random writes. All have 8k blocksize since that’s what Postgres uses by default.


[root@link ~]# cat random-write-test.fio
[random-write]
rw=randwrite
size=5G
direct=1
directory=/msa70/fio/data/
numjobs=5
group_reporting
name=random-write-direct
bs=8k
runtime=10

[root@link ~]# cat sequential-write-test-buf.fio
[sequential-write]
rw=write
size=5G
direct=0
directory=/msa70/fio/data/
numjobs=5
group_reporting
name=sequential-write-buffered
bs=8k
runtime=10

First I ran fio against a freshly created ext3 volume on our HP MSA70 with 24x 10kRPM SAS disks in RAID 10. The mkfs command used was mke2fs -j -m0 /dev/cciss/c0d0. The controller card is a HP P800. Here’s the output:

fio - ext3 full results

[root@link ~]# time fio random-write-test.fio random-write-test-buf.fio sequential-write-test.fio sequential-write-test-buf.fio
random-write-direct: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
random-write-direct: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
random-write-buf: (g=1): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
random-write-buf: (g=1): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
sequential-write-direct: (g=2): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
sequential-write-direct: (g=2): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
sequential-write-buffered: (g=3): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
sequential-write-buffered: (g=3): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
fio 1.57
Starting 20 processes
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 1 (f=1): [________________W___] [19.8% done] [0K/0K /s] [0 /0  iops] [eta 03m:14s]
random-write-direct: (groupid=0, jobs=5): err= 0: pid=2447
  write: io=596856KB, bw=56946KB/s, iops=7118 , runt= 10481msec
    clat (usec): min=67 , max=944937 , avg=645.32, stdev=4647.69
     lat (usec): min=67 , max=944937 , avg=645.46, stdev=4647.69
    bw (KB/s) : min=  144, max=47152, per=22.71%, avg=12933.03, stdev=4985.93
  cpu          : usr=0.26%, sys=1.48%, ctx=74614, majf=0, minf=134
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/74607/0, short=0/0/0
     lat (usec): 100=12.22%, 250=80.24%, 500=6.59%, 750=0.20%, 1000=0.16%
     lat (msec): 2=0.07%, 4=0.01%, 10=0.10%, 20=0.05%, 50=0.08%
     lat (msec): 100=0.10%, 250=0.14%, 500=0.04%, 750=0.01%, 1000=0.01%
random-write-buf: (groupid=1, jobs=5): err= 0: pid=2452
  write: io=6377.3MB, bw=472902KB/s, iops=59112 , runt= 13809msec
    clat (usec): min=8 , max=4490.8K, avg=77.54, stdev=3751.21
     lat (usec): min=8 , max=4490.8K, avg=77.66, stdev=3751.21
    bw (KB/s) : min=    7, max=438688, per=34.13%, avg=161414.56, stdev=63682.89
  cpu          : usr=2.27%, sys=24.01%, ctx=1596, majf=0, minf=134
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/816288/0, short=0/0/0
     lat (usec): 10=0.06%, 20=83.59%, 50=16.19%, 100=0.04%, 250=0.04%
     lat (usec): 500=0.03%, 750=0.01%, 1000=0.01%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=0.01%
     lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.01%
sequential-write-direct: (groupid=2, jobs=5): err= 0: pid=2457
  write: io=2832.0KB, bw=286105 B/s, iops=34 , runt= 10136msec
    clat (usec): min=75 , max=1506.6K, avg=157500.69, stdev=96237.55
     lat (usec): min=75 , max=1506.6K, avg=157500.81, stdev=96237.54
    bw (KB/s) : min=    6, max=  444, per=19.58%, avg=54.61, stdev=13.65
  cpu          : usr=0.00%, sys=0.00%, ctx=572, majf=0, minf=140
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/354/0, short=0/0/0
     lat (usec): 100=4.80%, 250=33.05%, 500=0.56%
     lat (msec): 50=1.13%, 100=18.36%, 250=25.99%, 500=9.04%, 750=4.24%
     lat (msec): 1000=1.41%, 2000=1.41%
sequential-write-buffered: (groupid=3, jobs=5): err= 0: pid=2462
  write: io=255624KB, bw=19933KB/s, iops=2491 , runt= 12824msec
    clat (usec): min=7 , max=9411.7K, avg=1443.28, stdev=33091.95
     lat (usec): min=7 , max=9411.7K, avg=1443.38, stdev=33091.95
    bw (KB/s) : min=   31, max=43687, per=29.58%, avg=5896.30, stdev=2696.81
  cpu          : usr=0.05%, sys=0.89%, ctx=600, majf=0, minf=136
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/31953/0, short=0/0/0
     lat (usec): 10=13.51%, 20=77.52%, 50=8.68%, 100=0.07%, 250=0.08%
     lat (usec): 500=0.03%, 750=0.01%
     lat (msec): 4=0.01%, 10=0.01%, 20=0.02%, 50=0.01%, 250=0.01%
     lat (msec): 750=0.02%, 2000=0.01%, >=2000=0.03%

Run status group 0 (all jobs):
  WRITE: io=596856KB, aggrb=56946KB/s, minb=58313KB/s, maxb=58313KB/s, mint=10481msec, maxt=10481msec

Run status group 1 (all jobs):
  WRITE: io=6377.3MB, aggrb=472902KB/s, minb=484251KB/s, maxb=484251KB/s, mint=13809msec, maxt=13809msec

Run status group 2 (all jobs):
  WRITE: io=2832KB, aggrb=279KB/s, minb=286KB/s, maxb=286KB/s, mint=10136msec, maxt=10136msec

Run status group 3 (all jobs):
  WRITE: io=255624KB, aggrb=19933KB/s, minb=20411KB/s, maxb=20411KB/s, mint=12824msec, maxt=12824msec

Disk stats (read/write):
  cciss!c0d0: ios=798/24542244, merge=0/937067, ticks=15866/140262853, in_queue=140448875, util=89.71%

real    18m35.082s
user    0m4.328s
sys     5m16.479s
[root@link ~]#

That’s a lot of data; the most interesting numbers are these:

random-write-direct: (groupid=0, jobs=5): err= 0: pid=2447
  write: io=596856KB, bw=56946KB/s, iops=7118 , runt= 10481msec
random-write-buf: (groupid=1, jobs=5): err= 0: pid=2452
  write: io=6377.3MB, bw=472902KB/s, iops=59112 , runt= 13809msec
sequential-write-direct: (groupid=2, jobs=5): err= 0: pid=2457
  write: io=2832.0KB, bw=286105 B/s, iops=34 , runt= 10136msec
sequential-write-buffered: (groupid=3, jobs=5): err= 0: pid=2462
  write: io=255624KB, bw=19933KB/s, iops=2491 , runt= 12824msec

I then reformatted the RAIDed LUN with ext4 with mkfs.ext4 -m0 /dev/cciss/c0d0 and ran the same tests again:

fio - ext4 full results

[root@link ~]# time fio random-write-test.fio random-write-test-buf.fio sequential-write-test.fio sequential-write-test-buf.fio
random-write-direct: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
random-write-direct: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
random-write-buf: (g=1): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
random-write-buf: (g=1): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
sequential-write-direct: (g=2): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
sequential-write-direct: (g=2): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
sequential-write-buffered: (g=3): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
sequential-write-buffered: (g=3): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
fio 1.57
Starting 20 processes
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 5 (f=5): [_______________WWWWW] [21.5% done] [0K/126.1M /s] [0 /15.9K iops] [eta 02m:30s]
random-write-direct: (groupid=0, jobs=5): err= 0: pid=2379
  write: io=589392KB, bw=58933KB/s, iops=7366 , runt= 10001msec
    clat (usec): min=77 , max=2522.4K, avg=781.82, stdev=9768.99
     lat (usec): min=77 , max=2522.4K, avg=781.95, stdev=9768.99
    bw (KB/s) : min=  515, max=44176, per=22.05%, avg=12996.62, stdev=5332.75
  cpu          : usr=0.26%, sys=2.86%, ctx=73715, majf=0, minf=134
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/73674/0, short=0/0/0
     lat (usec): 100=2.54%, 250=92.13%, 500=4.76%, 750=0.14%, 1000=0.01%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.07%, 20=0.04%, 50=0.07%
     lat (msec): 100=0.09%, 250=0.11%, 500=0.03%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.01%
random-write-buf: (groupid=1, jobs=5): err= 0: pid=2384
  write: io=6707.1MB, bw=670727KB/s, iops=83840 , runt= 10241msec
    clat (usec): min=7 , max=1582.5K, avg=57.04, stdev=1910.33
     lat (usec): min=7 , max=1582.5K, avg=57.15, stdev=1910.33
    bw (KB/s) : min=   30, max=518784, per=25.08%, avg=168240.26, stdev=84472.16
  cpu          : usr=2.51%, sys=25.56%, ctx=35766, majf=0, minf=134
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/858614/0, short=0/0/0
     lat (usec): 10=2.14%, 20=91.72%, 50=5.11%, 100=0.31%, 250=0.15%
     lat (usec): 500=0.19%, 750=0.15%, 1000=0.09%
     lat (msec): 2=0.08%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
     lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%
sequential-write-direct: (groupid=2, jobs=5): err= 0: pid=2389
  write: io=839760KB, bw=83959KB/s, iops=10494 , runt= 10002msec
    clat (usec): min=70 , max=300244 , avg=473.09, stdev=3815.84
     lat (usec): min=71 , max=300244 , avg=473.22, stdev=3815.84
    bw (KB/s) : min= 8198, max=30528, per=20.27%, avg=17014.88, stdev=1677.78
  cpu          : usr=0.32%, sys=3.15%, ctx=104986, majf=0, minf=140
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/104970/0, short=0/0/0
     lat (usec): 100=73.60%, 250=26.18%, 500=0.01%, 750=0.01%
     lat (msec): 50=0.01%, 100=0.01%, 250=0.18%, 500=0.02%
sequential-write-buffered: (groupid=3, jobs=5): err= 0: pid=2394
  write: io=1730.7MB, bw=163683KB/s, iops=20460 , runt= 10827msec
    clat (usec): min=7 , max=7649.9K, avg=220.41, stdev=12535.19
     lat (usec): min=7 , max=7649.9K, avg=220.52, stdev=12535.19
    bw (KB/s) : min=   27, max=331055, per=35.92%, avg=58788.88, stdev=27863.44
  cpu          : usr=0.42%, sys=5.54%, ctx=4119, majf=0, minf=136
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/221525/0, short=0/0/0
     lat (usec): 10=50.94%, 20=47.34%, 50=1.45%, 100=0.18%, 250=0.05%
     lat (usec): 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
     lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.01%

Run status group 0 (all jobs):
  WRITE: io=589392KB, aggrb=58933KB/s, minb=60347KB/s, maxb=60347KB/s, mint=10001msec, maxt=10001msec

Run status group 1 (all jobs):
  WRITE: io=6707.1MB, aggrb=670726KB/s, minb=686824KB/s, maxb=686824KB/s, mint=10241msec, maxt=10241msec

Run status group 2 (all jobs):
  WRITE: io=839760KB, aggrb=83959KB/s, minb=85974KB/s, maxb=85974KB/s, mint=10002msec, maxt=10002msec

Run status group 3 (all jobs):
  WRITE: io=1730.7MB, aggrb=163683KB/s, minb=167611KB/s, maxb=167611KB/s, mint=10827msec, maxt=10827msec

Disk stats (read/write):
  cciss!c0d0: ios=52/276341, merge=0/553601, ticks=224/1731709, in_queue=1735311, util=94.47%

real    0m42.218s
user    0m1.812s
sys     0m19.028s
[root@link ~]#

Summary numbers:

random-write-direct: (groupid=0, jobs=5): err= 0: pid=2379
  write: io=589392KB, bw=58933KB/s, iops=7366 , runt= 10001msec
random-write-buf: (groupid=1, jobs=5): err= 0: pid=2384
  write: io=6707.1MB, bw=670727KB/s, iops=83840 , runt= 10241msec
sequential-write-direct: (groupid=2, jobs=5): err= 0: pid=2389
  write: io=839760KB, bw=83959KB/s, iops=10494 , runt= 10002msec
sequential-write-buffered: (groupid=3, jobs=5): err= 0: pid=2394
  write: io=1730.7MB, bw=163683KB/s, iops=20460 , runt= 10827msec

This is just one run of each test, so not very scientific, but it does look pretty clear that ext4 outperforms ext3 (I actually did run the test several times but didn’t collect the results, they were pretty consistent though). What the test doesn’t show, but you can see from the time output is that each ext3 test took over 4 minutes to run while the ext4 tests each took about 10-12 seconds. In the ext3 tests, each “file1: Laying out IO file(s) (1 file(s) / 5120MB)” line took nearly a minute; in the ext4 tests they were essentially instant.

This is nothing groundbreaking, I was just playing around with fio and figured I’d post the results in case anyone else was curious. I did notice some oddities though, like the fact that random buffered writes appear to be faster than even sequential buffered writes, on both ext3 and ext4 – on ext4, showing 670 MB/s random buffered writes and 83k IOPS. That doesn’t seem to make much sense, but I’ll leave the tea-leaf reading to someone else.

Addendum: I ran the test with XFS, just for fun. Overall, it was significantly faster than ext3, but not as fast as ext4, though sequential buffered writes were much faster than either ext3 or ext4.

fio - xfs full results

[root@link ~]# mkfs.xfs -f -b size=4k /dev/cciss/c0d0
meta-data=/dev/cciss/c0d0        isize=256    agcount=4, agsize=107506378 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=430025510, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=209973, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@link ~]# mount -v -t xfs /dev/cciss/c0d0 /msa70/
/dev/cciss/c0d0 on /msa70 type xfs (rw)
[root@link ~]# mkdir -p /msa70/fio/data
(reverse-i-search)`f': mkdir -p /msa70/^Co/data
[root@link ~]# time fio random-write-test.fio random-write-test-buf.fio sequential-write-test.fio sequential-write-test-buf.fio
random-write-direct: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
random-write-direct: (g=0): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
random-write-buf: (g=1): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
random-write-buf: (g=1): rw=randwrite, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
sequential-write-direct: (g=2): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
sequential-write-direct: (g=2): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
sequential-write-buffered: (g=3): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
...
sequential-write-buffered: (g=3): rw=write, bs=8K-8K/8K-8K, ioengine=sync, iodepth=1
fio 1.57
Starting 20 processes
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
random-write-buf: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-direct: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
sequential-write-buffered: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 5 (f=5): [_______________WWWWW] [91.3% done] [0K/143.4M /s] [0 /17.1K iops] [eta 00m:26s]
random-write-direct: (groupid=0, jobs=5): err= 0: pid=2662
  write: io=416920KB, bw=41688KB/s, iops=5210 , runt= 10001msec
    clat (usec): min=83 , max=1592.6K, avg=1013.62, stdev=10137.19
     lat (usec): min=83 , max=1592.6K, avg=1013.77, stdev=10137.19
    bw (KB/s) : min=    7, max=23184, per=22.46%, avg=9363.32, stdev=2852.49
  cpu          : usr=0.31%, sys=1.61%, ctx=106802, majf=0, minf=134
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/52115/0, short=0/0/0
     lat (usec): 100=0.06%, 250=4.55%, 500=87.85%, 750=4.23%, 1000=0.70%
     lat (msec): 2=2.01%, 4=0.13%, 10=0.24%, 20=0.07%, 50=0.01%
     lat (msec): 100=0.03%, 250=0.06%, 500=0.03%, 750=0.02%, 2000=0.02%
random-write-buf: (groupid=1, jobs=5): err= 0: pid=2667
  write: io=6557.9MB, bw=27876KB/s, iops=3484 , runt=240892msec
    clat (usec): min=7 , max=237984K, avg=1631.10, stdev=257906.92
     lat (usec): min=7 , max=237984K, avg=1631.22, stdev=257906.92
    bw (KB/s) : min=    0, max=628832, per=1222.74%, avg=340851.04, stdev=70996.06
  cpu          : usr=0.13%, sys=1.33%, ctx=24945, majf=0, minf=134
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/839397/0, short=0/0/0
     lat (usec): 10=18.02%, 20=79.39%, 50=2.19%, 100=0.18%, 250=0.07%
     lat (usec): 500=0.03%, 750=0.01%, 1000=0.04%
     lat (msec): 2=0.02%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
     lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%
     lat (msec): >=2000=0.01%
sequential-write-direct: (groupid=2, jobs=5): err= 0: pid=2677
  write: io=519056KB, bw=51895KB/s, iops=6486 , runt= 10002msec
    clat (usec): min=81 , max=242925 , avg=825.67, stdev=3980.13
     lat (usec): min=81 , max=242925 , avg=825.81, stdev=3980.13
    bw (KB/s) : min=   59, max=20029, per=19.03%, avg=9873.41, stdev=2128.50
  cpu          : usr=0.23%, sys=1.43%, ctx=129921, majf=0, minf=140
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/64882/0, short=0/0/0
     lat (usec): 100=60.60%, 250=38.63%, 500=0.01%
     lat (msec): 10=0.10%, 50=0.11%, 100=0.23%, 250=0.32%
sequential-write-buffered: (groupid=3, jobs=5): err= 0: pid=2682
  write: io=7688.1MB, bw=770776KB/s, iops=96347 , runt= 10215msec
    clat (usec): min=6 , max=1727.2K, avg=53.67, stdev=1633.96
     lat (usec): min=6 , max=1727.2K, avg=53.78, stdev=1633.96
    bw (KB/s) : min= 5340, max=704608, per=21.28%, avg=164049.10, stdev=87065.00
  cpu          : usr=2.15%, sys=19.54%, ctx=16284, majf=0, minf=136
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w/d: total=0/984185/0, short=0/0/0
     lat (usec): 10=41.97%, 20=56.42%, 50=1.10%, 100=0.23%, 250=0.10%
     lat (usec): 500=0.05%, 750=0.02%, 1000=0.03%
     lat (msec): 2=0.01%, 4=0.01%, 10=0.02%, 20=0.01%, 50=0.01%
     lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 2000=0.01%

Run status group 0 (all jobs):
  WRITE: io=416920KB, aggrb=41687KB/s, minb=42688KB/s, maxb=42688KB/s, mint=10001msec, maxt=10001msec

Run status group 1 (all jobs):
  WRITE: io=6557.9MB, aggrb=27876KB/s, minb=28545KB/s, maxb=28545KB/s, mint=240892msec, maxt=240892msec

Run status group 2 (all jobs):
  WRITE: io=519056KB, aggrb=51895KB/s, minb=53140KB/s, maxb=53140KB/s, mint=10002msec, maxt=10002msec

Run status group 3 (all jobs):
  WRITE: io=7688.1MB, aggrb=770776KB/s, minb=789274KB/s, maxb=789274KB/s, mint=10215msec, maxt=10215msec

Disk stats (read/write):
  cciss!c0d0: ios=8/836060, merge=0/549215, ticks=11/36458428, in_queue=36468678, util=98.88%

real    4m31.948s
user    0m2.692s
sys     0m24.351s
[root@link ~]#

XFS abridged:

random-write-direct: (groupid=0, jobs=5): err= 0: pid=2662
  write: io=416920KB, bw=41688KB/s, iops=5210 , runt= 10001msec
random-write-buf: (groupid=1, jobs=5): err= 0: pid=2667
  write: io=6557.9MB, bw=27876KB/s, iops=3484 , runt=240892msec
sequential-write-direct: (groupid=2, jobs=5): err= 0: pid=2677
  write: io=519056KB, bw=51895KB/s, iops=6486 , runt= 10002msec
sequential-write-buffered: (groupid=3, jobs=5): err= 0: pid=2682
  write: io=7688.1MB, bw=770776KB/s, iops=96347 , runt= 10215msec

IOPS graphed:

Bandwidth:

VMWare 5′s new licensing model.

Evan Hoffman — Sun, 17 Jul 2011 14:20:56 +0000

After reading up on the new VMware licensing&pricing model I understand the uproar. Limiting vRAM is a reasonable constraint, but 32GB per socket for Enterprise? 48 GB for “Enterprise Plus”? If you have a dual CPU server with 144 GB (easily configurable last year), with 4.1 you’d only need 2 enterprise licenses to use all 144 GB, since in 4.1 an “Enterprise” license covered 1 CPU (up to 6 cores) and up to 256 GB memory on the host.

But with 5.0 you’ll either have to buy 5 Enterprise licenses (160 GB) or 3 Enterprise Plus licenses (144 GB) just to use the full 144 GB. I guess VMware has done away with memory overcommit as a selling point? They used to tell us it was recommended to go up to 2:1 so we could safely put ~140 GB of VMs on a 72 GB machine – with the new model that’s completely gone.

To put it in monetary terms, on the machine with 144 GB ram from above, the cost for 4.1 would be $2875 * 2 = $5750. To stick with Enterprise it would be $2875 * 5 = $14,375, or $3495 * 3 = $10,485. A gigantic price jump. I haven’t read up on any features of vSphere 5, but I don’t think any feature can make up for at minimum nearly doubling the cost, with loss of a major selling point (memory overcommit). I mean, you can still overcommit as long as you’re willing to pay for the overcommitted memory. Also, the above numbers are per-host, so if you’ve got a 5-host cluster you’re looking at a $25,000 price hike.

VMWare has long been one of my favorite products, but this is making me consider alternatives. Almost 100% of the feedback I’ve read about this change has been negative. Seems like a huge mistake on VMware’s part.

Migrating a PostgreSQL DB to a new machine without doing a dump & restore

Evan Hoffman — Mon, 11 Jul 2011 18:39:16 +0000

Long ago, before I stepped into my role as de facto DBA in my current job, dump & restore of our Postgres database (dumping the entire DB to a text file, formatting the data disk, and restoring the data) was a pretty regular event. This was needed because we didn’t regularly vacuum the DB, which in turn was due to vacuums taking forever (which in the end ended up being due to crappy underlying hardware). We started doing monthly global vacuums (they took so long that monthly was about all we could handle) and we discovered we no longer needed to do dump & restores. When we finally upgraded to Postgres 8.2, which introduced autovacuum, things got even better, since we didn’t have to manage vacuuming via cron jobs.

In the years between then and now our Postgres DB has ballooned to such a size that a D&R isn’t something we could feasibly do in a scheduled maintenance window. The last time I did one was when we bade farewell to the decrepit database in September, 2007. At that point our database consumed 730 GB on disk. Looking through my email archives, we began the dump at 6:30 PM on a Friday night and it completed at 3:48 AM Saturday. The restore started around 9 AM and ran until around 1 PM (I assume it went so much faster than the dump due to the new DB being significantly better hardware-wise). Building indices took until 9:27 PM Saturday. We then ran a global “ANALYZE” to generate DB stats; the analyze ran from 10 PM until 1 AM Sunday. We then had most of Sunday to process the backlog of data that accumulated since Friday afternoon when we took the database offline.

So, with a 730 GB DB, the entire procedure took 9 hours (dump) + 4 hours (restore) + 8 hours (index rebuild) + 3 hours (analyze), so about 24 hours.

However, as I said, in the years since then our database has grown as we house more and more data, currently at about 1220 GB. It might have been possible to do the migration via dump & restore in the scheduled window, but I wasn’t looking forward to it. Instead, I decided to try a different option: copying the data files directly from the old server to the new one. If this worked it would eliminate almost all the overhead and the move would be complete in however long it took to copy the data from one host to the other.

Reasons

We had a couple reasons for doing this upgrade. Performance wasn’t really one of them though; we were pretty confident that the performance of the DB was as good as we were likely to get with platter disks, and our SAN doesn’t currently have SSDs. The old DB has dual Xeon 5160 2-core CPUs @3.0ghz, 32 GB memory, and a RAID 5 OS volume. The database resided on an HP MSA70, with 24x 10krpm 146 GB SAS drives (+1 hot spare) in RAID 10 for a 1.6 TB logical volume. At the time I debated RAID 6 vs RAID 10 but in the end I opted for the performance of RAID 10 vs the capacity of RAID 6 and it worked out well.

But one of the reasons we decided to upgrade was that the drives in the DB were starting to die and the warranty had expired on them, and each disk cost about $300 to replace. That was a pretty big liability and I expected the disks to begin dying more frequently.

Another reason for upgrading was the benefits of having the data on the SAN, especially snapshotting. We’d been doing daily backups for a while but doing a dump of the DB while it’s in use makes it take forever, causing degraded performance while it’s running. Snapshot isn’t a perfect solution, but at least it’s an option.

Another reason for wanting to move the DB to the SAN was for DR purposes; if we setup SAN-SAN replication to another site, with the DB on the SAN, we get that backed up for free.

And probably the biggest reason, we were up to 1.25 TB used out of 1.6, over 80% full. We’d probably be good for another few months, but for me 80% is pretty full.

Prerequisites

In order for this to work, the version of Postgres on both machines has to be of the same minor version. In my case, the source DB (server A) was running 8.2.5 (the newest at the time the box was built), so I built 8.2.18 (source RPM) on the target (server B). Server B is pretty beefy: HP DL360 G7 with dual Xeon X5660 6-core CPUs @2.8 GHz, 96 GB PC3-10600 ECC mem, QLogic QLE-4062 iSCSI HBA connected to a 4TB volume on our Compellent SAN (tier 1, RAID 10 across 32x 15krpm FC disks). Both machines of course need to be of the same architecture, in my case x86_64. It might work across Intel/AMD, but I’m not sure about that; fortunately I didn’t have to worry about that.

Moving the data

When we did dump & restores, we dumped directly to a commonly-mounted NAS, which worked well, since we wouldn’t start the restore until the dump was complete, and we didn’t want to consume disk on the target with a gigantic dump file (in addition to spinning the disks with reading the dumpfile while attempting to write to them; the contention causes everything to go much slower). There wasn’t really any need to use a NAS as an intermediary in this case though, it would just double the amount of time needed to get the data from A to B.

I created an NFS export on B:

1	/data/pgsql 10.0.0.35(rw,no_root_squash)

And mounted it on A with these options in /etc/fstab:

1	10.0.0.36:/data/pgsql /mnt/gannon/nfs nfs rw,rsize=32768,wsize=32768,nfsvers=3,noatime,udp 0 0

I tried TCP vs UDP mounts and found UDP was faster.

I then copied the data over with my favorite Unix tool, rsync:

1	time rsync -atp --delete --progress /var/lib/pgsql/data /mnt/gannon/nfs/ > /home/evan/rsync.log

Dry run

In January I did a dry run of the procedure. I had tried copying data over without stopping postgres on A, so as not to cause a service interruption, but it didn’t work; data was changing too rapidly. By the time the rsync completed, the files it had copied over earliest had already been modified again. I ended up scheduling some downtime for a weekend and did the copy. With the above NFS settings I was able to transfer data at around 50 MB/s over our gigabit switches, the whole thing took 3-4 hours. When it came up, everything seemed to be fine. I was pretty happy, because 3-4 hours is a whole lot better than 24+.

Day of Reckoning

I finally did the real migration this past weekend. I started it at 8 PM and (after an rsync snafu) completed it around 4 AM. The snafu was caused by my use of the “--delay-updates” flag, which I later learned copies modified files to a /.~tmp~/ directory, and when all of them are copied, moves them into place in an attempt to make it a more “atomic” operation. I didn’t realize this was what was happening, and I got a little freaked out when I saw the disk usage for the target growing 100 GB larger than the source. I cancelled the rsync and ran it again, stupidly dropping the –delay-updates flag, which with the –delete flag caused it to delete all the stuff in .~tmp~ that it had already copied over. It deleted like 300 GB of stuff before I freaked out and cancelled it again. I cursed myself a few times, then manually moved the contents of .~tmp~ up to the parent to salvage what had already been transferred, and ran the rsync once again to move the remaining data. So it probably would have been done much sooner had I not cancelled it and then deleted a bunch of my progress.

You may notice that the rsync flags above don’t include -z. With huge binary files being transferred over a fast LAN I don’t think there’s much reason to use -z, in fact when I added -z the throughput plummeted.

After copying the data, I moved the virtual IP of the DB to the new machine, moved the cron jobs over, started postgres on B and everything worked. I finished all of my cleanup around 6 AM Saturday, though like I said, I would have been done much sooner had I not deleted a bunch of my progress. Even still, this is a lot better than the dump & restore scenario, and has the added benefit of being reversible. I’m planning to upgrade postgres on A to 8.2.18 and leave it as a standby server; if a problem arises with B, the data can be moved back relatively quickly.

Conclusion

Well, I don’t have any great insight to put here, but so far this has worked out. My next DB project is upgrading from 8.2 to either 8.4 or the 9.x series, but that’s going to require a lot more planning since client drivers will likely need to be updated, and I’m not sure if queries might need to be altered.

The end (I hope).

Remount all NFS mounts to another cluster IP

Evan Hoffman — Thu, 07 Jul 2011 21:51:44 +0000

1	mount \| grep 10.0.0.73 \| cut -f3 -d\ \| xargs mount -o remount,rw,rsize=32768,wsize=32768,nfsvers=3,bg,intr,tcp,retrans=0,addr=10.0.0.74

Not much to say. This seems to work even when the disk is in use. Using it with Isilon to remount off of a quiesced node.

Teaching myself node.js: Part 1

Evan Hoffman — Tue, 14 Jun 2011 04:50:59 +0000

I’ve been meaning for a while to learn node.js. I read a couple of books and plenty of blog posts but in the end I find, as usual, I can only learn by doing. I have most of the fundamentals to get started, I just need to have a project. The typical tutorials I’ve found focus on creating a blog; I’m thinking of something more along the lines of a Quora question/answer site. I’ll be trying to write this as I go, which will hopefully prod me into actually doing it this time. (I have a tendency to bounce around different tenses and I expect this will be worse than usual – stuff will likely be written in past/present/future at various points).

Installing node is pretty easy, I just downloaded the source (I used 0.4.7), configured & compiled it. For MongoDB I use the 10gen CentOS/Fedora repository. With node installed, I installed npm to facilitate adding other packages. I started mongod with

/etc/init.d/mongodb start

With everything installed and mongod running I thought about the schema of my object. One of the coolest things about MongoDB is the fact that it’s “schemaless” and you can basically put anything into the DB, so you just need to think about what your objects look like and you can basically shove them into Mongo in that exact format. I’ve never been a fan of ORM stuff for SQL and the Mongo document-store type of DB seems like it eliminates the need for that almost entirely, which is great. So here’s what I was thinking would be the central object in this q-a site:

{
        "_id" : ObjectId("4defd2713fce2e9bc35d09f0"),
        "author" : "Bob",
        "date" : ISODate("2011-06-08T19:49:48.809Z"),
        "body" : "Is it hot in here?",
        "votes" : 0,
        "answers" : [
                {
                        "author" : "evan",
                        "date" : ISODate("2011-06-08T19:48:54.388Z"),
                        "body" : "I think so.",
                        "votes" : 0
                },
                {
                        "author" : "Jim",
                        "date" : ISODate("2011-06-08T19:51:04.714Z"),
                        "body" : "Could be.",
                        "votes" : 0
                },
                {
                        "author" : "Aaron",
                        "date" : ISODate("2011-06-08T19:51:33.313Z"),
                        "body" : "No",
                        "votes" : 0
                }
        ],
        "tags" : [
                "question",
                "silly",
                "stupid",
                "microsoft"
        ]
}

So this is a “question” and it has an author, a date, an array of tags, a title, a vote count, a “body” (maybe a longer version of the question), and an array of “answers”. Answers are embedded objects, each with an author, date, body, and votes. This seems like a decent place to start.

The way the site would work is someone posts a question and other people answer it. The answers would just be attached to the post in the “answers” array. Other users can vote on answers and the answer with the most votes would be the “winner”. Not exactly groundbreaking, but seems like it should be pretty easy to implement.

I used npm to install the following packages to aid in development:

$ npm install express jade mongoskin underscore

Express is a framework that handles request routing and jade is a templating framework. I used this quick walkthrough to setup a basic express/jade app that works and demonstrates most of what I need to know to get this started. Another good tutorial is http://dailyjs.com/2010/11/08/node-tutorial-2/.

After some mucking around I figured out how to use Express properly and it made things much easier. I used mongoskin as the MongoDB library and jade for the template and managed to rig up something that responds to a “list” request at http://server:3000/questions by listing all the questions in the “questions” collection. My code as of this point is available on github here. The views/list.jade has the template for the question list, and the code that calls the template is in app.js.

When viewing in a web browser, it currently looks like this (YMMV based on what’s in your mongodb, of course):

nodeQuestionList.jpg

That’s enough for today. So far I’m enjoying node.js now that I think I can finally wrap my head around the callback stuff. Hopefully I can keep up the momentum.

PostgreSQL query to determine the largest tables in the database

Evan Hoffman — Tue, 17 May 2011 14:25:32 +0000

This is just a handy query I use from time to time to see which tables are growing madly, and how much of the growth is index bloat.

SELECT
schemaname,
tablename,
pg_size_pretty(pg_relation_size(schemaname || '.' || tablename)) AS size_p,
pg_total_relation_size(schemaname || '.' || tablename) AS siz,
pg_size_pretty(pg_total_relation_size(schemaname || '.' || tablename)) AS total_size_p,
pg_total_relation_size(schemaname || '.' || tablename) - pg_relation_size(schemaname || '.' || tablename) AS index_size,
(100*(pg_total_relation_size(schemaname || '.' || tablename) - pg_relation_size(schemaname || '.' || tablename)))/CASE WHEN pg_total_relation_size(schemaname || '.' || tablename) = 0 THEN 1 ELSE pg_total_relation_size(schemaname || '.' || tablename) END || '%' AS index_pct
FROM pg_tables
ORDER BY siz DESC LIMIT 20;

This returns schema, table, size on disk (in human-readable and byte format – for sorting), and total size on disk including indices, and the percentage of the total size that comprises the indexes.

Sample result from our OpenFire Jabber server:

 schemaname |      tablename       |   size_p   |    siz    | total_size_p | index_size | index_pct 
------------+----------------------+------------+-----------+--------------+------------+-----------
 public     | ofconversation       | 71 MB      | 159236096 | 152 MB       |   84623360 | 53%
 public     | ofconparticipant     | 38 MB      |  95395840 | 91 MB        |   55394304 | 58%
 public     | ofpresence           | 10184 kB   |  10452992 | 10208 kB     |      24576 | 0%
 public     | ofpubsubitem         | 5336 kB    |   7733248 | 7552 kB      |    2269184 | 29%
 public     | ofid                 | 4008 kB    |   4120576 | 4024 kB      |      16384 | 0%
 public     | ofpubsubsubscription | 912 kB     |   1458176 | 1424 kB      |     524288 | 35%
 public     | ofpubsubaffiliation  | 728 kB     |   1196032 | 1168 kB      |     450560 | 37%
 public     | ofoffline            | 832 kB     |    942080 | 920 kB       |      90112 | 9%
 pg_catalog | pg_depend            | 320 kB     |    794624 | 776 kB       |     466944 | 58%
 pg_catalog | pg_attribute         | 384 kB     |    761856 | 744 kB       |     368640 | 48%
 pg_catalog | pg_proc              | 376 kB     |    753664 | 736 kB       |     368640 | 48%
 pg_catalog | pg_rewrite           | 72 kB      |    270336 | 264 kB       |     196608 | 72%
 pg_catalog | pg_description       | 136 kB     |    245760 | 240 kB       |     106496 | 43%
 pg_catalog | pg_operator          | 112 kB     |    237568 | 232 kB       |     122880 | 51%
 pg_catalog | pg_class             | 88 kB      |    212992 | 208 kB       |     122880 | 57%
 pg_catalog | pg_type              | 64 kB      |    155648 | 152 kB       |      90112 | 57%
 pg_catalog | pg_statistic         | 72 kB      |     98304 | 96 kB        |      24576 | 25%
 pg_catalog | pg_amop              | 24 kB      |     90112 | 88 kB        |      65536 | 72%
 pg_catalog | pg_conversion        | 16 kB      |     90112 | 88 kB        |      73728 | 81%
 pg_catalog | pg_constraint        | 8192 bytes |     81920 | 80 kB        |      73728 | 90%
(20 rows)
 
Time: 8.917 ms

JavaScript: The Good Parts

Evan Hoffman — Tue, 26 Apr 2011 21:20:21 +0000

I just finished reading JavaScript: The Good Parts, one of the best programming books I’ve read. The ending is fantastic:

We see a lot of feature-driven product design in which the cost of features is not properly accounted. Features can have a negative value to consumers because they make the products more difficult to understand and use. We are finding that people like products that just work. It turns out that designs that just work are much harder to produce than designs that assemble long lists of features.

Features have a specification cost, a design cost, and a development cost. There is a testing cost and a reliability cost. The more features there are, the more likely one will develop problems or will interact badly with another. In software systems, there is a storage cost, which was becoming negligible, but in mobile applications is becoming significant again. There are ascending performance costs because Moore’s Law doesn’t apply to batteries.

Features have a documentation cost. Every feature adds pages to the manual, increasing training costs. Features that offer value to a minority of users impose a cost on all users. So, in designing products and programming languages, we want to get the core features—the good parts—right because that is where we create most of the value.

We all find the good parts in the products that we use. We value simplicity, and when simplicity isn’t offered to us, we make it ourselves. My microwave oven has tons of features, but the only ones I use are cook and the clock. And setting the clock is a struggle. We cope with the complexity of feature-driven design by finding and sticking with the good parts.

It would be nice if products and programming languages were designed to have only good parts.

Renaming a single-label domain to a FQDN

Evan Hoffman — Thu, 07 Apr 2011 21:56:54 +0000

Long ago — eons, perhaps — before I had anything to do with the Windows environment here, someone created the AD domain in my company as a single-label domain (e.g. instead of “example.com” our domain is just “example”). Over the years this has led to lots of “fun” on the part of Windows admins who’ve worked here as the implications of this choice became more apparent.

Since I inherited this system about a year ago, I haven’t really bumped up against any problems stemming from the single-label domain issue… until now. I recently attempted to add a new Windows 2008r2 file server to our DFS replication group/namespace. This totally failed for some mysterious reason. Well, I shouldn’t say “totally” failed, as I was able to add it to the DFS replication group, but unable to add it to the DFS namespace. In my attempt to debug the namespace issue, I deleted the namespace and attempted to recreate it, but just kept getting this error: The namespace cannot be queried. The specified domain either does not exist or could not be contacted.. I couldn’t do anything with the namespace – even clicking on it in the DFS Management console brought up an error. After some searching I found that this was likely due to having a single-label domain. I wasn’t sure why the error was happening even on Windows 2003 machines though, maybe joining a 2008r2 box to the domain made some schema changes? I tried a few suggestions like editing the hosts file but nothing seemed to resolve this.

Fortunately, we didn’t really need DFS namespaces and were able to just direct everybody to the fileserver via its DNS name, though as you can imagine this was clumsy. However, since this has been a problem since time immemorial, I figured it was time to see if it was fixable. After some quick searching, I found RENDOM. However, after even more searching I discovered this TechNet article which says:

The domain rename operation is not supported in Microsoft Exchange Server 2007 or Exchange Server 2010. DNS domain rename is supported in Exchange Server 2003. However, renaming of the NetBIOS domain name is not supported in any version of Exchange Server. Other non-Microsoft applications might also not support domain rename.

Well. We’re running Exchange 2010. So now what? I guess we’re going to have to create a second domain and migrate over to it. We’d already discussed this as a likely way of implementing the rename anyway, since it didn’t seem like “RENDOM” had any rollback procedure – it either just works (hahaha) or semi-works and semi-fails, leaving a wake of destruction throughout AD. Building a second domain seems like a lot of work, but at least we can move users over one at a time, and we get the side benefit of starting fresh, outgrowing the 5+ years of crud that’s accumulated in our AD.

Guess we’ll see what happens. Neither option seems like much fun. I guess the alternative is do nothing, but Microsoft clearly doesn’t think very highly of single-label domains, and anyone who asks about them gets looked at funny. At least it gives us something to do!

More fun parsing BIND query logs

Evan Hoffman — Mon, 13 Dec 2010 22:21:59 +0000

“But wouldn’t it be cool if it also…” That phrase usually triggers a lot of wasted cycles in my brain, though it sometimes comes up with something neat. I added a super lame graph to the DNS QPS parser. Makes it really easy to see peaks & troughs in usage:

2010-12-12 00:00 to 00:59 => 3237, rate 0.899 queries/sec	***********
2010-12-12 01:00 to 01:59 => 2709, rate 0.752 queries/sec	**********
2010-12-12 02:00 to 02:59 => 2473, rate 0.687 queries/sec	*********
2010-12-12 03:00 to 03:59 => 2171, rate 0.603 queries/sec	********
2010-12-12 04:00 to 04:59 => 2048, rate 0.569 queries/sec	*******
2010-12-12 05:00 to 05:59 => 1918, rate 0.533 queries/sec	*******
2010-12-12 06:00 to 06:59 => 2120, rate 0.589 queries/sec	********
2010-12-12 07:00 to 07:59 => 2477, rate 0.688 queries/sec	*********
2010-12-12 08:00 to 08:59 => 2994, rate 0.832 queries/sec	**********
2010-12-12 09:00 to 09:59 => 3704, rate 1.029 queries/sec	*************
2010-12-12 10:00 to 10:59 => 4297, rate 1.194 queries/sec	***************
2010-12-12 11:00 to 11:59 => 4744, rate 1.318 queries/sec	****************
2010-12-12 12:00 to 12:59 => 5106, rate 1.418 queries/sec	******************
2010-12-12 13:00 to 13:59 => 5311, rate 1.475 queries/sec	******************
2010-12-12 14:00 to 14:59 => 5083, rate 1.412 queries/sec	*****************
2010-12-12 15:00 to 15:59 => 4855, rate 1.349 queries/sec	*****************
2010-12-12 16:00 to 16:59 => 5179, rate 1.439 queries/sec	******************
2010-12-12 17:00 to 17:59 => 4959, rate 1.377 queries/sec	*****************
2010-12-12 18:00 to 18:59 => 4693, rate 1.304 queries/sec	****************
2010-12-12 19:00 to 19:59 => 4792, rate 1.331 queries/sec	****************
2010-12-12 20:00 to 20:59 => 4799, rate 1.333 queries/sec	****************
2010-12-12 21:00 to 21:59 => 5068, rate 1.408 queries/sec	*****************
2010-12-12 22:00 to 22:59 => 4672, rate 1.298 queries/sec	****************
2010-12-12 23:00 to 23:59 => 4514, rate 1.254 queries/sec	****************
2010-12-13 00:00 to 00:59 => 3917, rate 1.088 queries/sec	**************
2010-12-13 01:00 to 01:59 => 3458, rate 0.961 queries/sec	************
2010-12-13 02:00 to 02:59 => 2874, rate 0.798 queries/sec	**********
2010-12-13 03:00 to 03:59 => 2715, rate 0.754 queries/sec	**********
2010-12-13 04:00 to 04:59 => 2690, rate 0.747 queries/sec	*********
2010-12-13 05:00 to 05:59 => 2719, rate 0.755 queries/sec	**********
2010-12-13 06:00 to 06:59 => 2831, rate 0.786 queries/sec	**********
2010-12-13 07:00 to 07:59 => 3416, rate 0.949 queries/sec	************
2010-12-13 08:00 to 08:59 => 4962, rate 1.378 queries/sec	*****************
2010-12-13 09:00 to 09:59 => 6943, rate 1.929 queries/sec	************************
2010-12-13 10:00 to 10:59 => 8296, rate 2.304 queries/sec	****************************
2010-12-13 11:00 to 11:59 => 8938, rate 2.483 queries/sec	******************************
2010-12-13 12:00 to 12:59 => 8926, rate 2.479 queries/sec	******************************
2010-12-13 13:00 to 13:59 => 8950, rate 2.486 queries/sec	******************************
2010-12-13 14:00 to 14:59 => 2173, rate 0.604 queries/sec	********

You can see pretty quickly that 4-5 AM (Eastern) is the period of lowest activity, as you’d probably expect for a US-based US-centric site. Modified perl script below.

Perl script to calculate DNS queries per second in BIND (named)

Evan Hoffman — Tue, 07 Dec 2010 22:53:43 +0000

I’m pricing out DNS providers and was asked what our current queries-per-second currently are. Sadly I had no idea. After lots of Googling I decided there was really no good way to get this information so I decided to parse the logfile myself.

First, I turned on logging with timestamp in named.conf:

logging
{
...
        # Query logging 2010-12-07
        channel query-log {
                file "data/queries.log" versions 3 size 10m;
                print-time yes;
        };
        category queries { query-log; };
};
options
{
...
        # Query logging 2010-12-07
        querylog yes;

};

Then I restarted named and had to figure out how to parse the file. My first guess was Perl and it didn’t disappoint. Here’s the hideous mess of code:

parsebind.pl

#!/usr/bin/perl
 
#use Date::Parse;
use Time::ParseDate;
 
my $line_num = 0;
my $first_line = 0;
my $oldest_record = 0;
my $line = '';
my $date = 0;
 
my %query_counter = {};
my @abbr = qw( Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec );
 
while (<>) {
	chomp;
	$line = $_;
 
	#my ($date, $time, $ip, $hostname, $in, $type, $crap) = $line  =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/;
	if ($line =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/) {
		$date = parsedate("$1 $2");
#		print "$1 $2 => $date\n";
		if ($oldest_record == 0) {
			$oldest_record = $date;
			$first_line = $line_num;
			print "First line:\t$line_num\tdate\t".localtime($date)."\n";
		}
		my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($date);
		$year += 1900;
		$mon += 1;
		my $key = $year . '-' . sprintf('%02d',$mon) . '-' . sprintf('%02d',$mday). ' '.sprintf('%02d',$hour).':00 to '.sprintf('%02d',$hour).':59' ;
 
		if ($query_counter{$key}) {
			$query_counter{$key}++;
		} else {
			$query_counter{$key} = 1;
		}
	}
	$line_num++;
}
 
print "Last line:\t$line_num\tdate\t".localtime($date)."\n";
 
my $elapsed = $date - $oldest_record;
my $net_records = $line_num - $first_line;
my $rate = ($net_records * 1.0) / $elapsed;
 
print "Rate for $elapsed seconds: ".sprintf('%0.2f',$rate)." per second\n";
print "\n\n";
 
foreach my $key (sort keys %query_counter) {
	$rate = ($query_counter{$key} * 1.0) / 3600;
	print "$key => $query_counter{$key}, rate ".sprintf('%0.3f',$rate)." queries/sec\n";
}
 
exit 0;

And the beautiful output:

[root@ns3 ~]# perl ~evan/parsebind.pl /var/named/chroot/var/named/data/queries.log
First line:	308	date	Tue Dec  7 17:56:02 2010
Last line:	41289	date	Tue Dec  7 22:44:01 2010
Rate for 17279 seconds: 2.37 per second

2010-12-07 17:00 to 17:59 => 533, rate 0.148 queries/sec
2010-12-07 18:00 to 18:59 => 9185, rate 2.551 queries/sec
2010-12-07 19:00 to 19:59 => 8618, rate 2.394 queries/sec
2010-12-07 20:00 to 20:59 => 8075, rate 2.243 queries/sec
2010-12-07 21:00 to 21:59 => 8762, rate 2.434 queries/sec
2010-12-07 22:00 to 22:59 => 5808, rate 1.613 queries/sec
HASH(0x112382a0) => , rate 0.000 queries/sec
[root@ns3 ~]#

Not pretty but good enough to get a sense of QPS. Hope it helps someone.

Edit: It occurred to me that you may have more than one domain on a single bind server, and you may want the stats for a single one (rather than all aggregated together, which the above script does). The below version includes the second-level domain in the hash key for the final report:

#!/usr/bin/perl
 
#use Date::Parse;
use Time::ParseDate;
 
my $line_num = 0;
my $first_line = 0;
my $oldest_record = 0;
my $line = '';
my $date = 0;
 
my %query_counter = {};
my @abbr = qw( Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec );
 
while (<>) {
	chomp;
	$line = $_;
 
	#my ($date, $time, $ip, $hostname, $in, $type, $crap) = $line  =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/;
	if ($line =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/) {
		$date = parsedate("$1 $2");
		my ($domain) = ($4 =~ m/([\w\d\-]+\.[\w]+)$/);
		$domain = lc($domain);
#		print "$1 $2 => $date\n";
		if ($oldest_record == 0) {
			$oldest_record = $date;
			$first_line = $line_num;
			print "First line:\t$line_num\tdate\t".localtime($date)."\n";
		}
		my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($date);
		$year += 1900;
		$mon += 1;
		my $key = $domain .' '.$year . '-' . sprintf('%02d',$mon) . '-' . sprintf('%02d',$mday). ' '.sprintf('%02d',$hour).':00 to '.sprintf('%02d',$hour).':59' ;
 
		if ($query_counter{$key}) {
			$query_counter{$key}++;
		} else {
			$query_counter{$key} = 1;
		}
	}
	$line_num++;
}
 
print "Last line:\t$line_num\tdate\t".localtime($date)."\n";
 
my $elapsed = $date - $oldest_record;
my $net_records = $line_num - $first_line;
my $rate = ($net_records * 1.0) / $elapsed;
 
print "Rate for $elapsed seconds: ".sprintf('%0.2f',$rate)." per second\n";
print "\n\n";
 
foreach my $key (sort keys %query_counter) {
	$rate = ($query_counter{$key} * 1.0) / 3600;
	print "$key => $query_counter{$key}, rate ".sprintf('%0.3f',$rate)." queries/sec\n";
}
 
exit 0;

I suppose this doesn’t account for PTR (reverse DNS) lookups, but oh well.

Edit: I added some lame ASCII graphing functionality to the script, check it out in this post.