We wanted to setup a loadbalanced web cluster in AWS for expansion. My first inclination was to use ELB for this, but I soon learned that ELB doesn’t let you allocate a static IP, requiring you to refer to it only by DNS name. This would be OK except for the fact that our current DNS provider, Dyn, requires IP addresses when using their GSLB (geo-based load balancer) service.

Rather than let this derail the whole project, I decided to look into the software options available for loadbalancing in EC2. I’ve been a fan of hardware load balancers for a while, sort of looking down at software-based solutions without any real rationale, but in this case I really had no choice so I figured I’d give it a try.

My first stop was Nginx. I’ve used it before in a reverse-proxy scenario and like it. The problem I had with it was that it doesn’t support active polling of nodes – the ability to send requests to the webserver and mark the node as up or down based on the response. As far as I can tell, using multiple upstream servers in Nginx allows you to specify max_fails and fail_timeout, however a “fail” is determined when a real request comes in. I don’t want to risk losing a real request – I like active polling.

This led me to HAProxy. I’d never used HAProxy before but it seemed to be ideally suited to this (since it’s exclusively a load balancer). The option httpchk even allows for active polling of nodes – yay!

Unfortunately, HAProxy doesn’t support SSL. From the HAProxy site:

People often ask for SSL and Keep-Alive support. Both features will complicate the code and render it fragile for several releases. By the way, both features have a negative impact on performance :

Having SSL in the load balancer itself means that it becomes the bottleneck. When the load balancer’s CPU is saturated, the overall response times will increase and the only solution will be to multiply the load balancer with another load balancer in front of them. the only scalable solution is to have an SSL/Cache layer between the clients and the load balancer. Anyway for small sites it still makes sense to embed SSL, and it’s currently being studied. There has been some work on the CyaSSL library to ease integration with HAProxy, as it appears to be the only one out there to let you manage your memory yourself.

Poop! I figured out a workaround however, by using both Nginx and HAProxy on the same instance. HAProxy listens on port 80 and 8443 (so that it can relay decrypted SSL traffic to the nodes on a separate port, so that the nodes are aware that it was originally SSL traffic). Nginx is configured as a reverse proxy, listens on port 443 only, and has the SSL cert & key. The upstream for the Nginx is just localhost:8443 – HAProxy.

This was pretty easy to setup and works very well. I benchmarked HAProxy on an EC2 t1.micro instance (in front of two m1.large instances running our webapp) using ab -n 5000 -c 50 -t 60 and found it actually performed better than one of our hardware load balancers. That was pretty eye-opening (and sad).

The HAProxy and Nginx configs are below, in the hopes that it helps someone. The main warning I’d give is that using this will cause the logs on your nodes to interpret all requests as coming from the IP of the load balancer. I had to rewrite some code to have the app use the X-Forwarded-For address rather than the REMOTE_ADDR, but other than that this has been working out pretty well.

/etc/nginx/nginx.conf
Main thing is to make sure the server isn’t listening on port 80 (since HAProxy needs to).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

user              nginx;
worker_processes  1;
 
error_log  /var/log/nginx/error.log;
 
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    keepalive_timeout  65;
 
    #
    # The default server
    #
    server {
        listen       81;
        server_name  _;
 
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
 
        error_page  404              /404.html;
        location = /404.html {
            root   /usr/share/nginx/html;
        }
 
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
 
    }
 
    # Load config files from the /etc/nginx/conf.d directory
    include /etc/nginx/conf.d/*.conf;
 
}

/etc/nginx/conf.d/ssl-offloader.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

upstream haproxy {
        server localhost:8443 ;
}
 
server {
        listen       443;
        server_name f.q.d.n 1.2.3.4 ; # I put the FQDN and IP here, but maybe "_" will work too
#  server_name  _;
 
        ssl                  on;
        ssl_certificate      /etc/nginx/ssl-cert/cert.pem;
        ssl_certificate_key  /etc/nginx/ssl-cert/cert.key;
 
        ssl_session_timeout  5m;
 
        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;
 
        location / {
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
 
                proxy_pass http://haproxy/;
                proxy_redirect default;
                proxy_redirect http://$host/ https://$host/;
                proxy_redirect http://hostname/ https://$host/;
 
                proxy_read_timeout 15s;
                proxy_connect_timeout 15s;
        }
 
}

/etc/haproxy/haproxy.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2
 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
 
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
 
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    3s
    timeout queue           1m
    timeout connect         2s
    timeout client          5s
    timeout server          5s
    timeout http-keep-alive 1s
    timeout check           10s
    maxconn                 3000
 
       stats enable
       stats auth evan:change_me_brother
 
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main_http *:80
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_http
 
frontend main_https *:8443
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_https
 
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend web_http
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:80 check port 80
    server  node2 192.168.1.30:80 check port 80
    server  node3 192.168.1.40:80 check port 80
 
 
backend web_https
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:8443 check port 8443
    server  node2 192.168.1.30:8443 check port 8443
    server  node3 192.168.1.40:8443 check port 8443

I got my wife an iPad 2 for Christmas and she soon started complaining about the Wifi dropping its connection. I suggested she try turning off the “auto join” wifi setting, but that didn’t help. She’d be doing something and get the “Sorry, there’s no internet connection” error every 5-10 minutes. We’ve had FiOS for quite a while and we have 8 or 9 other devices connected (including Macs & iPhones) to the router without issue, so this seemed weird. I was starting to think it was a problem with the iPad, but we went to a friend’s house and used his wifi (with a Netgear router) and the iPad had no issues.

Back home, I logged into the router and tried assigning her iPad a static IP through DHCP. I had her release and renew and she got the new IP but the problem continued. Since we ruled out a problem with the iPad and I knew there was nothing “wrong” with the router, I figured I’d check and see if there are any reported issues with iPads and the Verizon router. Sure enough, there are. The first thing I clicked on, Fix for Verizon FIOS vs. iPad Wi-Fi Issues, suggested changing the wifi channel from “Automatic” to “6″ (it also suggests switching from WEP to WPA2-PSK, which I’ve always been using). I did that and it hasn’t dropped the wifi connection at all in the past 3 hours. Very odd issue. If I could get into the Actiontec (or the iPad for that matter) I’d like to check the logs and see what’s actually happening, but a win’s a win.

So, I’ve written about Compellent a few times from a price perspective, mostly on the disk side. I was recently contacted by our vendor with quotes for two new Compellent controllers. “What’s this all about?” I asked. “Why don’t we have a call with Compellent to discuss?” he replied. I rolled my eyes a little but figured it was worth hearing them out, since our Compellent SAN is at the heart of our infrastructure.

We currently have two controllers setup in failover mode. The first was bought in 2008 and the other in 2010 to add redundancy. Earlier this year we upgraded to the latest software version in preparation for moving our production DB onto the SAN, to allow us a nice window before we had to perform another upgrade (which would now risk DB downtime… I like failover but I don’t trust it enough to have a DB up during a failover), so I was kind of skeptical about any sort of upgrade to begin with.

On the call, the Compellent reps explained that they’ve dropped Fibre Channel connectivity between the controller and the disk enclosure, and the purpose of the upgrade is to give us SAS. In addition, they no longer sell SATA (!). I asked why we couldn’t simply add SAS cards to our existing controllers and was told that our current controllers are PCI-X, so can only support up to 3Gb/s SAS, while the new controllers have PCI-e and support 6Gb/s. And they want to ensure that we have the best possible performance. Pretty sure someone said the new controllers “have the future built in” to them.

One of the features we really liked about Compellent from the beginning was the fact that it was basically a software solution on top of commodity hardware. They stressed this point repeatedly. “When new technology comes out, we can just add a new card into your existing controller.” I think the example at the time was 10-gig Ethernet, but it seems like the same logic would apply to SAS. I understand that PCI-X doesn’t support 6Gb/s SAS, but it’s a tough pill to swallow that if we want to expand our SAN at all now, on top of whatever the actual expansion costs, we’re going to need to plunk down some serious money to upgrade the controllers, which really seems like a net-zero for us. We’re not going to ditch our existing FC enclosures so we’re going to be limited to 4Gb/s anyway. If they’re only selling SAS, well, that sucks for us, but ok. But why can’t we just throw a $500 PCI-X 3Gb/s card in to expand? So we’re not running at peak performance. I doubt that would be our performance bottleneck anyway. Plus, swapping out controllers is a huge operation for us.

I know at some point we’re going to have to bite the bullet and do this upgrade, but it just irks me. On the bright side, I guess, we don’t have to do a “forklift upgrade,” and the disks/enclosures will all still work. But we have a long way to grow before we need to expand, so fortunately I can put this off for a while.

After my 10-year-old basement Linux server died this week from a power outage, I took the sad step of giving up on it. It’s died before and I’ve patched it back together with a new power supply here or an addon PCI SATA card there, but I finally decided to throw in the towel since I had a newer old computer that had been idle for several years. The one that died was an Athlon K7 750 MHz with 512 MB ram. The new one is an Athlon 2 GHz (3200+) with 1 gig. For my uses, specs don’t really matter that much, but it’s nice to have more power for free.

I put CentOS 6 on it and configured Samba and copied all the data off the old machine and was back up and running within a few hours. Since I forward ports through my FiOS router to this box I did my standard lockdown procedure, including adding myself to the AllowUsers in sshd_config. Afterwards I took a look in /var/log/secure and saw the typical flood of dictionary attacks trying to get in as root or bob or tfeldman or jweisz. I have iptables configured to rate-limit SSH connections to 2 per 5 seconds per IP so the box doesn’t get DoSed out of existence, but some stuff does make it through to sshd.

Looking through /var/log/secure, I got to thinking it would be interesting if there was some way to visualize the attacks in a handy graph. Then I remembered, oh, wait, I can do that.

I wrote a perl script to parse out the attacks from /var/log/secure and insert them into a Postgres DB. This turned out to be pretty easy. Then I thought it would be more interesting to tie the IP of each attack to its originating country. I’ve used MaxMind’s GeoIP DB pretty extensively before, but I was looking something free. That’s when I remembered that MaxMind has a free GeoIP DB: GeoLiteCity. I grabbed it and yum-installed the Perl lib and added the geo data to the attack DB. Rather than worry about normalizing the schema I just shoved the info into the same table. Life is easier this way, and it’s just a for-fun project.

So I got that all working and parsed it against the existing /var/log/secures via

[root@lunix2011 ~]# zcat /var/log/secure-20111117.gz | perl parse-secure.pl

I wrote ssh.php to see what’s in the table:

ssh.php list of hacking attempts

So now that the data was all in place, time to move on to the graphs, which is what I really wanted to do. Last time I wanted to graph data programmatically I used JPGraph, which does everything in PHP and is super versatile. But I wanted something… cooler. Maybe something interactive. A little Googling turned up Highcharts which is absolutely awesome, and does everything in JavaScript. I basically modified some of their example charts and pumped my data into them and got the charts below.

Pie chart of attacks grouped by country for the past 30 days:

Pie chart by country

Bar graph of attacks per day:

Bar graph of daily attacks

So, that’s that. Code is in github if anyone wants to play around with it. I’ve cronned parse-secure.pl to run every 5 minutes so the data gets updated automatically.

I’m currently working on moving a Tomcat-based application into EC2. The code was written for Java 5.0. While Java 6 would probably work, I’d like to keep everything as “same” as possible, since EC2 presents its own challenges. I spun up a couple of t1.micro instances and copied everything over, including the Java 5 JDK, jdk-1_5_0_22-linux-amd64.rpm. Installing from RPM was easy, but the EC2 instance defaults to using OpenJDK 1.6:

[root@ec2 ~]# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.10) (amazon-52.1.9.10.40.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

There were a couple of things I had to do to get the system to accept the Sun JDK as its “real” java.

Alternatives

Red Hat’s “alternatives” system is designed to allow a system to have multiple versions of a program installed and make it easy to choose which one you want to run. Unfortunately I’ve found the syntax a bit strange and always have to Google it, so I figured I’d document it here for posterity.

So here’s the default:

[root@ec2 ~]# alternatives --config java
 
There is 1 program that provides 'java'.
 
  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
 
Enter to keep the current selection[+], or type selection number:

Here’s how to add Sun java, assuming the java binary is in /usr/java/jdk1.5.0_22/jre/bin/java (where the RPM puts it).

[root@ec2 ~]# alternatives --install /usr/bin/java java /usr/java/jdk1.5.0_22/jre/bin/java 1
[root@ec2 ~]# alternatives --config java
There are 2 programs which provide 'java'.
 
  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
   2           /usr/java/jdk1.5.0_22/jre/bin/java
 
Enter to keep the current selection[+], or type selection number: 2
[root@ec2 ~]# java -version
java version "1.5.0_22"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_22-b03, mixed mode)

Yay! Unfortunately this doesn’t help with the other problem I had with Tomcat, which was that EC2 instances set the JAVA_HOME var to OpenJDK as well (/usr/lib/jvm/jre). Fortunately this is an easy fix as well.

Setting JAVA_HOME

The JAVA_HOME var is set in /etc/profile.d/aws-apitools-common.sh. Comment out this line:

export JAVA_HOME=/usr/lib/jvm/jre

Create a new file, /etc/profile.d/sun-java.sh, and put this in it:

export JAVA_HOME=/usr/java/jdk1.5.0_22/jre

Also in that file I added the following to instruct the JVM to process all dates in America/New_York, since that’s the timezone all of our other servers use, and it makes reading log files easier when all dates are in the same tz:

export TZ=America/New_York

(I found I had to do this even after pointing /etc/localtime to the correct zoneinfo – Java was stuck on UTC even after the rest of the system was using America/New_York.)

I’m not usually one for introspection, but I found this a few years ago and it’s stuck with me.

A careful man I want to be;
A little fellow follows me.
I do not dare to go astray
For fear he’ll go the self same way.

I cannot once escape his eyes,
Whate’er he sees me do, he tries.
Like me he says he’s going to be;
The little chap who follows me.

He thinks that I’m so very fine,
Believes in every word of mine.
The base in me he must not see;
The little chap who follows me.

I must remember as I go
Through summer’s sun and winter’s snow,
I’m building for the years to be;
The little chap who follows me.

Linux supports hot-adding disks but whenever I add a new vdisk in VMware the new disk doesn’t show up unless I reboot, which defeats the purpose of hot-add. This command forces a rescan of the bus:

echo "- - -" > /sys/class/scsi_host/host0/scan

dmesg shows the new disk has been found:

  Vendor: VMware    Model: Virtual disk      Rev: 1.0 
  Type:   Direct-Access                      ANSI SCSI revision: 02
 target0:0:2: Beginning Domain Validation
 target0:0:2: Domain Validation skipping write tests
 target0:0:2: Ending Domain Validation
 target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
 sdd: unknown partition table
sd 0:0:2:0: Attached scsi disk sdd
sd 0:0:2:0: Attached scsi generic sg3 type 0

Now, why there’s no “rescan_sata” command is something I can’t fathom, but that’s Linux for you.

In an attempt to teach myself Objective C, and because I couldn’t find anything that did what I wanted, I wrote a little utility to display the currently-playing iTunes track in the Mac taskbar. Originally I had it display the full track name right in the taskbar but it was too much text for such a small space (especially on a 1440×900 screen), so now you click a little musical note and it shows you the info in a menu.

Here’s a screenshot:

The code is all in github. If you’re looking for a similar utility, and are brave enough to try my first-ever Obj-C app, you can download it here. But the freshest version will probably be in the github project.

As an aside, I was surprised at how easy it was to cobble this together having never written ObjC before. I found some good examples that I mostly ripped off, but it was still remarkably easy to have the app listen to iTunes for track changes, etc.

I’m currently in the process of moving our DNS over to another provider and I was curious as to whether the old or new provider offers faster lookups. dig shows query times, but I didn’t want to just run that over and over. I decided to write something to do this, in Java since I like Java. I found this post, which has the meat of the work done already. I also read some of Sun’s JNDI/DNS lookup info, which was pretty dense. All I want to do is specify the name server’s IP and do the lookup. I don’t even really care about the result, just how long the query takes.

The thing I wrote only looks up A records, but can easily be modified to do CNAMEs or whatever. Here’s how you call it:

$ java -jar DNSTester.jar 4.2.2.2 www.google.com 25
Resolved www.google.com to 74.125.235.19 against NS 4.2.2.2
Performed 25 lookups in 233.29 milliseconds.  Average 9.3316ms per lookup.
 
$ java -jar DNSTester.jar 8.8.4.4 www.google.com 25
Resolved www.google.com to 74.125.226.146 against NS 8.8.4.4
Performed 25 lookups in 450.034 milliseconds.  Average 18.00136ms per lookup.

Code is in github here. Jar is available here.

Over a year ago, the hard drive in my primary desktop at home bricked itself and rather than going through the hassle of reinstalling Win7 on the new disk, I decided to go with FC12. I’ve been pretty happy with it in general, since I’ve always been partial to Red Hat and use CentOS primarily at work.

Last week I got the great idea to upgrade to FC14. In hindsight I can’t even recall what led me to try this, but it didn’t end well. I tried the “preupgrade” procedure, which appeared to do the entire upgrade from FC12 to FC14 in place. I left it overnight, and when I looked at it the next day it looked like it was done. I was in FC14 and everything looked ok. But then I tried syncing my photos over NFS and discovered nfs wasn’t running on my desktop. When I tried starting it, it failed. After some trial and error, I used the Google and found that this is just what happens when upgrading to FC14 due to changes between FC12 and FC14, namely the introduction of systemd.

In all the threads I read, the “solution” was a clean install of Fedora. I tried doing this without formatting my / (root) partition, since that had 500 gigs of my stuff on it, but it kept failing. What I ended up doing was downloading partedmagic, which is a totally awesome partitioning tool. If you’re familiar with Partition Magic, this is similar but Linux-based and free. I burned the iso to disc, booted to it, and shrunk my / partition from 900 GB to 850 GB, and created a new 50 GB partition at the end of my disk without losing any of my data:

FC15 Partitions

Once this was done, which took surprisingly little time, I did a net install of FC15. I opted for a net install rather than downloading the ISO because I feel that with FiOS it’s actually faster than reading a DVD, and avoids having to run “yum update” afterwards.

So, I ended up with FC15 clean-installed to the new “/” partition. I moved everything around so the old partition is mounted at /docs and has all my stuff in it. I’d heard that FC15 was causing an uproar but until I logged into Gnome 3 myself I didn’t really understand the fuss. It’s going to take some getting used to, but after adding the tint2 taskbar it’s not too awful.

But back to systemd. In FC15 I wanted to ensure Samba started at boot, since that’s how I share docs between my VMs and host. Chkconfig is still there, but based on my problems with NFS and systemd on FC14 I decided to look into it a bit and see if there’s a “new” way to enable stuff at startup. There is!

Instead of:

# chkconfig smb on

The command is:

# systemctl enable smb.service

Of course, when I did this it apparently fell back to using chkconfig for smb:

[root@evan-fedora ~]# systemctl enable smb.service
smb.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig smb on

It does say in the Fedora wiki that systemd respects chkconfig and vice versa, so I guess this post is kind of pointless and I should have just linked to the wiki. But, whatever.

Edit Jan 24, 2012: Deleted all the crap from this story and just left the recommended Apache and Nginx SSL cipher suites for maximum security without SSLv2 and without BEAST vulnerability (at least according to Qualys).

Apache httpd

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
SSLHonorCipherOrder on

nginx

        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;

Source:

• Qualys
• SSL checker

This just started happening again, with these errors appearing in the event viewer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.
 
Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/17/2011 6:47:07 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' terminated
unexpectedly. The process id was '3108'. The process exit code was
'0x800703e9'.
 
Log Name: Application
Source: Application Error
Date: 9/17/2011 6:46:30 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time
stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time
stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

After reviewing the IIS logs and the event logs, I think it has to do with the WebReady document viewer – the thing in OWA that renders and lets you view .doc attachments within the browser rather than forcing you to open Word or Excel. I think users were attempting to open corrupted files and that was causing it to crash. I’ve disabled Webready in EMC (Server Config -> CAS) and I’ll see what happens.

So I’ve gone back to Linux from Mac, due to the SSD issues I had with my Macbook Pro basically making VMware unusable. A Win7 VM would grind the guest and host to a halt on the 7200 RPM SATA OEM drive, and the SSD wouldn’t work, so I put the SSD back in my HP and installed CentOS 6 x86_64. Not really ideal, but at least it works.

CentOS 6 ships with Firefox 3.6.9, which is really old by now. Fine for a server, but I wanted FF6. I grabbed the Firefox 6.0.1 bz2 from VoxCast, unzipped it and copied it to /usr/lib64/firefox-6. Then I ran yum remove firefox to remove 3.6.9 and avoid any issues. I tried the instructions from my older post on this subject but for whatever reason it didn’t work – I’m guessing because I’m not using the distro’s Firefox RPM.

I followed these instructions to get the Flash 11 64-bit plugin installed. It still wasn’t working though.

I ended up creating a symlink:

ln -s /usr/lib64/mozilla/plugins/libflashplayer.so /home/evan/.mozilla/plugins/libflashplayer.so

When I started up Firefox after creating the symlink, Flash worked.

Additionally, to set Firefox 6 as the default browser, run gnome-default-applications-properties, select Custom, and paste /usr/lib64/firefox-6/firefox %s in the Command: field.

Update: I upgraded my home desktop PC from FC12 to FC14 last night and used the above procedure to install Firefox 6.0.2 on it with Flash 11. So, in case anyone was wondering, the above works for Fedora as well.

With hurricane Irene passing through this past weekend I quickly shopped for an AM/FM radio. I found this one: Sony ICF-S10MK2 Pocket AM/FM Radio. Cheap, runs on 2 AA batteries, and worked great. A++, would buy again. I actually bought 2, one for my mom. I ordered them on Friday and paid the $8 to bump each one up to overnight with Prime, and they were both delivered Saturday morning. Unfortunately I wasn’t home to receive it, and the package weathered the storm on my front step.

Times like these you learn the real value of low-tech. iPhone, useless. The only way of getting information was this wonderful $10 AM/FM radio. Thanks Sony and WLNG!

A few weeks ago I switched from my trusty old HP nc8430 to a Macbook Pro (MC118LL/A) that was left spare when another employee left. I mostly enjoyed using Linux but I was tired of dealing with weird quirks like having X lock up, essentially forcing me to do a hard reboot.

To transition, I copied my documents from Linux to Mac, then turned off the Linux laptop. Surprisingly I found I didn’t need to turn Linux back on at all.

Last week, I decided to put the final nail in Linux’s coffin by taking the SSD (Corsair CMFSSD-128GBG2D)out of it and putting it in my Macbook. The Macbook was pretty fast (Core 2 Duo @ 2.5 GHz) but some things were noticeably slower on its 7200RPM disk than on Linux with an SSD, especially running Windows VMs.

I booted Linux to Knoppix and zeroed out the disk, then removed it. I backed my Mac up with Time Machine, shut it down, then undid the ~12 tiny screws, removed the bottom plate of the Macbook and popped the SSD in. I booted from the Mac DVD, restored from Time Machine and went home (it took ~4 hours to restore).

When I got in the next day, the restore was complete, though I had to click “Restart” to finish, which was annoying. Everything worked fine, and I was pretty impressed. The machine was kind of sluggish due to Spotlight indexing but once that was done I was pretty amazed at the transformation. Every app opened in under 1 second. Windows VMs were super snappy. Things were going well.

But then I started noticing periods of extended hanging. In the middle of some task, I’d get the beachball and the whole computer would become unresponsive (cursor would spin & move around but I couldn’t click anything). This would last about 30-60 seconds. I assumed it was some behind-the-scenes optimization, or some residual spotlight indexing.

Unfortunately, it hasn’t gone away. Earlier today I copied a 3 GB zip file from our file server to my laptop and it beachballed me on and off (about 60-70% of the time) for about 15 minutes as it copied. What’s odd is that the transfer speeds were pretty good, it appeared to be my computer itself that was bottlenecking it. After the download completed, I attempted to unzip it and was beachballed again. I checked Activity Monitor and it was peaking at 30 MB/s, but had extended periods of zeroes. I ran iostat and got basically the same information:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

EvanMBP:~ root# iostat -Kw 3
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    9.33   3  0.03     0.00   0  0.00   9  4 87  0.25 0.28 0.32
    0.00   0  0.00     0.00   0  0.00  13  5 82  0.23 0.27 0.32
   20.00   0  0.01     0.00   0  0.00  15  5 80  0.23 0.27 0.32
   36.00   1  0.02     0.00   0  0.00  16  6 77  0.21 0.27 0.32
    0.00   0  0.00     0.00   0  0.00  12  5 83  0.21 0.27 0.32
   20.02  60  1.18     0.00   0  0.00  17 10 74  0.19 0.26 0.32
   24.60 363  8.72     0.00   0  0.00  18 13 69  0.17 0.26 0.31
   25.40 307  7.60     0.00   0  0.00  14 11 75  0.17 0.26 0.31
   21.95 426  9.14     0.00   0  0.00  15 12 73  0.32 0.29 0.32
   82.50 352 28.35     0.00   0  0.00  17 11 73  0.32 0.29 0.32
  809.70  84 66.41     0.00   0  0.00  22  9 69  0.29 0.28 0.32
    0.00   0  0.00     0.00   0  0.00   9  5 86  0.27 0.28 0.32
    0.00   0  0.00     8.89  11  0.09   9  5 86  0.27 0.28 0.32
    0.00   0  0.00     0.00   0  0.00   9  5 87  0.33 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  12  7 81  0.33 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  14  8 78  0.38 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.35 0.29 0.33
    0.00   0  0.00     0.00   0  0.00  10  6 84  0.35 0.29 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 84  0.32 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  11  6 84  0.32 0.29 0.32
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  10  5 84  0.30 0.28 0.32
    0.00   0  0.00     0.00   0  0.00  11  6 84  0.27 0.28 0.32
    0.00   0  0.00     0.00   0  0.00  11  5 84  0.27 0.28 0.32
    0.00   0  0.00     0.00   0  0.00   9  5 85  0.25 0.27 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.25 0.27 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.31 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.45 0.31 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.45 0.31 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.41 0.31 0.33
    0.00   0  0.00     0.00   0  0.00  10  5 85  0.41 0.31 0.33
  384.49  10  3.85    10.00   0  0.00  12  6 82  0.32 0.29 0.32
  291.73 126 35.78     0.00   0  0.00  23 13 64  0.38 0.31 0.33
  236.65 338 78.20     0.00   0  0.00  30 17 53  0.34 0.30 0.33
  397.61  21  8.02     0.00   0  0.00  15  8 77  0.34 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.32 0.30 0.32
    0.00   0  0.00     0.00   0  0.00  12  7 81  0.32 0.30 0.32
    0.00   0  0.00     0.00   0  0.00  12  6 82  0.29 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.35 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  13  6 80  0.35 0.30 0.33
    0.00   0  0.00     0.00   0  0.00  11  6 83  0.32 0.30 0.32
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  13  8 78  0.32 0.30 0.32
    0.00   0  0.00     0.00   0  0.00  11  7 82  0.29 0.29 0.32
    0.00   0  0.00     0.00   0  0.00  10  5 86  0.35 0.30 0.33
  148.94 124 18.03     0.00   0  0.00  18  8 73  0.35 0.30 0.33
  267.73 121 31.72     0.00   0  0.00  17  8 76  0.32 0.30 0.32
  355.54 162 56.35     0.00   0  0.00  22  8 69  0.32 0.30 0.32
  738.07  38 27.38     0.00   0  0.00  16  6 78  0.30 0.29 0.32
  512.42  67 33.52     0.00   0  0.00  20  7 73  0.27 0.29 0.32
  835.74  61 50.05     0.00   0  0.00  19  7 73  0.27 0.29 0.32
  536.83  69 36.17     0.00   0  0.00  17  6 76  0.49 0.33 0.34
  543.89  83 43.90     0.00   0  0.00  20  8 72  0.49 0.33 0.34
  720.70  59 41.74     0.00   0  0.00  18  7 76  0.45 0.33 0.33
  541.23 124 65.70     0.00   0  0.00  22  9 70  0.41 0.32 0.33
  260.54 210 53.37     0.00   0  0.00  22  9 70  0.41 0.32 0.33
  806.93  73 57.78     0.00   0  0.00  20  8 72  0.46 0.33 0.34
  874.98  43 37.02     0.00   0  0.00  13  7 80  0.46 0.33 0.34
    0.00   0  0.00     0.00   0  0.00  11  4 85  0.42 0.33 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.39 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   9  4 87  0.39 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.44 0.33 0.33
          disk0           disk1       cpu     load average
    KB/t tps  MB/s     KB/t tps  MB/s  us sy id   1m   5m   15m
    0.00   0  0.00     0.00   0  0.00  11  5 84  0.44 0.33 0.33
    0.00   0  0.00     0.00   0  0.00  13  6 81  0.48 0.34 0.34
    0.00   0  0.00     0.00   0  0.00  10  4 86  0.44 0.34 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.44 0.34 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.41 0.33 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.41 0.33 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 87  0.38 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   7  4 89  0.35 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 87  0.35 0.32 0.33
    0.00   0  0.00     0.00   0  0.00   7  3 89  0.32 0.31 0.33
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.32 0.31 0.33
    0.00   0  0.00     0.00   0  0.00   7  3 90  0.29 0.31 0.33
  347.11  92 31.29     0.00   0  0.00  20  8 72  0.43 0.34 0.33
   49.98 656 32.03     0.00   0  0.00  32 10 59  0.43 0.34 0.33
  113.45 351 38.90     0.00   0  0.00  40 15 45  0.47 0.35 0.34
  819.41  34 27.20     0.00   0  0.00  26 12 63  0.47 0.35 0.34
  686.99  50 33.76     0.00   0  0.00  18  7 76  0.52 0.36 0.34
  878.17  23 20.01     0.00   0  0.00  20  8 72  0.47 0.35 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 87  0.47 0.35 0.34
    0.00   0  0.00     0.00   0  0.00   8  4 88  0.44 0.34 0.34

You can see in the MB/s column, lots of “0.00″ followed by some bursts of ~30 MB/s. The zeros didn’t actually print at the time, but flooded the screen in bursts when the bottleneck cleared up. It seems to me like it might be some problem with queueing or caching, or maybe the SATA controller on this Mac just isn’t up to the task of SSDs. I’m not sure, but at this point I’m afraid I might have to go back to the 7200 RPM Seagate that came with the Mac. 30-second hangups are far more annoying than having lots of things be slower. Kind of a strange amortization, if you think about it. Anyway, I’ll keep looking into it, now that I know how to reproduce the problem (unzip a huge file).

Updated: A quick Google search for “beach ball mac SSD” turned up this thread which seems to be about this same problem, with a different model SSD. Also referenced in this thread on Apple.com. It feels like the problem may be due to an “old” SSD.

Updated again: Here’s someone having the problem with a Corsair 128 GB SSD: http://forum.corsair.com/v3/showthread.php?t=91061.

Updated again: According to this post, this appears to be a problem with the SATA controller in the 2009 Macbooks. Bah.

The FCC recently conducted a study of some of the top broadband ISPs in the country and measured customers’ actual bandwidth as compared to what the ISPs advertised. FiOS really came out on top.

The report is available on the FCC site. The bottom line, though, is that Verizon FiOS averaged nearly 120% of advertised speed (i.e., more than was advertised) and Cablevision was between 50% and 75% of advertised speeds. Latency (ping) was also heavily in FiOS’s favor.

FCC - Fios vs Cablevision

More than 78,000 consumers volunteered to participate in this study and a total of approximately 9,000 consumers were selected as potential participants and were supplied with specially configured routers. The data in this Report is based on a statistically selected subset of those consumers—approximately 6,800 individuals—and the measurements taken in their homes during March 2011. The participants in the volunteer consumer panel were recruited with the goal of covering ISPs within the U.S. across all broadband technologies, although only results from three major technologies—DSL, cable, and fiber-to-the-home—are reflected in
this Report. To account for network variances across the United States, volunteers were recruited from the four Census Regions: Northeast, Midwest, South, and West. Within each Census Region, consumers were selected to represent broadband performance in three typical speed ranges: less than 3 Mbps, between 3 and 10 Mbps, greater than 10 Mbps.

The testing methodology itself required innovation on both the consumer, or “client,” side and on the ISP, or “server,” side. The server-side infrastructure, which comprised reference measurement points that were distributed geographically across nine different U.S. locations, was made available to SamKnows for the project by M-Lab, a non-profit organization that supports Internet research activities. Each consumer participant’s broadband performance was measured from a hardware gateway in his or her household to the off-net test node that had the lowest latency to the consumer’s address.

On the “client” side of the test, consumers self-installed a measurement gateway that was provided by SamKnows. These gateways, or “Whiteboxes,” were installed between the consumer’s computer and Internet gateway and came pre-loaded with custom testing software. The “Whitebox” software was programmed to automatically perform a periodic suite of broadband measurements while excluding the effects of consumer equipment and household broadband activity. This approach permitted a direct measure of the broadband service an ISP delivered to a consumer’s household.

Anyone who’s experienced both Cablevision and Verizon can probably corroborate these results. Unless Cablevision significantly improves their services (DVR/TV as well as Internet), at the same price there’s no way I’d go back to Cablevision; I’d even stick with Verizon if it meant spending $10-$20 more per month, the service is that much better.

RT has its own internal accounting & tracking system for logging activity, but I was interested in even more granular stuff, like seeing who looked at which tickets. I figured it wouldn’t be that hard to log this in Apache. Well, I was kind of right, in that it wasn’t “hard,” but it took me a long time to find the right place to do it. I did finally get it though.

httpd.conf

In httpd.conf I created a new LogFormat:

LogFormat "%h %l %{RTUSER}e %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined-rt

So instead of the HTTP-auth user, it puts the RT user in this field. Make sure to update your VirtualHost config to use the combined-rt LogFormat.

/usr/share/rt3/html/autohandler

In /usr/share/rt3/html/autohandler, right under this section:

# If we've got credentials, let's serve the file up.
if (    ( defined $session{'CurrentUser'} )
    and ( $session{'CurrentUser'}->Id ) )
{

Add this:

        $ENV{'RTUSER'} = $session{'CurrentUser'}->UserObj->EmailAddress;

This populates the RTUSER environment variable with the currently-logged-in user’s email address.

Restart httpd and the RT user’s email address should now appear in access_log. Note that it will only appear for Perl pages (not gifs/jpgs or other static content, since Perl doesn’t process those):

10.0.0.10 - evan@example.com [08/Aug/2011:17:30:34 -0400] "GET /rt3/index.html HTTP/1.1" 200 46809 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"
10.0.0.10 - - [08/Aug/2011:17:30:34 -0400] "GET /rt3/NoAuth/images//css/rolldown-arrow.gif HTTP/1.1" 200 83 "https://help.example.com/rt3/NoAuth/css/3.5-default/main-squished.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"

A few people have emailed me asking me to integrate the perl code snippet into I wrote to strip illegal headers when sending email via Amazon SES into something actually usable. I’ve done so! I haven’t really tested this beyond sending some test emails, but here it is. Use this at your own risk, I make no warranty, blah blah blah.

This fix requires editing ses-send-email.pl, so I’d advise making a backup copy, though I imagine you can always get a fresh version from Amazon if necessary.

Open ses-send-email.pl in a text editor and find the read_message method. It should look like this:

Delete that and paste this in its place:

I tried it on the command line and it gave the output I expected – I haven’t tried integrating it with sendmail/postfix since I haven’t encountered this problem. Here’s the test message I attempted to send:

From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!
 
Now we're out of the header, into the body.  Grand!

From, To, and Subject are required headers. Chicken & Cats are illegal, X-Zombies should be ok since it’s X-ified. Here’s what happened when I tried to send with the unmodified ses-send-email.pl:

[Tue Aug 02 14:14:51 evan@EvanMBP 62 amazon-email]$ ./ses-send-email.pl --verbose -k aws-credentials -r
From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!
 
Now we're out of the header, into the body.  Grand!
 
 
<ErrorResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidParameterValue</Code>
    <Message>Illegal header 'Chicken'.</Message>
  </Error>
  <RequestId>5ffad294-bd33-11e0-b6e3-affca9ad1eb5</RequestId>
</ErrorResponse>
Illegal header 'Chicken'.

Illegal header error. Now with the modified version:

[Tue Aug 02 14:15:17 evan@EvanMBP 63 amazon-email]$ ./ses-send-email-x-headers.pl --verbose -k aws-credentials -r
From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!
 
Now we're out of the header, into the body.  Grand!
 
 
<SendRawEmailResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/">
  <SendRawEmailResult>
    <MessageId>000001318bb51442-c2cfb780-0604-4363-925a-54a57015e567-000000</MessageId>
  </SendRawEmailResult>
  <ResponseMetadata>
    <RequestId>64e75a47-bd33-11e0-9d09-8f08f31615ad</RequestId>
  </ResponseMetadata>
</SendRawEmailResponse>

No errors, and I received the email below (extraneous SMTP headers added by Barracuda/Exchange removed):

From: <evan@example.com>
To: <evan@example.com>
Subject: Email
X-Chicken: yummy
X-ASG-Orig-Subj: Email
X-Cats: yucky
X-Zombies: kill them!
Date: Tue, 2 Aug 2011 18:15:25 +0000
Message-ID: <000001318bb51442-c2cfb780-0604-4363-925a-54a57015e567-000000@email.amazonses.com>
Content-Type: text/plain
MIME-Version: 1.0
 
Now we're out of the header, into the body.  Grand!

Illegal headers have been X-ified. Hope this helps someone.

In a recent “debate” with a friend, I looked for historical data about the US public debt. I found Google Public Data, which has info about the annual budget deficit/surplus, but apparently (oddly) doesn’t have the debt. Odd because this info is available on the Treasury website.

I copied & pasted the data into Google Docs and made a chart of my own and some basic comparisons over time. Here are some of them rendered as images, though the scale makes the horizontal access kind of useless. I tried embedding the interactive Flash-based graph but it didn’t work. Oh well.

Having migrated our DB to a new machine, I was left with a pretty good machine unused. I decided to rebuild it and try out Postgres 9.0.

I downloaded and installed CentOS 6.0 x86_64, built Postgres 9.0.4 from SRPMS (makes me feel better building it myself), installed it and did initdb, dumped the primary DB to our NAS using psql 9.0 (pg_dump [db] | gzip > /path/to/file.gz) and began the restore.

Not much of a story but it took 33 hours to restore the DB, with the bulk of the time spent building indices. When the restore was complete I did an analyze verbose which took another 2 hours, for a total of 35 hours, not counting the time it took to create the dump file – about 14 hours, but this was with the DB under heavy load, made heavier by the dump, so I don’t know how long it would really take. I’m glad I didn’t have to do this to migrate, but I’m not relishing the idea of doing a real upgrade to 9.0 if it’s going to take this long. For the restore I disabled autovacuum and set fsync = off. It’s an ext4 filesystem on a 24-disk RAID 10 of 10krpm SAS drives.

DB size on disk after a clean restore was 1156 GB, as per pg_database_size(), versus 1237 GB on disk for the one that was migrated via bit-copy, so there’s a sizable chunk of crud accumulated in there.

Also, in doing this I discovered the logger command. One of the things that I’ve always found annoying about doing a pg restore is that the output is just a bunch of statements like this:

SET
SET
SET
CREATE SCHEMA
ALTER SCHEMA
CREATE SCHEMA
ALTER SCHEMA
SET
CREATE TABLE
ALTER TABLE
CREATE FUNCTION
ALTER FUNCTION

This is just what’s returned from the SQL commands, but it’s not very informative, and most importantly it doesn’t show you the timestamp so you have no idea how long anything’s taking. With logger, you can just pipe the output to it and it’ll be logged with syslog. So my restore command looked like this:

zcat /nfs/db/pgdump-90.sql.gz | psql -Upguser db 2> ~/restore.20110723-1916.log | logger -t PGRESTORE

This made it trivial to get the messages out of /var/log/messages with grep:

[root@link log]# grep PGRESTORE messages | head
Jul 24 03:36:43 link PGRESTORE: ALTER TABLE
Jul 24 03:36:43 link PGRESTORE: SET
Jul 24 03:36:44 link PGRESTORE: ALTER TABLE
Jul 24 03:36:44 link PGRESTORE: ALTER TABLE
Jul 24 03:38:45 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE
Jul 24 03:39:13 link PGRESTORE: SET
Jul 24 03:39:13 link PGRESTORE: ALTER TABLE

In addition to having the timestamps it eliminates the accumulation of the random log files that accumulate like droppings from myriad one-off scripts. I modified about 20 cron jobs to pipe their output to logger rather than junk files in /tmp or my homedir. I can’t believe I never heard of logger before! Hooray for logger!