For a few months I’d been using the WooThemes “Mainstream” theme and really liked it. I made a point of always keeping it updated, as I do with everything WordPress. A few minutes ago, however, on a whim I did a

netstat -a

on my webserver, and saw a bunch of connections from my server to setlinks.ru:http. I quickly grepped the wordpress directory for “setlinks” and sure enough it looks like some trojaned code made it in. Under my

/wp-content/themes/mainstream/cache/

dir there are these directories:

drwxr-xr-x 3 apache apache 4096 Dec 16 00:07 setlinks_fa356/
drwxr-xr-x 3 apache apache 4096 Dec 16 00:08 692ad897a15978e7cfd099ace86a56bf/
drwxr-xr-x 2 apache apache 4096 Dec 16 00:08 12483e2d235715e4ad4c76c8cf04f0fd76c8c397/

Under 692ad897a15978e7cfd099ace86a56bf there are a bunch of PHP scripts, including sape.php, which has a bunch of crap in it linking to db.linkfeed.ru.

So anyway, rather than investigate fully right now I’m scrapping the WooTheme altogether. So the site is going to look more boring, but oh well.

About 6 years ago my mom gave me her Canon Digital Rebel (EOS 300D) and I loved it. Prior to that I’d only used point-and-shoots and the quality of pictures I got with the DSLR was amazing.

After a few years I realized I really wasn’t able to take the photos I envisioned in my mind. This was certainly due to having now idea how to use the camera and just relying on full auto to take pictures. Finally, this past winter, I decided to try and teach myself how to take good photos, or at least ones that were closer to what I pictured in my mind. A few people told me, “take a class!” But who has time for a class? I decided to use my good friend YouTube. A couple of 10-minute videos were extremely helpful: this one on aperture:

And this one on shutter speed:

Those two videos taught me enough to realize that the kit lens on my Canon Digital Rebel, the original 18-55mm f/4-5.6, was kind of a piece of crap. I agonized for weeks over what kind of lens I should get, with the decision made for me by my wife getting me the Canon 55-250mm IS lens, which I wanted for taking photos of my son’s soccer games, for Christmas.

While the 55-250mm lens is fantastic, it’s really situational – not really useful for everyday shots. I ordered the other one I wanted, the Canon 50mm f/1.8, a few days later. After playing with the 50mm for a few days I regretted having waited so long to buy it. It made taking photos a joy. Being able to do have actual depth of field and interesting-looking photos with such an inexpensive lens was fantastic.

The Rebel was still showing its age, however. Some of the major annoyances with it were its lack of USB 2.0 (which made copying photos off it very slow) and use of CF rather than SD card (which meant I couldn’t stick the memory card directly into the computer to copy them off). I have a card reader, but it was still very inconvenience. It also had a fairly long interval between taking photos.

After complaining about my camera for (literally) years, my lovely wife and my mom got me the Canon T3i body for my birthday this year. I can say without reservation that it is the best gift I have ever received. With the T3i and the new lenses I’ve been able to take some amazing photos. While I still take plenty of lousy ones, I have been able to take some gems. Some of my favorites from the past month or so are below. I’m still a total amateur but the camera and lenses make it easy to take great shots. None of these are edited except for some cropping on some of them.

Based on reviews on Amazon and Newegg, I opted for the Wintec FileMate 16 GB SD card. I have a 32 GB Class 10 Transcend card that I use in my Kodak Zx3 movie camera and I’ve tried both in the T3i and the Wintec card is significantly faster than the Transcend, despite both being labeled as Class 10. I can hold the shutter button down on the camera and take 30-40 shots at maximum resolution & JPEG quality without any buffering/lag. Really a great card at a great price.

Now the proud owner of an awesome camera and 3 different lenses, I figured it was time to get a decent bag to hold it all. After some more weeks of agonizing, I settled on the Case Logic SLRC-202 and so far I really like it. It has a nifty “hammock” to support the camera body itself and two compartments for lenses. I managed to cram the charger, USB cable and driver CD in as well, so I have everything I need all the time.

In 2000 or 2001 I started playing EverQuest. I was relatively late to the game, joining after the 3rd expansion (Luclin) was already out. I played it for a few years, at one point obsessively. In truth, I was never really that good, but I was in a guild that I loved and for the most part we all had fun. Like in other MMOs, when you start playing EverQuest you need to choose which server to play on. Basically I picked a random one to create my character on: Tarew Marr (most servers in EQ were named after the deities in the game).

For anyone unfamiliar with MMO games (as I was at the time), your server choice affects almost every aspect of your gameplay experience. Each game server is essentially a parallel universe with the same buildings, NPCs (characters), quests, and items, but completely different sets of actual human beings inhabiting them. Servers develop their own personalities as a result of the players inhabiting them. Like any community, there are nice people, morons, trolls, elite players, people who totally sucked at the game, grownups, children, racists and any other variety of person you can imagine. This, of course, is exactly the appeal of any MMO. You can play a single-player game in complete safety, but when you add 2000 random strangers in to the mix things get pretty interesting. There was plenty of drama, and Tarew Marr was actually one of the crazier servers.

What’s probably unsurprising in 2012, but I found pretty strange in 2001, is that there was an “offline” community for Tarew Marr in the form of a phpBB message board. Once again, I found the site pretty late but the forum was addictive. Server goings-on were discussed there and other general topics were incorporated daily. It was a self-selecting group of people with ages ranging from 15 to 50 (more or less) from all over the world whose only real commonality was playing Everquest on the same server. People came and went but the community was relatively cohesive. Eventually the phpBB forum became more interesting than the game it was created to serve. As people stopped playing EQ and moved to other games (or just gave up gaming entirely), they kept their roots in the forum. There was plenty of drama in the forum as well, with the requisite crazy people, trolls and idiots making life interesting for everybody.

When World of Warcraft came out, I too quit EverQuest. I played WoW for a few years and eventually quit that as well, all the while regularly checking the forum, which had long since lost its gaming focus, and just became an “internet community” of unique people. The site itself underwent several incarnations as the domain in one case was sold by the admin, and in another the subsequent admin got tired of running the site and shut it all down. The most recent incarnation was hosted by a member of the community who acquired the MySQL dump of the DB from the previous admin. Once everything was back up and running, it was nice to be “back.” It’s hard having a community ripped away.

Things went ok for a few years. The newest admin was a good guy who knew what he was doing. In response to a flood of spam accounts that were created, he ended up disabling registrations (each account had to be manually approved by him). Since our site was closed to the outside world (you had to be logged in to see anything) and the game that was the original focus of the site was basically dead, this pretty much guaranteed we’d never have any new members in our community. This was kind of sad, but there wasn’t really anything anyone could do about it.

Then one day last year I went to the site and it was down with a message from Hostgator saying “Hey, your account’s expired.” It was like that for a few days. Eventually people got concerned and after some digging, it was discovered that the admin had killed himself. Another member of the forum contacted Hostgator and paid to turn the site back on. He tried to get admin access to the Hostgator account but apparently that requires the original user’s credit card info, so basically we were out of luck. We all contributed via paypal and kept the site up indefinitely (in the now-deceased admin’s name) until we could hopefully get admin access to either phpBB or MySQL and get a mysqldump and move it to a new site.

Unfortunately, this didn’t happen in time, and this morning the domain expired in Godaddy. I don’t know if Godaddy will allow someone else to pay the bill, but I figured now was a good time to see if there was a way to download an archived version of the site. That way, even if the site is gone forever, at least the history will exist somewhere. I tried this a few months ago with some flags to wget --mirror but wget would get stuck in this endless loop appending new session IDs and save hundreds of copies of each page/post/topic.

What I ended up doing that’s actually successful is iterating over the topic IDs starting at 1 up to the maximum (that I know of) and saving the output to disk. My script for doing this is below. It might not be ideal but it does get the job done for me.

But on a more “meta” level, this is a really strange situation. There’s a community of 30-40 people relatively used to each other. We could move to another site, but many people didn’t want to do this because they didn’t want to lose the history. Maybe now that I have the site archived, people will feel more comfortable moving. I’m still able to access the site because, being the nerd that I am, I did a dig against Hostgator’s nameservers, which still had the correct IP address of the site, and like other shared hosting platforms you can access the site via http://(ip address)/~(username)/ , and that’s how my perl script is able to fetch the data. But the other 30-40 people aren’t going to have any idea how to do that, so even if we move to a new site now, most people won’t know what that new site is. I suppose we could buy an ad in AdWords so if people search for the old site name the new one comes up, but that seems like a mess.

What a strange 21st-century problem. I imagine this has happened before in other online communities; the founder dies, or just leaves and the rest of the community is left just kind of drifting. Anyway, here’s my script. In order to fetch actual data as a logged-in user, I used LiveHTTPHeaders to see what headers my browser was actually sending and copied them directly into the script. You’ll need at least your SID. For whatever reason, when I logged in via IP address, my browser wasn’t sending any Cookie: headers, so those aren’t included in the script.

When I first started monkeying around with Nginx about a year ago, I approached it as a typical Apache fanboy. I’ve used Apache for 10+ years and it’s been a champ most of the time. Anyone familiar with Apache knows that you can do just about anything with it. It’s really the Swiss-army knife of webservers.

So it was with some skepticism that I installed nginx and started playing around with it. The first thing that struck me was the simplicity of the configuration. The second was the speed. I read this story on Ars Technica which detailed one guy’s experience replacing Apache with Nginx for serving zillions of static files – something at which Nginx excels. The third thing that struck me about Nginx was the philosophy of the project. From “Why Use It”

Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache.

– Chris Lea

So, to put in less pejorative terms, Apache is the Swiss-army knife while Nginx is an Xacto knife.

I quickly became a fan of Nginx, but I wasn’t going to dump Apache. Apache’s still great, and sometimes it’s just not worth the effort to gut a site and rebuild it with Apache. But after some more reading about Nginx features and use-cases, I started to realize I didn’t necessarily drop Apache to leverage some of Nginx’s awesomeness.

One of the most common uses for Nginx is as a reverse-proxy, sometimes as a load-balancer. But Nginx also makes it really easy to use it as a caching reverse-proxy. This is commonly done for caching static content in front of another webserver (sometimes Apache), letting you leverage the awesome performance of Nginx without losing any of Apache’s functionality. This is what I really wanted to do… I had a couple of Apache-based sites that were getting swamped, in some cases apparently due to the volume of image requests – prime candidate for reverse-proxying. Another was a WordPress blog on a severely underpowered server.

The config for caching reverse-proxy is really trivial; there are examples all over the internet, but here’s the gist:

proxy_cache_path  /var/lib/nginx/cache/staticfiles  levels=1:2   keys_zone=staticfilecache:60m inactive=90m  max_size=50m;
proxy_temp_path /var/lib/nginx/proxy;
proxy_connect_timeout 30;
proxy_read_timeout 120;
proxy_send_timeout 120;
 
upstream blah {
        server  127.0.0.1:88;
}
 
server {
 <blah blah>
        location ~* /evan/.+\.(jpg|png|gif|jpeg|css|js|mp3|wav|swf|mov|ico)$ {
                        proxy_cache_valid 200 120m;
                        expires 864000;
                        proxy_pass http://blah;
                        proxy_cache staticfilecache;
        }
}

This instructs Nginx to cache static content (images, js, audio, video) requests that result in a 200 OK from the upstream for 120 minutes. The proxy_cache_path directive describes the “caching area,” specifying the on-disk path for cached objects, how many objects to cache (60m), the maximum size of the cache (50 MB), and after how much inactivity to delete them (90 minutes). That’s a whole lot of awesome in a couple of config lines.

So, this all works, but I wanted to cache the actual WordPress content pages as well, since each page requires hitting the MySQL DB, and most of the sites I manage are rarely updated. I created another key zone (caching area) for PHP content and told it to cache for 20 minutes. There were some problems with this though:

1. I didn’t want to cache (or serve cached versions of) the WordPress admin pages. It’s important to distinguish between caching (the temporary storage on the proxy of the content generated by the source server) and serving of cached content, since Nginx lets you specify settings for both actions independently. The easiest workaround I found for this was putting a .htaccess file in the wp-admin directory with the following:

   SSLRequireSSL
   Header set Cache-Control "no-cache"
   Header set X-Accel-Expires "0"
   Header set Expires "Wed, 1 Jun 2011 20:00:00 GMT"

   Any of these headers should be enough to force Nginx not to cache the contents; I have them all in there just to be safe. SSLRequireSSL forces https for admin.

2. I didn’t want to cache (or serve cached versions of) any page while I was logged in. When logged in, there’s a black bar at the top of the page with links to “edit post” or “go to dashboard,” and I wasn’t it them in some cases (I was being shown the non-logged-in cached page), and in some cases it would cache the logged-in version of the page, showing it to non-logged-in people, both bad situations. Based on this example, I added this to my Nginx config:

   map $http_cookie $logged_in {
       default 0;
       ~wordpress_logged_in 1; # Wordpress session cookie
   }
   server {
   <bla bla>
           location / {
                   proxy_pass http://blah/;
                   proxy_redirect  http://blah/  http://$host/;
    
                   proxy_cache_bypass $logged_in;
                   proxy_no_cache $logged_in;
    
                   proxy_cache php;
                   proxy_cache_valid 200 30m;
                   expires 5m;
           }
   }

   If logged in, both proxy_cache_bypass and proxy_no_cache are set to “1″, so all my logged-in request bypass the cache entirely.

3. I have the WPTouch plugin installed so iPhone users see a more iPhone-friendly version (rather than the regular site shrunk down to the iPhone’s screen size. I didn’t want desktop users seeing cached iPhone versions of the pages, and vice versa. To solve this, I set a $mobile var if “iphone” or “android” appear in $http_user_agent and incorporate that value into the proxy_cache_key. This way every page is still cached, but mobile users see the mobile version, and desktop users see the desktop version:

           set $mobile "_not_mobile_";
           if ($http_user_agent ~* "iPhone") {
                   set $mobile "mobile";
           }
           if ($http_user_agent ~* "Android") {
                   set $mobile "mobile";
           }
           location / {
                   proxy_pass http://blah/;
                   proxy_redirect  http://blah/  http://$host/;
    
                   proxy_cache_key "$mobile.$scheme$host$request_uri";
    
                   proxy_cache_bypass $logged_in;
                   proxy_no_cache $logged_in;
    
                   proxy_cache php;
                   proxy_cache_valid 200 30m;
                   expires 5m;
           }

So far this config has been working out pretty well. Uncached pages still load slow, but subsequent cached loads are pretty quick. I tweak it regularly, but here’s what I have currently, with Apache listening on 8888:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85

# Wordpress reverse-proxy config
proxy_cache_path  /var/lib/nginx/cache/staticfiles  levels=1:2   keys_zone=staticfilecache:60m inactive=90m  max_size=50m;
proxy_cache_path  /var/lib/nginx/cache/php levels=2:2 keys_zone=php:30m inactive=60m max_size=50m;
proxy_temp_path /var/lib/nginx/proxy;
proxy_connect_timeout 30;
proxy_read_timeout 120;
proxy_send_timeout 120;
 
proxy_cache_key "$scheme$host$request_uri";
# http://wp-performance.com/2010/10/nginx-reverse-proxy-cache-wordpress-apache/
 
map $http_cookie $logged_in {
    default 0;
    ~wordpress_logged_in 1; # Wordpress session cookie
}
 
upstream apache {
        server  127.0.0.1:8888;
}
 
server {
        proxy_cache_valid 200 20m;
 
        listen 80 default_server;
        server_name _;
 
        access_log  /var/log/nginx/combined-access.log combined;
 
        proxy_set_header X-Real-IP  $remote_addr;
 
        proxy_set_header Host $host;
 
        proxy_set_header X-Forwarded-For $remote_addr;
 
        proxy_set_header X-NginX-Proxy true;
 
        set $mobile "_not_mobile_";
        if ($http_user_agent ~* "iPhone") {
                set $mobile "iphone";
        }
        if ($http_user_agent ~* "Android") {
                set $mobile "android";
        }
 
        location / {
                proxy_pass http://apache/;
                proxy_redirect  http://apache/  http://$host/;
        }
 
        location /evan/ {
                proxy_pass http://apache/evan/;
                proxy_redirect  http://apache/  http://$host/;
 
                proxy_cache_key "$mobile.$scheme$host$request_uri";
 
                proxy_cache_bypass $logged_in;
                proxy_no_cache $logged_in;
 
                proxy_cache php;
                proxy_cache_valid 200 30m;
                expires 5m;
        }
 
        location /evan/wp-admin/ {
                proxy_pass http://apache/evan/wp-admin/;
                proxy_redirect  http://apache/  http://$host/;
        }
 
        location ~* /evan/.+\.(jpg|png|gif|jpeg|css|js|mp3|wav|swf|mov|ico)$ {
                        proxy_cache_valid 200 120m;
                        expires 864000;
                        proxy_pass http://apache;
                        proxy_cache staticfilecache;
        }
 
        location = /50x.html {
                root   /var/www/nginx-default;
        }
 
        # No access to .htaccess files.
        location ~ /\.ht {
                deny  all;
        }
 
        }

A few years ago I wrote a utility in Java to find all JPG files in a directory and move them into a date-based directory structure like /YYYY/MM/DD/ based on the date the photo was taken, extracted from the exif metadata in the file. Well, apparently that was a huge waste of time, as I just discovered that exiftool, an awesome perl utility I’ve used for years to edit/extract the metadata on the command line, can also do this natively. So my entire program can be replaced with this simple command:

$ exiftool -r '-FileName<CreateDate' -d /targetDir/%Y/%Y-%m/%Y-%m-%d/%Y-%m-%d.%%f.%%e /media/EOS_DIGITAL/

This will copy the files directly off the SD card mounted at /media/EOS_DIGITAL/ into the proper structure in /targetDir/.

Recently we noticed weird behavior downloading files from certain sites. The transfer would start out fast (around 10 MB/s), then after a couple of seconds it would plummet to around 9 KB/s. It didn’t happen for every file or every site: downloads from S3 buckets were still particularly fast. But some files that I remember being particularly fast were now showing this weird fast/slow/fast/slow behavior, for example the Sun JDK and ISOs from rit.edu that used to saturate our pipe were now getting all cRAzY.

After some poking around I decided to test HTTP versus FTP to see if it could be an application/protocol-level issue. The easiest way to do this was to find a file available via both FTP and HTTP and download it via both protocols. This is where mirrors.rit.edu came in handy. I used cURL to download it and noticed that via HTTP it was much slower than over FTP:

[evan@boba 16:07:03 ~]$ curl -O ftp://mirrors.rit.edu/pub/centos/6/isos/x86_64/CentOS-6.2-x86_64-netinstall.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  227M  100  227M    0     0   9.8M      0  0:00:22  0:00:22 --:--:-- 7816k
[evan@boba 16:07:33 ~]$ rm CentOS-6.2-x86_64-netinstall.iso 
[evan@boba 16:07:39 ~]$ curl -O http://mirrors.rit.edu/centos/6/isos/x86_64/CentOS-6.2-x86_64-netinstall.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  227M  100  227M    0     0  5686k      0  0:00:40  0:00:40 --:--:-- 6269k

22 seconds via FTP at 9.8MB/s average, 40 seconds over HTTP at 5.6 MB/s average (which was one of the better HTTP runs).

This was affecting all machines on our network, and had nothing to do with the per-machine iptables rules (verified by flushing all rules). The only thing I could think of that might affect all machines, but only HTTP and not FTP would be something like packet inspection. Well, turns out that http packet inspection is on by default on the ASA. So I disabled it as described here:

Zeus(config)# conf t
Zeus(config)# policy-map global_policy
Zeus(config-pmap)# class inspection_default
Zeus(config-pmap-c)# no inspect http
Zeus(config-pmap-c)# write mem
Building configuration...

Since then HTTP transfers have been consistently fast.

I’ve been using MRTG and routers2.cgi for years to graph the various aspects of a server that warrant monitoring. I’ve long known that they used something called rrdtool to do… well, something, but never had a need or desire to figure out exactly what that was.

But, having just moved my site to a new server, I was curious how the server would handle the load. Rather than setting up some behemoth like Nagios or Zabbix, which are full monitoring/alerting suites, I just wanted graphing. As I said, in the past I’ve used MRTG or routers2.cgi for this but both of them were overkill for me in this case. Since both of them used rrdtool, I figured that was a good place to look.

The two metrics I want to record are server load and in/out bandwidth. The first step is to create the RRDs (round robin databases). This was done via these commands:

# rrdtool create /mrtg/load.rrd --start N DS:load1:GAUGE:600:0:100 DS:load5:GAUGE:600:0:100 DS:load15:GAUGE:600:0:100 RRA:AVERAGE:0.5:2:800
 
# rrdtool create /mrtg/eth1.rrd --start N DS:in:COUNTER:600:0:10000000000 DS:out:COUNTER:600:0:10000000000 RRA:AVERAGE:0.5:2:800

A good explanation of what these various fields mean is here. In short, each “DS:” section defines a “column” (for fellow RDBMS users) in the database. The first one has 3 “columns,” named load1, load5, load15, each of which will contain GAUGE data. The second one contains two COUNTER fields, representing the bytes in/out for interface eth1.

To actually get the data I poll snmpd via this bash script:

#!/bin/bash
 
rrdupdate /mrtg/load.rrd N:\
`/usr/bin/snmpget -v 2c -c public -Oqv localhost laLoad.1`:\
`/usr/bin/snmpget -v 2c -c public -Oqv localhost laLoad.2`:\
`/usr/bin/snmpget -v 2c -c public -Oqv localhost laLoad.3`
 
rrdupdate /mrtg/eth1.rrd N:\
`/usr/bin/snmpget -v 2c -c public -Oqv localhost ifInOctets.3`:\
`/usr/bin/snmpget -v 2c -c public -Oqv localhost ifOutOctets.3`

I have that run every 5 minutes via cron. Then to generate the actual graph, I run this script via cron:

#!/bin/bash
 
rrdtool graph /var/www/html/graphs/load.png \
        -N \
        -E \
        --start now-30hours \
        --title "Load Averages" \
        --width 300\
         --x-grid MINUTE:60:HOUR:2:HOUR:4:0:%H\
        --height 200 \
        -u 1.0 \
        --lower-limit 0\
        --vertical-label "Load Avg" \
        --full-size-mode \
-a PNG --title="Load Avg" \
'DEF:load1=/mrtg/load.rrd:load1:AVERAGE' \
'VDEF:load1last=load1,LAST' \
'DEF:load5=/mrtg/load.rrd:load5:AVERAGE' \
'DEF:load15=/mrtg/load.rrd:load15:AVERAGE' \
'AREA:load15#33CC33:15 Min Load Avg ' \
'LINE1:load1#0000ff:1 Min Load Avg ' \
'GPRINT:load1:AVERAGE:"Load1 Avg\:%3.2lf"' \
'GPRINT:load1last:Drawn at %Y-%m-%d, %H\:%M:strftime' 
#'LINE1:load5#ff00ff:5 Min Load Avg ' \
 
 
rrdtool graph /var/www/html/graphs/eth1.png \
        -N \
        -E \
        --start now-30hours \
        --title "eth1 traffic" \
        --width 300\
         --x-grid MINUTE:60:HOUR:2:HOUR:4:0:%H\
        --height 200 \
        -u 1000000 \
        --lower-limit 0\
        --vertical-label "bps" \
        --full-size-mode \
-a PNG --title="eth1 traffic" \
'DEF:eth1in=/mrtg/eth1.rrd:in:AVERAGE' \
'CDEF:eth1inbits=eth1in,8,*' \
'VDEF:eth1last=eth1in,LAST' \
'DEF:eth1out=/mrtg/eth1.rrd:out:AVERAGE' \
'CDEF:eth1outbits=eth1out,8,*' \
'AREA:eth1inbits#33CC33:eth1 in ' \
'LINE1:eth1outbits#0000ff:eth1 out' \
'GPRINT:eth1last:Drawn at %Y-%m-%d, %H\:%M:strftime'

The final graphs look decent, though not very fancy, but I’ll play around with it a bit more:

eth1 graph

load graph

I’ll be moving this site to a new hosting provider over the next few days, so there will likely be some downtime. Not that this site gets that many repeat visitors, but figured I’d give a heads-up so Google can cache this ahead of time.

We wanted to setup a loadbalanced web cluster in AWS for expansion. My first inclination was to use ELB for this, but I soon learned that ELB doesn’t let you allocate a static IP, requiring you to refer to it only by DNS name. This would be OK except for the fact that our current DNS provider, Dyn, requires IP addresses when using their GSLB (geo-based load balancer) service.

Rather than let this derail the whole project, I decided to look into the software options available for loadbalancing in EC2. I’ve been a fan of hardware load balancers for a while, sort of looking down at software-based solutions without any real rationale, but in this case I really had no choice so I figured I’d give it a try.

My first stop was Nginx. I’ve used it before in a reverse-proxy scenario and like it. The problem I had with it was that it doesn’t support active polling of nodes – the ability to send requests to the webserver and mark the node as up or down based on the response. As far as I can tell, using multiple upstream servers in Nginx allows you to specify max_fails and fail_timeout, however a “fail” is determined when a real request comes in. I don’t want to risk losing a real request – I like active polling.

This led me to HAProxy. I’d never used HAProxy before but it seemed to be ideally suited to this (since it’s exclusively a load balancer). The option httpchk even allows for active polling of nodes – yay!

Unfortunately, HAProxy doesn’t support SSL. From the HAProxy site:

People often ask for SSL and Keep-Alive support. Both features will complicate the code and render it fragile for several releases. By the way, both features have a negative impact on performance :

Having SSL in the load balancer itself means that it becomes the bottleneck. When the load balancer’s CPU is saturated, the overall response times will increase and the only solution will be to multiply the load balancer with another load balancer in front of them. the only scalable solution is to have an SSL/Cache layer between the clients and the load balancer. Anyway for small sites it still makes sense to embed SSL, and it’s currently being studied. There has been some work on the CyaSSL library to ease integration with HAProxy, as it appears to be the only one out there to let you manage your memory yourself.

Poop! I figured out a workaround however, by using both Nginx and HAProxy on the same instance. HAProxy listens on port 80 and 8443 (so that it can relay decrypted SSL traffic to the nodes on a separate port, so that the nodes are aware that it was originally SSL traffic). Nginx is configured as a reverse proxy, listens on port 443 only, and has the SSL cert & key. The upstream for the Nginx is just localhost:8443 – HAProxy.

This was pretty easy to setup and works very well. I benchmarked HAProxy on an EC2 t1.micro instance (in front of two m1.large instances running our webapp) using ab -n 5000 -c 50 -t 60 and found it actually performed better than one of our hardware load balancers. That was pretty eye-opening (and sad).

The HAProxy and Nginx configs are below, in the hopes that it helps someone. The main warning I’d give is that using this will cause the logs on your nodes to interpret all requests as coming from the IP of the load balancer. I had to rewrite some code to have the app use the X-Forwarded-For address rather than the REMOTE_ADDR, but other than that this has been working out pretty well.

/etc/nginx/nginx.conf
Main thing is to make sure the server isn’t listening on port 80 (since HAProxy needs to).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

user              nginx;
worker_processes  1;
 
error_log  /var/log/nginx/error.log;
 
pid        /var/run/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
 
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
 
    access_log  /var/log/nginx/access.log  main;
 
    sendfile        on;
    keepalive_timeout  65;
 
    #
    # The default server
    #
    server {
        listen       81;
        server_name  _;
 
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
 
        error_page  404              /404.html;
        location = /404.html {
            root   /usr/share/nginx/html;
        }
 
        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
 
    }
 
    # Load config files from the /etc/nginx/conf.d directory
    include /etc/nginx/conf.d/*.conf;
 
}

/etc/nginx/conf.d/ssl-offloader.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

upstream haproxy {
        server localhost:8443 ;
}
 
server {
        listen       443;
        server_name f.q.d.n 1.2.3.4 ; # I put the FQDN and IP here, but maybe "_" will work too
#  server_name  _;
 
        ssl                  on;
        ssl_certificate      /etc/nginx/ssl-cert/cert.pem;
        ssl_certificate_key  /etc/nginx/ssl-cert/cert.key;
 
        ssl_session_timeout  5m;
 
        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;
 
        location / {
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
 
                proxy_pass http://haproxy/;
                proxy_redirect default;
                proxy_redirect http://$host/ https://$host/;
                proxy_redirect http://hostname/ https://$host/;
 
                proxy_read_timeout 15s;
                proxy_connect_timeout 15s;
        }
 
}

/etc/haproxy/haproxy.cfg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2
 
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
 
    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
 
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    3s
    timeout queue           1m
    timeout connect         2s
    timeout client          5s
    timeout server          5s
    timeout http-keep-alive 1s
    timeout check           10s
    maxconn                 3000
 
       stats enable
       stats auth evan:change_me_brother
 
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main_http *:80
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_http
 
frontend main_https *:8443
        option forwardfor except 127.0.0.1  
        option httpclose
        default_backend         web_https
 
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend web_http
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:80 check port 80
    server  node2 192.168.1.30:80 check port 80
    server  node3 192.168.1.40:80 check port 80
 
 
backend web_https
    balance     roundrobin
#       option httpchk GET / HTTP/1.1\r\nHost:\ host.com
        option httpchk
    server  node1 192.168.1.20:8443 check port 8443
    server  node2 192.168.1.30:8443 check port 8443
    server  node3 192.168.1.40:8443 check port 8443

I got my wife an iPad 2 for Christmas and she soon started complaining about the Wifi dropping its connection. I suggested she try turning off the “auto join” wifi setting, but that didn’t help. She’d be doing something and get the “Sorry, there’s no internet connection” error every 5-10 minutes. We’ve had FiOS for quite a while and we have 8 or 9 other devices connected (including Macs & iPhones) to the router without issue, so this seemed weird. I was starting to think it was a problem with the iPad, but we went to a friend’s house and used his wifi (with a Netgear router) and the iPad had no issues.

Back home, I logged into the router and tried assigning her iPad a static IP through DHCP. I had her release and renew and she got the new IP but the problem continued. Since we ruled out a problem with the iPad and I knew there was nothing “wrong” with the router, I figured I’d check and see if there are any reported issues with iPads and the Verizon router. Sure enough, there are. The first thing I clicked on, Fix for Verizon FIOS vs. iPad Wi-Fi Issues, suggested changing the wifi channel from “Automatic” to “6″ (it also suggests switching from WEP to WPA2-PSK, which I’ve always been using). I did that and it hasn’t dropped the wifi connection at all in the past 3 hours. Very odd issue. If I could get into the Actiontec (or the iPad for that matter) I’d like to check the logs and see what’s actually happening, but a win’s a win.

So, I’ve written about Compellent a few times from a price perspective, mostly on the disk side. I was recently contacted by our vendor with quotes for two new Compellent controllers. “What’s this all about?” I asked. “Why don’t we have a call with Compellent to discuss?” he replied. I rolled my eyes a little but figured it was worth hearing them out, since our Compellent SAN is at the heart of our infrastructure.

We currently have two controllers setup in failover mode. The first was bought in 2008 and the other in 2010 to add redundancy. Earlier this year we upgraded to the latest software version in preparation for moving our production DB onto the SAN, to allow us a nice window before we had to perform another upgrade (which would now risk DB downtime… I like failover but I don’t trust it enough to have a DB up during a failover), so I was kind of skeptical about any sort of upgrade to begin with.

On the call, the Compellent reps explained that they’ve dropped Fibre Channel connectivity between the controller and the disk enclosure, and the purpose of the upgrade is to give us SAS. In addition, they no longer sell SATA (!). I asked why we couldn’t simply add SAS cards to our existing controllers and was told that our current controllers are PCI-X, so can only support up to 3Gb/s SAS, while the new controllers have PCI-e and support 6Gb/s. And they want to ensure that we have the best possible performance. Pretty sure someone said the new controllers “have the future built in” to them.

One of the features we really liked about Compellent from the beginning was the fact that it was basically a software solution on top of commodity hardware. They stressed this point repeatedly. “When new technology comes out, we can just add a new card into your existing controller.” I think the example at the time was 10-gig Ethernet, but it seems like the same logic would apply to SAS. I understand that PCI-X doesn’t support 6Gb/s SAS, but it’s a tough pill to swallow that if we want to expand our SAN at all now, on top of whatever the actual expansion costs, we’re going to need to plunk down some serious money to upgrade the controllers, which really seems like a net-zero for us. We’re not going to ditch our existing FC enclosures so we’re going to be limited to 4Gb/s anyway. If they’re only selling SAS, well, that sucks for us, but ok. But why can’t we just throw a $500 PCI-X 3Gb/s card in to expand? So we’re not running at peak performance. I doubt that would be our performance bottleneck anyway. Plus, swapping out controllers is a huge operation for us.

I know at some point we’re going to have to bite the bullet and do this upgrade, but it just irks me. On the bright side, I guess, we don’t have to do a “forklift upgrade,” and the disks/enclosures will all still work. But we have a long way to grow before we need to expand, so fortunately I can put this off for a while.

After my 10-year-old basement Linux server died this week from a power outage, I took the sad step of giving up on it. It’s died before and I’ve patched it back together with a new power supply here or an addon PCI SATA card there, but I finally decided to throw in the towel since I had a newer old computer that had been idle for several years. The one that died was an Athlon K7 750 MHz with 512 MB ram. The new one is an Athlon 2 GHz (3200+) with 1 gig. For my uses, specs don’t really matter that much, but it’s nice to have more power for free.

I put CentOS 6 on it and configured Samba and copied all the data off the old machine and was back up and running within a few hours. Since I forward ports through my FiOS router to this box I did my standard lockdown procedure, including adding myself to the AllowUsers in sshd_config. Afterwards I took a look in /var/log/secure and saw the typical flood of dictionary attacks trying to get in as root or bob or tfeldman or jweisz. I have iptables configured to rate-limit SSH connections to 2 per 5 seconds per IP so the box doesn’t get DoSed out of existence, but some stuff does make it through to sshd.

Looking through /var/log/secure, I got to thinking it would be interesting if there was some way to visualize the attacks in a handy graph. Then I remembered, oh, wait, I can do that.

I wrote a perl script to parse out the attacks from /var/log/secure and insert them into a Postgres DB. This turned out to be pretty easy. Then I thought it would be more interesting to tie the IP of each attack to its originating country. I’ve used MaxMind’s GeoIP DB pretty extensively before, but I was looking something free. That’s when I remembered that MaxMind has a free GeoIP DB: GeoLiteCity. I grabbed it and yum-installed the Perl lib and added the geo data to the attack DB. Rather than worry about normalizing the schema I just shoved the info into the same table. Life is easier this way, and it’s just a for-fun project.

So I got that all working and parsed it against the existing /var/log/secures via

[root@lunix2011 ~]# zcat /var/log/secure-20111117.gz | perl parse-secure.pl

I wrote ssh.php to see what’s in the table:

ssh.php list of hacking attempts

So now that the data was all in place, time to move on to the graphs, which is what I really wanted to do. Last time I wanted to graph data programmatically I used JPGraph, which does everything in PHP and is super versatile. But I wanted something… cooler. Maybe something interactive. A little Googling turned up Highcharts which is absolutely awesome, and does everything in JavaScript. I basically modified some of their example charts and pumped my data into them and got the charts below.

Pie chart of attacks grouped by country for the past 30 days:

Pie chart by country

Bar graph of attacks per day:

Bar graph of daily attacks

So, that’s that. Code is in github if anyone wants to play around with it. I’ve cronned parse-secure.pl to run every 5 minutes so the data gets updated automatically.

I’m currently working on moving a Tomcat-based application into EC2. The code was written for Java 5.0. While Java 6 would probably work, I’d like to keep everything as “same” as possible, since EC2 presents its own challenges. I spun up a couple of t1.micro instances and copied everything over, including the Java 5 JDK, jdk-1_5_0_22-linux-amd64.rpm. Installing from RPM was easy, but the EC2 instance defaults to using OpenJDK 1.6:

[root@ec2 ~]# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.10) (amazon-52.1.9.10.40.amzn1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

There were a couple of things I had to do to get the system to accept the Sun JDK as its “real” java.

Alternatives

Red Hat’s “alternatives” system is designed to allow a system to have multiple versions of a program installed and make it easy to choose which one you want to run. Unfortunately I’ve found the syntax a bit strange and always have to Google it, so I figured I’d document it here for posterity.

So here’s the default:

[root@ec2 ~]# alternatives --config java
 
There is 1 program that provides 'java'.
 
  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
 
Enter to keep the current selection[+], or type selection number:

Here’s how to add Sun java, assuming the java binary is in /usr/java/jdk1.5.0_22/jre/bin/java (where the RPM puts it).

[root@ec2 ~]# alternatives --install /usr/bin/java java /usr/java/jdk1.5.0_22/jre/bin/java 1
[root@ec2 ~]# alternatives --config java
There are 2 programs which provide 'java'.
 
  Selection    Command
-----------------------------------------------
*+ 1           /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
   2           /usr/java/jdk1.5.0_22/jre/bin/java
 
Enter to keep the current selection[+], or type selection number: 2
[root@ec2 ~]# java -version
java version "1.5.0_22"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_22-b03, mixed mode)

Yay! Unfortunately this doesn’t help with the other problem I had with Tomcat, which was that EC2 instances set the JAVA_HOME var to OpenJDK as well (/usr/lib/jvm/jre). Fortunately this is an easy fix as well.

Setting JAVA_HOME

The JAVA_HOME var is set in /etc/profile.d/aws-apitools-common.sh. Comment out this line:

export JAVA_HOME=/usr/lib/jvm/jre

Create a new file, /etc/profile.d/sun-java.sh, and put this in it:

export JAVA_HOME=/usr/java/jdk1.5.0_22/jre

Also in that file I added the following to instruct the JVM to process all dates in America/New_York, since that’s the timezone all of our other servers use, and it makes reading log files easier when all dates are in the same tz:

export TZ=America/New_York

(I found I had to do this even after pointing /etc/localtime to the correct zoneinfo – Java was stuck on UTC even after the rest of the system was using America/New_York.)

I’m not usually one for introspection, but I found this a few years ago and it’s stuck with me.

A careful man I want to be;
A little fellow follows me.
I do not dare to go astray
For fear he’ll go the self same way.

I cannot once escape his eyes,
Whate’er he sees me do, he tries.
Like me he says he’s going to be;
The little chap who follows me.

He thinks that I’m so very fine,
Believes in every word of mine.
The base in me he must not see;
The little chap who follows me.

I must remember as I go
Through summer’s sun and winter’s snow,
I’m building for the years to be;
The little chap who follows me.

Linux supports hot-adding disks but whenever I add a new vdisk in VMware the new disk doesn’t show up unless I reboot, which defeats the purpose of hot-add. This command forces a rescan of the bus:

echo "- - -" > /sys/class/scsi_host/host0/scan

dmesg shows the new disk has been found:

  Vendor: VMware    Model: Virtual disk      Rev: 1.0 
  Type:   Direct-Access                      ANSI SCSI revision: 02
 target0:0:2: Beginning Domain Validation
 target0:0:2: Domain Validation skipping write tests
 target0:0:2: Ending Domain Validation
 target0:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
SCSI device sdd: 1048576000 512-byte hdwr sectors (536871 MB)
sdd: Write Protect is off
sdd: Mode Sense: 03 00 00 00
sdd: cache data unavailable
sdd: assuming drive cache: write through
 sdd: unknown partition table
sd 0:0:2:0: Attached scsi disk sdd
sd 0:0:2:0: Attached scsi generic sg3 type 0

Now, why there’s no “rescan_sata” command is something I can’t fathom, but that’s Linux for you.

In an attempt to teach myself Objective C, and because I couldn’t find anything that did what I wanted, I wrote a little utility to display the currently-playing iTunes track in the Mac taskbar. Originally I had it display the full track name right in the taskbar but it was too much text for such a small space (especially on a 1440×900 screen), so now you click a little musical note and it shows you the info in a menu.

Here’s a screenshot:

The code is all in github. If you’re looking for a similar utility, and are brave enough to try my first-ever Obj-C app, you can download it here. But the freshest version will probably be in the github project.

As an aside, I was surprised at how easy it was to cobble this together having never written ObjC before. I found some good examples that I mostly ripped off, but it was still remarkably easy to have the app listen to iTunes for track changes, etc.

I’m currently in the process of moving our DNS over to another provider and I was curious as to whether the old or new provider offers faster lookups. dig shows query times, but I didn’t want to just run that over and over. I decided to write something to do this, in Java since I like Java. I found this post, which has the meat of the work done already. I also read some of Sun’s JNDI/DNS lookup info, which was pretty dense. All I want to do is specify the name server’s IP and do the lookup. I don’t even really care about the result, just how long the query takes.

The thing I wrote only looks up A records, but can easily be modified to do CNAMEs or whatever. Here’s how you call it:

$ java -jar DNSTester.jar 4.2.2.2 www.google.com 25
Resolved www.google.com to 74.125.235.19 against NS 4.2.2.2
Performed 25 lookups in 233.29 milliseconds.  Average 9.3316ms per lookup.
 
$ java -jar DNSTester.jar 8.8.4.4 www.google.com 25
Resolved www.google.com to 74.125.226.146 against NS 8.8.4.4
Performed 25 lookups in 450.034 milliseconds.  Average 18.00136ms per lookup.

Code is in github here. Jar is available here.

Over a year ago, the hard drive in my primary desktop at home bricked itself and rather than going through the hassle of reinstalling Win7 on the new disk, I decided to go with FC12. I’ve been pretty happy with it in general, since I’ve always been partial to Red Hat and use CentOS primarily at work.

Last week I got the great idea to upgrade to FC14. In hindsight I can’t even recall what led me to try this, but it didn’t end well. I tried the “preupgrade” procedure, which appeared to do the entire upgrade from FC12 to FC14 in place. I left it overnight, and when I looked at it the next day it looked like it was done. I was in FC14 and everything looked ok. But then I tried syncing my photos over NFS and discovered nfs wasn’t running on my desktop. When I tried starting it, it failed. After some trial and error, I used the Google and found that this is just what happens when upgrading to FC14 due to changes between FC12 and FC14, namely the introduction of systemd.

In all the threads I read, the “solution” was a clean install of Fedora. I tried doing this without formatting my / (root) partition, since that had 500 gigs of my stuff on it, but it kept failing. What I ended up doing was downloading partedmagic, which is a totally awesome partitioning tool. If you’re familiar with Partition Magic, this is similar but Linux-based and free. I burned the iso to disc, booted to it, and shrunk my / partition from 900 GB to 850 GB, and created a new 50 GB partition at the end of my disk without losing any of my data:

FC15 Partitions

Once this was done, which took surprisingly little time, I did a net install of FC15. I opted for a net install rather than downloading the ISO because I feel that with FiOS it’s actually faster than reading a DVD, and avoids having to run “yum update” afterwards.

So, I ended up with FC15 clean-installed to the new “/” partition. I moved everything around so the old partition is mounted at /docs and has all my stuff in it. I’d heard that FC15 was causing an uproar but until I logged into Gnome 3 myself I didn’t really understand the fuss. It’s going to take some getting used to, but after adding the tint2 taskbar it’s not too awful.

But back to systemd. In FC15 I wanted to ensure Samba started at boot, since that’s how I share docs between my VMs and host. Chkconfig is still there, but based on my problems with NFS and systemd on FC14 I decided to look into it a bit and see if there’s a “new” way to enable stuff at startup. There is!

Instead of:

# chkconfig smb on

The command is:

# systemctl enable smb.service

Of course, when I did this it apparently fell back to using chkconfig for smb:

[root@evan-fedora ~]# systemctl enable smb.service
smb.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig smb on

It does say in the Fedora wiki that systemd respects chkconfig and vice versa, so I guess this post is kind of pointless and I should have just linked to the wiki. But, whatever.

Edit Jan 24, 2012: Deleted all the crap from this story and just left the recommended Apache and Nginx SSL cipher suites for maximum security without SSLv2 and without BEAST vulnerability (at least according to Qualys).

Apache httpd

SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
SSLHonorCipherOrder on

nginx

        ssl_protocols  SSLv3 TLSv1;
        ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;
        ssl_prefer_server_ciphers   on;

Source:

• Qualys
• SSL checker

This just started happening again, with these errors appearing in the event viewer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/18/2011 11:16:33 AM
Event ID: 5011
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' suffered a
fatal communication error with the Windows Process Activation Service.
The process id was '3760'. The data field contains the error number.
 
Log Name: System
Source: Microsoft-Windows-WAS
Date: 9/17/2011 6:47:07 AM
Event ID: 5009
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
A process serving application pool 'MSExchangeOWAAppPool' terminated
unexpectedly. The process id was '3108'. The process exit code was
'0x800703e9'.
 
Log Name: Application
Source: Application Error
Date: 9/17/2011 6:46:30 AM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: exch2010fe1
Description:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time
stamp: 0x4a5bd0eb
Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time
stamp: 0x4a5bdfe0
Exception code: 0xe053534f
Fault offset: 0x000000000000aa7d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

After reviewing the IIS logs and the event logs, I think it has to do with the WebReady document viewer – the thing in OWA that renders and lets you view .doc attachments within the browser rather than forcing you to open Word or Excel. I think users were attempting to open corrupted files and that was causing it to crash. I’ve disabled Webready in EMC (Server Config -> CAS) and I’ll see what happens.