Looking for the paradigm » perl

Graphing SSH dictionary attacks with HighCharts

Evan Hoffman — Fri, 18 Nov 2011 19:06:03 +0000

After my 10-year-old basement Linux server died this week from a power outage, I took the sad step of giving up on it. It’s died before and I’ve patched it back together with a new power supply here or an addon PCI SATA card there, but I finally decided to throw in the towel since I had a newer old computer that had been idle for several years. The one that died was an Athlon K7 750 MHz with 512 MB ram. The new one is an Athlon 2 GHz (3200+) with 1 gig. For my uses, specs don’t really matter that much, but it’s nice to have more power for free.

I put CentOS 6 on it and configured Samba and copied all the data off the old machine and was back up and running within a few hours. Since I forward ports through my FiOS router to this box I did my standard lockdown procedure, including adding myself to the AllowUsers in sshd_config. Afterwards I took a look in /var/log/secure and saw the typical flood of dictionary attacks trying to get in as root or bob or tfeldman or jweisz. I have iptables configured to rate-limit SSH connections to 2 per 5 seconds per IP so the box doesn’t get DoSed out of existence, but some stuff does make it through to sshd.

Looking through /var/log/secure, I got to thinking it would be interesting if there was some way to visualize the attacks in a handy graph. Then I remembered, oh, wait, I can do that.

I wrote a perl script to parse out the attacks from /var/log/secure and insert them into a Postgres DB. This turned out to be pretty easy. Then I thought it would be more interesting to tie the IP of each attack to its originating country. I’ve used MaxMind’s GeoIP DB pretty extensively before, but I was looking something free. That’s when I remembered that MaxMind has a free GeoIP DB: GeoLiteCity. I grabbed it and yum-installed the Perl lib and added the geo data to the attack DB. Rather than worry about normalizing the schema I just shoved the info into the same table. Life is easier this way, and it’s just a for-fun project.

So I got that all working and parsed it against the existing /var/log/secures via

[root@lunix2011 ~]# zcat /var/log/secure-20111117.gz | perl parse-secure.pl

I wrote ssh.php to see what’s in the table:

ssh.php list of hacking attempts

So now that the data was all in place, time to move on to the graphs, which is what I really wanted to do. Last time I wanted to graph data programmatically I used JPGraph, which does everything in PHP and is super versatile. But I wanted something… cooler. Maybe something interactive. A little Googling turned up Highcharts which is absolutely awesome, and does everything in JavaScript. I basically modified some of their example charts and pumped my data into them and got the charts below.

Pie chart of attacks grouped by country for the past 30 days:

Pie chart by country

Bar graph of attacks per day:

Bar graph of daily attacks

So, that’s that. Code is in github if anyone wants to play around with it. I’ve cronned parse-secure.pl to run every 5 minutes so the data gets updated automatically.

Logging RT username in Apache access_log

Evan Hoffman — Mon, 08 Aug 2011 21:34:18 +0000

RT has its own internal accounting & tracking system for logging activity, but I was interested in even more granular stuff, like seeing who looked at which tickets. I figured it wouldn’t be that hard to log this in Apache. Well, I was kind of right, in that it wasn’t “hard,” but it took me a long time to find the right place to do it. I did finally get it though.

httpd.conf

In httpd.conf I created a new LogFormat:

LogFormat "%h %l %{RTUSER}e %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined-rt

So instead of the HTTP-auth user, it puts the RT user in this field. Make sure to update your VirtualHost config to use the combined-rt LogFormat.

/usr/share/rt3/html/autohandler

In /usr/share/rt3/html/autohandler, right under this section:

# If we've got credentials, let's serve the file up.
if (    ( defined $session{'CurrentUser'} )
    and ( $session{'CurrentUser'}->Id ) )
{

Add this:

        $ENV{'RTUSER'} = $session{'CurrentUser'}->UserObj->EmailAddress;

This populates the RTUSER environment variable with the currently-logged-in user’s email address.

Restart httpd and the RT user’s email address should now appear in access_log. Note that it will only appear for Perl pages (not gifs/jpgs or other static content, since Perl doesn’t process those):

10.0.0.10 - evan@example.com [08/Aug/2011:17:30:34 -0400] "GET /rt3/index.html HTTP/1.1" 200 46809 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"
10.0.0.10 - - [08/Aug/2011:17:30:34 -0400] "GET /rt3/NoAuth/images//css/rolldown-arrow.gif HTTP/1.1" 200 83 "https://help.example.com/rt3/NoAuth/css/3.5-default/main-squished.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0.1) Gecko/20100101 Firefox/5.0.1"

Amazon SES: “illegal headers” with ses-send-email.pl (followup)

Evan Hoffman — Tue, 02 Aug 2011 18:39:45 +0000

A few people have emailed me asking me to integrate the perl code snippet into I wrote to strip illegal headers when sending email via Amazon SES into something actually usable. I’ve done so! I haven’t really tested this beyond sending some test emails, but here it is. Use this at your own risk, I make no warranty, blah blah blah.

This fix requires editing ses-send-email.pl, so I’d advise making a backup copy, though I imagine you can always get a fresh version from Amazon if necessary.

Open ses-send-email.pl in a text editor and find the read_message method. It should look like this:

Delete that and paste this in its place:

I tried it on the command line and it gave the output I expected – I haven’t tried integrating it with sendmail/postfix since I haven’t encountered this problem. Here’s the test message I attempted to send:

From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!
 
Now we're out of the header, into the body.  Grand!

From, To, and Subject are required headers. Chicken & Cats are illegal, X-Zombies should be ok since it’s X-ified. Here’s what happened when I tried to send with the unmodified ses-send-email.pl:

[Tue Aug 02 14:14:51 evan@EvanMBP 62 amazon-email]$ ./ses-send-email.pl --verbose -k aws-credentials -r
From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!

Now we're out of the header, into the body.  Grand!

    Sender
    InvalidParameterValue
    Illegal header 'Chicken'.

  5ffad294-bd33-11e0-b6e3-affca9ad1eb5

Illegal header 'Chicken'.

Illegal header error. Now with the modified version:

[Tue Aug 02 14:15:17 evan@EvanMBP 63 amazon-email]$ ./ses-send-email-x-headers.pl --verbose -k aws-credentials -r
From: evan@example.com
To: evan@example.com
Subject: Email
Chicken: yummy
Cats: yucky
X-Zombies: kill them!

Now we're out of the header, into the body.  Grand!

    000001318bb51442-c2cfb780-0604-4363-925a-54a57015e567-000000

    64e75a47-bd33-11e0-9d09-8f08f31615ad

No errors, and I received the email below (extraneous SMTP headers added by Barracuda/Exchange removed):

From: <evan@example.com>
To: <evan@example.com>
Subject: Email
X-Chicken: yummy
X-ASG-Orig-Subj: Email
X-Cats: yucky
X-Zombies: kill them!
Date: Tue, 2 Aug 2011 18:15:25 +0000
Message-ID: <000001318bb51442-c2cfb780-0604-4363-925a-54a57015e567-000000@email.amazonses.com>
Content-Type: text/plain
MIME-Version: 1.0

Now we're out of the header, into the body.  Grand!

Illegal headers have been X-ified. Hope this helps someone.

Amazon SES – “Illegal Header” errors

Evan Hoffman — Mon, 16 May 2011 16:37:49 +0000

Update 8/2/2011: please see the followup post for a possible workaround.

A few people have inquired about the “Illegal header” error when attempting to relay email through SES. “Oncle Tom” pointed to a thread on Amazon’s forums about a similar problem which led to a list of headers Amazon will accept. The list is below; if you need to add headers outside this list, you can do so using “X-Headers.”

Some people have posted sample code in the thread with modifications to ses-send-email.pl to replace “illegal” headers read from STDIN with an equivalent X-Header. I wrote something generic that takes the list of legal headers and replaces anything not on that list with its X-Header equivalent. Disclaimer: I haven’t tested it, but it seems like this type of solution would be more adaptable since you don’t have to keep fixing it every time a new application attempts to send email with a new header. I haven’t worked this into ses-send-email.pl (since I’m not having any problems, I don’t want to touch it ) but I fed in an email header and the output looked correct.

Input email:

MIME-Version: 1.0
Received: by 10.142.136.15 with HTTP; Mon, 16 May 2011 09:32:44 -0700 (PDT)
Date: Mon, 16 May 2011 12:32:44 -0400
Delivered-To: evandhoffman@gmail.com
Message-ID: 
Subject: Hi friend.
From: "Evan D. Hoffman" 
To: "Evan D. Hoffman (Personal)" 
Content-Type: text/plain; charset=ISO-8859-1

Email Test.

Output email:

[evan@EvanMBP ~]$ perl replace-headers.pl test-email.txt
MIME-Version: 1.0
Received: by 10.142.136.15 with HTTP; Mon, 16 May 2011 09:32:44 -0700 (PDT)
Date: Mon, 16 May 2011 12:32:44 -0400
X-Delivered-To: evandhoffman@gmail.com
X-Message-ID: 
Subject: Hi friend.
From: "Evan D. Hoffman" 
To: "Evan D. Hoffman (Personal)" 
Content-Type: text/plain; charset=ISO-8859-1

Email Test.

Hope this helps someone.

Here’s the legal header list, from http://docs.amazonwebservices.com/ses/2010-12-01/DeveloperGuide/index.html?AppendixHeaders.html

Accept-Language
Bcc
Cc
Comments
Content-Type
Content-Transfer-Encoding
Content-ID
Content-Description
Content-Disposition
Content-Language
Date
DKIM-Signature
DomainKey-Signature
From
In-Reply-To
Keywords
List-Archive
List-Help
List-Id
List-Owner
List-Post
List-Subscribe
List-Unsubscribe
Message-Id
MIME-Version
Received
References
Reply-To
Return-Path
Sender
Subject
Thread-Index
Thread-Topic
To
User-Agent

Integrating Amazon Simple Email Service with postfix for SMTP smarthost relaying.

Evan Hoffman — Thu, 28 Apr 2011 16:23:47 +0000

So, we’ve outgrown the 500 outbound messages/day limit imposed by Google Apps’s Standard tier. A wise friend suggested SendGrid, but I figured it was worth looking into what options Amazon provides. I found SES and am in the process of setting it up. Hopefully I can set it up as a drop-in replacement, obviating the need for code changes to use it. SES is attractive for us because:

Free Tier
If you are an Amazon EC2 user, you can get started with Amazon SES for free. You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk. Many applications are able to operate entirely within this free tier limit.

Note: Data transfer fees still apply. For new AWS customers eligible for the AWS free usage tier, you receive 15 GB of data transfer in and 15 GB of data transfer out aggregated across all AWS services, which should cover your Amazon SES data transfer costs. In addition, all AWS customers receive 1GB of free data transfer per month.

Free to try? Sounds good.

After signing up, the first thing I did was download the Perl scripts. Create a credentials file with your AWS access key ID and Secret Key (credentials can be found here when logged in). The credentials file (aws-credentials) should look like this:

AWSAccessKeyId=022QF06E7MXBSH9DHM02
AWSSecretKey=kWcrlUX5JEDGM/LtmEENI/aVmYvHNif5zB+d9+ct

Make sure to chmod 0600 aws-credentials. To ensure it’s working, run:

$ ./ses-get-stats.pl -k aws-credentials -s

If it doesn’t return anything it should be working correctly.

Next, you need to add at least one verified email address:

$ ./ses-verify-email-address.pl -k aws-credentials --verbose -v support@example.com

Amazon will send a verification message to support@example.com with a link you need to click to verify the address. Once you click, it’s verified. It’s important to note that initially your account will only be able to send email to verified addresses. According to this thread, you need to submit a production access request to send to unverified To: addresses. I did this and got my “approval” email about 30 minutes later.

To send a test email:

$ ./ses-send-email.pl --verbose -k aws-credentials -s "Test from SES" -f support@example.com evan@example.com
This is a test message from SES.

(Press ctrl-D to send.)

The next step is integrating the script with sendmail/postfix. The first thing I did was move my scripts to /opt/ (out of /root/) and attempt to run them with absolute pathnames (rather than ./ses-send-email.pl) and I got perl @INC errors:

[root@web2 ~]$ mv amazon-email/ /opt/
[root@web2 ~]$ /opt/ses-get-stats.pl -k aws-credentials -s
-bash: /opt/ses-get-stats.pl: No such file or directory
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s
Can't locate SES.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /opt/amazon-email/ses-get-stats.pl line 23.
BEGIN failed--compilation aborted at /opt/amazon-email/ses-get-stats.pl line 23.

The problem is that SES.pm isn’t in perl’s include path. To solve this, I tried adding the directory to the PERL5LIB environment var:

[root@web2 amazon-email]$ PERL5LIB=/opt/amazon-email/
[root@web2 amazon-email]$ echo $PERL5LIB
/opt/amazon-email/
[root@web2 amazon-email]$ cd
[root@web2 ~]$ export PERL5LIB
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k aws-credentials -s
Cannot open credentials file <aws-credentials>. at /opt/amazon-email//SES.pm line 54.
[root@web2 ~]$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -s
Timestamp               DeliveryAttempts        Rejects Bounces Complaints      
2011-04-27T20:27:00Z    1                       0       0       0               
[root@web2 ~]$

This worked for setting all users’ PERL5LIB … but didn’t allow postfix to send the message. After a couple more attempts at doing this “the right way,” I just ended up dropping a symlink to SES.pm in /usr/lib/perl5/site_perl and the @INC error went away.

After following Amazon’s instructions for editing main.cf and master.cf, I still was unable to send mail through Postfix, even though I could send directly through the perl scripts. I kept getting this error:

Apr 28 11:26:32 web2 postfix/pipe[27226]: A2AD33C9A6: to=<user@domain.com>, relay=aws-email, delay=0.35, delays=0.01/0/0/0.34, dsn=5.3.0, status=bounced (Command died with status 1: "/opt/amazon-email/ses-send-email.pl". Command output: Missing final '@domain' )

Google led me to this blog post which led me to this other blog post which illuminated the problem: apparently the Postfix pipe macro ${sender} uses the user@hostname of the mail sender. Since the hostname of an EC2 machine is usually something crazy like dom11-22-33-44.internal, this is not likely a validated sending email address. So the solution proposed by Ben Simon was to create a regex to map user@internal to user@realdomain.com and have postfix map everything. This didn’t work for me or the bashbang.com guys, who changed it to map from user@internal to validuser@realdomain.com. I found that you can eliminate the need for the mapping entirely by changing the master.cf entry to this:

  flags=R user=mailuser argv=/opt/amazon-email/ses-send-email.pl -r -k /opt/amazon-email/aws-credentials -e https://email.us-east-1.amazonaws.com -f support@example.com ${recipient}

The only difference between the above line and Amazon’s suggestion is that this replaces “-f ${sender}” with “support@example.com” which is a validated email address.

After this I was able to relay email successfully through SES. Whew!

Update 5/26/2011: We’ve been relaying through SES without issues for a few weeks now. I recently ran ses-get-stats.pl to see how many messages we’re actually sending and it’s a lot lower than expected. I’m still glad we moved to SES though, since it has no hard cap like Google Apps does:

$ /opt/amazon-email/ses-get-stats.pl -k /opt/amazon-email/aws-credentials -q
SentLast24Hours Max24HourSend   MaxSendRate     
317             10000           5

More fun parsing BIND query logs

Evan Hoffman — Mon, 13 Dec 2010 22:21:59 +0000

“But wouldn’t it be cool if it also…” That phrase usually triggers a lot of wasted cycles in my brain, though it sometimes comes up with something neat. I added a super lame graph to the DNS QPS parser. Makes it really easy to see peaks & troughs in usage:

2010-12-12 00:00 to 00:59 => 3237, rate 0.899 queries/sec	***********
2010-12-12 01:00 to 01:59 => 2709, rate 0.752 queries/sec	**********
2010-12-12 02:00 to 02:59 => 2473, rate 0.687 queries/sec	*********
2010-12-12 03:00 to 03:59 => 2171, rate 0.603 queries/sec	********
2010-12-12 04:00 to 04:59 => 2048, rate 0.569 queries/sec	*******
2010-12-12 05:00 to 05:59 => 1918, rate 0.533 queries/sec	*******
2010-12-12 06:00 to 06:59 => 2120, rate 0.589 queries/sec	********
2010-12-12 07:00 to 07:59 => 2477, rate 0.688 queries/sec	*********
2010-12-12 08:00 to 08:59 => 2994, rate 0.832 queries/sec	**********
2010-12-12 09:00 to 09:59 => 3704, rate 1.029 queries/sec	*************
2010-12-12 10:00 to 10:59 => 4297, rate 1.194 queries/sec	***************
2010-12-12 11:00 to 11:59 => 4744, rate 1.318 queries/sec	****************
2010-12-12 12:00 to 12:59 => 5106, rate 1.418 queries/sec	******************
2010-12-12 13:00 to 13:59 => 5311, rate 1.475 queries/sec	******************
2010-12-12 14:00 to 14:59 => 5083, rate 1.412 queries/sec	*****************
2010-12-12 15:00 to 15:59 => 4855, rate 1.349 queries/sec	*****************
2010-12-12 16:00 to 16:59 => 5179, rate 1.439 queries/sec	******************
2010-12-12 17:00 to 17:59 => 4959, rate 1.377 queries/sec	*****************
2010-12-12 18:00 to 18:59 => 4693, rate 1.304 queries/sec	****************
2010-12-12 19:00 to 19:59 => 4792, rate 1.331 queries/sec	****************
2010-12-12 20:00 to 20:59 => 4799, rate 1.333 queries/sec	****************
2010-12-12 21:00 to 21:59 => 5068, rate 1.408 queries/sec	*****************
2010-12-12 22:00 to 22:59 => 4672, rate 1.298 queries/sec	****************
2010-12-12 23:00 to 23:59 => 4514, rate 1.254 queries/sec	****************
2010-12-13 00:00 to 00:59 => 3917, rate 1.088 queries/sec	**************
2010-12-13 01:00 to 01:59 => 3458, rate 0.961 queries/sec	************
2010-12-13 02:00 to 02:59 => 2874, rate 0.798 queries/sec	**********
2010-12-13 03:00 to 03:59 => 2715, rate 0.754 queries/sec	**********
2010-12-13 04:00 to 04:59 => 2690, rate 0.747 queries/sec	*********
2010-12-13 05:00 to 05:59 => 2719, rate 0.755 queries/sec	**********
2010-12-13 06:00 to 06:59 => 2831, rate 0.786 queries/sec	**********
2010-12-13 07:00 to 07:59 => 3416, rate 0.949 queries/sec	************
2010-12-13 08:00 to 08:59 => 4962, rate 1.378 queries/sec	*****************
2010-12-13 09:00 to 09:59 => 6943, rate 1.929 queries/sec	************************
2010-12-13 10:00 to 10:59 => 8296, rate 2.304 queries/sec	****************************
2010-12-13 11:00 to 11:59 => 8938, rate 2.483 queries/sec	******************************
2010-12-13 12:00 to 12:59 => 8926, rate 2.479 queries/sec	******************************
2010-12-13 13:00 to 13:59 => 8950, rate 2.486 queries/sec	******************************
2010-12-13 14:00 to 14:59 => 2173, rate 0.604 queries/sec	********

You can see pretty quickly that 4-5 AM (Eastern) is the period of lowest activity, as you’d probably expect for a US-based US-centric site. Modified perl script below.

Perl script to calculate DNS queries per second in BIND (named)

Evan Hoffman — Tue, 07 Dec 2010 22:53:43 +0000

I’m pricing out DNS providers and was asked what our current queries-per-second currently are. Sadly I had no idea. After lots of Googling I decided there was really no good way to get this information so I decided to parse the logfile myself.

First, I turned on logging with timestamp in named.conf:

logging
{
...
        # Query logging 2010-12-07
        channel query-log {
                file "data/queries.log" versions 3 size 10m;
                print-time yes;
        };
        category queries { query-log; };
};
options
{
...
        # Query logging 2010-12-07
        querylog yes;

};

Then I restarted named and had to figure out how to parse the file. My first guess was Perl and it didn’t disappoint. Here’s the hideous mess of code:

parsebind.pl

#!/usr/bin/perl
 
#use Date::Parse;
use Time::ParseDate;
 
my $line_num = 0;
my $first_line = 0;
my $oldest_record = 0;
my $line = '';
my $date = 0;
 
my %query_counter = {};
my @abbr = qw( Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec );
 
while (<>) {
	chomp;
	$line = $_;
 
	#my ($date, $time, $ip, $hostname, $in, $type, $crap) = $line  =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/;
	if ($line =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/) {
		$date = parsedate("$1 $2");
#		print "$1 $2 => $date\n";
		if ($oldest_record == 0) {
			$oldest_record = $date;
			$first_line = $line_num;
			print "First line:\t$line_num\tdate\t".localtime($date)."\n";
		}
		my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($date);
		$year += 1900;
		$mon += 1;
		my $key = $year . '-' . sprintf('%02d',$mon) . '-' . sprintf('%02d',$mday). ' '.sprintf('%02d',$hour).':00 to '.sprintf('%02d',$hour).':59' ;
 
		if ($query_counter{$key}) {
			$query_counter{$key}++;
		} else {
			$query_counter{$key} = 1;
		}
	}
	$line_num++;
}
 
print "Last line:\t$line_num\tdate\t".localtime($date)."\n";
 
my $elapsed = $date - $oldest_record;
my $net_records = $line_num - $first_line;
my $rate = ($net_records * 1.0) / $elapsed;
 
print "Rate for $elapsed seconds: ".sprintf('%0.2f',$rate)." per second\n";
print "\n\n";
 
foreach my $key (sort keys %query_counter) {
	$rate = ($query_counter{$key} * 1.0) / 3600;
	print "$key => $query_counter{$key}, rate ".sprintf('%0.3f',$rate)." queries/sec\n";
}
 
exit 0;

And the beautiful output:

[root@ns3 ~]# perl ~evan/parsebind.pl /var/named/chroot/var/named/data/queries.log
First line:	308	date	Tue Dec  7 17:56:02 2010
Last line:	41289	date	Tue Dec  7 22:44:01 2010
Rate for 17279 seconds: 2.37 per second

2010-12-07 17:00 to 17:59 => 533, rate 0.148 queries/sec
2010-12-07 18:00 to 18:59 => 9185, rate 2.551 queries/sec
2010-12-07 19:00 to 19:59 => 8618, rate 2.394 queries/sec
2010-12-07 20:00 to 20:59 => 8075, rate 2.243 queries/sec
2010-12-07 21:00 to 21:59 => 8762, rate 2.434 queries/sec
2010-12-07 22:00 to 22:59 => 5808, rate 1.613 queries/sec
HASH(0x112382a0) => , rate 0.000 queries/sec
[root@ns3 ~]#

Not pretty but good enough to get a sense of QPS. Hope it helps someone.

Edit: It occurred to me that you may have more than one domain on a single bind server, and you may want the stats for a single one (rather than all aggregated together, which the above script does). The below version includes the second-level domain in the hash key for the final report:

#!/usr/bin/perl
 
#use Date::Parse;
use Time::ParseDate;
 
my $line_num = 0;
my $first_line = 0;
my $oldest_record = 0;
my $line = '';
my $date = 0;
 
my %query_counter = {};
my @abbr = qw( Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec );
 
while (<>) {
	chomp;
	$line = $_;
 
	#my ($date, $time, $ip, $hostname, $in, $type, $crap) = $line  =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/;
	if ($line =~ m/^([\d]{2}-[\w]{3}-[\d]{4}) ([\d]{2}:[\d]{2}:[\d]{2}\.[\d]{3}) client ([\d\.]+)#[\d]*: [\w]+: ([\w\d\.]+) ([\w]+) ([\w]+) ([.]*)/) {
		$date = parsedate("$1 $2");
		my ($domain) = ($4 =~ m/([\w\d\-]+\.[\w]+)$/);
		$domain = lc($domain);
#		print "$1 $2 => $date\n";
		if ($oldest_record == 0) {
			$oldest_record = $date;
			$first_line = $line_num;
			print "First line:\t$line_num\tdate\t".localtime($date)."\n";
		}
		my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($date);
		$year += 1900;
		$mon += 1;
		my $key = $domain .' '.$year . '-' . sprintf('%02d',$mon) . '-' . sprintf('%02d',$mday). ' '.sprintf('%02d',$hour).':00 to '.sprintf('%02d',$hour).':59' ;
 
		if ($query_counter{$key}) {
			$query_counter{$key}++;
		} else {
			$query_counter{$key} = 1;
		}
	}
	$line_num++;
}
 
print "Last line:\t$line_num\tdate\t".localtime($date)."\n";
 
my $elapsed = $date - $oldest_record;
my $net_records = $line_num - $first_line;
my $rate = ($net_records * 1.0) / $elapsed;
 
print "Rate for $elapsed seconds: ".sprintf('%0.2f',$rate)." per second\n";
print "\n\n";
 
foreach my $key (sort keys %query_counter) {
	$rate = ($query_counter{$key} * 1.0) / 3600;
	print "$key => $query_counter{$key}, rate ".sprintf('%0.3f',$rate)." queries/sec\n";
}
 
exit 0;

I suppose this doesn’t account for PTR (reverse DNS) lookups, but oh well.

Edit: I added some lame ASCII graphing functionality to the script, check it out in this post.

Blocking comment spammers by IP

Evan Hoffman — Fri, 13 Aug 2010 15:59:27 +0000

I use Akismet to block comment spam, but it still annoys me that it even exists. Last night I put a simple IP ban into my httpd config. But who to block?

I used a grep & Perl to get a rough guess of which IPs were submitting the most comments (working on the assumption that one IP address submits many spam comments) It took me about 20 minutes to write this mess but it does what I wanted to do:
[root@lunix ~]# zgrep POST /var/log/httpd/evanhoffman-access_log-201008??.gz | grep comment | perl -ne 'chomp; $_ =~ m/(?:\d{1,3}\.){3}\d{1,3}/; print "$&\n";' | perl -e '%a = (); while (<>) { chomp; $a{$_} += 1; } while (my ($key, $value) = each (%a)) { if ($value > 1) { print "$value\t=>\t$key\n";}}' 2 => 218.6.9.140 180 => 91.201.66.34 2 => 213.5.67.41 2 => 188.187.102.74 [root@lunix ~]#

That’s pretty hard to read. Here’s a quick explanation of each piece:

zgrep POST /var/log/httpd/evanhoffman-access_log-201008??.gz

Use zgrep to search for the string “POST” in all of the gzipped Apache logs for August. Pipe the results (the matching lines) to the next part:

grep comment

grep for the string “comment”. This isn’t really scientific, but I feel safe in assuming that if “POST” and “comment” both appear in the HTTP request, it’s probably someone posting a comment. Pipe the matches to…

perl -ne ‘chomp; $_ =~ m/(?:\d{1,3}\.){3}\d{1,3}/; print “$&\n”;’

This is a perl one-liner that uses a regular expression to match an IP address in a given line and print it out. The original regex I used was \d+\.\d+\.\d+\.\d+, this one was slightly fancier but did the same work in this case. It’s worth noting that this will only print out the first match in the given line, but since the requester’s IP (REMOTE_ADDR) is the first field in Combined Log Format, that’s fine this case.

The output (the IPs from which comment posts have been made) is piped to…

perl -e ‘%a = (); while (<>) { chomp; $a{$_} += 1; } while (my ($key, $value) = each (%a)) { if ($value > 1) { print “$value\t=>\t$key\n”;}}’

This is another perl one-liner. Basically, it maintains a hash of String=>count pairs, so each time it sees a string it increments a “counter” for that line. Then when it’s done receiving input (i.e. all the data has been processed) it prints out the contents of the hash for keys that have a value > 1 (i.e. IPs that have POSTed more than 1 comment).

The output shows pretty clearly where the spam is coming from:

2 => 218.6.9.140
180 => 91.201.66.34
2 => 213.5.67.41
2 => 188.187.102.74

180 submits from 91.201.66.34. Out of curiosity I looked up that IP in whois:

[root@lunix ~]# whois 91.201.66.34
[Querying whois.ripe.net]
[whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: This output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '91.201.64.0 - 91.201.67.255'

inetnum:        91.201.64.0 - 91.201.67.255
netname:        Donekoserv
descr:          DonEkoService Ltd
country:        RU
org:            ORG-DS41-RIPE
admin-c:        MNV32-RIPE
tech-c:         MNV32-RIPE
status:         ASSIGNED PI
mnt-by:         RIPE-NCC-END-MNT
mnt-by:         MNT-DONECO
mnt-by:         MNT-DONECO
mnt-lower:      RIPE-NCC-END-MNT
mnt-routes:     MHOST-MNT
mnt-routes:     MNT-PIN
mnt-domains:    MHOST-MNT
source:         RIPE # Filtered

organisation:   ORG-DS41-RIPE
org-name:       DonEko Service
org-type:       OTHER
address:        novocherkassk, ul stremyannaya d.6
e-mail:         admin@pinspb.ru
mnt-ref:        MNT-PIN
mnt-by:         MNT-PIN
source:         RIPE # Filtered

person:         Metluk Nikolay Valeryevich
address:        korp. 1a 40 Slavy ave.,
address:        St.-Petersburg, Russia
e-mail:         nm@internet-spb.ru
phone:          +7 812 4483863
fax-no:         +7 901 3149449
nic-hdl:        MNV32-RIPE
mnt-by:         MNT-PIN
source:         RIPE # Filtered

% Information related to '91.201.66.0/23AS21098'

route:          91.201.66.0/23
descr:          Route MHOST IDC
origin:         AS21098
mnt-by:         MHOST-MNT
source:         RIPE # Filtered

[root@lunix ~]#

Not much info other than the IP is based in Russia. Well, anyway, I IP blocked 91.0.0.0/8 (sorry, Russia), so if you’re in that subnet you’re probably seeing a 403 now.

Edit: It occurred to me that I can accomplish the same thing while being less draconian if I wrap the Deny in a clause. This way everyone can still see the site but certain IP ranges won’t be able to POST anything:


Order Allow,Deny
Allow from all
Deny from 218.6.9.
Deny from 173.203.101.
Deny from 122.162.28.
Deny from 91.
Deny from 213.5

Running MRTG cfgmaker across your entire subnet?

Evan Hoffman — Wed, 27 Jan 2010 17:00:30 +0000

I realized recently that I had a bunch of newly-provisioned VMs that weren’t being monitored by MRTG (one of the tools we use to monitor network usage and other fun stats). Rather than manually run cfgmaker against all the new machines, I decided to script my way out of this.

Step 1: Build a list of hosts to monitor

For this I used an nmap ping test to generate a list of “up” hosts:

nmap -sP 10.0.0.0/24 -oG 10.0.0.0-255.255.255.0

This creates a list of IPs that are up in a format like this:

Host: 10.0.0.60 () Status: Up Host: 10.0.0.61 () Status: Up Host: 10.0.0.63 () Status: Up Host: 10.0.0.64 () Status: Up Host: 10.0.0.65 () Status: Up

Step 2: Parse out the IP address and run cfgmaker

Here’s the command I ran:

$ for i in `perl -ne 'chomp; next if $_ =~ /^#/; my @a = split(/ /); print "$a[1]\n";' 10.0.0.0-255.255.255.0`; do /usr/bin/cfgmaker --global 'LogFormat: rrdtool' --global 'WorkDir: /mrtg/data/mrtg' --global 'Options[_]: growright,bits' --global 'XSize[_]: 600' --global 'YSize[_]: 400' public@$i > /etc/mrtg/mrtg-$i ; done

This parses out the IP address from the input file and for each address, runs cfgmaker and puts the output in /etc/mrtg/mrtg-[ipaddress] .

Taking this a step further, I used gethostbyaddr to resolve the IPs to hostnames, so the MRTG files have the proper hostnames in them:

$ for i in `perl -MSocket -ne 'chomp; next if $_ =~ /^#/; my @a = split(/ /); my $ip = inet_aton($a[1]); my $host = gethostbyaddr($ip, AF_INET); print "$host\n";' 10.0.0.0-255.255.255.0`; do /usr/bin/cfgmaker --global 'LogFormat: rrdtool' --global 'WorkDir: /mrtg/data/mrtg' --global 'Options[_]: growright,bits' --global 'XSize[_]: 600' --global 'YSize[_]: 400' public@$i > /etc/mrtg/mrtg-$i ; done

Then all of the mrtg .cfg config files are in /etc/mrtg/, but who wants to put in all those cron entries to run each one? Not me. Simple fix?

cat /etc/mrtg/*.cfg > /etc/mrtg-all.cfg

Then you only have one cron entry.

Victory! Change Active Directory Password via LDAP through browser

Evan Hoffman — Wed, 13 Jan 2010 17:51:43 +0000

I had to give up on PHP and go to Perl, but it turned out not to be so bad. Users can now change their Active Directory passwords via a self-service web page that doesn’t require admin credentials. The Perl code is below. Authentication to the script is done via .htaccess LDAP authentication, so the REMOTE_USER env variable is assumed to contain the user’s username (sAMAccountName) by the time this script is called. There is a simple check for $ENV{HTTPS} to ensure the script is called via SSL, and AD requires password changes to be done via ldaps, so the whole thing should be encrypted end to end.

(Edited 5/14/2010 to replace the inlined Perl script with a link to the script as a text file.)

(Edited 10/6/2011 to replace link to script with link to gist)

Flac to MP3 (flac2mp3.pl)

Evan Hoffman — Mon, 07 Nov 2005 04:00:44 +0000

I wrote a crappy perl script to read all the .flac files in a directory and convert them to MP3 using LAME, extracting the meta-data and inserting it as ID3v2 tags. I figured this might be somewhat useful to others, so here you go: ~~flac2mp3.pl~~