I planned and budgeted for this this year but it still pisses me off. Look, I realize these guys have to make money, but the bottom line is they’re charging over $1.40/GB for SATA storage. There are 7200 RPM 2TB SATA drives on Amazon and Newegg ranging in price from $130 to $220. That’s as low as $0.065/GB. There was even a 2TB WD “Green” drive (sub-7200 RPM, I think 5900 RPM) on Slickdeals today for $99. That’s 5 cents per gig.

Like I said, I realize these guys are running a business and they want to make money, but this type of price gouging is just infuriating. Assuming a 2TB SATA disk retails for $200, and they tack on 100% markup, so $6400 for 16 disks. $5000 for the enclosure. $10,000 for their “special sauce.” That’s about $21,000 for 32TB, or $0.65/GB, which is still about a pretty healthy profit. This doesn’t take into account the insane annual support contracts you’re paying for in perpetuity.

I don’t know. I guess I’m just cheap. I’ve complained about this before, and I complained about it again this time to our reseller. The SAN vendor is Compellent, and while the product is very good, the pricing certainly doesn’t scale the way they made it sound during the sales pitch. They kept saying how they’re able to leverage commodity hardware in order to keep costs low. Well, maybe they meant their own costs. It’s certainly not getting passed on to us.

Though maybe I’m jaded because this is the first SAN I’ve worked with. I’ve heard lots of horror stories about SANs and carving luns and having to restripe arrays and worrying about hot disks, etc. With Compellent I don’t have to worry about any of that. It is a nice product. I just wish it was cheaper. My reseller’s response to my complaint was basically “It’s definitely expensive, but at least it’s good; all SAN vendors mark up their prices wildly but the product often sucks.”

Anyway, I guess when you have shareholders to please, if you can milk people, you’d better do it. I suppose my advice would be to buy what you need up front and don’t expect to be able to leverage falling costs of storage. The savings don’t get passed on to you.

The first step was to setup a VPN between the office and the datacenter so that users in the office would be able to connect seamlessly to the Exchange server once it was moved. This turned out to be relatively easy. The next step was basically to move the Exchange server. This originally seemed like it would be an easy thing to do — having a long history with PostgreSQL I figured I could do essentially a “dump and restore” – run some command that would backup contents of the mail database to a file and then restore it to a new machine. Well, I quickly learned that wasn’t possible, at least not given the factors involved.

Microsoft suggests two ways of moving an Exchange server to new hardware: 1) replacing a machine in-place with another one that takes its name and doing a restore, and 2) setting up the new server “next to” the old one and moving mailboxes over one at a time. I ruled out the first method because it seemed like a total crapshoot with no easy “rollback” mechanism. Plus I had no idea how long it would take to do a restore of our Exchange server – total mailbox size at the time was over 300 GB, and it took about 28 hours just to do the backup, so it seemed like it could easily take over 72 hours, meaning even if we started it Friday at 6 PM, it wouldn’t complete by Monday morning, and people would come in to work to find they had no email. No good.

This left the second option – setting up another server and moving mailboxes one at a time. This seemed pretty simple, except for the fact that people frequently use Outlook Web Access (webmail) to check their mail when out of the office, and ActiveSync to get mail on their phones. We tested the 2-server setup a while back and while mail gets routed properly, and users in the office are able to connect to both Exchange servers without problems, when they try accessing their mail from outside the office it fails. This is because if A is the old server (which people use for webmail) and B is the new server, if you log in to webmail (server A) but your mailbox is homed on server B, webmail will issue you a 302 redirect to http://B . If that’s not a valid URL outside your office (as is the case with us) it won’t work. If we could move everybody’s mailboxes from A to B overnight, and then make webmail point to B rather than A, that would solve the problem, but again, we had no way to know how long that would take, and I didn’t want to risk making anyone’s mail unavailable.

The plan I then came up with was to set up an Exchange frontend server in our office in front of our existing Exchange server. The frontend server would handle all the OWA/ActiveSync stuff and abstract that away from the backend server (where the mailboxes live). I could then set up an Exchange server in our office in a VM, migrate mailboxes over to them one at a time, and when it was done, copy the VM Exchange server to an external USB drive and drive it to the datacenter (about 25 miles away) and import the VM to our VMware production cluster, fiddle with its IP address and voila – the Exchange server would be moved.

But then I had a better idea: set up the frontend server and the new backend server in the datacenter in the VMware cluster from the get-go. Then when people accessed webmail they’d be hitting a server in the datacenter, which would connect to the Exchange server in the office transparently and relay them their mail. I could then move each mailbox from A to B with B being in the datacenter and the move taking place over the VPN.

Well, this is what I ended up doing, and there have been some wrinkles in the process, but so far it’s generally been working as I expected. I moved my mailbox to the new server today, and the move itself went fine – took about 90 minutes to move my 1.5 GB mailbox. It wasn’t quite a seamless process – the mailbox was moved but I couldn’t send or receive mail from the other server or the Internet in general. I managed to fix outbound SMTP pretty quickly (we relay mail through a smarthost in the datacenter) but inbound wasn’t working because the old server and new one couldn’t communicate for some reason, and all mail was being delivered to the old server. Among the things I did in attempting to solve this problem were create a new routing group for the servers in the datacenter (since we only had one Exchange server before, we only had one routing group), and then setup a Routing Group Connector between the two. This seemed like it should have resolved it but it didn’t. From server A, I could “telnet B 25″ and the connection would succeed, but if I issued a HELO I got 500 5.3.3 Unrecognized command. Same thing happened if I tried B -> A. After hours of checking settings, I came across a post on Experts Exchange that suggested the problem may be with the firewall (Cisco ASA) inspecting SMTP traffic. This was something that had flitted around in my head for a couple of seconds but I didn’t actually check it. In the end though, that’s what it was – the ASA in the datacenter was mangling the SMTP packets somehow and preventing the two from communicating. Once I issued the “no inspect esmtp” line, the whole day’s worth of mail came flooding through to my inbox (now on server B).

For some reason, however, mail was still not going B->A. I spent a while trying to figure out why – looking in logs, doing “telnet A 25″ and everything seemed fine. The mail queue kept showing queued messages though and an error like “remote server didn’t respond to the connection.” What ended up solving it, though, was deleting the Routing Group Connector associated with the datacenter routing group and re-adding it. For whatever reason, that cleared it right up.

So as of right now, we have Office and Datacenter, with Office having Exchange server A, and Datacenter having Exchange servers B and C – B being the new backend and C being the new frontend. DNS has been updated so webmail points to C, and C connects to A or B to get the user’s mail for OWA/ActiveSync. It works, it’s fast, I’m mostly happy.

I should probably mention that we discussed moving to Google Apps in the midst of this project. I was about 70% in favor of it, but in the end it seemed too expensive. We’ve already paid for our Exchange licenses and a hardware message archiver. Google’s price for Google Apps is $83/person per year if you include their 10-year archival option. If you don’t already have infrastructure in place, that might be cheap, but when you’re comparing it to “$0″ (and yes, I realize projects like the one I mentioned above aren’t free), it is a lot when you have ~100 users. In addition, most people at my company weren’t comfortable with the privacy/legal implications of having Google host our mail in the cloud – not to mention lots of people are Outlook addicts. They offered 25 GB storage per user, which was pretty compelling, and I personally love the Gmail interface, but in the end we opted to stick with Exchange for the time being.

1	Fire Island	$96,228
2	East Williston	$93,001
3	Island Park	$88,992
4	Quogue	$88,975
5	Great Neck	$87,444
6	Roslyn	$87,277
7	Mineola	$85,978
8	Lawrence	$85,400
9	East Rockaway	$84,130
10	Bridgehampton	$83,568
11	West Hempstead	$83,546
12	Jericho	$83,150
13	Carle Place	$83,088
14	Locust Valley	$82,553
15	Port Washington	$82,520

16	Oyster Bay-East Norwich	$82,513
17	Amagansett	$81,440
18	Fishers Island	$81,434
19	Plainview-Old Bethpage	$81,163
20	Syosset	$79,065
21	Garden City	$79,030
22	Cold Spring Harbor	$78,599
23	Glen Cove	$78,485
24	Valley Stream 30	$78,089
25	Southampton	$78,050
26	Oceanside	$77,448
27	Uniondale	$77,132
28	Wantagh	$76,480
29	East Hampton	$76,436
30	Shelter Island	$76,272
31	Herricks	$75,539
32	Hempstead	$74,950
33	Valley Stream 24	$74,743
34	Seaford	$74,479
35	North Bellmore	$74,424
36	Patchogue-Medford	$74,316
37	East Islip	$73,405
38	Westbury	$73,326
39	New Hyde Park-Garden City Park	$73,284
40	Sachem	$72,846
41	Levittown	$72,832
42	Franklin Square	$72,812
43	Lynbrook	$72,547
44	Bethpage	$72,467
45	Elwood	$72,202
46	Port Jefferson	$71,984
47	Comsewogue	$71,980
48	Floral Park-Bellerose	$71,952
49	Mount Sinai	$71,773
50	Babylon	$71,674
51	Freeport	$71,575
52	Baldwin	$71,566
53	Bellmore	$71,527
54	Copiague	$71,465
55	Sayville	$71,396
56	Rockville Centre	$71,325
57	Center Moriches	$71,016
58	Riverhead	$70,193
59	Farmingdale	$69,838
60	Malverne	$69,449
61	Southold	$69,432
62	Westhampton Beach	$69,408
63	Massapequa	$69,269
64	Bayport-Blue Point	$69,202
65	South Country	$68,981
66	Springs	$68,752
67	North Merrick	$68,752
68	Longwood	$68,752
69	Bellmore-Merrick	$68,750
70	East Quogue	$68,727
71	William Floyd	$68,627
72	West Babylon	$68,419
73	Sewanhaka	$68,216
74	Plainedge	$68,143
75	Island Trees	$67,988
76	Miller Place	$67,624
77	Sag Harbor	$67,266
78	Elmont	$66,602
79	Montauk	$66,463
80	Brentwood	$66,241
81	Merrick	$66,170
82	Amityville	$65,522
83	Islip	$65,042
84	Hauppauge	$64,877
85	Harborfields	$64,711
86	Hicksville	$64,638
87	South Huntington	$64,543
88	Northport-East Northport	$64,483
89	Huntington	$64,385
90	Commack	$64,302
91	Central Islip	$63,864
92	West Islip	$63,727
93	Roosevelt	$63,298
94	Bay Shore	$63,094
95	Three Village	$62,802
96	Mattituck-Cutchogue	$62,718
97	Half Hollow Hills	$62,588
98	Smithtown	$62,547
99	North Babylon	$62,505
100	Middle Country	$60,868
101	Wainscott Common	$60,774
102	Lindenhurst	$60,231
103	Rocky Point	$60,146
104	Hampton Bays	$59,478
105	Eastport-South Manor	$59,021
106	Deer Park	$58,541
107	Kings Park	$58,483
108	Tuckahoe	$58,362
109	Remsenburg-Speonk	$58,034
110	Wyandanch	$57,469
111	Greenport	$55,657
112	Shoreham-Wading River	$55,436
113	East Moriches	$55,108
114	Oysterponds	$54,324
115	Little Flower	$53,784
116	New Suffolk	$42,150

I still hate MySQL though. Postgres is so much better.

On another note, I got my 3rd FiOS bill – still running a credit. That’s $43 for 3.5 months of great service. Good luck competing with that, Optimum.

MAYBE!!!