Saturday, July 19, 2008

Contractual Obligations

by Roger in California

Never have non-technical people enter and sign agreements or services contracts with technical vendors. It's basically the worst idea in the world. You never want anyone who can't read code vetting a contract with these kind of people because they're the bastard offspring of sharks and vultures and will absolutely rob you if you don't know what's going on. All of their lawyers have technical backgrounds and their service obligations are defined by ex-engineers. They're playing for keeps.

And by "read code" I mean any kind of code - legal code, accounting code, building code, computer code, an engineering calculations sheet, it doesn't matter. The vast majority of individuals that have never had to deal with rigorously defined language or numbers will simply not know how to interpret it or synthesize their own. They will scan. They will browse. They'll get the gist of it. They'll do everything but actually read the damn contract and think about what it's actually saying.

I'll give you an example to set this context for you a little bit. Here at Well Funded Startup Corp we have a hosting contract with a company that I won't mention. Actually, yes I will, it's Rackspace. Everybody's favorite Texas colocation company. Anyway, we have a hosting contract with these people that was "negotiated" (I'm using this term loosely) by the former COO. To get an idea of how terrible this contract is, just think of the worst case scenario, and then scale up all the cost and acceptable downtime parameters that pop into your head by one order of magnitude.

* Over $1000 per month per server. Single Xeons with 2GB of RAM and no RAID. Read that twice and let it sink in for a second. We have more than 10 of these matchboxes sitting in a rack somewhere. We've already paid the cost of these machines almost 10 times over, by my estimates.

* No direct access to load balancer or firewall. We have to file tickets with their "fanatical support" technicians to do things like block spam-generating subnets.

* No internal backhaul network. Yes, that's right, our database traffic and internal traffic (version control checkouts, file transfers, etc) were going out on the same ethernet interface as our web traffic. You can imagine how confused we were when we first installed a graphing package to monitor inbound versus outbound traffic. Which brings me to my next point:

* No monitoring solutions. If you don't specifically request it, rackspace doesn't provide you with so much as uptime monitoring. If your machines go down... good luck. Hope you're by a computer.

* Arrogant technicians. Have you ever had an MSP's technician login as root on YOUR server and start moving things around? No? Well, you haven't had the pleasure of being at Rackspace then. How they actually did this is beyond me. It wouldn't surprise me in the least to find out they are running backdoored kernels.

* 10 days grace period for "issue" resolution. That isn't a typo. Our servers could explode, or some dimwit cagemonkey could drop his screwdriver into our web cluster's power supply, and we have no legal recourse as long as they bring it back before the 240th hour. Two hundred and fortieth. Hour. This nice little clause pretty much puts you at the top of the list for:

* Atrocious support. We requested that they upgrade a disk controller on a file server and they gave us a service window. 30 minutes later, the machine is still down. I had to call them three times before they finally admitted that the tech had "grabbed the wrong card" and had to run back to the depot to grab the right one. Meanwhile, our machine sat powered off outside of the service window. Unbelievable.

At this point you're probably saying: even the nightmarish task of migrating data centers isn't worth bad service, 10 days of downtime, and outrageous costs every month. Well, of course you'd say that, because you don't know about the big fat cherry on top of this delicious cow dung pie:

* Contractual lock-in. Yeah. 24 months. Inquired about the enforceability of this clause through legal of our parent company and they told us we were pretty much screwed.

If you've ever negotiated a really good deal on hosting and wondered "How the hell do these companies make money?", wonder no longer. They make money when idiots with zero technical background call them up and ask for a sweet hosting deal after clicking on the stupid ad they saw on Ziff Davis or wherever the hell. The "good deal" customers subsidize the bulk purchase costs for hardware and energy so they can take advantage of the "bad deal" customers. This is how the hosting business works.

I'm pretty sure when sales guys get these accounts on the phone they immediately reaches into the "for idiots" contract pile and start thinking about what they're going to do with his $5k commission. That goes a long way in Texas.

1 comment:

sapphirepaw said...

"Arrogant technicians. Have you ever had an MSP's technician login as root on YOUR server and start moving things around?"

Well, that one's pretty much standard. One of my previous employers had a GoDaddy leased server. They force all outbound mail through their relay (any other destination on port 25 returns "no route to host"), which aggressively checks it for spam, and spits back 550 errors if it doesn't like it. Whenever we'd open a ticket with them, they wanted root access... so they could run "date | mail" to determine if our mail server was correctly configured. (The fact that the bounce message names secureserver.net as the rejecting MTA apparently makes no difference.) They never did figure out that we replaced Sendmail with Postfix.

ServerBeach (a Peer1 company) is also known to run a special "sbadmd", the ServerBeach Admin Daemon, which is basically an extra SSHD on a nonstandard port, which accepts root logins and some particular private key. We haven't needed assistance from them yet, but it makes me nervous. The prize for cracking that key is root access to any SB machine...