Rackspace Cloud Database Critical Failure Mode
We are spinning up a new startup. It’s about time to go live. For this startup, we are using Rackspace and their cloud offerings. We do quite a bit of business with them (up to $1K per month) and it was just easy here.
I’m prepping for an internal final review of the first MVP (Minimum Viable Product). Among many tasks, I decide to prune all of our test data and then empty a few history tables we don’t need data for anymore. In the middle of a stack of “delete from X” queries, the database goes offline. Connection lost and I’m unable to reconnect via command line?!
Okay. Let’s log in to the control panel and see what it says:
Unexpected. Okay, let’s dial up Fanatical Support and see what is up…
It could be rude to post the transcript. To cut a 30 minute chat short, the good word from Rackspace is that the database became blocked because “you are almost out of disk space.”
Notice the screenshot above, it shows 1.9 GB used of 2 GB. Now, a cloud database that has a critical failure mode when disk space is close to, but not completely used up, is un-freakin-believable. Even if disk space was completely used up, it’s an arbitrary limit that was created for pricing. The database shouldn’t go down if we hit it. It’s especially ridiculous when the failure happened as I was deleting records (not inserting!) to clear up disk space. (Likely because MySQL needed to use some additional space for referential integrity checks or something) Finally, this requires manual intervention from support personnel to remedy the situation. My two options given were to increase the disk space of the cloud database, or give support the queries I need to run to finish clearing up disk space. I’d be happy to resize disk space, if it was temporary. However, insanely, you cannot reduce disk space of a cloud server after increasing it.
Even though we are in testing, think forward to golive and beyond. The database disk usage goes to within 5% of the limit and will then go dark. Complete and critical failure of the app. No warning.
Would you trust your 3 AMs to this? Would you trust your business to this? I’m appalled that Rackspace would bring this to market.
We have another startup on Heroku. What happens when you hit a pricing limit with Heroku databases? You get emailed for up to two weeks before they start denying inserts. Even then, the database is still responding to read requests.
Credit where credit is due. The support rep that was talking with me for 30 minutes was great. Even emailed me afterwards when I callously dismissed his last attempt at help. I don’t get that from Heroku.