Today’s data loss

As you have surely noticed, there are bets missing from last week today. Here’s the reason why.

First, some background: Last week we switched new a new server to be able to implement all the things we want to make and to meet the demands from the growing amount of users and data. We moved to Digital Ocean, one of the most respected hosts available for solutions like ours.

Today we were planning to upgrade our equipment there to make things even speedier and better. This was obviously planned for ahead and we did it early in the morning when most of you have better things to do than entering bets into a spreadsheet. 😉

Things went well, until we suddenly noticed that we weren’t able to access the filesystem, which is needed to reboot the system after the upgrade. Not even their own technicians were able to figure out what had happened, so we all decided to start up another server and move the database there.

That’s when the engineers realised the disc was corrupt and therefore we couldn’t read from it, so the database had to be restored from a backup. We were prepared for some data loss, but it turns out the weekly snapshots are made every Friday, so we had the worst possible timing, today being a Thursday. We tried out a few different options to find the best restore point, but had to settle for a week old version…

To sum up, we had the most possible bad luck at the worst possible time. Things like these just don’t happen at large hosts like these and if they do, they sure don’t happen at the worst possible time. But it did for us.

Having a disc collapse at a cloud storage, while upgrading the equipment, the day before weekly snapshots are made, it’s just less likely than Leicester winning the league and the Champions League the same year (yeah, we know it could happen, but I wouldn’t place my money on it).

All in all, it means that all bets registered last week are gone and all data tat comes with it. There’s no way for us to restore it since the disc is broken so if you need the data in, the only way is to enter it again manually, which sucks, big time, we know. We’re heavy users of the service ourselves, so we’re all in the same situation.

To make things feel at least a little better, all Pro subscribers has got a free month added to their account already. If you’re not a Pro subscriber yet, you will automatically get an extra month for free if you sign up before Monday (you can cancel anytime).

We know it doesn’t really make up for the lost data, but at least it’s something. The risk for something like this happening again is just… Well, at least almost non-existing.

If there’s anything you’re wondering about, feel free to contact us in all the usual channels.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s