Hi everyone. Unfortunately, yesterday evening the server responsible for our User database suffered a hard drive failure. Database fail-safes kicked in and kept all accounts online and accessible. However, custom themes which are stored as separate meta data needed to be restored and imported separately, making themes appear reset until the data was restored this morning.
While we were thrilled that the site survived the crash, we were very disappointed with the time needed to restore custom themes. As engineers who take a tremendous amount of pride in our work, we are meticulous in ensuring the reliability of our products. This particular delay was the result of our misunderstanding how the user database storage engine (InnoDB) interacted with our incremental back-ups. What should have been a 30-minute recovery kept all of us up until 4AM. Huge thanks to brilliant team at Rackspace (Will Kelly is our hero), and the impeccable Marco Arment.
As always, it breaks our hearts to inconvenience our users. And we’re putting more measures in place to ensure the reliability of our service.
Thank you for your continued support and understanding while we continue to expand our infrastructure. April will be bringing some very cool updates to Tumblr.
