Exercising our infrastructure

Hi everyone. Unfortunately, yesterday evening the server responsible for our User database suffered a hard drive failure. Database fail-safes kicked in and kept all accounts online and accessible. However, custom themes which are stored as separate meta data needed to be restored and imported separately, making themes appear reset until the data was restored this morning.

While we were thrilled that the site survived the crash, we were very disappointed with the time needed to restore custom themes. As engineers who take a tremendous amount of pride in our work, we are meticulous in ensuring the reliability of our products. This particular delay was the result of our misunderstanding how the user database storage engine (InnoDB) interacted with our incremental back-ups. What should have been a 30-minute recovery kept all of us up until 4AM. Huge thanks to brilliant team at Rackspace (Will Kelly is our hero), and the impeccable Marco Arment.

As always, it breaks our hearts to inconvenience our users. And we’re putting more measures in place to ensure the reliability of our service.

Thank you for your continued support and understanding while we continue to expand our infrastructure. April will be bringing some very cool updates to Tumblr.

33 responses to “Exercising our infrastructure

  1. Your focus on users is impressive and appreciated!
    Glad to be a part of this fine community.

  2. I second Mini ZuD’s comment. A lot of companies still neglect to keep their user base adequately informed of problems, even in this age of “Web 2.0”.

    Fortunately I had a personal backup of my edited theme so wasn’t greatly inconvenienced by the hard drive crash.

    Keep up the good work. Looking forward to the updates.

  3. My posts on the subject pretty much sums up my take on all of this:

    Long story short: Kudos for the speedy info and speedy recovery. I look forward to the new features!

    Get some sleep.

  4. Hey, you guys stayed up untl 4 a.m. supporting a free product. How mad at you can we really be? ;-P

    Seriously, thanks for getting things straightened out quickly. And, thanks for Tumblr. For me, it has made blogging fun again!

    Marc :-)

  5. Champion! I’m happy to say that I, for one, appreciate all of your efforts! :-)

  6. Your dedication and the proof that your users come first is very reassuring. Thank you for you hard work.

  7. Wooo! Sorry you had to go through that, great to hear you’re back up and running! Keep up the good work! Hearts and bunnies!

    Okay, maybe not hearts and bunnies, but we appreciate it! :D

  8. It was startling to see all the Tumblr logs I visit (and my own) looking exactly alike. ::chuckle:: But hardly inconveniencing. I didn’t skip a bit at all.

    A huge mahalo to you guys for your brilliant product and extraordinary, user-centric service.

    It’s clear you love what you do. :-)

  9. Aye. I come from Second Life, where Linden Lab routinely has issues like these they don’t seem to know how to fix. It’s nice to see a company that listens to their users, is so prompt to fix things, and keeps them up to date.

  10. As someone who was up until 4AM this week bringing an entire data center back online after both a power and UPS failure, I have great empathy for you! You guys did a great job in keeping our logs up and running and getting things squared away fast. You can bet I’ll be signing up for a pro account once they become available.

  11. thanks a lot for the effort.
    it’s great you were able to fix it, although it’s a shame you had to work so late… thanks

  12. Great post! I appreciate it as both a Tumblr and as someone who’s interested in major hosting.

    Just curious… what kind of package/deal/setup do you have with Rackspace?


  13. Get some sleep – and all your work is appreciated. Take care.

  14. Thank you for your impressive work.
    You’re terrific !

  15. Thank you guys for keeping us informed and for your hard work in restoring everything! Much appreciated! :)

  16. for some reason, shortly after the restore, all of my non-link text has gone italicized? Me or not me?

  17. A belated thanks for the time you guys put in to getting it all back up and running.

    The only things I lost were a couple of minor CSS changes I had obviously made since the last back-up.

    Much appreciated, and for a free service, this was excellent… :)

  18. No problem, I wanted a new design anyway. :) I really did actually.

    Thanks to the entire Tumblr team for their hard work.

  19. Are we having database issues again, by chance? My custom theme just re-arranged itself, and not in a good way! Yikes!

  20. Hey Sarah! It looks like you have a stray <div> tag in your post: http://sarahlane.tumblr.com/post/1007822
    That seems to be breaking the formatting.

    Our HTML filtering is supposed to be protecting against that automatically, so we’ll check out what’s up.

  21. Hmmm, it happened when I posted a video from Dailymotion. Deleted the video, reposted, and everything seems to be ok now. Nothing to see here.

  22. Thanks for building & supporting tumblr :) I trully love it! What new features are brewing? (Please say support for visitor comments!)


  23. Honestly, I gave up and left Tumblr when everything broke down. I was actually in the middle of editing!

    After calming down, I realized Tumblr is currently the best, fastest, and most convenient way of “tumblng” with our thoughts.

    Thanks for continually taking care of us, your loyal tumblrs.

    And more power to you!

  24. Request: upload images to Tumblr when a user adds one to a regular post.

  25. Request: please add a link to a feedback from from the tumblr dashboard.

  26. Request: next/prev navigation links at each post/tumble permalink page.

  27. I don’t really see anywhere to leave suggestions, but I just wanted to make a couple of them.

    1- I love tumblr. I dislike writing blog posts because I am an obsessive editor, but with tumblr I feel okay writing a couple sentences about every link I submit. My first suggestion is that you make a way to integrate tumblr into blogs. I don’t want to convert my whole blog to a tumblr address, but I would like a WordPress plugin that checked if tumblr posts were available and used them as entries.

    2- Make the submission process easier. Easier? Isn’t it extremely easy? Yes, but what might be nice would be a toolbar of some kind (I use flock). I use tumblr so often that it would be nice to be able to add a comment in a toolbar and select the type of comment and away you go.

    Thanks for the great tool.

  28. Big fan!
    I like simple and effective tools.
    Tumblr is a near perfect match for what I’ve been looking for.

    I only have one item to add to the growing “wishlist”:

    Is it possible to add a navigation element that can filter posts by content-type?

    i.e. “Home (all) | Posts | Quotes | Videos | Pictures | Links | Conversations”

    The “occasional blogger’s” posts get lost in the other tumbles – it’d be great to lasso them as either a reader or author.


  29. Leave a reply

  30. 8jvidw8ug4a9pmwo

  31. Ciekawy blog, dodalem go do ulubionych, bede tu wpadal czesciej

  32. whoah this blog is fantastic i really like reading your
    articles. Stay up the good work! You realize, a lot
    of persons are searching around for this information, you could aid them greatly.

  33. Hey there would you mind letting me know which hosting company
    you’re working with? I’ve loaded your blog in 3 different
    internet browsers and I must say this blog loads a lot quicker
    then most. Can you suggest a good hosting provider at a honest price?
    Thanks a lot, I appreciate it!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s