It’s Not About The Best Product, It’s About The Best Users

16 Flares Filament.io Made with Flare More Info'> 16 Flares ×

Guys, do you have a minute? First off, we are really sorry for all the troubles that happened in the past 48 hours. There were many things that went wrong, Buffer was wonky and caused problems for many of you. We tried to fix a lot of these things and there are still a few bugs in the system that we are dealing with right now. This was all our fault and many of these things shouldn’t have happened.

You Are Amazing

Yes, YOU are amazing. Despite all the troubles and despite all the ongoing glitches, you stayed strong and saw it through with us. The kind reminders and bug spots that we received on Twitter and via email is something that helped us tremendously to get things out of the system more quickly. Seeing so many of you that showed their support, wished us all the best to get the problems fixed and worked with us, was just unbelievable. So I guess, all we can say is thank you. You rock and we will work hard to make it up to you with our upcoming features.

What The Hell Happened Then?

So why all this mess? Where we just flicking a few switches on the servers whilst being drunk? Not quite, here is what happened:

We moved from MySQL, a database system, over to another one. All the mechanisms that send your Tweets, store your Twitter accounts, your analytics and more has changed. We moved to a new database system called MongoDB. What this meant in terms of development was to write the whole code for Buffer anew so it works with this new system.

The reason we swapped to this new system were mainly growing pains and issues as we are expanding to other networks. Buffer is about to expand to Facebook, Google+ and Linkedin. In order to do this, we had to significantly ramp up our servers and do all the expansion work with a new database. Another key issue was that we just had so many users, that the site slowed down for many of you. MongoDB, the new system, will handle a lot of our scaling issues a lot better than the existing one.

So yes, this was the actual reason and we continue to battle the water that is leaking in at many holes. In a way, we are a ship that just got hit by a major storm. The water is everywhere, but we have great captains on the steering wheel and the best users and hands in the world that help us get the water out of the boat.

It is also important to mention here that something like this will NOT happen at any point in the future again. This problem with a switch to another database is something that really will never be an issue again for us. So, the biggest storm is past us.

We have the best users on the web

A quote from an important entrepreneur that stuck with me ever since went something like this:

“It’s not about who has the best product, it is about who has the best users that help you make the product awesome.”

I couldn’t agree more with this. Would it not be without you, we would have never seen this through and be as far along as we are right now.

So here is a quick thank you note from Joel and me for you:

Please let us know below any thoughts you have, any bugs you spot or any ideas for Buffer features in the future. We would love to hear from you and will make sure to respond to every single comment.

  • Jandroid

    Now that’s customer service: Major doo happens. They “own” it. And they apologize and thank users for patience/support in return. Yes, they still have some work to do, but I am glad to hear they are still working on it. And, like any “construction” project /repair, it sounds like it will be worth the headaches we had in the end, since it will expand/improve the service a bunch. (Looking forward to FB extension). This is becoming a vital Twitter feed app for me, running 3 Twitter accounts (1 personal, 1 biz, 1 volunteer non-profit org). Keep up the great work guys. Let’s bring this thing home.

    • http://twitter.com/LeoWid Leo Widrich

      Jandroid, thanks so much for this great comment. Yes, it is just like you say, there is no point in pointing fingers as we clearly made a few mistakes and we are just keen to get all things working as quickly as possible again. 

      Awesome, it is great to hear that Buffer is so useful for you and yes, we want to help you improve your business performance on Social Media as fast as we can! :) 

    • http://joel.is Joel Gascoigne

      Hi Jandroid, your kind words mean so much to me and I’m sure Leo too. Thanks so much for understanding, since we certainly messed up here and we know how much problems we caused for you and many others.

      You’ve mentioned the headaches will be worth it and I’m very excited about cranking out some neat features (including Facebook support) and continuing to improve what we have.

      Thanks again, amazing people like you make this a really fun journey :)

  • http://jonnyrowntree.com Jonny Rowntree

    First off, you guys are the best for keeping us all up to date on whats happening in the best way possible, Twitter. Second, sorting out what you have done in 48 hours?!? You guys aren’t just the best. You literally kill the small startup developers out there who are trying to succeed a large scale operation.

    If I had the cash, I’d be on the next flight to SF to congratulate you both. All the very best for the future and get some sleep!

    Jonny

    • http://twitter.com/LeoWid Leo Widrich

      Jonny, thanks so much for stopping by here and leaving your kind words of encouragement. I greatly appreciate your support and yes, the last 48 hours were rough, even rougher I can imagine for some of our users though, so we are keen to get i all back to normal asap. :) 

      • http://jonnyrowntree.com Jonny Rowntree

        Leo, its always a pleasure! You have to make important changes to your business infrastructure in order to see it through the next level of users. We know it will all be worth it!

    • http://joel.is Joel Gascoigne

      Thanks Jonny, we felt very bad when things started to go wrong and worked hard to fix it as quick as we could, but we were down for a number of hours and it is not good at all, and it would be very understandable for people to be annoyed. I was surprised that the reality of it was that everyone expressed their support and helped us to identify bugs so we could fix major issues quickly. It was such a nice reminder how lucky we are to have such amazing people who’ve joined us and are so connected with us along this journey :)

    • http://joel.is Joel Gascoigne

      Thanks Jonny, we felt very bad when things started to go wrong and worked hard to fix it as quick as we could, but we were down for a number of hours and it is not good at all, and it would be very understandable for people to be annoyed. I was surprised that the reality of it was that everyone expressed their support and helped us to identify bugs so we could fix major issues quickly. It was such a nice reminder how lucky we are to have such amazing people who’ve joined us and are so connected with us along this journey :)

  • http://twitter.com/willsteward William Steward

    If there’s one thing I learned with crisis management, it’s to never say “never again”. Perhaps say you’ve learnt from it and will minimise the risk, but the worst is over/never again can be asking for trouble. ;) 

    I did the same thing a few years ago when I was running File Den, had a couple of disk failures at the same time – which caused data loss for some of our users. We said it’d never happen again, improved our redundancy, and had an even greater number of disks fail at the same time corrupting RAID arrays etc which resulted in 1 week or so restore times for ~1/3 of our userbase I think it was. We couldn’t afford to backup the sheer amount of data we had for all our free users (and definitely couldn’t afford something like AWS for the sheer amount of data we were pushing around), so utilised multiple servers with multiple RAID arrays, making sure data was on at least 2 servers at once etc, but still had problems. 

    We had a lightning strike do a load of damage to equipment too, moved on and recovered from that, to have almost a year later another lightning strike do exactly the same thing. 

    Every time we told our users “never again”, but instead we should have told them what we had learned, exactly what we did to prevent it in the future, and how it’s been made less likely – in the end though this crap happens, and props to you guys for coming out quickly & giving an explanation. The best way to face any problem is head on!

    Let us know how MongoDB works out for you guys!

    • http://twitter.com/SmartSoftMarket Smart Soft Market

      Joel/ Leo
      First, you get great support from your users because you totally earned it.Second, I totally agree with William. I wish you every success, I hope your service grows and grows and you become gazillionaires. But as you grow, stuff will go wrong, it always does. Learning is the key.My suggestion: after this fire is put out, spend a little bit of time reflecting on what you learnt. Draw up a to do list and pick 1 or 2 items e.g. give yourselves the ability to add a service status message on the interface

      Thank you again for a great product

    • http://twitter.com/LeoWid Leo Widrich

      William,

      Thanks a lot for the great comment. Yes, you are absolutely right, the never is something I should update, there will always be more troubles coming up. 

      Wow, that is a crazy story there and very reassuring for us at the same time too. One just always has to be prepared for the worst. A lightning strike is something I can’t even imagine how to deal with. Really fantastic you guys saw that one through.

      Thanks for the encouragement and sharing of your learning, will get back to this once we are hit by a major issue again. :) 

  • http://twitter.com/WeAssistVentura Daniel Richman

    Dudes, your apology is accepted. Keep up the amazing work. 

    • http://twitter.com/LeoWid Leo Widrich

      Hi Daniel, thanks so much, that is fantastic to hear, proud to have you and so many other amazing users on board! :) 

  • http://www.rijkwillemse.nl/ Rijk Willemse

    Thanks guys!
    Good example of how to deal with mishap!
    Good luck
    Rijk

    • http://twitter.com/LeoWid Leo Widrich

      Rijk, thanks so much for the kind support here! :) Will try to get all back to normal asap. 

  • http://www.propelly.com Cec Plascott

    How many other startups would go through so much effort to apologise to users? I can’t really name any! Fixing all the problems in 48 hours? Jeeez…Oh, and props to you guys for being completely open and honest when communicating the problems you were having.

    • http://twitter.com/LeoWid Leo Widrich

      Cec, thanks so much for stopping by here, really appreciate that. Haha, well I think it was quite unusal to break so many things at a time, so yes, keen to get all things working again asap. :) 

  • http://twitter.com/Twylah Twylah.com-Kelly Kim

    So happy you’re getting everything back on track, Leo & Joel! 

    LOVE buffer, love the “buffer guys,” and really appreciate the honesty.  

    Many thanks for all you do for us.

    • http://twitter.com/LeoWid Leo Widrich

      Kelly, it is always a pleasure to see your friendly face appear in our comments. And I really appreciate your understanding here. We are keen to put the ship back on course and make it sail faster than ever! :) 

  • Anonymous

    It’s amazing to see such transparency and honesty. Great work guys :)

    • http://twitter.com/LeoWid Leo Widrich

      Mark, thanks so much for your encouragement, yes, it is just like you say. Keeping things transparent and honest is the most important thing I believe. 

  • http://twitter.com/JamesRing James Ring

    I was obviously one of the fortunate ones that didn’t have any problems.

    I just wanted to drop by and show some gratitude for the great service you provide. Often people are quick to say what’s wrong but say nothing when they have a good experience.

    I would just like to say once again outstanding customer service by both @LeoWid:twitter & @joelg87:disqus. Keep up the great work and am looking forward to the Buffer for Facebook. 

  • http://twitter.com/fizzylemon Serrina B

    No worries!  I love your transparency – I think everyone understands that you are human beings just doing the best you can.  Also, if you don’t feel any bumps along the way, that means you weren’t driving fast enough.  :-)  Keep up the good work.

  • Claire Berlin

    So impressed by your integrity and commitment to customer service- Buffer is hands down my favorite social media tool and been so instrumental in producing positive change in my organization. 

    We appreciate your transparency- keep up the good work!