At around 3am GMT today, our infrastructure provider suffered a severe technical outage which caused many of its clients (including the likes of Netflix, Heroku and Instagram) to go down. Unfortunately this included GoSquared, along with other analytics providers.
Even with the backup and failover systems we have in place, the outage was simply too widespread to be mitigated completely, affecting the entire North Virginia US-East Data center and everyone hosting services in the multiple availability zones within it.
We worked hard to bring back normal GoSquared service to all users this morning, but despite our best efforts the severity of the problem meant we suffered an interruption in service lasting up to 6 hours for some sites. The team and I would like to express our sincerest apologies for this downtime and for any inconvenience caused. We take events like this very seriously and always formulate ways to avoid the same issues repeating themselves in the future.
GoSquared Infrastructure: Looking forward
Over the past few months, we’ve been working tirelessly to improve our systems so that we are able to effortlessly endure scenarios like this with minimal downtime. We’ve made huge improvements to the underlying hosting and infrastructure on which our services are built, engineering many layers of failover, backup and redundancy to assure reliability. The whole engine is constantly evolving under the hood, and we’re proud of how we’ve managed to do that whilst providing good service to our users.
As part of these ongoing improvement works, we’re working on multi-region redundancy allowing us to route services through datacenters in a completely different geographic region, should one region encounter difficulties. This arrangement is not quick to build but the progress we have made in our general system design allow us to achieve this viably.
GoSquared is constantly getting stronger and smarter. We always ensure we make problems like this a thing of the past.
