A Ring of Failure

Photo by Marc Renken on Unsplash

Generally speaking, failure is just embarrassing.

But what if your failures resulted in profound havoc at banks, airlines, and companies, creating billions of dollars worth of damage as well as devastating disruption?

Chills.

A wise person learns from the mistakes of others.

So let’s accumulate some wisdom and go over the biggest failures in the history of software.

Ya, Boo

Yahoo has had a notable history of both data breaches and hacker attacks.

In December 2014, Yahoo's security team reported that hackers had obtained the credentials of at least 500 million Yahoo accounts. The hack took place in 2014 but was only made public in 2016.

Thankfully, sensitive financial data like bank accounts and passwords were left uncorrupted.

The question remains: Why did it take so long to confirm the hack and its scale?

Facebook Takes a N’app

In 2019, Facebook suffered a severe outage that tampered with the Facebook family of apps, including Instagram, WhatsApp, and Messenger.

The company was back on track after 24 hours and blamed the issue on ‘routine maintenance.’ As Facebook reports, it was a server configuration change that triggered a cascading series of issues.

Now let’s imagine the magnitude of frustration when business owners couldn’t access their digital Holy Grail.

Glitch Airways

In November 2019, a computer glitch left thousands of British Airways passengers stuck on the ground for a maximum of 24 hours. The delays and further cancellations had been caused by what the airline describes as a "technical issue."

But that was not an isolated incident. In August of the same year, another IT glitch caused the cancellation of more than 100 flights and torpedoed the plans of 35,000 passengers into disarray.

Oh, talk about chaos.

Github et tu?

Yep, even super techies are not invincible.

On January 31st, 2017, GitHub suffered a major backup restoration failure for one of its products: The online service GitLab.com.

The failure was the result of accidental data removal from their primary database server. The aggravating factor was that the backups had not been made for a while due to a configuration error.

As a result, the company lost some production data including modifications to database data such as projects, comments, user accounts, and others. A total of 300 GB of customer data was lost.

Amazin' Outage

Thousands of online services store their data in the data centers of AWS.

This means that even a small tech glitch can take down Netflix, Slack, Airbnb, Pinterest, Reddit, Quora, and even NASA.

Hence, many of them were impacted during a prolonged AWS outage that took place in November 2020.

The company states that this incident occurred after a "small addition of capacity" to its front-end fleet of Kinesis servers.

Have a Plan

You can’t get away from phishing attacks or ransomware.

But you can at least mitigate the risks by rolling back your product or feature in the case of an embarrassing emergency.

Feature flags help you safely test features in production and control each individual’s experience with user segmentation.

Credit for the above piece goes to Tatsiana Isakova, Hang Ngo, and Ellen Stevens.