Auth0 Silently Loses Some Indexes

The Downtime Project
Auth0 Silently Loses Some Indexes
/

Auth0 experienced multiple hours of degraded performance and increased error rates in November of 2018 after several unexpected events, including a migration that dropped some indexes from their database.

The published post-mortem has a full timeline and a great list of action items, though it is curiously missing a few details, like exactly what database the company was using. 

In this episode, you’ll learn a little about what Auth0 does and what happened during the outage itself. You’ll also hear about the things that went well and went poorly, and some speculation about whether something like Honeycomb.io could have reduced the amount of time to resolve the outage.  

One Subtle Regex Takes Down Cloudflare

The Downtime Project
One Subtle Regex Takes Down Cloudflare
/

On July 2, 2019, a subtle issue in a regular expression took down Cloudflare (and with it, a large portion of the internet) for 30 minutes.

Monzo’s 2019 Cassandra Outage

The Downtime Project
Monzo's 2019 Cassandra Outage
/

Monzo experienced some issues while adding servers to their Cassandra cluster on July 29th, 2019. Thanks to some good practices, the team recovered quickly and no data was permanently lost.

Gitlab’s 2017 Postgres Outage

The Downtime Project
Gitlab's 2017 Postgres Outage
/

On January 31st, 2017, Gitlab experienced 24 hours of downtime and some data loss. After it was over, the team wrote a fantastic post-mortem about the experience. Listen to Tom and Jamie walk through the outage and opine on the value of having a different color prompt on machines in your production environment.

Slack vs TGWs

The Downtime Project
Slack vs TGWs
/

Slack was down for about 1.5 hours on the first day everyone was back in their (virtual) office in 2021, Jan 4th. Fortunately for everyone, they published a great post-mortem on their blog about what happened. Listen to Tom and Jamie walk through the timeline, complain about Linux’s default file descriptor limit, and talk about some lessons learned.

Introduction

The Downtime Project
Introduction
/

Welcome to The Downtime Project! Here’s a quick episode where Tom and Jamie talk about why they created the show and what you can expect.