Only Incidents in Production

A podcast about what large-scale incidents teach us, read from the public record.

Public incident reports are the most honest engineering writing our industry produces, and they are also the least carefully read. Every few months an incident takes down something that millions of people depend on; in the best cases, the organization that ran it publishes an account of what happened, and that account is read once, argued about for a week, and forgotten. Only Incidents in Production is built around those case studies: one public incident per episode, read carefully, with the people who wrote about it or lived through it.

While it is easy to treat a candid postmortem as an admission, I'd argue it is closer to a gift: the report and the people who wrote it are the reason the rest of us get to learn anything at all, and this show treats them accordingly. The questions that matter to anyone running an enterprise platform are sociotechnical ones rather than forensic ones: where the failure domain sat, what the organization knew and when, and what a remediation program looks like once the incident channel goes quiet. This is not a show about who was at fault, and not a show about what should have been obvious in hindsight.