Write the runbook before the incident
The best time to explain how a system fails is while nothing is on fire.
Every system I have been paged for had an explanation somewhere. Usually it was in one person’s head, and usually that person was asleep.
A runbook is that explanation written down while it is still calm. It does not have to be long. It has to answer three questions for someone who has never seen the system before: how do I know it is broken, what do I try first, and who do I wake up if that does not work.
Write it while you build
The moment to write a runbook is the day you ship the thing, not the day it breaks. That is when you still remember why the queue has a dead letter table, what the retry limit is for, and which alert is noisy on Mondays.
Keep it next to the code
A runbook in a wiki drifts. A runbook in the repository, changed in the same pull request as the behaviour it describes, stays true for much longer.
The checklist I start from
- What does healthy look like, in one graph?
- Which alert fires first, and what does it mean?
- The first three things to check, in order
- How to turn the feature off without a deploy
- Who owns it, and how to reach them at night
- When this runbook was last tried for real
The last item matters most. A runbook nobody has followed in a year is a story about how the system used to work.