Write the runbook before the incident

Every system I have been paged for had an explanation somewhere. Usually it was in one person’s head, and usually that person was asleep.

A runbook is that explanation written down while it is still calm. It does not have to be long. It has to answer three questions for someone who has never seen the system before: how do I know it is broken, what do I try first, and who do I wake up if that does not work.

Write it while you build

The moment to write a runbook is the day you ship the thing, not the day it breaks. That is when you still remember why the queue has a dead letter table, what the retry limit is for, and which alert is noisy on Mondays.

Keep it next to the code

A runbook in a wiki drifts. A runbook in the repository, changed in the same pull request as the behaviour it describes, stays true for much longer.