Member-only story
Nothing Broke. What Did You Do All Week?
On reliability engineering, invisible work, and the radical ambition of eating dinner while it’s hot.
Fix an outage at two in the morning and you can have quite a story for Monday.
There was a call. There were people on it who normally communicate through other people. Someone said “business impact” with the gravity usually reserved for a medical diagnosis. You found the problem, brought the service back and went to bed feeling dreadful but, at least, useful.
Catch the same problem on Wednesday afternoon, before it hurts anyone, and Monday’s update is considerably less cinematic.
“Fixed a configuration issue.”
“Great. Anything else?”
I find this maddening. The better version of the week can produce the worse story about your contribution.
Take a service that’s gradually running out of connections. Someone notices the pattern, tracks down the leak, tests a fix and watches what happens after deployment. It’s careful work. There are false starts and an unglamorous amount of staring at graphs.
Then Friday arrives and the service keeps working.
No emergency call. No customer apology. Nobody has to explain to their partner why the laptop is coming to dinner.
