Podcast about DevSecOps

#74 - From Preparation To Execution: Handling An Active Incident

One keyboard, a shared incident record and scheduled updates help responders keep control.

14 min read

What keeps an incident from spiraling out of control? How can you organize your team on the spot? We continue our series on incident response, moving from preparation to real-time actions. Mattias shares key points from his course. Listen to learn how we handle incidents step by step.

Discuss the episode or ask us anything on LinkedIn


Four people are typing and nobody knows who changed what

An alert fires. Connections are piling up against the production database, the site is crawling, and by the time you open the channel four people are already in it. Two of them are making changes. Someone’s manager is asking in a direct message whether it’ll be up in three hours. An hour later the service recovers, and nobody can say which of the two changes did it or what the graph looked like when it started.

That hour is what this is about. #73 Incident Response: Key Preparations You Need covered the half you get to decide in advance: backups you’ve restored, a risk list, the plans, the rehearsals. This is the part where that preparation meets people who are tired, worried about their job, and improvising. We’ve been in incidents where somebody cried and in incidents where somebody got angry. Both are normal, and neither improves the decisions being made.

Coordination failures delay recovery as surely as technical ones do. Too many hands on the system, no record of what was tried, the person deepest in the problem answering status questions, and nobody thinking about who takes over at midnight. Those are the four worth fixing, the fix is to assign the roles early, and none of it needs new tooling.

Stop, think, react

Start with the instinct that costs the most: the urge to do something to the system immediately.

What do you do in the first minute of an incident?

Nothing to the system. You stop what you were doing, work out what the alert actually means, and act after that.

The pause is short. Five minutes, sometimes one. What it buys you is the difference between the service that’s screaming and the service that’s broken. High load on the database is the standard example. Sit with it for a moment and the shape of it says something about the web tier too, and the useful question becomes what’s sending that traffic rather than how to make the database swallow it. Jump straight in and you spend the first half hour tuning the thing that’s loudest.

The question to ask in that minute is how big this really is. Is anyone getting hurt by it? Answer that before you touch anything, because the answer is often smaller than the alert made it sound, and the response you pick next follows from it. Don’t rush. Make a plan first, even a bad one you’ll replace in ten minutes.

Ask what the business is losing

The pause is only useful if you spend it on the right question, and the right question isn’t technical.

How do you decide whether this is all hands on deck?

By the downstream effect on the business, not by how red the graph is.

A database running hot with the business ticking along underneath is something to work on. It isn’t a reason to pull six people off what they were doing. Scale the response to the impact, and say out loud what the impact is: customers can’t check out, a batch didn’t run, data left the building. That sentence is also what everyone outside the room will be given, so it’s worth getting right in the first ten minutes.

The assessment decides the next step too. Operational incidents and security incidents pull in different people and different obligations, and the escalation path for each one is in the incident response plan you wrote before today. Open it. The plan is where the communication channel to the business is named and where the all hands effort gets kicked off, which is exactly the sort of thing nobody invents well under pressure.

One channel, and it’s also the record

Once you’ve decided it’s real, the next job is giving it somewhere to live.

Where does the incident live?

In one place everyone can find: a room with a door if you’re in the office, a dedicated channel and a huddle if you’re not.

The physical version is worth prepping, because it’s cheap. A room with screens, machines with the tooling already installed, and the coffee. The remote version is a channel created for this incident and nothing else, so the whole history stays in one place instead of scattered through a general channel between deploy notifications. Put a video call next to it and record it, record the screen too, and you’ve written most of the post-incident report while it was happening.

Google’s SRE book makes this the incident commander’s main job: keep a living incident document, which “can be messy, but must be functional” (Managing Incidents). In practice it’s lighter than it sounds. The second person into the incident shouldn’t reach for the keyboard. They should screenshot the graph with a timestamp, paste it in, and keep doing that.

Do the same at every check-in. A few lines on what was agreed, what it ruled out, and who owns what next. People are rushed and stressed and they mishear each other, so the recap is what keeps two responders from walking away with different ideas of the plan.

One pair of hands on the keyboard

A record only tells you what happened if one person at a time is making it happen.

How many people should be changing things?

One, as we’d run it. Everyone else brings their ideas to that person.

With two people changing things in parallel you lose causality. The error rate drops, and you can’t say whether it was the config change, the restart, or neither. That single keyboard is our coordination rule, and it’s tighter than the one Google writes down. The SRE book draws its line around a group rather than a person, “The operations team should be the only group modifying the system during an incident”, and spends the rest of the chapter separating who repairs from who commands and who communicates. We’d keep the repair work itself down to one set of hands. Say it plainly in the room. Hands off the keyboard, tell me what you want to check, and we’ll check it.

That arrangement already has a name outside incidents. Mob programming puts the whole team in front of one computer, and the rule it inherits from Llewellyn Falco’s strong-style pairing is that “for an idea to go from your head into the computer it MUST go through someone else’s hands”. The typist isn’t deciding. The room is deciding and the typist is executing.

One piece of mob programming transfers badly. Rotating the driver on a timer is the point of the practice, and mid-incident it may cost more than it returns: another machine, another shell, another set of aliases, and keyboard proficiency is a real skill difference. So don’t rotate for its own sake. Pick whoever moves fastest between servers and keep them there while that’s working. Deliberate handovers are a different thing, and you’ll want them: when the typist is tiring, or the problem moves into something a colleague knows better, hand over on purpose and say so in the channel.

Give the people standing around a runbook

Keeping one person on the keyboard works only if everyone else has something better to do than narrate.

What do the people who aren’t typing actually do?

They run the checks the keyboard doesn’t have time for, out of the runbooks written before today.

The default shape of an incident is one person working and three giving advice over their shoulder. Break that up by handing out tasks with the procedure attached: here’s an IP, here’s the runbook for checking it, put what you find in the channel. Diagrams out where people can see them. Now the group is widening the search instead of crowding the fix. Agree the scope of each check before it starts, keep changes out of it so nothing happens to the system that the one keyboard didn’t do, and have every result land in the channel where it joins the record.

Pull in whoever you’re missing while you’re at it. Sometimes you need someone from another team, the data team or a particular development team, and finding them, waking them and getting them onto the call is a job someone should own rather than something three people attempt at once.

The same slot takes an AI assistant well. Point retrieval at your own knowledge base, your old postmortems and your incident channels, and ask it what looks like this and what fixed it last time. Keep that inside the company. Your incident history is a description of how to hurt you, and it isn’t something to paste into a public tool.

Someone has to own the talking

None of this survives contact with an audience unless one person is standing between the responders and everyone who wants an update.

Who talks to management during an incident?

One named person whose only job is that, so nobody fixing anything answers the same question twice.

The job is to listen, collect status from the responders, translate it into business terms, and publish on a fixed interval. PagerDuty’s open-source incident response documentation names the job separately from the commander and from the people doing repairs (the roles), and its steps for each role put internal updates to the executive team at “roughly every 30mins”. The same docs are blunt about what the commander isn’t: “Delegate all repair actions, the Incident Commander is NOT a resolver.”

It’s a good first incident role for someone new. They learn the system by listening to people debug it, and they’re useful on day one. Pair them with someone who understands what’s actually broken, so the technical impact survives the translation and whoever speaks to customers knows what they’re saying. And keep the interruptions off the responders. Asking an engineer every three minutes whether it’s done moves nothing forward.

The words that cost more than the outage

Which brings up the other half of communication: what nobody in the room should be saying on their own account.

What should responders not post in public?

Anything about the incident, because external impact goes out through the people who own that, and the word “breach” goes out through legal.

The pull to narrate is real, and the companies that do it well decided to in advance. GitLab is the reference case. On 31 January 2017, as their postmortem tells it, an engineer restoring replication “proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary”, and around 300 GB went with it. GitLab then ran the recovery in public: a shared document of the timeline, the restore streamed on YouTube to about 5,000 viewers, status updates on Twitter. That worked because the company chose it and controlled it, and the same postmortem notes the document started out private and carried the engineer’s name.

An employee posting from their own account during an incident is a different thing. They have part of the picture, no view of the legal position, and no way to take it back. “Breach” is the word that costs most. It carries a specific meaning in regulation, and whether this incident meets it gets decided by people reading the law, not by the person reading the logs. Leave the terminology to them, and get them the facts early, because the clocks don’t wait on the vocabulary: the HIPAA notification duty, to take one regime, runs from discovery of a qualifying breach rather than from anyone’s choice of word (HHS). Legal controls what you call it. Legal also needs to assess what happened while the incident is still running, so their review doesn’t eat a deadline. We went through those definitions and their clocks in #73.

Plan for the incident that doesn’t end

The last thing to arrange is the one that looks unnecessary at hour one.

When do you start planning for a long incident?

Around two hours in with no end in sight, because when you need the night shift it’s too late to organize one.

Few people debug well for four hours straight. People have kids to collect, and they’ll leave whether or not the incident is over. So while the responders work, someone else is answering different questions. Who relieves these people, and when. Where dinner comes from. Whether there’s a taxi home at three in the morning, which matters more than it sounds if you’ve ever finished an incident at that hour and found no way to get home.

Google’s SRE book gives this its own role, supporting the responders with “filing bugs, ordering dinner, arranging handoffs”. It also asks for the handover to be explicit rather than assumed. The outgoing commander states it, in those words, and doesn’t leave the call “until receiving firm acknowledgment of handoff”, and everyone else is told who’s leading now.

Hold the report until people have slept. We’ve been in a meeting where the site was back but some functionality still needed repair, and the discussion turned to how to communicate. You’re still in the storm. Get out of it first, and let the post-incident work be its own day’s work.

What this means for teams

A lot of how an incident goes is decided by people who aren’t looking at a terminal, and the earlier those roles are assigned the less the coordination costs you. One record, one keyboard, one voice outward, and someone thinking about hour six.

Four things to do this week:

  • Add the next update time to your status template, and make “no change, next update at HH:MM” an acceptable update.
  • Write the rule that one person makes changes into the incident response plan, next to the list of responders.
  • Set up a one-command way to create a per-incident channel with the video call in it, so the record builds itself.
  • Decide in writing who may describe impact publicly and who may use the word “breach”, before the night you need the answer.

Common questions, answered

How do we run an incident when only two of us are on call?

Split the two jobs anyway: one person works the system, the other keeps the record and talks to everyone outside. It’s tempting for both of you to debug, and it costs you the timeline and leaves the questions landing on whoever is deepest in the problem.

The second person’s work is cheap and specific. Screenshot the graphs with timestamps, paste what was tried into the incident channel, and post a status with the time of the next one.

How often should we post status updates during an incident?

On a fixed interval, with the time of the next update in every one. PagerDuty’s incident response docs use roughly every 30 minutes for internal executive updates, which is a reasonable place to start if you have no number of your own.

Keep the content in business terms: what customers can’t do right now, and what you know about when that changes. “No change since the last update” is a valid post, and it’s the one that keeps people from asking.

Can we use an AI assistant during an incident?

Yes, as long as it’s pointed at your own material and stays inside the company. Retrieval over your knowledge base, your past postmortems and your old incident channels answers the question you actually have at 2am: has this happened before, and what fixed it?

Keep the incident detail out of public tools. Your incident history is a description of how to hurt you. Treat the assistant as one more person in the room who isn’t on the keyboard, checking things and reporting back into the channel.

Resources

  • Managing Incidents, Google SRE book, chapter 14: the role split, the living incident document and how to hand off command.
  • Different roles during an incident, PagerDuty: six named roles, and the rule that the commander delegates every repair action.
  • During an incident, PagerDuty: the per-role steps, the 30-minute internal update cadence and keeping the call free of distraction.
  • Postmortem of database outage of January 31, GitLab, February 2017: what public-by-default looks like when the company chooses it, including what they held back.
  • Breach Notification Rule, US Department of Health and Human Services: one worked example of a notification duty that runs from discovery, not from what anyone calls the event.
  • Llewellyn’s strong-style pairing, Llewellyn Falco, June 2014: the one-keyboard rule the incident version borrows.
  • Mob Programming, The Art of Agile Development, second edition: where putting the whole team in front of one computer comes from.

Since we recorded

The third part of the series landed on 27 February 2025: #75 Learning from the Crisis: Post-Incident Actions covers the report, the postmortem and what you do with them. NIST published SP 800-61 Revision 3 in April 2025, superseding Revision 2 and reframing incident response as a CSF 2.0 Community Profile rather than the old four-phase life cycle. The suggestion at the end of the episode, putting retrieval over your own incident history, turned into its own conversation in #99 AI SRE and the End of 3 AM On-Call with Birol Yildiz, on AI-written status updates and why postmortems and documentation are the easiest place to start.