#73 - Incident Response: Key Preparations You Need
Backups, clear communication roles and rehearsals give responders a foundation before an incident starts.
17 min read
Incident response can be complex, but where do you start? Andrey, Mattias, and Paulina dive into the preparation steps you need to take. Mattias shares his expertise from teaching an incident response course. What’s their top recommendation? Listen and find out!
Most incidents come from inside the building
The incident that eats your week is probably not ransomware. One of ours opened as high load on a website that was going down, with no obvious cause. Connections were stacking up against the production database. We asked around, and a developer said they had changed an index, and that it shouldn’t have any effect on production. It was a table lock, and the site was down for the hours it held.
The loud ones happen too. A content update from a security vendor put 8.5 million Windows devices into a crash loop in July 2024, which CrowdStrike later traced to a mismatch between the fields its sensor expected and the fields the update supplied. In late December 2024, researchers showed at 38C3 that movement data for 800,000 electric cars, and the contact details of their owners, had been sitting unprotected online at Volkswagen. The way in was a heap dump from an exposed Java application, with credentials in the memory it dumped. Different causes, same morning: something is broken, people are shouting, and whatever you prepared beforehand is all you have.
How bad an incident gets is mostly decided before it starts. Split the work into pre-incident, incident and post-incident. This is the pre-incident half, the part that decides what you have to stand on when the other two arrive.
Backups are the part of ransomware nobody puts in the headline
Start with the control that decides how the loudest kind of incident ends.
Why is the backup the first thing to fix?
Because a tested backup the ransomware can’t reach is what gets your availability back without depending on the attacker.
We once ran a small web store on WordPress with WooCommerce. One morning it was a red page demanding Bitcoin. The web folder was encrypted, all of it, though we could still log into the server. We had a backup, so we removed the encrypted folder, restored, and the store came back. The log showed a single request to wp-admin from a Tor node, and that was the whole story.
A backup only counts if ransomware can’t reach it and you have restored from it recently. The CISA and MS-ISAC #StopRansomware Guide says to maintain offline, encrypted backups and to regularly test their availability and integrity in a disaster recovery scenario, because many variants go looking for reachable backups and delete or encrypt them. A backup job that reports green and a backup you have actually restored are two different facts.
Time the restore when you run it, and be clear about which number you got. The stopwatch measures how long the data takes to come back. Getting the service back to customers also takes the dependencies that have to come up with it, the cutover, and the checks that prove it works.
Restoring also doesn’t close the incident the way it used to. Crews take the data out first, encrypt second, and then threaten to publish what they took, which CISA calls double extortion. Paying doesn’t buy deletion, and we wouldn’t trust them not to share it anyway. Your backup buys availability back. It buys nothing on disclosure, and disclosure is the half with lawyers and deadlines in it.
Name what can hit you before you rank it
Backups answer one scenario well. The next job is the list of scenarios, and it is wider than most teams’ threat model.
How do you pick which risks to work on?
Score each one by probability times impact, then work the top three, because nobody gets to address them all.
The categories are worth separating, because they fail differently. Cybercrime is the obvious one: ransomware, an intrusion, credentials in someone else’s hands. Vendor failure is the one where you own the consequences without owning the fix. Kinesis went dark in us-east-1 on 25 November 2020, and AWS’s own summary says a small capacity addition pushed the front-end fleet past the maximum number of threads an operating system configuration allowed. Every service that depended on Kinesis degraded with it. Then there is losing access to your people rather than your systems. We had a consultant in Kazakhstan in 2022 we could not reach, because the government cut internet access during protests. And there is the self-inflicted kind, which is most of them.
Rate the impact in the three buckets the business already thinks in. Reputation: would anyone trust us after this. Money: the fine, the credits, the lost contract. Work: if the data center burns down, can we build a new one and hire the people before the customer leaves. Low, medium, high is enough precision to sort the list.
Those buckets are also how the work gets funded. The honest pitch is that we want our users’ data to stay theirs and the platform to stay up; the practical one is that the standards you are audited against mandate incident response and disaster recovery. Contracts are the third lever: slow recovery against a B2B SLA has a price. Check what an SLA buys, though. Under the Amazon S3 SLA, a service credit is “your sole and exclusive remedy”, and your own customer contracts may be written harder than your provider’s.
The expensive incidents are the ones nobody modeled. NotPetya went through Maersk’s Windows estate in 2017, and Gavin Ashton, who worked on identity there, wrote afterwards that “by rights, the directory should have been gone” and that they were “incredibly lucky”. The recovery story we remember is one domain controller in an African office that happened to be offline when the worm hit, and became the seed for the rebuild. That detail is secondhand and we never verified it. The lesson holds either way: copies that replicate each other also replicate the damage.
Three documents, and the one teams write last
A ranked list of risks is only worth the time if something downstream changes because of it. Three documents carry that weight.
What does each of the three plans answer?
The business continuity plan says how the business keeps working, the disaster recovery plan says how the systems come back, and the incident response plan says who does what while it is happening.
Business continuity is the one the business people engage with, because the questions are theirs. How do we keep working if people can’t get to the office. How do we keep working if nobody has internet. Disaster recovery is the technical twin: what gets restored, onto what, in what order, and how long that takes. Both go stale fast. A plan written half a year ago describes a system you have since changed.
The incident response plan comes after those, and it is where templates multiply. They all do the same four things: detect, determine the scope, contain, and feed what you learned back to the beginning. NIST’s Computer Security Incident Handling Guide splits the life cycle the same way, into preparation, detection and analysis, containment, eradication and recovery, and post-incident activity. Pick a template, put your own names in it, and move on to what’s specific to you.
Protect the loop. If what you learn doesn’t come back and change the plan, the organization isn’t learning, and you may as well skip the exercise.
Your incident response plan is missing a person
Plans are good at naming the people who fix things. The role they leave out is the one that decides whether the fixing gets interrupted.
Who is missing from most incident response plans?
The person who owns communication, so that the people working the incident are not the ones answering questions about it.
You know what the channel looks like. Is it done? Is it done? When will it be up? Can it be up in three hours? What do I tell the client? Can I deploy now? Every one of those lands on whoever is deepest inside the problem.
Someone also has to own the business impact question: how bad this is for customers and for us. That answer is what the communicator passes on, and it is what the reporting clocks below run on.
Then there are the three audiences. Customers are easy in B2C, because you have their email addresses already. B2B is harder, because the notification contacts are buried in contracts nobody has open at two in the morning. Pull them into one list now, or run a status page your partners already subscribe to. Management needs its own briefing, from one person, so that nobody improvises a version of events outward. And employees need to be told not to post details on social media. They don’t have the full picture, and a well-meant mischaracterization becomes the company’s legal problem. People reach for the word breach without knowing it is a defined term.
“Breach” is a word with a clock attached
That word is worth being pedantic about, because regulators have defined it and attached deadlines to it.
Why does the word you use in public matter?
Because “breach” is a defined term, and using it in public before anyone has assessed the facts commits the company to a claim its lawyers have to unpick.
What the word doesn’t do is start the clock. The obligations run on what happened and when you found out, so staying quiet about the label buys you nothing.
Under HIPAA, 45 CFR 164.402 defines a breach as an acquisition, access, use or disclosure of protected health information that isn’t permitted under the privacy rule and that compromises its security or privacy. The presumption is the sharp edge. Once PHI has been used or disclosed impermissibly, that disclosure “is presumed to be a breach”. The way out is narrow: you have to demonstrate “a low probability that the protected health information has been compromised”, through a risk assessment of at least four factors. An intruder who never reached PHI is an intrusion, with no impermissible disclosure for the presumption to attach to. HHS also lists three exceptions to the definition, among them a good-faith, in-scope access by a workforce member that goes no further. Where it is a breach of unsecured PHI, and PHI encrypted to the HHS guidance isn’t unsecured, individuals get notified without unreasonable delay and no later than 60 calendar days after discovery.
Europe’s clocks don’t wait for anyone’s data to leak. DORA entered into application on 17 January 2025, five days before this article’s release, and requires financial entities to report major ICT-related incidents to their competent authority whether or not anything was disclosed. The delegated regulation the Commission adopted on 23 October 2024, from the standards the European Supervisory Authorities submitted in July 2024, sets three clocks. Initial notification is due within 4 hours of classifying an incident as major, and no later than 24 hours after you became aware of it. Where classification lands beyond those 24 hours, the 4 hours run from classification instead. An intermediate report follows within 72 hours of the initial notification, and the final report within a month of the intermediate report or the latest updated one. That text was adopted but had not entered into force when we recorded. NIS2 is the one about to land on companies that never thought of themselves as regulated: member states had until 17 October 2024 to transpose it into national law.
Practice it, or all you have is a document
Everything above is a hypothesis until someone runs it.
How do you practice without waiting for a real incident?
Put a scenario on the calendar, record who took part and what broke, and fix the worst thing you found.
The frameworks ask for the exercise, not just the plan, usually once or twice a year, with a record of the scenario, the participants and how it went. CISA publishes over 100 tabletop exercise packages with objectives, scenarios and discussion questions, which beats starting from nothing. The record isn’t the reason to do it. You don’t think well in crisis mode, and the exercise is where the weak points show up: the one person everything depends on, the step nobody can follow, the contact list that is three jobs out of date. It sounds boring. So does going for a run, and you feel better afterwards.
Netflix’s Chaos Monkey took the other route and generates incidents for you, terminating instances in production at random. That is a bigger commitment than a tabletop, and it only exercises the failures it can cause.
Terminology differs between companies, so don’t take our wording as canonical, but the distinction is useful. A runbook is step by step: the database is out of disk, so log into the host, find what is filling it, reclaim the space with the supported command, log out. Put the real commands in it, and check the dangerous ones before someone runs them under pressure. MySQL binary logs are the classic trap. The manual has you run SHOW REPLICA STATUS on every replica first, to find the earliest file any of them is still reading. Then run PURGE BINARY LOGS TO 'that-file', which deletes the files preceding the target and leaves the target itself in place. Name a later file and a replica that happens to be disconnected loses the log it was reading, so it cannot resume when it returns. Don’t reach for rm instead: deleting the files that way leaves the .index file out of sync, and later purges fail. A playbook sits higher up and describes how you work, what you run and which vendors are in the picture. The runbook is the one that saves you at three in the morning.
If you only fix three things: endpoints, training, updates
Preparation expands to fill any amount of time, so here is the short list for a team that has none.
Where do you start if you can only fix three things?
Know your endpoints, train your people, and keep your software updated.
Endpoints first, meaning anywhere data comes in: the web applications, the management interfaces, the APIs you expose, the dependencies on third parties, and the laptops if you count them. Know what is connected and what each one does. The heap dump at Volkswagen was an endpoint nobody meant to publish.
Training second, and not only for developers. A developer learns by doing that code which is fine at 10 rows behaves differently at 20,000. Security is the same muscle pointed at different questions: what happens if the data isn’t what I expect, if it arrives as a one-gigabyte blob, if it comes from a client I don’t control. Managers need their own version of the training, because awareness is what turns into budget and time. The habit generalizes past code, too. Checking whether a shop your kid found on TikTok has existed for more than six months and has a support address is the same due diligence, done at home.
Updates third. You can’t defend against every threat out there, but the companies whose software you run are fixing theirs continuously, and keeping pace with them is the part you control.
The same preparation works outside work
One more, and it is the one people don’t expect.
Does any of this apply outside work?
Yes, because your family has a recovery plan whether or not anyone wrote it down.
If something happened to you permanently, would your partner know how to get into your password manager, how to reach the bank, which contacts need to be told? That is a disaster recovery plan with a different asset list, and most of us have never tested it.
The threat side has changed, too. Attackers aggregate what you publish and feed it to a model, and what comes back is a convincing message, voice or video. The FBI’s Internet Crime Complaint Center warned in December 2024 that criminals use generative AI to run fraud at larger scale, including short audio clips that imitate a relative asking urgently for money. Caller ID doesn’t settle it: numbers are spoofable, and SIM swapping is easy enough that it happens all the time.
Agree that phrase with the people who would call you in an emergency. If a call feels wrong, ask for it, or hang up and reach the person on a channel you chose yourself.
What this means for teams
Preparation is unglamorous, and it is the part of an incident you get to decide in advance. The rest you find out on the day.
Four things to do this week:
- Restore one backup into a scratch environment and time it end to end. A green backup job is not evidence.
- Write two names next to “communications” in the incident response plan, so there is a second one when the first is on a plane.
- Pull the incident notification contacts out of your B2B contracts into one list your on-call can open in thirty seconds.
- Score your risks by probability and impact, take the top one, and write the runbook for it.
Common questions, answered
How do I know my backups will survive ransomware?
Keep at least one copy offline or otherwise unreachable from the systems it protects, and restore from it on a schedule rather than trusting the job status. CISA’s #StopRansomware Guide asks for offline, encrypted backups and regular tests of their availability and integrity, because ransomware looks for reachable backups and encrypts or deletes them.
Time the restore while you are at it, and note which number you got. The stopwatch covers the data coming back. Getting the service back also takes the dependencies, the cutover and the checks that prove it works.
How do I reach B2B customers during an incident when the contacts are in contracts?
Extract them now, into one list your on-call can open without legal’s help. Contractual notification contacts and timelines are scattered across agreements, and nobody is reading PDFs at two in the morning.
A status page or a mailing list partners subscribe to in advance carries a lot of the load, because you update everyone once instead of answering the same question individually.
Is every security incident a data breach?
No, and each regulator defines its own labels, so let the people who own that call make it before anyone says “breach” in public. Under HIPAA, an intrusion that never touched protected health information is an intrusion; the breach presumption attaches to an impermissible use or disclosure of PHI, and holds unless one of the three statutory exceptions applies or a documented risk assessment of at least four factors shows a low probability of compromise.
Waiting on the label buys no time. HIPAA’s clock runs from discovery of a breach of unsecured PHI, and DORA’s runs from detection and classification of a major ICT-related incident, with or without data leaving the building. Start the assessment and the reporting timeline together, and keep one person responsible for anything that goes out.
Related episodes
- #21 Surviving AWS Outage: the Kinesis failure that took CloudWatch and ECS with it, and the assumptions it broke.
- #37 Surviving AWS outage, revised for 2021: the disaster recovery ladder, from backup and restore up to multi-region.
- #24 Ways To Protect Yourself From Data Breaches: what to do about the breaches that happen at someone else’s company.
Resources
- #StopRansomware Guide, CISA and MS-ISAC, September 2023: what an offline, tested backup has to look like, and what double extortion changes.
- NIST SP 800-61 Rev. 2, Computer Security Incident Handling Guide: the four-phase life cycle your incident response plan template is copying.
- CISA Tabletop Exercise Packages: ready-made scenarios and discussion questions, so your first exercise doesn’t start from a blank page.
- 45 CFR 164.402: the HIPAA definition of a breach, including the presumption and the three exceptions to it.
- Commission Delegated Regulation on reporting major ICT-related incidents under DORA, adopted 23 October 2024: the reporting clocks as the Commission adopted them, before the text entered into force.
- Summary of the Amazon Kinesis Event in US-EAST-1, 25 November 2020: a vendor failure with a published root cause, which is what a good post-incident report looks like.
- Amazon S3 Service Level Agreement: what an SLA actually pays out, and why it is not damages.
- FBI IC3 alert I-120324-PSA, 3 December 2024: generative AI in financial fraud, and the family code word.
Since we recorded
We finished the series in the next two episodes: handling an active incident in #74 and the post-incident work in #75, then took the practice question further in our chaos engineering pair, #76 and #77. The guide we lean on above has been replaced: NIST withdrew SP 800-61 Revision 2 on 3 April 2025 and published Revision 3, which recasts incident response as a Cybersecurity Framework 2.0 community profile rather than a standalone handling guide. Europe’s reporting clocks kept getting more concrete after DORA started to apply, and we walked through them alongside NIS2 and the AI Act in #87.
