#99 - AI SRE and the End of 3 AM On-Call
AI can draft the status update and post-mortem before it gets permission to change production.
18 min read
Could AI handle the worst parts of incident response before you even join the call? Mattias and Paulina talk with Birol Yildiz about AI-written status updates, fast root cause analysis, and the path from read-only help to autonomous fixes. They also explore why post-mortems and documentation may be some of the best places to start.
The first part of a 3 AM page isn’t repair work
You get paged at three in the morning. You find the laptop, you squint at an alert, and then comes the part nobody budgets for: working out what actually broke. Which service is this. What shipped yesterday. Is a customer affected yet, does the status page need a sentence, and who’s going to write that sentence. By the time you know enough to act, you haven’t touched the problem. You’ve been reading.
Reading is the part a machine can do while you’re still asleep. That’s the whole argument about AI in incident response, and it’s narrower than the marketing around it: which pieces of the night shift are you willing to hand over, in what order, and what has to be true before you widen an agent’s access. Get the order wrong and you’ve handed write access to production to something your team hasn’t learned to trust.
Birol Yildiz, co-founder and CEO of ilert, has been working on this since 2010, when his master’s thesis project was getting an SMS to a human after Nagios noticed something. The customer was a software vendor selling to North American law firms whose users were highly paid lawyers expecting support at any hour, and back then, in his words, “there weren’t APIs for everything.” He later ran 24/7 services at REWE digital, one of Europe’s largest grocery retailers, and launched the current version of ilert in 2019. The company now integrates with more than 150 monitoring and ticketing tools, which is a useful vantage point: it means seeing what alerts look like across a lot of very different stacks.
Most alert noise is a design problem, not a tooling problem
Before any of this is an AI question, it’s an alerting question, and the answer starts with the person who writes the alert rule.
What belongs on a pager?
Something that needs a human now, and that a human can act on. If it fails either test, it isn’t a page.
CPU at 80% is the classic bad page. On its own it tells you nothing about whether the service is healthy, and nothing about whether anyone needs waking. Your fleet doesn’t have to idle at 10% for your comfort. Google’s SRE book is blunt on the principle: “Every page should be actionable”, and “Every page response should require intelligence. If a page merely merits a robotic response, it shouldn’t be a page.” If the only human response is a command you could have scripted, script it and stop waking someone up to type it.
Old tools get blamed for this, and the blame usually lands in the wrong place. Nagios gives a service four states: OK, WARNING, CRITICAL and UNKNOWN. For a new active-check failure, Nagios normally retries before treating it as HARD; max_check_attempts includes the initial check.
The buckets aren’t what makes a page useless. The threshold is. check_disk takes -w and -c as either a percentage of free space or an absolute amount of it, per volume: -w 10% on /tmp, -w 100000 MiB free on /. Choose the wrong form for the volume and CRITICAL fires while you still have plenty of gigabytes free, and the person you woke has nothing to act on. The interesting signal is usually the WARNING anyway, because it tells you where things are heading, and it’s the one everybody has learned to scroll past.
We’ve all walked into the Slack channel where alerts arrive every few minutes and nobody looks. Ask about one and you get “that’s been going on for a while, we ignore it.” The cost isn’t the noise. It’s the next real alert landing in the same channel, and a customer telling you about the outage before your own monitoring gets your attention. That’s time you can’t get back once a human has to escalate on your behalf.
The structural fix is older than any of the tooling: the people who write the service carry its pager. A bad alert then hurts the person who can delete it.
Grouping takes fifty alerts down to two problems
Even well designed alerts arrive in bursts, because one broken thing rarely tells you only once.
How do you stop a single failure from paging you fifty times?
Group alerts before they reach a human: by time window, by fields the alerts already carry, and by what they mean.
The cheap layers come first. Time based grouping collapses everything from one source inside a window, or allows one open alert per source until it’s accepted or resolved. ilert shipped that alongside its AI assisted incident comms in August 2023, with windows as short as two minutes. If the payload already carries a correlation field, group on it. Neither needs a model.
Then there’s the case where two alerts mean the same thing in different words. A full disk gets reported one way by the application monitor and another way by the infrastructure agent. The text has nothing in common, the problem is one problem. ilert’s similarity grouping handles that by comparing an incoming event’s summary and details against unresolved alerts from the same source, and its docs are specific about the knobs: a similarity threshold that defaults to 0.75, and a grouping window chosen from a list that starts at two minutes, with five minutes recommended. Both are yours to set per alert source. And this is a different kind of AI from the one in the headlines. It’s semantic matching, in production since well before anyone said “agent”.
Similarity runs out at the point that hurts most. When a central component fails, the downstream services alert about their own symptoms, and those alerts really are different: a timeout here, a queue depth there, a 5xx rate somewhere else. One root cause, no shared vocabulary.
End to end traces would give you that map, and almost nobody has them across the whole stack. The fallbacks are a service directory that already knows your topology, or an eBPF collector that watches traffic from the kernel and infers the relationships. With the topology in hand, alerts landing close together can be read against it, and the downstream noise can be attached to the upstream cause instead of paging four teams.
Alert or incident? Don’t sort it at 3 AM
Once the noise is down, the next argument teams have is about vocabulary, and it’s usually the wrong argument to have at speed.
When does an alert become an incident?
When there’s business impact, or when fixing it needs more than one person.
The working split ilert uses is simple. An alert is something that needs immediate human attention, whether or not customers can tell yet. An incident is the coordination anchor: there’s impact, or you need a space where database, platform and support people can work the same problem.
The database volume at 95% shows why that distinction isn’t academic. Nothing is broken, no customer notices, and if the volume keeps filling, you could have an outage within hours on your most central component. It earns a page, somebody clears space, and the whole thing ends without ever becoming an incident.
What makes the model work is the other half: a low priority alert doesn’t wake anyone by design. It waits for the next business day. That’s the decision worth getting right, because it’s the one your on-call engineer feels. Everything else is labels.
At ilert, the status update came before the root cause work
ilert’s earliest generative AI feature wasn’t root cause analysis. It was prose.
What can AI write for you while the incident is still live?
The customer facing update. You type “payment API is broken” and get back a publishable sentence in your company’s register.
It saves a few minutes, which isn’t the main point. The main point is that it removes a context switch at the worst possible moment. You’re in firefighting mode, and the question “how do I phrase this professionally for customers” is the last thing you want in your head. ilert has had this since 2023: a basic technical overview in, an incident summary and a status page message out, with the affected services flagged.
The other 3 AM time sink isn’t AI at all, and it’s worth saying so plainly. Mobilizing more help should never involve opening a spreadsheet to work out who covers the database rota on a Tuesday. You ask for the database person, the platform resolves who that is, and if the primary doesn’t answer it escalates to the secondary. Any modern on-call tool does this out of the box. If yours doesn’t, fix that before you shop for an agent.
A root cause analysis waiting when you open the laptop
The next step up the ladder is handing over the reading, and it comes with a deadline.
How fast does an automated root cause analysis have to be?
Faster than you are. The notification takes about thirty seconds to reach you, you need another two minutes to be awake and looking, so the analysis has to be there inside two to four minutes or you start digging yourself.
What the agent does in that window is the triage you would have done. Fifty alerts come in, it sorts them into two clusters, and it investigates each cluster separately. There’s no recipe for root cause analysis, but there’s a question list every responder works through: where is the problem, what changed, what do the logs and metrics say, what shipped recently. Changes are the first thing to check because changes are the usual culprit. Google’s SRE book puts a number on it: “SRE has found that roughly 70% of outages are due to changes in a live system.”
That list is exactly what you do by hand in the dark: grep the logs, check the dashboards, scroll Slack to find out who deployed. None of it needs you specifically, and the agent isn’t asleep, so it can start the moment the alert fires rather than the moment you reach the desk.
Models may also help with a pattern that short window thresholds miss. Web logs carry 401s all day, so nobody alerts on them. A couple of failed attempts, a five minute pause, then a couple more, and the rate stays under the rule you wrote. Hand a day of logs to an LLM and ask what looks odd, and a single IP pacing itself across those 24 hours is the sort of thing it can surface. Trend based detection can probably find it too.
An agent only finds what you connected it to
The ceiling shows up as soon as the cause sits outside everything the agent can read.
Why did the agent miss the cause?
Because the failure was in the deployment, and the agent kept working the code.
A recent one from a client: a change that looked correct in the repository and behaved wrongly in production. The code read fine and nobody could see the problem in it. The deployment pipeline only analyzed changes up to a few hundred files, the pull request was bigger than that, so part of the deployment never triggered. Half the change shipped, half didn’t, and the two halves clashed. The agent couldn’t work it out either, and it failed in a telling way: it went looking for more problems in the code while the broken thing sat in the pipeline.
So before you judge an agent’s analysis, write down what it reads: logs, metrics, deployments, repositories, cluster state. Then write down what it doesn’t: your pipeline’s own limits, the vendor dashboard, the DNS record someone changed by hand, the third party that’s quietly degraded. Plausibility isn’t evidence, and the hypotheses worth having are the ones that come with links you can check.
From read-only to pre-approved actions
Which brings up the real question: how much rope, and when.
How much should an agent be allowed to do?
Start where it can’t change anything, then widen one class of action at a time.
Read only is the first rung. ilert’s first agent arrived in June 2025 as a read-only beta, reaching Grafana, Prometheus, GitHub and Kubernetes through MCP and returning recommendations rather than actions. Even off script, it can’t push a change. But read-only is a scope, not a safety property: it is access to your logs, your repositories and your cluster, and it deserves the scrutiny you give any other credential.
Scope isn’t the only risk. Volume is. An agent that polls your cluster can hurt it without changing a thing: Kubernetes’ write-up of streaming list responses says API Priority and Fairness protects CPU far better than memory, and a few large list requests can exhaust the API server. So give the agent its own credentials and its own rate limits, and bound what it reads: selectors instead of whole collections, and a client that streams lists where your version supports it. That lowers the odds of a runaway loop taking down the API server while you investigate something else. It doesn’t rule it out.
The next rung is propose and confirm: the agent asks to raise the memory limit on that pod, and a human clicks. After that, pre-approve a class of actions it may run alone: bump memory, add CPU. A rollback is a poor candidate for that list, because rolling back is itself a change with its own failure modes.
The last rung is the interesting one. The agent handles what it recognizes and pages you only for genuinely novel incidents. Birol Yildiz is candid about the gap: he believes it’s possible, has no answer yet for how to do it safely, and “I wouldn’t do it today.”
The reason to climb at all is the self-driving analogy: a car you have to watch is useful, one you don’t watch is far more useful, and the gap between them is all supervision cost.
Post-mortems are the cheapest place to start
If you want a first use that pays for itself without touching production, look after the incident rather than during it.
Where should a cautious team start with agents?
The incident report. It’s the task everyone postpones, nothing it produces changes a running system, and the raw material is already sitting in two systems.
One source is machine generated and already structured: the alert, the fifty events behind it, the three minutes it took to acknowledge, when the incident was opened, which status updates went out. The other is human and unstructured: a war room channel with forty people in it, a call recording, the decisions made in both. Combine them and you get a reconstruction: timeline, impact, actions taken. Call it 80% of a post-mortem document, written while you sleep.
That’s cheap, not free. The agent reads the war room channel, which carries whatever people pasted into it at 2 AM, so scope its access the way you’d scope anyone’s. And a reconstruction can be wrong in ways that read smoothly, so someone should check the timeline against the events before the document circulates.
Birol Yildiz’s verdict is unusually strong for a hedged field: “it’s irresponsible not to get the help of an LLM for those tasks.” Hard to argue when the alternative is a timeline rebuilt from memory a week later, or never written at all.
From there the agent can go further, if you connect it to your source control: suggest the measures, propose a higher load balancer timeout, sketch code changes for review. Then the meeting argues about which of those is right.
There’s a second reason this is the right entry point. Documentation nobody wants to write is where people most readily accept help, and willingness is the scarce resource in the autonomy question. The documentation problem is getting worse, because code now changes faster than the humans around it can describe. NIS2 and DORA add to that pile: by Birol Yildiz’s account the EU rules changed the bureaucracy rather than the demand for fast response, since companies with business critical services already wanted to fix things fast. We covered what they ask for in episode 87.
What this means for teams
Much of the 3 AM pain starts before any agent is involved: alerts that shouldn’t have fired, routing that should have been automatic, and a write-up nobody had time for. Agents help most with the work around the incident, which is also where they can do the least harm. Start there and the autonomy question arrives with evidence behind it.
The counter-argument is worth taking seriously: if agents do the investigating, do engineers lose the ability to investigate? It’s the same bargain as every abstraction before it. You don’t manage memory by hand anymore, and you’re still better off knowing that garbage collection exists. Keep one layer of understanding below what you delegate.
Four things to do this week:
- Pull every page from the last 30 days and ask of each one whether a human could act on it, and whether a human did. Downgrade what could have waited until morning, delete what nobody could act on, and find out why the rest went unanswered.
- Turn on grouping on your noisiest alert source, and set the window and threshold deliberately rather than taking the default on faith.
- Give an agent read-only access to logs, metrics and one repository, on its own credentials and its own rate limits, and let it draft post-mortems and status updates. Nothing it produces goes out unread.
- Write your autonomy ladder down before you need it: what it may read, what it may propose, what it may run unattended, and who signs off on moving a rung.
Common questions, answered
How do I stop alert fatigue without missing real incidents?
Delete the pages a human can’t act on, then group the ones that remain. The first pass is at the source: any alert whose response is a scripted command should trigger the script, and any alert that doesn’t need attention tonight should be low priority so it waits for the morning. The second pass is grouping, by time window, by a correlation field in the payload, or by similarity for alerts that describe the same failure in different words. What survives both passes should reach a phone.
How do I lower the risk of giving an AI agent access to production?
Start read only, on its own credentials and rate limits, scoped the way you’d scope any other credential. Read-only permissions stop an agent from changing your configuration, not from overloading what it reads, so bound its queries too. Logs, metrics, deployment history and one repository are a reasonable opening scope, though a cause sitting in your pipeline’s own limits, or in a system you never connected, stays invisible. When you want it to act, start with propose and confirm, where a human clicks before anything runs. Only then pre-approve a narrow class of actions, like raising a memory limit. A rollback is a poor candidate: it is a change with its own failure modes.
Do we still need a post-mortem meeting if AI writes the report?
Yes, and the meeting gets better. What the AI produces is a reconstruction: timeline, impact, what was done, assembled from the alert data and the war room conversation. The meeting is for the part that needs judgment, which is deciding what to change so this class of incident stops happening. A post-mortem meeting with no preparation is wasted time, and preparation is exactly the part people skip.
Related episodes
- Episode 73, Incident Response: Key Preparations You Need: the groundwork that has to exist before tooling helps, including roles, rehearsals and backups.
- Episode 74, From Preparation To Execution: Handling An Active Incident: running the live incident, with one keyboard, a shared record and scheduled updates.
- Episode 75, Learning from the Crisis: Post-Incident Actions: the post-mortem practice an agent is drafting into.
- Episode 98, Beyond AI SRE: released six days earlier, on repeatable operational work as the natural fit for agents.
- Episode 87, EU Compliance 101: AI Act, DORA, NIS2 explained: what the incident documentation and reporting duties actually require.
Resources
- ilert introduces Agentic Incident Response (June 16, 2025): the first agent shipped read only by default, reaching Grafana, Prometheus, GitHub and Kubernetes over MCP.
- Intelligent alert grouping with ilert AI: the settings behind similarity grouping, including the
0.75default threshold and the recommended five minute window. - ilert summer release notes (August 23, 2023): AI drafted incident communication and time based grouping, dated well before the agent wave.
- Google SRE book, Monitoring Distributed Systems: settles what belongs on a pager, including “Every page should be actionable”.
- Google SRE book, Introduction: the source of the 70% of outages caused by changes figure, in the Change Management section.
- Nagios Core, State Types: what SOFT and HARD states mean, and how
max_check_attemptscounts. - Streaming list responses in the Kubernetes API server (December 17, 2024): why a read-only client still needs bounded queries, and what a handful of large list requests did to API server memory.
- Birol Yildiz on LinkedIn: where he posts on incident response and agent autonomy, alongside his work at ilert.
Since we recorded
ilert’s AI SRE became generally available on September 14, 2026, on every paid plan and in trials. It shipped the first two rungs of the ladder described here, observe and propose, with “Nothing changes the state of your system without an engineer clicking approve”, while pre-approved actions and full autonomy stayed switched off. The analysis waiting for you when you wake didn’t ship: at GA you start the investigation yourself, from an alert, an incident or the agent’s chat, and investigations that begin on their own the moment an alert fires are called “the next step, not this one”. We came back to the other side of the trade in episode 111, on what happens when an agent holding real credentials acts past the guardrails somebody wrote down.
