A thread limit in one region stopped your deploys

On 25 November 2020, Amazon Kinesis Data Streams in us-east-1 stopped serving requests, and a lot of teams who have never written a line of streaming code had a very long day. CloudWatch couldn’t persist metrics, so PutMetricData and PutLogEvents returned errors and alarms slid into INSUFFICIENT_DATA. ECS and EKS lean on EventBridge, which was delayed, and AWS’s summary of the event says that “impacted provisioning of new clusters, delayed scaling of existing clusters, and impacted task de-provisioning”. Cognito ran into a latent bug in its buffering code and started failing authentication. Reactive autoscaling waited for metrics that weren’t coming.

The trigger was a small capacity addition to the Kinesis front-end fleet between 2:44 and 3:47 AM PST. Each front-end server keeps operating system threads open to every other server in the fleet, so a bigger fleet means more threads per server, and the new capacity pushed all of them past the OS limit. Cache construction stopped completing, the servers held shard maps they couldn’t use, and requests had nowhere to go. First alarms at 5:15 AM, root cause confirmed at 9:39 AM, first front-end servers taking traffic at 10:07 AM. The two recoveries that mattered to everyone else came much later and separately. Container provisioning remained disrupted into the afternoon: AWS says the majority of the ECS and EKS issues were resolved by 4:15 PM PST. CloudWatch metrics and alarms fully recovered at 10:31 PM PST that same day. That’s a working day lost in the region where a good chunk of AWS customers keep part of their estate, because it’s where new features land first.

There’s no blame to hand out here. Large systems fail, and four hours to find a thread limit inside a fleet that size is good work by whoever was on call. The useful question for the rest of us is the one that comes after the postmortem: your provider will do this again, you can’t fix their fleet, and your deploy pipeline still needs to work next time. Some of the answer is cheap habit, like making every machine disposable. The rest is breaking your own system on purpose, before it breaks itself.

It will happen again, so change the assumptions

Start with the part nobody likes: the next one is already scheduled, you just don’t have the date. AWS has published summaries like this before, including the February 2017 S3 event in the same region, where a mistyped input to a maintenance command took out more capacity than intended.

What should you change after an outage you couldn’t have prevented?

Your assumptions, because they’re the only part of the system you own. We can’t make AWS more reliable from the outside, and stopping use of a service we chose on its merits is rarely the answer either.

The assumption worth attacking first is that a dependency is either up or down. The thread limit is a good example of why that’s wrong. A default nobody set deliberately, inside an operating system nobody was thinking about, took down a managed service and then a monitoring service and then a container scheduler. We’ve hit the same class of limit from the other side: a backend autoscaler came up, every new machine opened connections to the same database, and the count ran into the machine’s ceiling of roughly 55,000 ports. That’s how you learn what the ceiling on that box was.

The second assumption is that you know your dependency list. During the outage one of us couldn’t unlock LastPass, which means no passwords, which means no access to anything else either. We never confirmed what sat underneath it, and the AWS summary does note that Cognito returned authentication and credential errors that day. You don’t find that kind of chain by reading your architecture diagram. You find it when it breaks, or when you break it yourself.

Regional isolation is the blast radius you get for free

Having named the assumptions, the next question is how far one bad day travels.

Does a multi-region setup actually buy you anything here?

It buys something different from the region boundary, and the two are worth keeping apart. The boundary held this failure in: one service, one region, most of a day. That’s a fact about this event rather than a property every service hands you.

STS is the one to check in your own code. Regional STS endpoints have been active by default since 16 November 2015, so there’s nothing to switch on for the regions your account has enabled. The IAM docs describe all AWS STS requests going to a single endpoint, https://sts.amazonaws.com. They recommend the regional ones instead. Your SDK may already do that: an AWS maintainer showed in March 2020 that the Java SDK 2.x calls the configured region’s STS endpoint unless the region is AWS_GLOBAL. None of that settles what your own client does, so log a request and read the host.

Containment isn’t availability either. If your workload only runs in us-east-1, a contained outage is still your outage, and what keeps you serving is a deployment elsewhere, prepared and exercised in advance.

The trade comes due when you cross that boundary. Cross-region on AWS is genuine effort: separate endpoints, separate state, replication you design and pay for. Google Cloud makes some of that easier with global services, and that convenience is billed elsewhere. On 14 December 2020, a quota problem in Google’s User ID Service made credential issuance and account metadata lookups for all Google user accounts fail for about 47 minutes.

Resilience is billed too: three virtual machines in three zones with a replicating database can run up more cross-zone traffic than the instances cost. Staying in one zone is a legitimate trade, as long as somebody chose it deliberately rather than during the outage.

The cheapest pattern from this outage is CloudWatch’s own, in two halves. AWS said it would persist three hours of metric data in a local store, and let users and services such as Auto Scaling read recent metrics “directly from the CloudWatch local metrics data store”. Buffering alone wouldn’t unblock Auto Scaling: data that replays once Kinesis returns arrives after the decision needed it. So if your code writes to Kinesis, SQS or another remote sink, buffer with a bound on how much you hold, and make that buffer readable while the sink is away.

Immutable infrastructure first, chaos after

Before you start breaking things on purpose, it’s worth taking the failures you can remove by construction, because each one is noise in every experiment you run later.

What does immutable infrastructure protect you from, and what does it miss?

It makes replacing a machine repeatable, and it says nothing about what happens between machines. When every instance and container comes from a template you rebuild regularly, the replacement is identical to the thing it replaced, so a degraded host under your virtual machine stops being a puzzle. Whether you actually recover is a separate question with its own answers: how fast you notice, whether something replaces the machine without you, whether there’s capacity to replace it into, and what happens to any state that was sitting on it. That property is also what makes Spot and preemptible instances usable, since you’ve already agreed that any single machine can disappear.

Then the list of what it doesn’t touch. It does nothing for a third-party service you consume, which is exactly what failed on 25 November. It does nothing for latency and packet loss on the network between your services. And it does nothing for resource starvation inside an instance that’s still perfectly healthy. T3 instances launch with unlimited bursting enabled by default, so a sustained load above baseline shows up as an extra charge per vCPU-hour rather than as a slowdown. Switch one to standard mode and the trade changes: as the credit balance runs down, CPU utilization is gradually lowered to the baseline level. Nothing is down, nothing is replaced, and your graphs look strange for hours.

That gap is the argument for chaos engineering. What’s left after you’ve made your machines disposable is dependencies, networks and limits. A unit test can fake a dependency error and check that your handler does the right thing with it, and on its own it can’t tell you how the deployed system behaves when the real dependency stops answering.

A chaos experiment starts with a hypothesis

The gap is clear enough, and the common mistake in filling it is to confuse the experiment with the damage.

What makes an experiment different from just killing a pod?

The hypothesis you wrote down before you killed it. If you already know the thing will break, breaking it teaches you nothing: you were right, it’s broken, and you spent an afternoon to find out. The principles of chaos engineering put it as four steps: define steady state as a measurable output, hypothesise that steady state holds while you inject a variable, inject real-world events, then try to disprove yourself.

We see a lot of teams skip straight to the injection and report back that it crashed and they knew it would. Run the experiment in dev or stage until you understand the behaviour, fix what it shows you, and then run it in production with the expectation that nothing visible happens. A production experiment that passes is the point, not a consolation prize.

Here’s the shape of a good question. On GKE, scale a replica set from three pods to four and Kubernetes adds a pod. Scale it back to three, and which pod goes? If you’re running a cache in those pods, the answer changes your performance after every scale-down. We went and found out by trying it, because we couldn’t find it written down anywhere, and the ordering for nodes turned out not to match the ordering for pods. On the AWS side you get more say: an Auto Scaling group has termination policies you can set, so oldest or newest is your decision rather than a discovery.

Delete things to find what nobody wrote down

Hypotheses are easiest to come by in the places where documentation ran out years ago, and there’s an old, blunt technique for those.

How do you find a dependency nobody remembers creating?

Take the thing away and see who comes to find you. One of us ran the ClearCase to Git migration at Ericsson about ten years ago: a decade of accumulated history, thousands of developers, and a ring of homegrown tooling where nobody could say what was still in use. So the migration ran on hypotheses. We think this is dead. Make it invisible to everyone, leave it in place, and wait two months. If nobody comes screaming, it was dead, and it gets deleted for real.

That’s chaos engineering with worse tooling and a longer feedback loop, and it was the only instrument available. It also keeps the rule that makes removal safe.

Pick your first targets for their lack of consequences. Stateless pods and stateless virtual machines are where to start, not the instance holding state nobody replicated. From there you can move to fault injection libraries that add latency or drop packets, which is where the dependency questions start getting answered.

Decide what counts as an error before you inject one

Deleting things only teaches you something if you can tell a healthy system from a sick one, and that turns out to be the hard part.

When should your service return 500?

When your service has failed, which is a decision you have to make rather than a fact you can look up. A 400 says the server believes the client got it wrong, and that belief is a judgement, not evidence that your side is fine. Investigate unexpected 400 responses, especially when a request you expect to succeed fails. A 500 is yours, and it should be rare enough to be worth reading. The awkward case is in between: your database is unreachable but you can still serve something from cache, and whether that’s a 200, a 503 or a degraded payload is a business decision about the minimum useful service you owe people.

Probes make that decision concrete, and Kubernetes splits it in two. A readiness probe decides whether a Pod is eligible for Service traffic; a failing liveness probe gets the container restarted. Put the database check in liveness and the platform will keep restarting a container that was coping on cache. Leave readiness permissive and it keeps routing requests to a Pod that can’t answer them.

This is where error budgets earn their place. The SRE book’s chapter on embracing risk frames the gap between your SLO and your measured availability as a budget you’re allowed to spend, and an experiment that might cost you a little availability is a reasonable thing to spend it on. It also makes the conversation with the business tractable: you’re not asking permission to break production, you’re asking to spend part of a number everyone already agreed to.

The prerequisite under all of it is monitoring. Chaos engineering without monitoring is shooting in the dark, the way a Kubernetes rollout without a build pipeline is mostly ceremony. Read your own error messages too. If one of them is unreadable to the person who has to act on it, you’ve found a good candidate for an experiment.

The tools are optional, the assumptions are not

With a definition of healthy and something watching it, the tooling question gets a lot smaller.

Where do you get your first hypotheses?

From your developers, by asking them what they think will happen. Tell them you’re going to kill a pod and they’ll tell you it’ll be fine. That confident answer is your hypothesis, written in their words, and testing it is a conversation about the system rather than an ambush.

So announce it, book the time, and pick one area. Nobody enjoys finding out from a graph that someone is running experiments on their service. Collect the assumptions first, test them one at a time, and write down what you learned, because a discovery nobody acts on is worth nothing.

For automating the runs, Chaos Toolkit is the open source option that looks most developed. Experiments are declared in JSON or YAML, with probes that measure and actions that break, which makes them reviewable and repeatable the way any other test is. Each run writes an execution journal, journal.json, recording what was done and what the probes saw. Reports are a separate step: install the chaostoolkit-reporting extension and chaos report turns one or more journals into an HTML or PDF document. The declarative syntax will feel familiar if you’ve written Ansible. Gremlin is the commercial end of the market, with a free tier to get started, and it’s hard to miss given how thoroughly it saturates search results for the term.

AWS now points at the practice itself. The Well-Architected Framework’s reliability pillar asks how you test reliability, tells you to inject failures regularly in pre-production and production, to form a hypothesis about how the workload will react, and lists Chaos Toolkit and Netflix’s Chaos Monkey among the tools. Netflix built that tool to kill instances in production at random, and their account of the April 2011 EC2 and EBS event is specific about what it bought them. The service ran through the morning without intervention, at a higher error rate and higher latency than usual. Then, as it became clear AWS wouldn’t fix the zone before evening peak, engineers manually moved traffic out of it, service team by service team, updating every ELB endpoint and the autoscaling groups behind them. Chaos Monkey had trained them for single instances dying, not for a whole availability zone, which is why the same post describes a bigger version they were starting to call Chaos Gorilla.

What this means for teams

Nothing in this is about Kinesis. The lesson of 25 November is that a default in an operating system can reach your deploy pipeline through four layers of managed service, and that your list of dependencies is longer and stranger than your diagram says. You won’t reason your way to that list.

The sequence that works is unglamorous. Make your machines disposable so single-machine failures stop being events. Define what healthy means and monitor it, so you can tell a successful experiment from an unsuccessful one. Then start asking what happens when a dependency goes away, in small controlled doses, starting somewhere nobody will be hurt by the answer.

This week:

• Write down what your service does when its most important dependency stops answering. If nobody can answer, you have your first hypothesis.
• Check the buffering on anything that ships data to a remote sink, decide how much you’re prepared to hold and for how long before you start dropping, and work out who needs to read that buffer while the sink is away.
• Ask the team which pod Kubernetes kills on scale-down, then scale something in a test namespace and see whether the answer was right.
• Book an hour with the developers, collect their assumptions about failure verbatim, and pick the cheapest one to test.

Common questions, answered

How do I keep deploying when one AWS region has an outage?

Deploy from a second region you prepared and exercised before the bad day, because nothing you do during one brings an unavailable provisioning API back. Then work out which control plane calls that second path actually needs, since most pipelines fail on one nobody thought about. In the November 2020 event, running containers kept running, while new ECS and EKS clusters were delayed and autoscaling stalled waiting for CloudWatch metrics that weren’t arriving. Write that list down instead of deriving it from the architecture diagram, which won’t have it, and check your own call sites for clients still pointed at a global endpoint or pinned to us-east-1.

How do I run a chaos experiment without upsetting my team?

Tell people first, and start with something stateless that you’ve already agreed is disposable. Book the time, name the area you’re testing, and write the hypothesis down before you touch anything, so the result is a shared answer rather than an accusation. Run it in dev or stage until you understand the behaviour, then in production, where the expectation is that nothing user-visible happens. Keep away from state nobody replicated, where a failed experiment adds data loss to the bill. An experiment against a stateless service can still interrupt the people using it.

Which chaos engineering tool should I start with?

Start with no tool: a terminal, a hypothesis and someone watching the dashboards will carry the first several experiments. When you want the runs repeatable, Chaos Toolkit is open source, declares experiments in JSON or YAML, and writes an execution journal for every run. Add the separate chaostoolkit-reporting extension and chaos report turns those journals into an HTML or PDF document. Gremlin is the commercial alternative, with a free tier if you’d rather buy than build.

Related episodes

• Monitoring Done Wrong or Dreaming for a Better Monitoring: the prerequisite for all of this, including why boilerplate monitoring gets rebuilt again and again instead of coming with sane defaults.
• Deleting Resources in the Cloud: deleting as a practice, with the checklist and the question of how you know something is really unused.
• Do You Need a Staging Environment?: where to run the experiment that isn’t ready for production yet, and how many environments you actually need.
• Infrastructure as Code: the foundation under immutable infrastructure, and where drift comes from when a change doesn’t go back into code.

Resources

• Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region: the thread limit, the trigger, the full timeline, how the failure reached CloudWatch, Cognito, ECS, EKS and Lambda, and the CloudWatch local metrics data store that dependent services will be able to read from.
• Google Cloud incident 20013: the 14 December 2020 authentication failure, and what a global service failing globally looks like in a postmortem.
• AWS Security Token Service (STS) Is Now Active by Default in All AWS Regions: 16 November 2015, which settles that the regional endpoints are already on and an administrator can switch most of them off again. Manage AWS STS in an AWS Region gives the documented default, the global endpoint, and doesn’t settle what your own client does. On Improve regional STS endpoint UX an AWS maintainer shows the Java SDK 2.x using the configured region’s STS endpoint unless the region is AWS_GLOBAL, so check the SDK and version you run.
• Principles of Chaos Engineering: the four steps, and the definition of steady state that separates an experiment from an outage you caused.
• AWS Well-Architected Framework, REL 12: How do you test reliability?: AWS’s own position that you should inject failures in production, plus the tool list.
• Chaos Toolkit: the experiment format, the extensions for AWS, Kubernetes and Prometheus, and the journal each run writes; reports come from the separately installed chaostoolkit-reporting extension.
• Lessons Netflix Learned from the AWS Outage: April 2011, written by the team that lived it, including the error rates they did see and the zone evacuation they had to do by hand.
• Site Reliability Engineering, “Embracing Risk”: how an error budget is formed from an SLO and what spending it is allowed to buy.

Since we recorded

AWS launched Fault Injection Simulator in March 2021; alarm-based stop conditions halt experiments but can’t restore terminated instances. Northern Virginia had another bad day on 7 December 2021 that disrupted provisioning and authentication while running instances carried on. The Katacoda tutorials in our show notes are gone from the open web: O’Reilly shut katacoda.com down on 15 June 2022 and kept the technology inside its own learning platform. We came back to all of this a year later in Surviving AWS outage, revised for 2021.