us-east-1 kept serving traffic and your week broke anyway

On 7 December 2021 at 7:30 AM PST, an automated scaling activity in us-east-1 set off a surge of connection attempts on the internal AWS network. That network carries monitoring, internal DNS and parts of the EC2 control plane. AWS’s summary of the event describes congestion between that network and the main one, retries that made it worse, and a latent bug that stopped clients backing off. Running EC2 instances were unaffected throughout and existing load balancers stayed healthy. What broke was uneven, and the unevenness is the lesson.

EC2 API calls started erroring at 7:33 AM and new instance launches did not recover until 2:40 PM. Route 53 answered DNS queries normally the whole time while its APIs were impaired until 2:30 PM. Console access was fully restored at 2:22 PM and STS at 4:28 PM, while Lambda APIs and function invocations operated normally start to finish. It wasn’t only management calls either. API Gateway returned elevated errors on customer requests until it largely recovered at 4:37 PM. Reaching S3 and DynamoDB through VPC endpoints was impaired, while direct access to both was fine. Creating support cases was impaired from 7:33 AM until 2:25 PM PST, so the way to ask AWS for help was degraded as well.

That is a different shape of failure from the one most disaster recovery plans are written for. Those plans say “we’re multi-AZ” and stop, which is a sentence about losing part of a region’s capacity. It says nothing about a day when every zone serves traffic and half your management calls come back as errors. For a lot of people the first sign was domestic: a kid who can’t watch Disney Plus, a parcel that doesn’t move. Ours was customers who couldn’t deploy, couldn’t scale and, for several hours, couldn’t log in. Eight days later we watched us-west-1 and us-west-2 go dark enough to notice. The assumption that a whole region doesn’t have a bad day is now retired.

Multi-AZ didn’t cover what actually broke

Start with the strategy almost everyone has already paid for, because this event tested it and the result wasn’t what teams expected.

Why didn’t spreading across availability zones help on 7 December?

Because no availability zone failed. An availability zone is one or more discrete data centres with their own power, networking and connectivity, and a region is built from several of them. The advice for the last decade has been to spread your workload across zones so that losing one costs you capacity rather than service. That advice holds. It covers a failure that didn’t happen on 7 December: the zones kept serving, existing instances kept answering, healthy load balancers stayed healthy, DNS records kept resolving.

What degraded was the layer you reach for when you want to react. New load balancer provisioning and instance registration slowed down behind the Elastic Load Balancing API latency. Autoscaling couldn’t add capacity it couldn’t launch. Your pipeline’s terraform apply came back with errors against a region that was, by the narrow measure, up.

This is the part worth taking to the team: write down which of your recovery steps need a control plane call, and which service each one calls. Scaling up and replacing a failed node both need an EC2 launch. A rollback may or may not, and the difference is your deployment strategy. Swap the artifact or the container image on instances that are already running, and you stay clear of the launch path. Roll back by replacing instances or standing up a fresh group, and you don’t. Every step that launches or replaces an instance assumed a path that was erroring for most of a working day.

us-east-1 isn’t just another region

Teams in Europe felt less of this, and the ones who felt it anyway are the interesting case.

Why does a Northern Virginia problem show up in an account that runs in Frankfurt?

Because specific named endpoints live there, not because every service secretly routes through the region. Sign-in is the one that bit hardest. AWS’s summary records elevated latency on OIDC credential requests to STS, which caused login failures in dependent services, and console access wasn’t fully restored until 2:22 PM PST. The AWS STS global endpoint sts.amazonaws.com is hosted in a single region, US East (N. Virginia), with no automatic failover to anywhere else. Regional endpoints such as sts.eu-west-1.amazonaws.com have existed since February 2015, and AWS recommends them over the global one for exactly this reason: lower latency and redundancy. If your SDK config still points at the global endpoint, that’s a Northern Virginia dependency you took by default rather than on purpose.

What we saw on the day fits that shape rather than a blanket outage. Dashboards in other regions opened fine until you switched the region selector to us-east-1, and then came delays and 500s. SSO worked for a while and then stopped. Root sign-in failed. Credentials we already held kept working, against services that were themselves healthy.

One more thing we hit that the post-mortem doesn’t mention: at some point the console in us-east-1 redirected to us-east-2. If you don’t notice the region has changed, your EC2 list is empty, and the first thought isn’t “wrong region”, it’s that everything is gone. Treat it as our experience rather than documented behaviour. Either way the lesson is cheap: check the region selector before you panic.

Some of this is the price of age, and credit is due before we take any of it apart. us-east-1 is the oldest region AWS runs, and it carries a decade and a half of things nobody wants to touch. New services land there first, and it holds up remarkably well for all that. The detailed public post-mortem landed within days, which is good for trust and better for the rest of us.

Put the number on the table before the outage

The technical answer to all of this is a second region. The reason most teams don’t have one isn’t technical.

How do you get a second region approved?

Price the downtime, not the architecture. We’ve built the multi-region proof of concept and found inter-region networking cost more than the business wanted to pay. So we went back to a single region with our eyes open: risk understood, accepted against the price. What changes after an event like this is that both sides of the trade have numbers. An hour down costs a figure somebody in finance can produce. The second region costs a figure you can produce. Money is the one language that moves the conversation, and the words that carry it are disaster recovery strategy and business continuity plan, not availability zones.

One disagreement on this, and it’s an honest one. Julien wants all infrastructure code in a single repository. When the day comes you change one place and redeploy somewhere else, instead of hunting through every repo for what’s deployed where and waiting on busy developers. Andrey won’t open that can of worms on air: there are other ways to reach the same result, and that argument deserves its own episode.

Multi-cloud is the other reflex, and multi-region comes first. A second cloud means writing for a second set of APIs, closer to coding for Windows and then for macOS than to deploying the same thing in another data centre. The primitives sound the same, disk and network and queue, and nothing about how you talk to them is. Geography may not help either: a Swedish workload has an AWS region at home, while Google Cloud’s nearest is across the water in Hamina, Finland.

Replicate the data, reserve the capacity

If a second running copy of production is out of budget, the question is what the smallest useful version is.

What has to be in the second region before you need it?

Your data, your images, your keys, and a claim on compute. AWS’s disaster recovery whitepaper lays out a ladder between plain backups and active/active, and the two rungs in the middle are not the same deal. With a pilot light, a small part of your infrastructure runs all the time to keep mutable data in sync while the rest stays off. Recovery means you provision a full-scale environment around that core. Warm standby keeps a scaled-down but functional version running. It costs more every month and asks less of you on the day: you scale it up rather than build it.

Either way the stateful pieces have to be current there already. Database snapshots copied over. Buckets covered by S3 replication. Container images copied by ECR cross-region replication, which landed in December 2020. Tables that need to be in both places on DynamoDB global tables.

Keys got less painful during 2021, within limits. Multi-Region KMS keys arrived in June 2021, and what they solve is client-side encryption: ciphertext encrypted under one replica decrypts under another, with no cross-region call or re-encryption step. A managed service that encrypts on your behalf keeps its own flow. Copying an encrypted RDS snapshot to another region still means naming a KMS key valid in the destination region. What changed is that the key you name can be a replica of the same logical key rather than an unrelated one. Each replica is still a separate KMS key with its own key policy, grants and aliases, so you configure permissions in both regions either way.

If your infrastructure as code genuinely covers everything, a provisioning run should have you serving again in an hour or two. Should is the right word. Nobody knows their recovery time until a restore exercise produces it: pick a quarter, build the second region from the repository and the snapshots, and time it.

You’re not the only clever team with a runbook pointing at the same neighbouring region, and the compute there is finite. On-Demand Capacity Reservations hold capacity for a specific instance type in a specific availability zone, and you pay the On-Demand rate whether or not you use it. That price is the point: it’s the difference between a plan and a hope.

Keep the failover switch outside the region you’re leaving

Replication and reservations only pay off if you can point traffic at them, which brings up the least glamorous dependency in the stack.

Where should the DNS record that moves your traffic live?

With someone who isn’t having the outage. Route 53 answered queries normally throughout 7 December, and its APIs were impaired for seven hours. Read that twice: the records kept resolving, and you couldn’t change them. The switch you’d reach for was in the room that was on fire.

What we’d do instead is split the zones. Internal and intermediate domains can stay in Route 53, with health checks and DNS failover doing their job and weighted or geoproximity routing moving traffic between regions. The apex domain your customers actually type is better off at another large DNS provider. When the console is unreachable you can still log in somewhere else and repoint it. AWS Global Accelerator gives you two static IPs and lets you move what sits behind them, which is the same idea with the same caveat: it’s inside AWS.

Caching may cushion a DNS provider’s bad ten minutes, and how much depends on the TTL and on whose resolver answers. The base rule is RFC 1034: a cached record is good for as long as its TTL says, and after that the resolver discards it and asks again. So a client holding an unexpired answer rides the outage out, and one whose copy expires partway through goes back to authoritative servers that aren’t answering. RFC 8767, from March 2020, loosens that for this exact case: a resolver may serve a stale answer when it cannot reach the authoritative servers. Whether the resolver in front of a given user does that is somebody else’s configuration, so plan on the TTL and treat serve-stale as a bonus. Caching also delays deliberate failover. Resolvers can keep serving the old answer until its TTL expires. Set short TTLs on failover records before an incident. None of that is an argument that DNS is easy: anyone running it at scale absorbs amplification attacks and traffic spikes measured in gigabits. As far as we could tell, the large providers are rarely the thing that’s down.

Your monitoring shouldn’t live inside the thing it watches

Failover plans assume somebody notices, and this event was unkind to that assumption at every scale.

Where should the thing that tells you you’re down run?

Outside the blast radius, which means outside the account and usually outside the cloud. AWS is candid about its own version of the problem. Congestion impaired the Service Health Dashboard tooling’s failover to its standby region, and AWS wasn’t successfully updating the dashboard until 8:22 AM PST, nearly an hour in. The status page was degraded by the thing it was supposed to describe. We’ve all seen the small version, a monitoring stack on the same cluster as production, quiet at exactly the moment you need it.

For external checks we’ve been using Better Uptime across several projects. What makes it worth naming is the packaging. Endpoint checks, a public status page and on-call paging come in one product, instead of Pingdom plus PagerDuty plus Statuspage and the integration work between them. It costs a fraction of what the incumbents charge. One feature we hadn’t seen before earns its keep. When an endpoint starts answering again, Better Uptime waits out a recovery window of several minutes before it closes the incident. A flapping service then produces one incident rather than two hundred SMS messages to acknowledge one by one.

It’s a young product and it shows. There’s no scripted browser check, so you can’t click through a signup flow the way Pingdom does, and no SNS integration when we looked. The delivery path isn’t perfect either. We’ve had incidents where the phone call never landed and the incident itself recorded a carrier problem. It fell back to email, and it supports app push as well. It kept working while AWS was down, which is the most we can honestly claim for it; none of us knows whether it runs multi-region. Whatever you pick, test the alert path end to end and on more than one channel. A monitoring system that detects the outage and then sends mail through a mail server that’s part of the outage has told nobody anything.

The jobs that fail by going quiet

External checks catch the things that answer requests. The other half of your estate doesn’t answer anything.

How do you find out a cron job stopped running?

Turn the check inside out and alert on silence. The job calls a webhook every time it finishes, the monitor expects that call on a schedule, and when the call doesn’t arrive the monitor opens an incident. That’s a heartbeat, and it covers the failure mode where nothing errors, nothing pages, and a backup simply stops happening.

If you run Prometheus the same pattern is a routing rule away. The Watchdog alert that ships in kube-prometheus has the expression vector(1), so it fires continuously. Having the rule is not the point, and on its own it does nothing for you. Configure Alertmanager to send Watchdog notifications to an external receiver that alerts you when they stop arriving. Dead man’s switch, same logic, applied to one more path than a single job.

One path, though, and not your alerting as a whole. A heartbeat only vouches for the route it travels, and Alertmanager configures each integration separately. So a Watchdog that keeps arriving by webhook says nothing about whether the email or SMS receiver next to it still delivers. To know the mail path works you need a check that goes through the mail path, and it has to report its failure somewhere that doesn’t.

The habit this replaces is worth naming, because a lot of us have it. Jobs post their status into a Slack channel, and every morning a human scans the channel for the one that didn’t post. That’s a cron job made of a person, and it fails on holidays.

What this means for teams

Nothing here is really about us-east-1. The lesson of 7 December is that the region boundary you were counting on holds for the data plane and leaks at named points. Those points are the sign-in path, the global STS endpoint, and the control planes you call when you want to change something. Add the request paths that cross the internal network, API Gateway and VPC endpoints among them. Multi-AZ was the right answer to the last question. This one has a price, and the business can only agree to pay it if someone puts both numbers in front of them.

Which parts of your estate get that price paid for them is a business question before it’s an engineering one. Run the exercise security people already run: identify what matters most, protect that hardest, accept more risk on the rest. The tier list that falls out buys you an order to restore in. It also buys permission to leave things broken, and permission to switch things off so there’s less to think about. Customers who are businesses with a contract need somebody awake for them. If you’re serving pictures of kittens, an hour off is an hour off and lunch is a defensible answer.

The good news is that the work got cheaper while nobody was looking. Replicating keys, images and data across regions in 2021 is configuration rather than a project, capacity is reservable, and the failover switch can live somewhere else. What’s left is deciding what deserves the treatment and testing whether the plan runs.

This week:

• List every control plane call your recovery depends on, name the service behind each, and ask what happens when that service errors for a working day.
• Find out who can change your apex DNS record without the AWS console, and whether that path has been used in the last year.
• Pick the job whose silence would hurt most, give it a heartbeat, and then stop it on purpose to confirm somebody gets paged.
• Ask the business what one hour of downtime costs, write the pilot light estimate next to it, and agree which flows come back first.

Common questions, answered

How do I get into my AWS account when console sign-in is failing?

Use credentials you already hold, from the command line, against the services that are still healthy. On 7 December existing access keys and temporary credentials kept working while issuing new ones and signing in to the console did not. Access is not function, though: EC2 API calls in us-east-1 returned elevated errors and latency, so credentials you held didn’t guarantee the call succeeded. Prepare the access half now: point your tooling at a regional STS endpoint instead of the global sts.amazonaws.com, which AWS has recommended since 2015.

Do I need multi-cloud to survive a region outage?

No, and it’s the most expensive place to start. A second region reuses your APIs, your tooling and your team’s knowledge, and covers the failure that actually happened. A second cloud means a second set of APIs, closer to porting an application between operating systems than to deploying it twice. There are good reasons to run on two clouds, and surviving a regional event is rarely the one that pays for it. Get the second region working, then decide whether you still want the second provider.

How do I check that my alerts actually reach someone?

Send a signal that should arrive constantly, and alert when it stops. That’s the dead man’s switch pattern. Route Prometheus’s always-firing Watchdog alert to a receiver outside your cluster that pages you when the notifications stop, or have a job call a heartbeat URL on a schedule. A heartbeat only vouches for the path it travels, so a Watchdog arriving by webhook says nothing about whether your mail route still delivers. Test each channel with its own check that reports failure through a channel it doesn’t depend on. A page that fails at the phone carrier is a page nobody got.

Related episodes

• Surviving AWS Outage: the November 2020 Kinesis event, where a single service went down and took CloudWatch with it, plus the chaos engineering habits that came out of it.
• Monitoring Done Wrong or Dreaming for a Better Monitoring: why boilerplate monitoring gets rebuilt on every project, and what sane defaults would look like.
• Three pillars of observability: logs, metrics and traces, where to start, and the mistakes that make the bill arrive before the insight.

Resources

• Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region: the trigger, the congestion, per-service impact and recovery times, and why the Service Health Dashboard couldn’t fail over.
• Disaster recovery options in the cloud: backup and restore, pilot light, warm standby and multi-site active/active, priced against RTO and RPO.
• On-Demand Capacity Reservations: what a reservation holds, that it’s scoped to one availability zone and instance type, and that you’re billed either way.
• Manage AWS STS in an AWS Region: the global endpoint is hosted in US East (N. Virginia) with no automatic failover, and AWS recommends regional endpoints instead.
• Encrypt global data client-side with AWS KMS multi-Region keys: June 2021, what multi-Region keys are for, and that each replica carries its own key policy and grants.
• RFC 1034, section 3.6 and RFC 8767: a TTL says how long a resolver may cache a record. The March 2020 exception lets a resolver keep serving a record past expiry when the authoritative servers are unreachable.
• Alertmanager configuration: receivers are configured per integration, so a webhook route and a mail route succeed or fail independently of each other.

Since we recorded

Northern Virginia had another long day in October 2025. DNS resolution for the regional DynamoDB endpoints failed late on 19 October, and a second wave hit Network Load Balancer health checks until the afternoon of 20 October. Better Uptime has since been folded into Better Stack, where it’s the Uptime product. We came back to this ground properly in our incident response series, starting with Incident Response: Key Preparations You Need, and again in Chaos Engineering Explained: Part 1.