The three ways AWS accounts actually get breached

Most AWS incidents aren’t clever. Three shapes cover the bulk of them. A publicly accessible resource, usually an S3 bucket or an Elasticsearch cluster. An access key that got committed and pushed, usually to GitHub. And an IAM role stolen through SSRF or remote code execution against an EC2 instance. The attacker reaches the metadata service at 169.254.169.254 and walks off with the instance profile’s credentials. The first shape keeps happening. In June, researchers at vpnMentor reported 845 GB across more than 20 million files sitting in unsecured S3 buckets shared by a group of niche dating apps. The files held photos, private chats and audio. The third shape is the one the Department of Justice describes in the Capital One case. The 2019 intrusion “occurred through a misconfigured web application firewall that enabled access to the data”. That release names the way in and not the request path behind it.

Knowing the three doesn’t tell you what to do on Monday. It’s harder still when what you inherited is twelve accounts nobody can name, a handful of IAM users with admin, and access keys older than the product. Scott Piper’s AWS Security Maturity Roadmap, published in May 2020, is the best answer we’ve read to that question. It runs to ten pages, it’s free, and it doesn’t ask for your email address, which is rare. Piper has spent three years consulting on AWS security through his own company, Summit Route, for clients from startups to Fortune 100s. He also wrote the free flaws.cloud CTFs and the open source CloudMapper and Parliament.

Eleven stages, and they build on each other: an inventory before a scanner, a scanner before detection, detection before you take options away from people. We’d put it in front of anyone starting a cloud security program on AWS. The one thing we don’t agree on among ourselves is how early the blocking should start.

You can’t secure the accounts you can’t name

Stage one is a spreadsheet, which sounds like a joke until you sit down to write it.

What does stage one actually produce?

A list of every AWS account in the company, each with its 12-digit id, its point of contact and its payer. Pull all of them into one AWS Organization, and add a separate account that belongs to whoever does security.

That list takes detective work: ask each group what they have, ask finance what’s being paid for, search the mail archive for signup confirmations. The paper is firm that the master account holds no resources of its own, no EC2 instances, no buckets. Finish stage one and you have at least three accounts: the one with the business in it, an empty master account, and the security account the logs land in.

The objection is that nobody arrives at a greenfield. You walk into an account someone else created years ago, with admin handed out freely and access keys over 500 days old. That’s fair, and it doesn’t change the order. A legacy estate is a journey. Pull the accounts into one Organization and an unknown number of unknown problems becomes a list you can rank, worst account first.

Backups are what’s left when an attacker has admin

Stage two is one line long, and it decides whether an incident is expensive or terminal.

Why is “have backups” a security stage and not an ops one?

Because destruction is the incident you can’t detect your way out of. The paper’s example is Code Spaces: an attacker destroyed its data and its backups, and the business shut down within 12 hours.

We once watched a vendor at a conference demonstrate ransomware hunting, complete with the Tor address and how they track the operators down. Good backups take the decryptor off the table: you restore, you lose the hours since the last backup, and you buy nothing from anyone. That ends one problem rather than the incident. You still have to find out how they got in and what left with them. Maze has been publishing victims’ data to force payment since late 2019, and no restore undoes that. The condition on the restore is that the backups sit somewhere the compromised credentials can’t reach. Keep them in the same account under the same keys and they get encrypted along with everything else, and you’re back watching the vendor’s demo.

So the target is a copy in a separate account, in another Region or off AWS entirely, written where the credentials running your workload can’t delete or encrypt it. Another Region on its own isn’t isolation: one set of account credentials works in every Region. S3 Object Lock gives you write-once copies where you want them. The two services the paper names split the work. AWS Backup runs the schedule and copies a backup to another Region, which it has done since January 2020, and the copy stays in the account that made it. The hop into another account is S3 replication’s, writing to a destination bucket a different account owns, or yours to build out of shared snapshots.

Turn the lights on before you start fixing anything

With the accounts listed and the data backed up, stage three is seeing what you’ve got.

What do you switch on first, in every account?

CloudTrail, GuardDuty and IAM Access Analyzer, all reporting into the security account, plus an IAM role in every account that lets the security team look around.

Configure CloudTrail from the master account as an organization trail: new accounts are covered the day they’re created, and a child account can’t switch it off. GuardDuty reads your CloudTrail events, VPC flow logs and Route 53 DNS query logs. It raises findings from threat intelligence and anomaly detection: traffic to a destination on its threat lists, behaviour its detectors are built for. A reverse shell calling a known command and control host surfaces without you writing a rule; an unremarkable one may not. Point the findings at a Lambda function posting to Slack and someone sees them that day.

Access Analyzer reports resources reachable from outside its zone of trust, and you pick that zone when you create the analyzer: one account, or the whole organization. Pick the organization and access between your own member accounts stops producing findings. That is the point and the limit: keep an account-scoped analyzer where you care who inside the company can reach what. Analyzers belong to a Region, so create one in every Region you use. The paper notes it’s free where GuardDuty isn’t, and both go on across an Organization from the master account through delegated admin.

Run a scanner once, to find out how much work is in front of you. Prowler is a pile of AWS CLI calls that checks an account against the CIS benchmarks plus additional security checks of its own. CloudMapper and ScoutSuite do the same job. Security Hub bundles AWS Config rules that cover similar ground, and as far as we can tell Prowler checks more. Nobody has to write software to crawl every API in every account. On Google Cloud, Forseti Security scans them on a schedule and evaluates rules against the results.

One more switch, the cheapest here: S3 Block Public Access, which AWS shipped in November 2018. Its settings override the ACLs and bucket policies underneath them.

This is the stage teams skip. The application works, the client is happy, you sit back with a beer, and nobody ever deploys the sensors.

The alerts you’re missing are in the data events

Seeing is stage three. Being told is stage four.

What’s the minimum detection worth having?

The paper names three: a GuardDuty finding from any account reaches a human, AccessDenied in CloudTrail can raise an alert, and you can search CloudTrail by principal and time window.

Access denied is the one people miss, and catching it on object reads costs money. CloudTrail records management events, the control plane, and separately records data events for calls like s3:GetObject and Lambda invokes. Data events are high volume: the first copy of management events in each region is free, data events run $0.10 per 100,000, and the bill follows.

Some of that traffic is name guessing. Some is our own users reaching into one account with another account’s credentials. What’s left is worth chasing: a request for an object nobody outside should know about might mean the name leaked somewhere else. A denied read won’t settle it. S3 answers 403 Access Denied rather than 404 Not Found for an object that isn’t there when the caller can’t list the bucket. The denial tells you someone asked, and nothing about whether the key exists. The control plane carries the same tell. A trickle of access denied is normal, and a ramp is a warning that somebody is working through a list.

Volume is why this has to be automated. You could once read the day’s logs end to end and sign them off; a cluster produces a thousand events while you do that. Aggregate first, dig second. Where the logs already sit in a warehouse, BigQuery ML fits a k-means model in SQL. You pick the features, connection counts and origins among them. The rows furthest from any cluster centre are worth a look, and a human should check them before the machine acts.

Our own gap is data exfiltration. We go as far as the cloud offerings take us. Below that line there’s plenty around: Falco for unwanted behaviour in a cluster, host intrusion detection and network sniffers on VMs. Our game there needs stepping up.

Blocking is harder than detecting, and that’s where we split

Detection tells you the bad thing happened. The next question is whether you could have made it impossible.

Should you stop a developer from opening port 22, or just find out that they did?

Mattias would detect it. Stopping an IAM user from launching an EC2 instance with 22 open to the world is hard, and standing up a detection that pings Slack is easy. Andrey would remove the option: let people launch instances, don’t let them create security groups, and give them a set of predefined ones to attach.

Policy as code is where this gets interesting, because it checks the change before the change exists. Sentinel does it inside the Terraform workflow, and Pulumi ships policy checks alongside what you deploy. Open Policy Agent will do it for a Terraform plan. Generate the plan, convert it to JSON, run opa eval against a Rego policy, and fail the pipeline when the plan holds something that shouldn’t be there. OPA is a general-purpose policy engine. Its creators built it to unify authorization across a wide range of systems. Kubernetes admission control is one integration among several, with Terraform plans and CI pipelines on the same list.

Where neither fits, automated remediation closes the loop: an AWS Config rule notices the open port and a remediation action closes it with nobody in the middle. Forseti’s enforcer does the same on Google Cloud. That route is the slowest, because the bad state exists for as long as it takes to notice it.

Stop issuing keys that never expire

Stage five is about people and the credentials they carry, and it opens with the bluntest instruction in the paper: remove all IAM users.

What replaces an IAM user?

A role assumed through SSO, so the credentials anyone holds expire on their own and removing a person is one change in one place.

IAM user access keys never expire. They end up in public repositories, and once the CLI is configured they sit in plain text in ~/.aws/credentials on a laptop that travels. GitHub has scanned pushes to public repositories for known credential formats since October 2018, and it sends each match to the provider that issued it. What happens next is the provider’s call: validate it, revoke it, mail the owner. We’ve watched a key go up by accident and the mail arrive within the hour. Replace the exposed key yourself rather than wait to learn which it was. Treat the scan as a safety net with holes: it covers public repositories and the formats partners registered.

AWS SSO is not fun to set up, and we’d still do it. If you already run HashiCorp Vault there’s a shorter path. Its AWS secrets engine issues credentials per request. One path creates an IAM user and deletes it when the lease ends; the other assumes a role and returns STS credentials that expire on their own. Take the second if you want the console too. STS credentials buy a sign-in URL at the AWS federation endpoint, which drops you into the console as the same short-lived identity. The IAM user path needs one more step, a GetFederationToken call with those keys, because the federation endpoint takes temporary credentials and nothing else. Either way, what gets clicked traces back to the same lease.

For the keys you can’t kill yet, the paper points at aws-vault, which keeps them in the OS keychain and hands a single command temporary credentials instead. envchain does the same trick for environment variables. Add pre-commit hooks to catch a key before it gets committed, with detect-secrets as the paper’s suggestion. The hook runs on the laptop and git commit --no-verify skips it, so it catches accidents rather than guarding the repository. The ambition is what cert-manager did for TLS: certificates used to be something you thought about, and now they renew themselves.

Shrink what’s reachable

Stage six takes options away from the whole organization at once.

What do service control policies actually buy you?

They cap what principals in your member accounts can do, the member-account root user included. Whole categories of action become impossible: the Regions you don’t operate in, deleting the security team’s audit role. Summit Route’s SCP examples are a starting set. An SCP doesn’t touch the organization’s master account, which is a reason to keep nothing in it, and it has no effect on principals outside your organization.

Here’s the second place we split. Mattias doesn’t want security standing between a developer and a Region. Someone with users in Europe who wants a bucket next to them shouldn’t need a ticket and a week of waiting. Andrey’s version is narrower than it sounds. An SCP is for the Regions and services nobody is using, and the day a team wants one, you turn it back on. Regions launched since March 2019 are opt-in, and the older ones are on by default. Nobody watches the default-on Regions you never open, which is exactly where somebody else would rather work. Protecting CloudTrail from being switched off is the case neither of us argues with.

Next, stop putting things directly on the internet. A load balancer in front of EC2, or CloudFront in front of a bucket, gives you somewhere to hang the controls. Shield Standard covers Elastic Load Balancing, CloudFront and Route 53 automatically, at no extra cost. The other two you configure: WAF needs a web ACL you associate with the load balancer, and access logs are off by default until you name a bucket.

The metadata service is the last piece, and it answers incident number three. Credentials for an instance role arrive through IMDS, and SSRF is a way of asking for them on the attacker’s behalf. IMDSv2 requires a PUT to open a session, then the returned token on every request after it. AWS calls that “always strictly more effective than requiring only a static header”, and an SSRF that can trigger only a GET comes back empty handed. Switching IMDS off entirely is stronger, though user data and the instance’s SSH public key come through the same endpoint. An instance launched without it never runs user data and has no key to fetch. Work out what reads 169.254.169.254 before you take it away. Migrate the fleet to IMDSv2, then enforce it with an SCP.

The last five stages

Stages seven through eleven each assume the stages before them are genuinely done.

What’s left after the first six stages?

Reproducibility, sharper detection, auto-remediation, network control and incident practice, in that order.

Stage seven is infrastructure as code, control over where your AMIs and packages come from, and a tagging strategy you can enforce with tag policies. Tagging sounds like paperwork until something is wired to it. We delete untagged resources after two days, which cleans up whatever got made by hand in the console and stops us paying for it. An intruder who does enough reconnaissance will tag their own resources to blend in, so it’s a tidiness control with a security side effect.

Stage eight adds honey tokens: a fake credential or a document nobody has a legitimate reason to open. Put it where an intruder would find it, and wire it to alert the moment it’s used. Stage nine is auto-remediation and tighter IAM policies, where the Config rules above earn their keep. Private subnets and restricted egress arrive at stage ten, because an open egress path is how a compromised instance reaches a command and control server for its next instruction. Incident preparation closes the list: limit the blast radius, then practice. What counts as a reportable breach has a legal definition attached, and the night you need that definition isn’t the night to go reading up on it.

What this means for teams

The paper’s real contribution is the order. An alert from an account nobody can enumerate has nowhere to land, which is why the inventory comes first. It sets the scope every later control applies to, from Block Public Access at stage three to the service control policies at stage six.

This week:

• Write the account list: name, 12-digit id, point of contact, payer. Ask finance what they’re paying for, and search the mail archive for signup confirmations.
• Turn on an organization trail from the master account, with GuardDuty and Access Analyzer reporting into one security account, then check that a GuardDuty finding actually reaches a person.
• Run Prowler or ScoutSuite once against your worst account. Fix the public buckets with real data in them, the public Elasticsearch and anything else exposed, and leave the rest of the report for later.
• Turn on S3 Block Public Access wherever no bucket needs to be public, and pick the date you’ll delete your last IAM user.

Common questions, answered

How do I find the AWS accounts my company forgot about?

Ask finance which accounts show up on a bill, then ask each team what they created, and search the company mail archive for AWS signup confirmations. Record each one with its name, 12-digit account id, point of contact and payer. Invite them into a single AWS Organization, so the next account arrives with your baseline already on it. An account missing from that list is also missing from your logging, your scanning and your guardrails.

How do I tell whether someone is probing my S3 buckets?

Turn on CloudTrail data events for the buckets that matter and alert on denied s3:GetObject calls, because the failed reads are the signal and management events don’t carry them. Data events cost $0.10 per 100,000, so start with the buckets holding something worth taking. A denied read won’t tell you whether the key exists. S3 returns 403 Access Denied for a missing object when the caller can’t list the bucket. Treat a run of denials as a reason to check what you might have disclosed elsewhere.

How do I stop an EC2 instance’s IAM role from being stolen through SSRF?

Require IMDSv2 on your instances, which makes software open a session with a PUT and present the returned token on every later request. An SSRF limited to GET requests comes back with nothing. Move the fleet over incrementally, enforce the result with a service control policy, and narrow the role while you’re there, since the role is what an attacker walks away with. Turning the metadata service off is an option for an instance that reads nothing from it, and it needs a check first. User data execution and the SSH public key arrive through the same endpoint, so an instance launched without it loses both.

Related episodes

• Infrastructure as Code: stage seven’s requirement, and why a change made in a console is a security problem rather than an ops annoyance.
• What Are We Working On: keys in repositories, short-lived credentials from Vault and who can read a bucket, stage five seen from the other side.
• Terraform in CI: why the account layer, Organizations, IAM and GuardDuty, deserves its own state file and review.

Resources

• AWS Security Maturity Roadmap, May 2020 edition: the paper itself, eleven stages in ten pages, free and without an email wall. This is the archived copy of the PDF as published on 21 May 2020, which is the one this article works through.
• Summit Route’s announcement of the roadmap: Piper’s own post on why he wrote it and who it is for.
• AWS SCP Best Practices: the service control policies to start from, and the accounts and principals they don’t reach.
• Defense in depth: the EC2 instance metadata service: why a session token beats a static header against SSRF, and that IMDS can be disabled outright.
• Amazon S3 Block Public Access: the account and bucket settings that override the ACLs and policies underneath them.
• Creating a trail for an organization: the trail new accounts inherit and child accounts can’t switch off.
• IAM Access Analyzer in AWS Organizations: what a zone of trust covers, and why an analyzer is per Region.
• Vault AWS secrets engine: the two ways Vault hands out AWS credentials, and what it cleans up when the lease ends.

Since we recorded

Piper revised the roadmap, and the edition that download link serves is dated January 2021, with the eleven stages folded into ten. Stage three’s cheapest switch became the default, since new buckets have come with Block Public Access on and ACLs disabled from April 2023. The metadata advice held: an account setting from March 2024 defaults new launches to IMDSv2 per Region, and account-level enforcement fails a launch that doesn’t set httpTokens to required. Resource control policies joined SCPs in November 2024, the subject of episode 72, and of the tools above Forseti Security is now archived.