#66 - Multi-Account Strategy And Landing Zones: Account Segmentation Approaches For Security And Efficiency On AWS
Each AWS account boundary adds isolation, but also work to keep logs, access and networking usable.
18 min read
In this episode of DevSecOps Talks, co-hosts Andrey, Julien, and Mattias are joined by AWS Consultant Fernando Gonçalves to explore the complexities of AWS organization and account segmentation. Get insights into the debate over development, stage, and production accounts versus micro-segmentation. Don’t miss Julien’s take on why he believes staging is a waste of time and money, as well as Fernando’s explanation of what the AWS Landing Zone is. Learn about the tools provided by AWS for multi-account management and the pros and cons of various account segmentation approaches.
Everything in one AWS account, until it isn’t
Most AWS environments start as one account. Dev and production sit in the same console, the CI role can reach both, and the only thing between a wrong terraform apply and your customers is whoever skimmed the plan. That setup works right up to the day it doesn’t, and that day is rarely a quiet one.
The opposite mistake is quieter. Split five applications into five accounts and your developers still need one place to read the telemetry and correlate a request across all of them. Slice everything up without doing the work that makes the slices usable, and what you get is a drop in productivity rather than a gain in safety.
So the useful question isn’t how many accounts you need. It’s which boundaries buy you something, and what each boundary obliges you to build afterwards. AWS consultant Fernando Gonçalves, who teaches AWS in Portuguese on his channel Cloud na Prática, worked through that order of operations with us, and it starts smaller than most people expect.
Split when it hurts, not before
The first split is almost always the same one, and it’s worth doing before you are certain you need it.
When should you create a second AWS account?
When two things in one account must not be able to touch each other, and production against everything else is the first pair that qualifies.
Call that the vertical cut: dev, something in the middle, production, with no path from one to the next. The horizontal cut comes later, inside a single environment, when you pull the front end, the back end and analytics into accounts of their own. Where the lines go is the part people put their thinking into. The expensive part is the friction each line adds for the people working inside it, and the work you then do to take that friction back out.
The containers are simple enough. An organization holds your accounts, organizational units are the folders you sort them into, and AWS lets you nest those five levels deep under the root. Depth is not a score, and AWS’s own whitepaper on organizing your environment is the shortest route to one that has worked for other people.
Staging is the environment nobody agrees on
The middle environment is where the vertical cut stops being obvious.
What is the environment between dev and production for?
It depends on who actually tests in it, and that question is worth asking before you pay for the account.
Dev and prod are called dev and prod everywhere. The thing in between gets called staging, test, QA, integration or pre-prod, and the name tends to track how much anyone believes in it. Julien won’t pay for it at all.
What he’d put in its place is named by audience: an ops test environment, where operations run their tests, and dev, where developers run theirs, because developers don’t trust the ops one. Two audiences, two environments, and a shared middle environment has to say which of them it serves.
In a multi-account setup that question comes with a price tag. Each environment you keep is at least one more account, with its own configuration recording and its own share of the detection bill.
The two accounts that pay for themselves
Past environments, the next accounts worth creating aren’t applications at all.
Which account should you create after dev and prod?
A log archive account, then a networking account. Both exist so that something stops being everybody’s problem.
The log archive account holds CloudTrail from every account in the organization, application logs if you want them there, and day-to-day access for almost nobody. That restriction is the feature: it’s the copy the audited can’t rewrite, and it’s where you point Datadog or whatever you run. Aggregation is the work the boundary creates. An organization trail is the least of it: one trail in the management account, covering every member account and the ones created later. Anything written per account and per region stays where it landed until you route it somewhere.
The networking account is the one AWS’s own guidance pushes you towards. The whitepaper puts shared networking in an Infrastructure OU, in accounts holding no application workloads, with the VPCs managed there and shared out, rather than every workload account running its own VPC and the mesh of peerings that follows. The money argues the same way: NAT gateways are billed per hour and per gigabyte processed, per gateway, so a VPC per account multiplies a fixed cost that one shared environment pays once per zone.
That also settles an argument you have probably had. Developers want admin in their account; security doesn’t want anyone moving routes. Shared subnets give you both, because participants can’t create, delete or associate route tables and can’t modify the subnets, while they do manage their own security groups and can write rules referencing groups owned by other accounts. Admin in your own account simply stops at resources your account doesn’t own.
Guardrails run downhill
Shared accounts decide where things live. Policy decides what anyone can do once they’re there.
What stops a developer with admin rights in their own AWS account?
A service control policy attached above that account, because an SCP caps what the IAM users and roles in that account can do, its root user included.
That last part is the one people miss. Two constraints have to hold at once. An SCP has to allow the action at every level from the organization root down through the account. Inside the account, an IAM user or role also needs an allow, from its own identity-based policy or the resource’s. AWS checks explicit denies across every applicable policy, and applies the SCP restrictions before any allow inside the account, so one explicit deny anywhere settles the matter. Deny organizations:LeaveOrganization above production and the root user of that account can’t walk it out of your organization.
Know which half of the problem that solves. An SCP binds the principals in the accounts it covers, so it does nothing about an outside caller your own resource policy lets in. The resource owner holds that half. An Allow conditioned on aws:PrincipalOrgID restricts that grant alone, and an ACL or another policy can still let an outsider in, so AWS recommends the same condition in a Deny instead. The deny also shuts out the AWS services that write to your bucket, CloudTrail log delivery included, so make the exception with aws:PrincipalIsAWSService. Service-linked roles are exempt from SCPs entirely, and SCPs don’t apply to the management account, one more reason to run nothing in it.
Google Cloud reached the same hierarchy from the other end. An organization holds folders, folders hold projects, and an allow policy on a folder is inherited by everything beneath it. A child can’t remove the binding it inherited, which is where the habit of granting narrowly comes from: at the project, or at the single database, rather than at the folder. IAM deny policies, generally available since November 2022, attach to an organization, folder or project. Google checks them before any allow policy, so a deny on the child blocks a permission the parent handed down.
The message does better than it used to: AWS started naming the policy type in access denied errors in 2021, so a supported error can say an SCP turned you away rather than the role’s own policy. Which policy, on which OU, is still yours to find.
Control Tower does the boring part
Knowing which accounts you want is one job. Creating them, repeatedly and identically, is another.
What does a landing zone actually give you?
A baseline that every new account lands in: logging, configuration recording, controls and access, set up the same way without anyone doing it by hand.
Before Control Tower, you built that baseline yourself. We did, back when Organizations arrived in 2017, and we provisioned every new account by hand. Control Tower is AWS’s managed version. Enable it from an account that does nothing else and it creates a log archive account and an audit account. CloudTrail logs land in the first. The second holds the Config aggregator and the access your security people work through. Account Factory then vends accounts into the OU you pick with the controls already applied.
Be precise about what arrives with it: the audit account’s name promises more than the service delivers. Control Tower enables AWS Config in enrolled accounts and aggregates it there. GuardDuty and Security Hub are still yours to turn on: delegate administration of each to the audit account, then let the service cover the organization itself with GuardDuty’s auto-enable preference and Security Hub’s central configuration.
Access is the part that quietly changes your week. Instead of IAM users in every account and policies deployed account by account, you define a permission set once and assign it to an account together with a group. That group can come from the external identity provider you already run. Everyone gets the same portal, every credential is temporary, and nobody chains role switches to reach production.
The cost is that Control Tower holds opinions, and you get few options against them.
Concretely: it puts account IDs in bucket names, which we would not, and the CloudTrail setup it deploys has not always matched what Security Hub checks expect, while its own SCPs stop you editing what it created. The escape hatches have arrived about one a year: since June 2023 you can self-manage account access rather than let it configure Identity Center, and organization trails are optional if you would rather own CloudTrail yourself.
What Control Tower won’t decide for you
Control Tower gives you a floor. The strategy standing on it is still yours to write.
What’s left to build once Control Tower is running?
Your account list, your OU structure, and everything a new account needs that isn’t a security service.
That last category is longer than it looks. A state bucket in each account if you run Terraform. An access log bucket replicating into the archive account, the subnet share from the networking account, budget alarms, a tagging convention somebody enforces. Control Tower invents none of it, and the two accounts it creates start your account list rather than finish it.
Managed services also come with knobs they don’t expose. AWS Config bills per configuration item recorded, and a busy account keeps producing them: every ECS task definition revision, every service update, every elastic network interface that comes and goes underneath. Periodic recording, released in November 2023, writes at most one item per resource per day instead of one per change, and can be set per resource type rather than for the whole recorder. Settle what audit detail you can lose first: daily recording keeps the state at the end of each window, drops the changes inside it, and can miss a resource created and deleted between recordings. Daily items also cost more each than continuous ones, so count the items by type too. As far as we could tell you can’t choose periodic recording from Control Tower at all. That’s the trade with a managed service: take the default, and keep a backlog item for the day the setting appears.
Which leaves buy or build. Consultancies sell landing zones on AWS Marketplace, Andrey’s company FivexL among them, so weigh what follows accordingly. In most cases one of these products costs less than a month of the consultant you’d otherwise hire to build one. A good chunk of that month goes to the laptop and the access requests before any work starts. How well it holds depends on how much of your list the product covers, so compare the quote against the scope. A product company does this once and never repeats the knowledge. If you set up a new organization every quarter, build your own.
Nobody knows where anything is deployed
There’s a tax on multi-account that shows up months after the structure is right.
How do you find which account a service runs in?
Query the AWS Config aggregator instead of crawling accounts yourself, then check what the aggregator doesn’t cover.
Developers hit this constantly: my service talks to that thing over HTTP, I need to change its configuration, and I have no idea which account it’s in. Control Tower turns Config on in the accounts it enrolls and aggregates their recordings into the audit account, so much of that inventory already exists, and the part people miss is that it’s queryable. Advanced queries take SQL-like syntax over recorded resources, much the way Steampipe queries a provider, so the Lambda functions and ECS services Config recorded are one query against one account’s API. AWS put a natural-language query generator into preview at re:Invent 2023 that writes the SQL for you.
What comes back is what Config recorded, which is not the same as what you run. An aggregator is a read-only view of the source accounts and regions it is authorized to see, and of the accounts in the organization that have Config enabled. Accounts you had before Control Tower are the first gap. Control Tower hasn’t configured recording in an account until you enroll it. That isn’t the same as nothing being recorded there, because plenty already run a recorder of their own. AWS publishes a procedure for enrolling accounts that have existing Config resources. Check the two separately: whether a configuration recorder runs in the account, and whether the aggregator reads that account as a source. Both determine coverage: the recorder must capture the resource, and the aggregator must include its account and region.
Policy coverage is wider than either: preventive controls on a registered OU reach enrolled and unenrolled accounts alike. An SCP landing on an account tells you nothing about what Config records there. The other gaps: a region where nobody turned the recorder on, a resource type the recorder skips, and any type you moved to daily recording, where a short-lived resource comes and goes between writes. None of them arrives in the result as an absence. Compare the accounts and regions you deploy into against the aggregator’s sources, and publish the gap list beside it.
One scheduled export a day, a JSON file and a static page on the other end beats the crawler you were about to write.
Accounts you throw away, accounts you move
None of this structure is permanent, and the shift worth making is to stop treating accounts as precious.
Can you still fix your account structure after you’ve shipped?
Yes, and most teams do it at least once. Accounts move between organizations, and an organization can be replaced under them.
Start with the cheap version, which AWS builds into its recommended structure: a Sandbox OU of accounts where builders explore, typically disconnected from your internal networks and internal services. A throwaway account keeps an experiment away from everything else, and when the experiment is over you close the account instead of hunting down what it left running. Closing is not instant deletion. A closed member account stays in the organization for a 90-day post-closure period, still counts against your account quota, and can be reopened in that window. After it, AWS permanently closes the account and deletes its content and resources, CloudTrail trails excepted.
The expensive version is the one you grow into. A Redshift cluster sits next to the production Kubernetes cluster because there was nowhere else to put it. Then the data scientists and operations people need access to that account. Had an analytics account existed as a placeholder, it would have landed there.
The mechanics are friendlier than the reputation. An account can move to another organization. Accounts that grew up separately can be invited into one organization and enrolled in Control Tower afterwards. That is often how a clean-up starts: new organization, old accounts invited in, workloads migrated out gradually. Do it one account at a time. An SCP above the new home starts applying the moment the account lands there, so something that worked yesterday can fail today.
When the provider deletes your subscription
One more reason to know where your things live, from the weeks before we recorded.
In May 2024 the Australian pension fund UniSuper lost its Google Cloud environment. Google’s account of the incident says an operator left a parameter blank while provisioning the fund’s Google Cloud VMware Engine private cloud. The blank gave the subscription a fixed one-year term. When the term ran out, that private cloud was deleted across both zones it ran in. The deletion didn’t reach any other private cloud. Backups the fund held in Google Cloud Storage were unaffected and went into the restoration, which ran over several days.
No customer-side IAM policy or SCP could have prevented this, and neither would a better contact list: Google’s report says no notification was sent to the customer, because the deletion followed automatically from the blank parameter.
Contact hygiene is worth doing on its own merits, not as a fix for this. Nothing propagates contacts for you: alternate contacts are per account, and the management account or a delegated administrator sets them for member accounts through the Account Management API, so your landing-zone automation is what has to do it. Make each one a mail group, so the ordinary notices outlive whoever created the account.
What this means for teams
Where your company sits decides how much of this to do now. Before product-market fit the project’s longevity is the open question, and a single account with everything in it is a defensible choice; we’ve seen plenty of MVPs run on one Lightsail instance or one EC2 box, and then disappear. Once something repeatable is earning money and a round has closed, the calculus flips: you now have something worth protecting from the script kiddies, if not yet from anyone targeting you specifically. Keep the MVP account running, create a fresh account, start an organization there, invite the old account in, and migrate workloads out of it as you go. If you’re past the seed rounds with a hundred or two hundred developers sharing one account, the job is bigger and the excuse is thinner.
Three things that fit in a week:
- Check whether your accounts are in an organization at all. If they aren’t, create one from a clean account and invite the rest in.
- Stand up a log archive account and send CloudTrail there before you split anything else.
- Move your people onto Identity Center permission sets, production first, then delete the IAM users they used to sign in with.
Common questions, answered
How do I give a team admin in their own AWS account without letting them change the network?
Put the VPC in a networking account, share its subnets with the team’s account, and give them admin in that account. Participants can launch workloads into a shared subnet and manage their own security groups, but they can’t create, delete or associate route tables, and can’t modify the subnets. Routing, NAT and network ACLs stay with the owning account, so those workloads use the shared environment’s NAT gateways instead of each account paying for its own.
Do I need AWS Control Tower if I only have two AWS accounts?
Create the organization now, and you can enable Control Tower later without starting over. An organization on its own gives you consolidated billing, cost split per account, SCPs and central contacts, and it costs nothing to have. Two accounts become six sooner than you expect, and retrofitting governance onto a running production account is where the SCP surprises live.
How do I stop AWS Config from getting expensive in a busy account?
Count the configuration items by resource type first, because the answer depends on which type is generating them. Daily recording writes at most one item per resource per day and can be set per resource type, so it helps where one resource changes many times a day. It costs detail: changes inside the window aren’t recorded, and a short-lived resource can be missed, so move a type only where you can lose that history. The per-item rate is higher than continuous recording, so switch the types your counts and your audit needs both justify. In a Control Tower landing zone, Config is part of what the service deploys, so check what you’re allowed to change before planning around it.
Related episodes
- #16 Do You Need a Staging Environment?: the environment-count argument in full, from before accounts put a line item on each one.
- #50 History of AWS networking and new ways to design your VPC setup: shared subnets, transit gateways, and the peering mesh they replace.
- #11 AWS Security Maturity Roadmap: the baseline to put in every account, including what forgotten accounts and long-lived keys cost you.
Resources
- Organizing Your AWS Environment Using Multiple Accounts: AWS’s own account and OU patterns, including where shared networking and sandbox accounts belong.
- Determining whether a request is allowed or denied within an account: where SCP restrictions sit against identity policies, resource policies and permissions boundaries in one evaluation.
- Bucket policy examples: why
aws:PrincipalOrgIDon anAllowdoes not keep outsiders out, and theDenyform that does. - Responsibilities and permissions for owners and participants: exactly who may touch subnets, route tables and security groups in a shared VPC.
- What are the shared accounts?: what Control Tower creates beside your management account and what it keeps there.
- Automate enrollment of accounts with existing AWS Config resources into AWS Control Tower: what happens to a Config recorder an account already runs when you enroll it.
- Sharing details on a recent incident: Google’s account of how a blank parameter deleted a customer’s private cloud, and what it says about notification.
- Fernando Gonçalves on LinkedIn and his Portuguese-language AWS channel Cloud na Prática.
Since we recorded
AWS added a second policy type above the account: resource control policies, launched on 13 November 2024. They cap what a resource’s own policy can grant, the way an SCP caps what an identity policy can grant. We took them apart in #72 AWS Resource Control Policies (RCPs). The question of how much of this a young company should take on came back in #67 Building MVP On AWS.
