Most AWS networks stop evolving the day they work

Sign up for AWS and you get a default VPC: a couple of public subnets, an internet gateway, a default security group. Launch an instance, it has internet, nothing to think about. Anyone who knows what they’re after throws that away and builds their own VPC, private subnets behind a NAT gateway and public subnets in front. That is usually where the design stops. One VPC per account, one account each for dev, stage and prod, a peering connection wherever two of them have to talk.

It holds for a year or two, and then the same three things show up. You want a second region for disaster recovery, so the VPCs need to reach each other. The data team needs the production database, and because nobody planned a place for them they land in the production VPC next to the backend. The bill grows a NAT gateway line nobody can explain. We work with scale-ups and we see this picture over and over again. The fix people reach for is another peering connection, every time.

There’s a layer above that most teams never look at, and AWS shipped it in 2018. One account owns the VPC, the subnets and the routing, and every other account launches workloads into those subnets. The common name for it is VPC sharing, which undersells it: what you share is subnets, and what does the sharing is AWS Resource Access Manager. Ownership of the network lands in one account, IAM decides who may administer what, security groups and network ACLs still decide what traffic flows, and one set of NAT gateways serves everybody.

From one big shared network to your own

Some of today’s defaults are decisions and some are leftovers, and the history tells you which is which.

What did a VPC give us that EC2-Classic didn’t?

Your own address space and your own routing, instead of a virtual machine with a public IP sitting in one large network shared with everybody else.

Early AWS sold what DigitalOcean and Linode sold: a VM, a public IP, and that was the product. You couldn’t see the other machines and you couldn’t shape the network around yours. AWS announced Amazon VPC on 26 August 2009 as a limited beta and opened it to every EC2 account that December. If the date in your head is 2013 instead, that’s the changeover you lived through: accounts created after 4 December 2013 are VPC-only, as AWS put it when it announced the EC2-Classic retirement in July 2021. That retirement is still running its course.

What you got in exchange maps onto the data center most of us came from. A virtual network with subnets you can split into public and private, an internet gateway for traffic in, a NAT gateway so private instances can reach out without being reachable, route tables and network ACLs. That’s a firewall, a DMZ and a database segment by another name, which is why people who have racked switches feel at home in a VPC within an afternoon.

The second VPC is where the thinking actually starts, and most of the time nobody does it until the second VPC is already there.

Peering is the path of least resistance, and it has a ceiling

The first reason for a second VPC is usually resilience. AWS is regional, for good and for bad, so a second region means a second network, and you want somewhere to send traffic when the first region starts to suffer. Whether those two networks also need to reach each other depends on how self-sufficient each one is. If the regions share nothing, leave them apart. Most don’t: databases replicate, services sync to one another, Kubernetes clusters stay connected, and each of those dependencies wants a path.

When does VPC peering stop being enough?

When the connections and route entries you have to maintain outgrow the patience of whoever maintains them, because peering is point to point.

VPC peering connects one VPC to another, works across regions, and takes about ten minutes. Then two constraints land on you. The CIDR block of each VPC has to not overlap with the one it’s peered to, so routing stays possible. And peering carries no transit: AWS is blunt that “transitive peering relationships are not supported”. Three VPCs peered to each other is a documented, supported setup, but if A is peered to B and B to C, A still cannot reach C through B. You either put an endpoint in B and hop, or you create the A-to-C connection yourself. Four VPCs that all need each other is six connections, each with route entries at both ends, and the tables stop being readable long before anything technical stops you.

There are quotas waiting at the end of that road too. The default is 50 active peering connections per VPC, adjustable up to a hard maximum of 125.

One VPC per account does buy you something real, so don’t give it up by accident. Blast radius is small. You can go into one VPC and experiment with a network ACL that blocks a traffic type, and if you get it wrong you’ve broken one account’s network and nobody else’s. That sandbox is worth keeping in mind when you start consolidating.

A managed transit gateway has limits of its own

The standard answer to the mesh is a transit gateway: one hub, attachments instead of n-squared connections, transitive routing that peering won’t give you. It solves the topology. It also hands you a managed service, and managed services have limits.

Does a transit gateway take the scaling problem off your plate?

It takes the routing problem off your plate and leaves you a bandwidth ceiling you don’t control, documented per VPC attachment per Availability Zone and not listed as adjustable the way the peering quota is.

That documented number is one risk. How fast the device grows towards it is a different one, and that part has been tested in public. Slack had a bad morning on 4 January 2021, and its postmortem names the mechanism. The trigger was the return from the holidays rather than a failover: “On the first Monday back, client caches are cold and clients pull down more data than usual.” Network problems plus the 7:00am PST traffic peak saturated the web tier, and Slack became unavailable. By around 9:15am PST it was “degraded, not down”. The network returned to normal at 10:40am PST, once AWS engineers had added gateway capacity by hand. Slack’s verdict on the device: “our TGWs did not scale fast enough.” The postmortem does not attribute the network failure to a transit gateway bandwidth quota; what it reports there is capacity that arrived too late. It does name an AWS quota, but in its provisioning service rather than in the network: “we also exceeded an AWS quota limit.” There’s a second lesson in the same text, about topology rather than throughput: “our dashboard and alerting services failed because they were running in a different VPC from their backend databases.”

At real scale people do make the move. Grammarly wrote up its multi-region rebuild: peering had become unmanageable across hundreds of accounts and VPCs, so they replaced it with a full mesh of transit gateways, sharded their IP space into small, medium and large blocks per region, and stood up a central networking account per environment. That last piece is the part worth stealing, and it doesn’t require the first two.

One VPC per account is also a NAT bill

Before the alternative, look at what the usual layout costs when nothing goes wrong at all.

Why does the NAT line grow faster than your traffic?

Because you pay per gateway, per hour, per Availability Zone, in every VPC, before a single byte moves.

In us-east-1 a NAT gateway is $0.045 per hour plus $0.045 per GB processed. You should be running at least two Availability Zones, so that’s two gateways per VPC, roughly $66 a month per VPC in standing charges. Now count your VPCs honestly. Vertically you have dev, stage and prod. Horizontally you have backend, and data engineering, and whatever analytics turned into. Eight VPCs with two AZs each is about $525 a month in gateway hours, and the data processing charge sits on top of that.

You can be cheap about it and run a NAT instance on EC2 instead, a box in a public subnet routing traffic out for the private ones. It’s much less money and it’s yours to patch, size and keep alive. The managed gateway is the expensive option precisely because it’s the one you don’t have to think about. Either way, the charge repeats per VPC, which means the cost follows your account layout rather than your traffic.

VPC sharing puts the whole network in one account

So stop putting a VPC in every account. Build the network in one account and share the subnets out.

What are you actually sharing when you “share a VPC”?

Subnets, through AWS Resource Access Manager, from the account that owns the VPC to the accounts that run workloads in it.

If you’ve ever written an S3 bucket policy, RAM will feel familiar. Some AWS resources carry a policy saying which account may read them and which may write: buckets, ECR repositories, SQS queues. A VPC doesn’t take that kind of resource policy, so RAM does the job from the outside. Your networking account creates the VPC, the subnets and the routing, then shares specific subnets with specific accounts. In the participant account those subnets simply appear, owned by another account id, and you launch into them. AWS described the model when it launched: the owner keeps central control of structure, routing and IP allocation, participants manage their own resources inside.

The control story is what makes it safe to consolidate, and it is read-only rather than blind. A participant can describe the shared VPC, its subnets, its route tables and its network ACLs, and cannot create, modify, delete or replace any of them. That is the split you want: a team debugging its own connectivity needs to read the routes it sits on. What a participant does not get is other accounts’ resources. It cannot attach, detach or modify a network interface it doesn’t own, and it cannot even describe flow logs for one.

The money follows. With one VPC behind many accounts you reuse one set of NAT gateways and VPC interface endpoints, and AWS says as much: “costs can be optimized through reuse of NAT gateways, VPC interface endpoints, and intra-Availability Zone traffic.” The same consolidation gives you one place where egress leaves, which is where a traffic inspection appliance goes if you need one, and the AWS Security Reference Architecture builds its network account on that idea.

RAM keeps growing past subnets: Lake Formation shares Glue Data Catalog databases and tables across accounts through it, and the list of shareable types grows every year.

Google Cloud got to the central-network idea earlier: Shared VPC, written up in June 2017, puts the networks in a host project and attaches service projects to them, so a central group owns routes, firewalls and subnet ranges while each team keeps its own project and IAM.

What sharing subnets costs you

None of that is free, and the costs land in places the diagram doesn’t show.

What breaks first when ten accounts share one subnet?

Your IP plan.

One VPC now has many more tenants than a VPC per account ever had, so size the subnets for all of them at once. If one of them is EKS, size them again: the VPC CNI gives pods addresses from your subnets, so one busy cluster eats a range that looked generous on the whiteboard. Running out of addresses doesn’t stop what’s already running. It stops what comes next, for every account in that subnet and not just the one that drained it. New launches fail, scale-outs stall, replacements don’t come back, right down to AWS’s own error text: “Subnet has insufficient free addresses to create this NAT gateway”.

Four more edges matter here. Sharing works only between accounts in the same AWS Organization, so you can’t hand a subnet to a customer or a vendor; a private path across a company boundary is still PrivateLink for one service, or peering or a Site-to-Site VPN for networks. Default VPCs and subnets can’t be shared, so this is a deliberate build. Cross-region is still your problem: a VPC lives in one region, so peering or a transit gateway comes back the moment you need two. And a share can be taken back, in one direction only. Unshare a subnet and “the participant will no longer be able to launch any new resources into shared subnets”. What is already there keeps running, because a VPC owner “cannot delete, modify or forcefully eject a participant’s resources.”

Be precise about what the IAM boundary buys you, because this is where people oversell it. A peering connection opens the gates wide between two VPCs, and while you can narrow what’s reachable across it, in practice almost nobody does. Shared subnets don’t change that. What they separate is administration: who may create, change and delete which resources, with participants unable to change the VPC-level configuration or work with each other’s resources. The filtering is unchanged. Security groups apply to traffic between two workloads in the same subnet, so the groups on each interface are what decide. Network ACLs do not apply there: they are evaluated at the subnet boundary and never see traffic that stays inside one. A host firewall on the instance can filter the same traffic, and it belongs to whichever account owns that workload.

IPv6 is the fix for the address math, when the software catches up

All of this address arithmetic exists because we’re still rationing IPv4.

Why are we still carving up IPv4 so carefully?

Because the layers above the network haven’t moved, and IPv4 hasn’t hurt enough yet to force them.

Two different scarcities get mixed up here. Public IPv4 is exhausted, with a date on it: the RIPE NCC made its final /22 allocation on 25 November 2019 and has run a waiting list since. The addresses inside your VPC are not those addresses. RFC 1918 reserves 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16 for private reuse, so nothing outside your organisation stops you giving a VPC a /16. What runs a subnet dry is local: ranges carved small years ago, overlaps you dodge for peering, and tenants that eat addresses faster than the whiteboard suggested.

Our read on the blockage higher up: software rather than hardware. AWS supports an IPv6 stack, but the layers above it have to follow, and Kubernetes was late. Dual-stack networking only reached GA in 1.23, in December 2021, and EKS added IPv6 clusters in January 2022, so pod scaling stops eating private IPv4 space. The front door moved first: ALB and NLB got end-to-end IPv6 support on 23 November 2021, so clients still arrive over IPv4 while the targets behind them are IPv6 only.

Your dependencies don’t all have to move with you. NAT64 and DNS64 shipped on 24 November 2021, so IPv6 resources in a VPC can reach IPv4-only services through the NAT gateway. EKS went another way at launch: pod traffic to IPv4 destinations is translated to the node’s VPC address and leaves through a NAT gateway, “while we work to leverage NAT64 capabilities”.

The security model needs a rethink too, and that part has no release note. IPv6 addresses are globally unique and publicly routable, so NAT no longer keeps your instances unreachable. For native IPv6 egress the replacement is the egress-only internet gateway, which allows outbound IPv6 traffic and “prevents the internet from initiating an IPv6 connection with your instances”. The NAT gateway stays in the diagram as the route to whatever IPv4 you still depend on, so you may end up watching two egress paths instead of one. Filtering stays put: AWS shipped IPv6 in VPC with applications secured “in the same easy manner available today through security groups, network ACLs and route tables”. A gateway is the route out, not the firewall.

What this means for teams

If your AWS network is one VPC per account joined by peering, nothing here says tear it up. It says the next VPC you’re about to create is a decision, and there’s a second option that was not available when you made the first one.

The cases worth moving on are the ones where the current layout costs you twice: accounts that need low-friction connectivity and separate ownership, horizontal splits like backend and analytics, and anywhere your NAT standing charge is multiplied by a VPC count rather than by traffic. Vertical isolation between dev, stage and prod shouldn’t change at all.

This week:

• Write down every CIDR you’ve allocated, per account and per region, and find the overlaps before a peering request does.
• Pull the NAT gateway hours out of last month’s bill and count how many of those gateways exist only because each account has its own VPC.
• Pick the one pair of accounts that most wants a flat network between them, and prototype a shared subnet from a networking account. Size it for the combined peak demand of every account that will launch into it, with headroom for replacements and growth.
• Check what your failover path actually crosses, and whether your dashboards sit on the far side of it.

Common questions, answered

How do I stop paying for a NAT gateway in every AWS account?

Put one VPC in a networking account, share its subnets with the workload accounts in one environment and region, and their egress goes through a single set of NAT gateways. In us-east-1 a gateway costs $0.045 per hour plus $0.045 per GB processed, charged per Availability Zone. The standing cost therefore scales with how many VPCs you run, not with how much traffic you send. Collapse three production accounts in one region onto shared subnets, and six gateways become two, with one egress path to inspect. Keep dev, stage and prod on separate networks, and consolidate only across the accounts inside one environment. The savings are real, though central control of routing is worth more over time.

Can I share a VPC subnet with a customer’s AWS account?

Not directly. VPC subnet sharing works only between accounts in the same AWS Organization, so a customer or vendor in their own organization can’t be a participant in your shared subnet. The options for a private path across a company boundary are the older ones: PrivateLink for exposing a single service, or peering and Site-to-Site VPN for connecting whole networks. If the arrangement is a managed service you run inside their environment, the account that runs it has to live in their organization, which is a commercial and governance conversation as much as a networking one.

What happens to my running resources if the VPC owner stops sharing the subnet?

Resources you already launched keep running, and you can’t launch anything new into that subnet. AWS made both halves deliberate: the participant loses the ability to create resources in the subnet, and the owner can’t delete, modify or forcefully eject what the participant is already running there. The asymmetry is quiet until the next scale-out or replacement instance fails, so treat an unshare as a migration agreed with the owning account, not a switch either side flips alone.

Related episodes

• Episode 15, Remote Work Security is the VPN versus zero trust conversation this one refers back to: access to a whole network versus access to a named service.
• Episode 37, Surviving AWS outage, revised for 2021 covers the disaster recovery thinking that sends most teams multi-region in the first place.
• Episode 23, How do we run Kubernetes in the cloud? goes through the cluster networking choices, including the VPC CNI address limits that make subnet sizing a cluster problem.

Resources

• VPC sharing: a new approach to multiple accounts and VPC management, AWS Networking blog settles the owner and participant model, the same-organization constraint, what an unshare does, and the claim that costs can be optimised by reusing NAT gateways and interface endpoints.
• Share your VPC subnets with other accounts, Amazon VPC User Guide settles exactly what participants may and may not do, resource by resource, down to describe-only access to the owner’s subnets, route tables and network ACLs; the network ACL page settles why an ACL is not what separates two workloads inside one subnet.
• AWS Resource Access Manager and its list of shareable resource types settle what else can be shared this way beyond subnets.
• VPC peering basics, Amazon VPC documentation settles that transitive peering is not supported, with the A, B and C example; the quota page settles the 50 default and 125 maximum connections per VPC.
• AWS launches NAT64 and DNS64 capabilities settles that IPv6 resources can reach IPv4-only services through a NAT gateway, so an IPv6 migration does not wait for every dependency.
• Slack’s outage on January 4th 2021, Slack Engineering settles the timeline, the holiday-return traffic spike and what a transit gateway scaling too slowly looks like from inside, in Slack’s own words. It does not attribute the network failure to a transit gateway bandwidth quota.
• Scaling AWS infrastructure to support multiple regions, Grammarly Engineering settles what the end state looks like at hundreds of accounts: central networking accounts, a transit gateway mesh, and sharded IP blocks per region.
• AWS Security Reference Architecture settles where a network account sits in AWS’s own recommended multi-account layout.

Since we recorded

In July 2023 AWS announced a charge of $0.005 per hour for every AWS-provided public IPv4 address, in use or idle, from 1 February 2024. That put a line item on the habit of giving every VPC its own public addressing, and made both IPv6 and shared egress easier to justify to finance. EC2-Classic finished its retirement a few months after we recorded: the last instance was shut down on 15 August 2023. We came back to the account layout side of this in Multi-account strategy and landing zones, with AWS consultant Fernando Gonçalves.