What you stop owning when the cloud runs your control plane

There are two ways to put Kubernetes on AWS or Google Cloud. You build the cluster yourself, with something like kops, and keep the masters on your own instances. Or you buy the control plane from EKS or GKE and keep only the nodes. The cluster your developers talk to looks the same either way. What changes is who gets paged when etcd misbehaves, and who answers the auditor who wants evidence about how the API server is hardened.

Then there’s the part you configure either way. Every cluster needs a CNI. Anything that has to reach your services from outside needs a route in, usually an ingress controller, sometimes a LoadBalancer Service or a NodePort. Any pod that calls a cloud API needs credentials from somewhere. Each of those has a default that works on day one and a limit that shows up months later. A node hits its pod limit and the scheduler has nowhere to put the next one. The TLS passthrough setting you need turns out never to have worked. Every pod on the box shares one IAM role, because that was the only thing wired up when the cluster was built.

Those two questions, who owns the masters and what you configure on the first day, account for most of the operational pain in the year that follows.

Buying a control plane is not about upgrades

Start with the control plane, because it decides how much of the rest is yours.

Is a self-managed cluster harder to keep up to date?

Less than the sales pitch suggests, and this is where we don’t agree with each other. Julien’s reason to rent is bandwidth. Kubernetes has a lot of moving pieces, and when a patch lands or an incident starts, someone has to be free to deal with it. Renting something with an SLA attached has its advantages. Andrey has run kops clusters for years and says upgrades were never the hard part. It does rolling updates of nodes, it installs the CNI, and it carried clusters through the etcd2 to etcd3 migration behind etcd-manager in kOps 1.12. Those release notes are blunt about it: the master upgrade is disruptive, so back up etcd and pick a maintenance window.

Autoscaling isn’t the dividing line either. The cluster autoscaler is a program that runs inside your cluster and talks to the cloud API, so it works much the same whoever built the cluster.

So the answer depends on what you get audited against. If nobody is asking you to produce evidence for the control plane, kops is a reasonable place to keep it, and it will keep upgrading your clusters. If someone is, you’re buying a hardening programme you’d otherwise staff yourself.

Price the managed control plane against your own masters

The regulated case is easy to argue. The cost case gets argued more often, and it is worth doing the arithmetic rather than reaching for a reflex.

What does a managed control plane cost next to running your own masters?

About $73 a month per cluster on EKS, and whether that beats your own masters depends on what you would run instead. AWS charges $0.10 per hour per cluster, half the launch price since January 2020. Google charges the same $0.10 per cluster per hour, with one zonal cluster per billing account exempt: 720 hours a month you subtract from your zonal usage, which covers a dev cluster and nothing regional, because regional clusters pay the fee in full. Price the alternative properly before you call the rest expensive: three master instances so etcd has a quorum, their EBS volumes, the load balancer in front of the API server, and the engineer who rolls them when a CVE lands.

We held off on EKS at first, when a cluster really was expensive. The two vendors reached the same number from opposite ends. EKS started high and came down. GKE started free and introduced a cluster fee later. Marketing-wise, announcing that you’re cutting a price reads better than announcing that you’re adding one, and the second one is the one customers remember.

Pods get real VPC addresses, and a ceiling with them

With the control plane bought, the network is the next decision, and the one with the most consequences.

Should you replace the CNI your provider installs?

Not until you can name the reason, and it is usually network policy or IP addresses. On EKS the AWS VPC CNI is the default, and it maps pods onto ENIs: a pod carries a VPC-routable address a load balancer can reach directly. Pods share the security groups of the node they land on, unless you configure security groups for pods, which needs supported instance types. GKE does the direct path through container-native load balancing, where network endpoint groups point at pod IPs.

The ENI model has a ceiling lower than people expect. The EKS AMI ships the table as eni-max-pods.txt: 17 pods on a t3.medium, 29 on an m5.large. Your DaemonSets spend that budget first, and a log shipper, a metrics agent and a CSI driver take three slots before your workloads land. When it runs out, new pods sit in Pending with Too many pods against every node.

Address exhaustion is a different failure, and the two get confused. The cap stops a pod being scheduled at all; exhaustion bites after a node is picked, when the pod sits in ContainerCreating with a FailedCreatePodSandBox event because ipamd had no secondary address for it. The CNI’s troubleshooting guide sends you to the subnet first. A node keeps handing out the addresses in the pool it already holds; what a drained subnet stops is the top-up, so networking fails for the next pod on a node with nothing spare. Then check the pool size, set by WARM_IP_TARGET and MINIMUM_IP_TARGET. On a node already running 30 pods, WARM_IP_TARGET of 30 keeps 30 free addresses on top of those, 60 in all; MINIMUM_IP_TARGET of 30 with WARM_IP_TARGET of 2 holds 32 on the same node.

You can keep the addresses and move policy elsewhere with CNI chaining, where the VPC CNI hands out addresses and Cilium attaches eBPF programs on top. We wanted network policy from it, and encryption if Cilium could do it. We got neither. The two plugins disagreed over the JSON they pass each other, the container sandbox never came up, and no traffic moved between nodes. EKS doesn’t let you pick the CNI at creation either, which makes this awkward to automate in Terraform. What came out of it was a patch to Cilium’s encryption page, which didn’t say clearly enough what gets encrypted.

Encryption isn’t identity, and some of it is already done

Encryption is the other reason people go shopping for a CNI, and it’s worth being precise about what you’re buying.

Do you need mTLS if the traffic is already encrypted?

They answer different questions, so you can have one without the other. Ordinary HTTPS already authenticates one end: the client checks the server’s certificate, and RFC 8446 makes authenticating the client the optional half. From the server’s side, which is where this bites inside a cluster, the pipe is encrypted both ways and you still have no idea who is sending you traffic. With mTLS the client presents a certificate too, so the remote side has an identity it can prove, and that identity is what an authorization policy can act on. In Consul Connect every sidecar gets a certificate, and policies then say this service may talk to that service. You’ve stopped writing IP addresses and started writing identities.

That identity is also what the auditor is after. As soon as you’re in an audited space you have to show that some services can’t reach others, and network policy is how you show it. The hard part isn’t the technology. It’s that someone who has read firewall rules all their working life now gets a wall of YAML instead, and that takes teaching before it takes configuration.

Before you buy a mesh for encryption alone, check what you already have. Some AWS instance families built on the Nitro System encrypt traffic between instances automatically, with no configuration from you. The EC2 user guide lists the types and the conditions attached to them. Both instances have to be in the same Region, and in the same VPC or peered VPCs. The traffic must not pass through a virtual network device or service such as a load balancer or a transit gateway. So it covers node-to-node traffic that goes straight across your VPC, and not the hop through a load balancer in front of it. The cheaper T3 sizes aren’t on the list at all. When we wanted the T family we ran Weave instead, as a DaemonSet providing an encrypted overlay between nodes. It doesn’t encrypt traffic between two pods on the same node, which sounds worse than it is. Reading that traffic takes privilege on the node rather than permission to install software: a capture needs a raw socket, and a raw socket needs CAP_NET_RAW (packet(7)).

Ingress: choose it for the routing you need

Traffic that’s encrypted and identified still has to get into the cluster, which is the next thing to configure.

Which ingress controller should you run?

The one whose routing features you’ll actually use, because the basics are much the same everywhere. We’ve run three of them.

Envoy, usually through Istio’s ingress gateway, takes its configuration as JSON or YAML and exposes a control endpoint you can talk to, which you should keep off the internet. What made it stick for us was the comparison with planning an Nginx config restart inside a container, back when that was its own small project.

ingress-nginx is the boring option, and it works. Putting the certificates on the controller rather than on the load balancer keeps TLS termination inside the cluster, so be clear about what the hop in front of it does. An ALB’s HTTPS listener terminates the client’s connection and opens its own to the target, so both legs can be encrypted, but it is not one TLS session from client to pod. For traffic that has to reach the cluster undecrypted, AWS’s own guidance is a Network Load Balancer with a TCP listener on 443, which passes encrypted traffic through to the targets. Inside the cluster, SSL passthrough is the one thing we couldn’t get working, for a service that wants to terminate TLS itself. Two attempts, gibberish arriving at the service, and nobody we asked had got it working either. The docs explain the shape of the problem: --enable-ssl-passthrough is off by default, it intercepts the HTTPS port and hands traffic to a local TCP proxy, it bypasses NGINX with a non-negligible performance penalty, and it depends on clients sending SNI. While you’re there, note that nginx-ingress and ingress-nginx are different Helm charts, which confuses the hell out of people.

Traefik is where we landed when developers wanted canary releases and weighted routing. It has ready rules for a 90/10 split, Prometheus metrics, OpenTracing, JSON access logs that parse straight into Elastic, and a Consul backend, so a curl to Consul updates the routing. Kong sits closer to an API gateway.

The integration you want someone else to run is IAM

That brings the buy-or-build question back, because one integration costs you the most to set up and keep running yourself.

Why is cloud IAM integration the strongest reason to buy?

Because the provider runs the identity plumbing for you, and your pods get short-lived cloud credentials out of it. EKS publishes an OpenID Connect endpoint, and you define roles that identities from that provider may assume, so a pod consumes STS credentials and you hand it no AWS keys at all. That’s IAM roles for service accounts, and Workload Identity is the same idea on GKE. The number of secrets in a cluster drops considerably when you turn it on, though prebuilt manifests and Helm charts sometimes get in the way of the annotations it needs.

This is about who assembles it, not about a capability EKS keeps to itself. AWS’s launch post walks through the same mechanism on a DIY cluster and names kops as the example, so you can run the OIDC provider, the signing keys and the admission webhook yourself. The question is whether you want to own that alongside everything else, and keep owning it.

Now price the alternatives on a cluster where nobody has built that. You can rely on the role attached to the node, which means every pod on that node shares it, and that’s undesirable the moment two teams share a cluster. You can put keys in secrets or ConfigMaps and own the rotation. Or you can issue dynamic credentials from Vault, which works and which we like, at the cost of a new dependency in the credential path: lose the connection to Vault, the pod can’t renew, the credentials expire, and now you have a different problem.

The everyday version of this is small. external-dns watches your ingress objects and writes Route 53 records, so it needs Route 53 permissions and nothing else. With the OIDC integration that’s one role bound to one service account, and no long-lived AWS access key needs to be stored in the workload.

One library, several languages

One release landed the same day we recorded, and it speaks to who builds the clusters above.

Pulumi made its EKS package available in TypeScript, Python, .NET and Go, from a component authored once, as the first step towards components that compile into every language they support. If that holds up, a platform team can write the cluster decisions in Go and a product team can consume them from Python, without either side learning a new DSL.

There’s a counter-argument we take seriously. A DSL limits what you can express, and that’s a feature: people write bad code more often than good code, and the boundaries keep them out of trouble. On the other side, a real language in a real editor gives you completion and type safety, which is a genuine saving and reason enough for some teams. Both can be true, and we’d like to see where Pulumi takes it.

What this means for teams

If you’re audited, buy the control plane. The hardening evidence for master nodes is the expensive part, and the fee is in the same range as the instances you’d otherwise run. If you’re not audited, a kops cluster is still a fine place to be, and the upgrade story is better than its reputation.

For the cluster itself, keep the provider’s CNI until you can name what’s wrong with it. Keep the ingress controller boring unless you need its routing. Turn on the cloud identity integration before anything else, because it removes secrets rather than adding a system to manage them.

This week:

• Work out the pods-per-node ceiling for the instance types in your node groups and count your DaemonSets against it, then check the free addresses left in those node subnets and what WARM_IP_TARGET and MINIMUM_IP_TARGET hold per node.
• Check whether your node instance types encrypt traffic between instances, and whether your traffic takes a path that qualifies, before anyone proposes a service mesh to do it.
• Find every pod that uses the node role or a static AWS key, and move one of them to IAM roles for service accounts or Workload Identity.
• Put a number on your own masters, instances, volumes and load balancer, and compare it with $0.10 per cluster per hour, less the zonal cluster Google exempts if you’re on GKE.

Common questions, answered

How do I tell whether new pods are blocked by the EKS pod limit or by IP exhaustion?

Run kubectl describe pod and read the events, because the two failures look different and take different fixes. A pod the scheduler can’t place stays Pending, and on a full node the reason reads Too many pods: that’s the kubelet’s pod cap, which the EKS AMI sets from eni-max-pods.txt, 17 on a t3.medium and 29 on an m5.large. Add nodes, or pick a bigger type. A pod that was placed and then couldn’t get an address fails later, and sits in ContainerCreating with a FailedCreatePodSandBox event instead. That one is about addresses, so check the free space in the node’s subnet and what WARM_IP_TARGET and MINIMUM_IP_TARGET hold per node.

How do I encrypt traffic between nodes without running a service mesh?

Check your instance types and the network path first, because AWS may be doing it already. Several Nitro-based families encrypt traffic between instances with no configuration from you. The conditions are narrow: both instances must be in the same Region, and in the same VPC or peered VPCs. Traffic must not pass through a virtual network device such as a load balancer or a transit gateway. The EC2 docs name the types and those conditions. Outside them, an overlay such as Weave runs as a DaemonSet and encrypts node-to-node traffic.

Should I migrate my kOps cluster to EKS?

Migrate if you want the control plane hardened and the identity plumbing run by someone else. Pod-level IAM is not the deciding factor: AWS’s launch post covers the same setup on a DIY cluster and names kops, so the difference is who operates the OIDC provider and the webhook. If you’re happy owning that, and nobody is asking you for control plane evidence, a working kops cluster handles upgrades, CNI installation and node rotation.

Related episodes

• Infrastructure as Code: where we first argued about DSLs against a general-purpose language, the argument Pulumi keeps reopening.
• Docker Runtime Security: what someone with a foothold on the node can already do, which is the backdrop to the packet capture line above.
• AWS Security Maturity Roadmap: why long-lived IAM keys are worth hunting down, in and out of clusters.
• All You Need to Know About Setting Up HashiCorp Vault: dynamic credentials, the option we reach for when a cluster has no cloud IAM integration.

Resources

• Pod networking in the Amazon EKS user guide: how the VPC CNI gives pods VPC addresses, and where the per-node limits come from.
• eni-max-pods.txt in the EKS AMI: the actual pods-per-node number for each instance type.
• VPC CNI troubleshooting tips: how a subnet with no free addresses shows up, and the pool each node keeps.
• Introducing security groups for pods: what it takes to give a pod its own security group instead of the node’s.
• EC2 encryption in transit: which instance families encrypt traffic between instances, and the Region, VPC and network path conditions that come with it.
• Cilium with the AWS VPC CNI plugin: what chaining mode does, and how Cilium attaches to the devices the VPC CNI sets up.
• TLS and SSL passthrough in ingress-nginx: the --enable-ssl-passthrough flag, its SNI requirement and its performance cost.
• Introducing fine-grained IAM roles for service accounts: the OIDC mechanism behind pod-level roles, on EKS and on DIY clusters.

Since we recorded

In August 2021 the VPC CNI added prefix delegation, which hands each ENI a /28 prefix instead of single addresses on Nitro instances and raises the pods-per-node limit sharply. Cilium graduated in the CNCF in October 2023. Our ingress-nginx habit aged badly: in November 2025 Kubernetes SIG Network announced the retirement of ingress-nginx, with best-effort maintenance until March 2026 and Gateway API as the recommended destination. We came back to both threads later, in All the Things You Wanted to Know About Pulumi and Do I Need a Service Mesh?.