The ask is encrypted traffic between your own services

A developer spins up Elasticsearch locally with Docker and curls it: no user account to create, no password to pass around, no certificate to install. That is the local experience worth protecting. The same service then runs in the cluster, where every call between workloads should be authenticated and encrypted, and where nobody should have to write TLS into the application or hand a credential to a laptop. Push that problem down into the platform and the developer’s day doesn’t change at all.

The answer everyone points at is a service mesh. Sidecar containers take over the traffic, do the handshake, and enforce rules about who may call whom. What doesn’t fit on the slide is the other half of the trade: you now own the network your developers stopped thinking about, plus a control plane, plus an upgrade path for both.

So the useful question isn’t which mesh. It’s what you’re trying to fix. If the whole ask is mutual TLS, there are smaller things that do mutual TLS. If you run on AWS with one address per workload, you may already have identity and encryption without a certificate anywhere. And if you do need the full product, it wants a team, not an afternoon.

Your workloads have IP addresses, and you wanted identities

A mesh is usually explained through its mechanics, so start one level up, at what it changes about the network.

What does a service mesh actually change?

It moves the conversation from “this IP may reach that IP on that port” to “this application may call that application”, and it gives each workload an identity the network can verify.

Rules written in IP addresses were workable when a host sat in a rack for a year. In Kubernetes the addresses are ephemeral, they get recycled, and the firewall rule you wrote last quarter now points at something else. Your workload does already have an identity: the service account. It’s mostly used against the Kubernetes API, not on the wire. Mutual TLS stretches that identity to the network, because each side presents a certificate and checks the other one, and then the policy can be written in names. Consul calls those rules intentions, enforced by identity at layer 4 or by request at layer 7, with the default following your ACL policy, so default_policy = "deny" gives you deny by default.

One correction on the mechanics, because it matters when you debug: the proxy isn’t a separate pod in the cluster. It’s a container inside your pod, and iptables rules in that pod’s network namespace redirect traffic through it on the way in and on the way out. Once it sits there it can also retry failed calls and break circuits, which is network trivia you stop reimplementing in every language your teams use.

Take over the network and you own the network

Both of those are real. Here’s the invoice.

What does running a mesh cost you?

Once the mesh carries your traffic, every network problem in the cluster is yours, including the ones the mesh brought with it.

Credit where it’s due: the big meshes do things nothing else does. We’ve attached a VM workload to a mesh with about twenty lines of YAML and had access working. Istio’s virtual machine install is the shape of that work: a WorkloadGroup in the cluster, istioctl x workload entry configure to generate the bootstrap files, then the root certificate, the token and the sidecar package on the VM itself. The network path underneath doesn’t disappear. The same page lists IP connectivity from the VM to the mesh’s ingress gateway as a prerequisite, and recommends L3 connectivity to every pod on top of that for performance.

Istio is also a big beast, and standing it up takes a while. The features that repay that work show up on a large estate, or on several clusters that have to reach each other, so the question is whether your requirements and your operating capacity look like that. On a small cluster run by people who have other jobs, the setup and the upgrades are most of what you experience. Upgrading it isn’t the most trivial thing. When traffic stops moving you had better be good at tcpdump, at getting onto the node, and at reading iptables chains, because the abstraction you gave your developers is now the thing in the way.

There’s a cheaper version of this. What we usually mean by “service mesh” is the control plane that configures all those sidecars, and you can take the data plane without it: deploy Envoy with a static config, and when the config changes, redeploy. No control plane to secure, upgrade or watch. When your config genuinely has to shift at runtime across a large fleet, that’s the moment a control plane starts earning its keep.

Start from the problem, then buy the smallest thing that solves it

Which makes the comparison table the wrong place to start.

How do you choose between the meshes?

You don’t start there. Write down the single problem you’re solving, and most of that table turns out to be irrelevant to you.

Header-based routing is a good feature, and it’s how you do canary releases at the network level. If your team is three developers getting things to production part-time, you won’t run canaries, so that feature scores zero. Mutual TLS is in all of them, so it doesn’t separate anything either. What’s left is who operates the thing. On AWS, App Mesh is the basic option, and its reach stops at AWS compute. A service joins it only if it runs on Fargate, ECS, EKS, Kubernetes on EC2, or EC2 with Docker. That compute can sit in your own data center: ECS, EKS and App Mesh have supported AWS Outposts since December 2019, so an Outposts rack is in reach while another cloud isn’t. Mutual TLS has been there since February 2021, with certificates either on the Envoy file system or served by a sidecar that implements Envoy’s Secret Discovery Service, such as SPIRE. It’s a hosted control plane that already fits the rest of the AWS estate.

This is the one place the three of us land differently. Mattias went shopping for a mesh because he wanted mutual TLS without developers handling certificates. Julien’s answer is not to buy a mesh for that at all. Andrey’s is that on AWS you may not need the certificates in the first place.

There are a few. SPIFFE and SPIRE, a specification and an implementation of it, have been in CNCF incubation since June 2020, and they’re capable rather than friendly. Smallstep’s step-ca will hand a workload a certificate in exchange for its Kubernetes service account token, through the K8sSA provisioner. Read their caveat before you build on it: the token carries no SANs, so the CA “does little to no validation on the CSR before signing it”, and a token holder can get a certificate for names that aren’t theirs. You still have to use the certificate correctly in code, and you may end up running a certificate authority, which is its own job. Outsource that part wherever you get the chance.

Security in layers, and where each layer stops

The reason you can often get away with less than a mesh is that the mesh was never your only layer.

Where does each layer actually stop?

Your VPC gives the workloads a private address space, reachable only the ways you configure it. A security group drops traffic the workload isn’t allowed to receive. Mutual TLS stops an attacker who is already inside the group from impersonating one of your services.

Be careful with that first layer. A VPC isn’t an automatic deny-all box: what can reach your workloads from outside depends on the gateways, routes and endpoints you attach to it, and on the rules you write. The rules are the explicit part, security groups at the resource level and network ACLs at the subnet level, with flow logs to show you where they are too loose or too tight (internetwork traffic privacy in Amazon VPC).

The resource-level layer got much better on Kubernetes. Since September 2020 EKS can assign EC2 security groups to pods on most Nitro-based instance types, so a pod gets its own elastic network interface and its own firewall. A security group is an identity too, and for east-west rules between workloads it does roughly the job intentions do inside a mesh. Encryption then comes off the hardware: on supported instance types the Nitro system encrypts traffic between instances with 256-bit AEAD. Both ends have to sit in the same Region, and in the same VPC or peered VPCs. The path matters as well, because traffic that goes through something like a load balancer or a transit gateway falls outside the guarantee (EC2 data protection documentation). Identity and encryption, no certificate in sight.

Say what that doesn’t cover. Nitro encrypts between instances, so two pods that land on the same node get nothing from it, and that case is one a mesh does cover: proxy-to-proxy mutual TLS encrypts the hop whether the two proxies sit on one node or two. The leg neither of them covers is the one inside the pod. Your application talks plaintext to its own sidecar over localhost, which the App Mesh documentation states plainly, since mutual TLS runs between Envoy proxies and not between your code and its proxy.

The sidecar sits at the wrong level

Thinking in layers is also what makes the sidecar itself look like a workaround.

Why does the sidecar bother us?

Because it’s bolted next to the workload, and the job it does belongs in the network stack underneath it.

The compute cost is the small part. Each injected sidecar adds a container to the pod, so you pay for its memory and CPU, and a monitoring bill priced per container picks up one more container per pod. Where the pod ran a single application container, that is twice the count the vendor was billing you for. We looked at that pricing a couple of years ago and it was enough to notice; it may well have changed since. The compute is a fair trade anyway, because TLS termination has to happen somewhere. What grates is the level of abstraction: packets take extra hops through a userspace proxy for something this low-level.

The eBPF projects are pushing it where we think it belongs. Cilium runs one agent per node rather than a container per pod, and in version 1.10 that agent can bring up WireGuard tunnels between nodes so traffic between Cilium-managed pods is encrypted in the kernel as it crosses from one node to another. Be precise about what that mode covers. It is transport encryption for pod traffic: host-level traffic stays unencrypted, packets that never leave the node aren’t encrypted either, and L7 policy enforcement and visibility aren’t supported alongside it in 1.10. Identity-based policy is a separate Cilium feature and still applies at L3 and L4. That isn’t a mesh. It’s the right level.

For what it’s worth, the sidecar world is less uniform than it looks. Istio, App Mesh and Consul all ship Envoy, so a lot of meshes are the same proxy with different configuration. Linkerd wrote its own Rust micro-proxy instead and published why. NGINX Service Mesh, at 1.0 by May 2021, injects a sidecar built on an NGINX Plus instance rather than Envoy.

Observability is the feature teams keep

One item on the buffet survives all of this, and it isn’t the one the security conversation starts with.

What do you get that you would otherwise instrument by hand?

A map of your network traffic: who calls whom, how often, how slowly, which hop is saturated, with no change to application code.

The wiring is genuinely cheap. With App Mesh you set ENABLE_ENVOY_XRAY_TRACING to 1 on the Envoy sidecar and run the X-Ray daemon next to it on UDP port 2000, and the traces show up in X-Ray. That’s a couple of lines of Terraform, and if you have no tracing at all it gets you a long way.

It isn’t the same thing as tracing, though, and the gap is worth knowing before you promise it. Proxies see traffic, not intent. With Istio the sidecars emit the spans, and something in your application still has to copy x-request-id and the x-b3-* headers from the inbound request onto its outbound calls, or the spans never join into a trace. We went through that in detail when we did the three pillars. So the free part is the hop map and the per-hop latency. A complete trace requires application context propagation, through supported instrumentation or changes to your code.

What stalled our rollout wasn’t the YAML

Which brings us to the failure we’ve run into, and it wasn’t technical.

What stalls a mutual TLS rollout?

In our case, a service that couldn’t do mutual TLS, and the exception made for it that nobody came back to remove.

It went like this. One team needed to call a service owned by another team, and the other team didn’t know how to implement mutual TLS. Access on that path was relaxed to let the calls through. A ticket called “implement mTLS” sat on the board for a month.

Be precise about what “relaxed” means, because two different switches get called the same thing, and the tools ship with both. Istio puts destination workloads in permissive mode by default, which accepts plaintext as well as mutual TLS, and its own security best practices recommend moving to strict mode where you can. The other switch is authorization, and the same page is blunt about it: mutual TLS “provides only authentication, not authorization”. Check which setting changed. Permissive transport accepts a plaintext call with no verified caller identity. Allow-all authorization keeps mutual TLS and removes the restriction on which authenticated workloads may connect.

So do the education before the rollout. Talk to the developers who will have to live with it, watch the reaction, and let them tell you how they would do it. Then roll it out gradually, service by service. It takes longer, and you get to roll one service back instead of the estate.

What this means for teams

A service mesh is a product you run, with its own control plane, its own upgrades and its own debugging skills. That’s worth it when the problem is big enough and somebody owns it full time. It’s a poor way to buy a single feature, and mutual TLS is a single feature. This week:

• Write your problem on one line before you compare anything. If the line is “we want mutual TLS”, go and look at SPIFFE and SPIRE, at step-ca, and at your cloud provider’s managed option before you look at Istio.
• Name the owner. If nobody is going to spend most of their week on the mesh, take the managed control plane, or take the proxy with a static config and no control plane at all.
• On AWS, check what you already have: security groups for pods on Nitro-based nodes, and encryption between qualifying instance types in the same VPC. You may be covered for identity and for traffic between nodes without issuing a single certificate.
• Pick the first pair of services and ask the other team whether they can do mutual TLS at all. Build the rollout around that answer.

Common questions, answered

How do I get mutual TLS between services in Kubernetes without running a service mesh?

Issue short-lived certificates from a certificate authority that trusts the identity Kubernetes already gives your workload. Smallstep’s step-ca does this with its K8sSA provisioner: a pod presents its service account token and gets a certificate. SPIRE does it through attestation and its own agent. Neither puts a proxy in front of your traffic.

The trade is that your application has to use the certificate itself, and somebody has to run the CA or buy a managed one. Read the limits first: step-ca does little validation of the signing request behind a service account token, so a stolen token gets a certificate for a name that isn’t its own.

Is AWS App Mesh enough if I only need encrypted traffic between services on EKS?

For mutual TLS between services plus basic traffic rules, yes, and there’s no control plane to run or upgrade. App Mesh added mutual TLS in February 2021, and takes certificates from the Envoy file system or a Secret Discovery Service sidecar such as SPIRE.

Two limits to plan around. The mesh reaches only AWS compute, which includes Outposts racks in your own data center but not another cloud. Encryption runs between Envoy proxies, so traffic between your application and its own sidecar stays plaintext on localhost.

Do I need a service mesh to get distributed tracing?

No, and a mesh won’t finish the job by itself. The sidecars emit spans for every hop, so you get a service map and per-hop latency without touching application code. A proxy still can’t tell which outgoing call belongs to which incoming request. The trace context has to travel with the request, which is the application’s job, whether an instrumentation library does it or you do.

For real traces, instrument the application and treat the mesh view as a bonus.

Related episodes

• Episode 23, How Do We Run Kubernetes In The Cloud? covers the plumbing underneath all of this: the VPC CNI, what broke when we chained Cilium onto it, and keeping an Envoy control endpoint off the internet.
• Episode 28, Scaling Security is where certificates at scale came up: who issues them, who rotates them, and why a mesh handing out certificates isn’t the same as mutual TLS being enforced.
• Episode 30, three pillars of observability has the longer version of the tracing argument, including the trace context your application has to propagate before mesh spans add up to a trace.

Resources

• AWS App Mesh now supports mutual TLS authentication, February 2021, settles where App Mesh gets its certificates: the Envoy file system, or a Secret Discovery Service sidecar such as SPIRE.
• Mutual TLS authentication, App Mesh documentation settles what is encrypted and what is not: proxy to proxy is, application to its own proxy is not.
• Amazon EKS now supports assigning EC2 security groups to Kubernetes pods, September 2020, settles that per-pod firewalls exist, on most Nitro-based instance types.
• Data protection in Amazon EC2 settles the conditions for encryption between instances: supported instance types, same Region, same or peered VPC, and no load balancer or transit gateway in the path.
• Istio security best practices settles what mutual TLS does and does not do, that permissive mode accepts plaintext by default, and that authentication is not authorization.
• Intentions, Consul documentation settles how identity-based policy reads outside Kubernetes, and how the default allow or deny behavior follows your ACL policy.
• WireGuard transparent encryption, Cilium 1.10 documentation settles what the kernel-level option covers in that release, and what it leaves out, including host-level traffic and L7 enforcement.
• Why Linkerd doesn’t use Envoy, December 2020, settles that the data plane is a choice, with the reasoning for a purpose-built sidecar proxy over a general-purpose one.

Since we recorded

The specific advice to start with App Mesh on AWS has expired: AWS discontinued support for App Mesh on 30 September 2026 and points customers at Amazon ECS Service Connect instead. The complaint about sidecars sitting at the wrong level got an answer from the mesh side: Istio’s ambient mode, generally available in version 1.24 in November 2024, drops the per-pod sidecar for a shared layer 4 proxy on each node, with layer 7 proxies only where you ask for them. The bigger call held up. Looking back across a hundred episodes, our read was that Istio as the default never happened and simpler patterns won.