Someone asks whether the container is safe enough

You want to move a service off its own virtual machine and into a container, and someone with a compliance document asks whether that’s a downgrade. The answer they have already heard is that you can break out of a container but not out of a VM, so the container is the weaker option and the conversation is finished before it starts.

That sentence carries a lot of weight for something with no numbers in it. Breaking out of a container has preconditions, and the VM you’re comparing against isn’t a clean room: it’s running a mail agent, a log shipper, a host intrusion detection daemon, a package manager and whatever else was installed the year it was provisioned. Which side of that trade you land on decides how you spend the next quarter, because it sets whether you harden images and cluster policy or keep building golden VM images.

We’ve argued about this one and we still don’t agree. Mattias says running containers in Kubernetes is more secure than running VMs, because of how little ends up inside a container and how briefly it lives. Andrey and Julien say that compares operating models rather than technologies, and that a container is a weaker boundary with homework attached. We agreed to disagree. The pieces we disagreed about are more useful than a verdict would have been.

What a breakout actually takes

Start with the claim the whole objection rests on, because it’s the softest part.

Can an attacker escape your container with one command?

It depends on the vulnerability, the container configuration and the access the attacker already has. In the breakouts we had read about, something attacker-controlled has to be inside the container first, and putting it there is its own job.

Our reading of the Docker breakout CVEs on air was a general one: the ones we had seen needed either the attacker’s own container, or a container running privileged. We named no single advisory, so here’s one to check it against: CVE-2019-5736, the runc flaw published in February 2019.

The advisory from runc maintainer Aleksa Sarai names two routes to it. The first is a container you create from an attacker-controlled image. The second needs write access to a container that already exists, plus somebody running docker exec into it. Both need a command running as root in that container, and in the second it’s yours: the attacker plants the file, an administrator runs it. The exploit reaches the host’s runc binary through /proc/self/exe and overwrites it, so the next invocation runs attacker code as host root. The default AppArmor policy did not block it. User namespaces did, when host root isn’t mapped into the container.

The advisory is narrower than our version on one point and wider on another. --privileged isn’t in it at all, and the second route doesn’t need a foreign image: the container already running can be one of yours.

The two routes ask different things of you. Running images you didn’t build is a decision somebody made on purpose and can unmake. The write route needs a path the container executes redirected at /proc/self/exe, so the host’s runc re-executes itself inside the container, and then a way to write to that binary. Unit 42’s walkthrough lands that write two ways: a second process in the container that overwrites runc as it exits, or a rebuilt libseccomp in an attacker-built image, whose constructor runs inside the runc process. An upload directory nothing executes from reaches neither.

None of that makes the boundary equal to a hypervisor. Escaping a container is architecturally easier than escaping a VM. The honest version is narrower than the slogan: one vulnerability, with prerequisites that had to line up, content the attacker controls inside the container and a root command that runs it. The next escape may need something else entirely.

The VM you’re comparing against isn’t empty

So if the boundary argument is weaker than it sounds, the next question is what sits behind each boundary.

What else is running on that hardened web server?

Postfix, syslog, a host intrusion detection agent, monitoring, a package manager and sudo, all of it reachable the moment someone gets a shell as the web user.

PCI DSS has asked for one primary function per server since well before this argument started. Version 2.0 clarified the intent of that existing requirement in 2010 and widened the definition of system components to cover virtual ones, and VM estates implement it by splitting roles across machines. Containers take the same idea further than requirement 2.2.1 ever pushed it. The log shipper, the metrics agent and the detection engine don’t go in the image. They run beside the workload, which means an attacker who lands in your application container finds the application and not much else.

The local tooling matters because it gets exploited on its own schedule. The sudo bug published on the oss-security list in January 2020, CVE-2019-18634, was a stack overflow reachable when pwfeedback was enabled in sudoers, which some distributions shipped on by default. Every convenience binary on that VM is a tool the attacker inherits for free.

Shipping less is also the part you can do today. DockerSlim strips an image down without you reasoning about every layer, and distroless base images leave out the shell and the package manager entirely. Neither makes an image clean: Sysdig reported in October 2019 that 52% of the images it scanned had known vulnerabilities. Smaller mostly means fewer things you didn’t choose.

Short lives come from your pipeline

The second half of the case for containers is about time, not size, and this is where we stopped agreeing.

Is “it dies on the next deploy” a container feature?

Not by itself. It’s a deployment model, and a VM can have it too.

Mattias’s version: a container arrives in production at the end of its life. It gets replaced on the next deploy, so it needs no editor, no package manager and no way to apply an update in place. Andrey’s objection is that this describes immutable infrastructure, not containers. You bake a VM image, roll a new one and throw the old one away, and you get the same property. He has also watched the other direction in large organizations: a team gets a VM, starts a container on it to run something like Mattermost for the team, and never kills it. That container has tools inside, because people go into it and change things.

Mattias’s answer is that he’s comparing the common case, not the possible one. The common case has numbers behind it. Sysdig found in 2019 that half of the containers it observed lived less than five minutes, up from a fifth the year before.

The replacement model does hand you a bill. Your web server now ships inside the application image, so an NGINX or Tomcat fix becomes a rebuild, a retest and a release of an application whose own code didn’t change. That is the same dependency you always had, instantiated once per service instead of once per fleet. What you get back is that one team can take the new base image today without waiting for every other application on the shared server to be ready.

Containers do not contain

Both of those arguments are about what’s inside the box. The box itself deserves a sentence.

What is a container isolating?

Namespaces isolate processes’ views of system resources; cgroups account for and limit resource use, while containers share the host kernel.

You do get generic defaults, and they beat nothing. Docker has shipped a seccomp profile for unprivileged containers since Engine 1.10 in February 2016, and the documented default disables around 44 of the 300-plus syscalls on the system. The docker-default AppArmor profile is the same kind of compromise, drawn to keep most applications working. Neither knows that your service only reads files, opens one socket and never calls mount.

Under Kubernetes, check that you’re getting even that much. The kubelet runs pods unconfined unless you ask for the runtime’s profile, and that default is not what KEP-135 sets out to change: the proposal graduates seccomp out of alpha by moving the annotations into stable securityContext fields, and leaves what an unannotated pod gets alone. So ask for the profile by name, with seccomp.security.alpha.kubernetes.io/pod: runtime/default on the pod or container.seccomp.security.alpha.kubernetes.io/<name> on one container.

Then there are the settings that give pieces of the boundary back, and they give back different pieces. --privileged is the broad one: the container gets the full capability set and the host’s devices. Running as root is narrower. The container’s root is uid 0 on the host unless you turned on user namespace remapping, but Docker drops most capabilities from it, so what it’s worth to an attacker depends on what else you mounted or granted. Host networking removes the network namespace and leaves the others: the process keeps its own mount and process namespaces, and gets the node’s network stack, including whatever is listening on the node’s loopback. Remapping uids is a separate switch, and not one you get by default: you turn it on for the daemon with --userns-remap, and Docker lists host networking among the features that mode doesn’t support. A container is as contained as the profile you wrote for it, and most of them don’t have one, because writing that profile means knowing every syscall the application makes on a bad day.

Someone has to say what the app may do

That profile is where the technical argument turns into an organizational one, which is the part that decides whether any of this gets done.

Who writes it, the developer or the platform team?

Often neither, which is how workloads end up running unconfined.

With a VM, operations defines the access and the application lives inside it. Move to dynamic workloads and the specification drifts toward whoever writes the workload, who reasonably says they’re building a feature and didn’t sign up to secure infrastructure. They also can’t answer the question honestly, because the syscalls come from libraries they didn’t write. The people securing the workload and the people writing it rarely work hand in hand.

Mattias’s counter is that you don’t ask every developer. Whoever sets the cluster up sets the standard for what’s allowed to run in it, and admission control makes the standard stick. Open Policy Agent Gatekeeper was written up on the Kubernetes blog in August 2019 for exactly this: a constraint that rejects pods asking for root, for privileged, or for the host network, written once and applied to every namespace.

That still leaves the per-application part, and we’d rather say so than pretend policy covers it. Least privilege needs somebody to answer what this program should be allowed to do, and finding the profile that says it is tedious work on any platform. The cluster policy gets you the floor. The profile is the ceiling, and it’s still manual.

Detection you can act on

Once you accept that the boundary is partly a policy you wrote, the next lever is noticing when something crosses it.

What does monitoring buy that isolation doesn’t?

A response, triggered on what the workload does, not on a boundary somebody had to get right in advance.

Falco, which Sysdig open sourced in 2016 and the CNCF took into its sandbox in 2018, watches syscalls and matches them against rules. The one everybody demonstrates is Terminal shell in container: a shell gets a terminal inside a running container, Falco fires, and the alert goes to a function that deletes the pod. Your developers keep deploying. Someone who pops a shell in production loses the container they’re standing in.

Try that signal on a VM and it’s noise, because a shell is how work gets done there. The cluster changes how the sensor gets there, too. Falco reads syscalls per node, so it needs an agent on every node, and it ships manifests that run it as a DaemonSet: one definition, and Kubernetes places and replaces the agents, including on nodes you add next month. On VMs that coverage is a rollout you maintain host by host.

We can’t prove the next bit, so take it as experience, not evidence. Where monitoring is bad, access control gets wild: people need production access to debug, nobody can say why a permission exists, and the list only grows. Where monitoring is good, you can ask why somebody opens a shell in every container, hear “because I need the logs”, and fix the actual problem by shipping the logs somewhere they can read them.

You’re not choosing, you’re stacking

Most of this argument assumes a fork in the road that doesn’t exist in anybody’s account.

Do you have to pick one?

No. In the deployments we were arguing about, the containers sit on VMs already, so the useful question is what each layer owns.

Bare metal is a real option, and Diamanti’s 2019 survey found 21% of respondents running containers on it, more among the teams spending most on containers. If your nodes are cloud instances, though, the hypervisor is in your stack whether you think about it or not, and the serverless end of the market stacks the layers on purpose. AWS open sourced Firecracker in November 2018, the microVM monitor behind Lambda and Fargate, and the NSDI paper from February 2020 describes the trade it makes: a separate kernel per workload, with a device model small enough to keep the host attack surface down. A Lambda environment is about as hardened as ephemeral gets. Little inside it, a mostly read-only file system, no access to the configuration.

It’s still worth looking at how people attack it, because the pattern generalizes. /tmp is writable, and an environment is reused between invocations. Unit 42 showed in September 2019 how an attacker with code execution inside a function can replace the runtime’s bootstrap so every later event passes through their code and gets copied out first. That foothold lives in the one execution environment they compromised and lasts only while AWS keeps it, which is why their attacker sends periodic requests to stop it being reclaimed. After that you’re into what the execution role can reach, and into exfiltration through channels nobody alarms on, resource tags among them.

Stacked properly, the layers split cleanly: the container boundary is about the application, the VM and the hypervisor are about operational security and resource allocation. That’s also why the host under your nodes keeps shrinking, and why the stripped-down host operating systems churn. CoreOS Container Linux got its end-of-life announcement in February 2020, with final updates due in May. The other direction is being tried too, with gVisor putting a user-space kernel under the container and Kata Containers giving each one a lightweight VM. Nobody building these treats namespaces as the last word either.

What this means for teams

The verdict you want out of this doesn’t exist, and asking for it sends you shopping instead of fixing things. Containers make the small, short-lived, single-purpose workload the easy path, and VMs make the strong boundary the default. Whichever one you run, the security comes from the work that neither gives you for free: knowing what’s in the image, knowing what the process may do, and noticing when it does something else.

Four things worth doing this week:

• Grep your manifests for privileged: true and hostNetwork: true. The first hands over the full capability set and the host’s devices, the second only the node’s network stack, and each one should have a reason written next to it.
• List the images you run that you didn’t build. Then look at where a request can write inside your own containers, and check whether the entrypoint, or any other path a root command will execute, sits somewhere writable. That redirect at /proc/self/exe is the prerequisite the runc bug has. How an attacker lands the payload afterwards varies by exploit, so don’t check for one set of mechanics and call the container clear.
• Require runAsNonRoot: true on your pods, which makes the kubelet refuse to start a container whose effective user resolves to uid 0. Back it with a non-root USER in the Dockerfile, checked where the image is built by a linter such as hadolint and its DL3002 rule, so the identity is already right before anyone writes a pod spec.
• Wire up one detection you can act on, starting with a shell in a running container, and decide now whether it pages someone or kills the pod.

Common questions, answered

How do I stop containers in my cluster from running as root?

Set runAsNonRoot: true and let the kubelet enforce it: it resolves the effective user and refuses the container when that is uid 0, or when the image’s USER is a name it can’t resolve. The Dockerfile’s USER is a second safeguard, not a prerequisite: a runAsUser in the spec overrides it.

Container root is uid 0 on the host unless you turned on user namespace remapping. Docker drops most capabilities from it, so it isn’t a root shell on the node, and it’s still the account an attacker would rather land in. Expect breakage from images that write to root-owned paths or bind a privileged port.

Do I still need to patch the operating system if my app ships in a container?

Yes, in two places. The userland in your base image, the runtime and any bundled web server travel with your artifact, so a fix for any of them is a rebuild, a scan and a deploy of an application whose code didn’t change.

The kernel and the node’s packages are in neither your image nor your build, and no rebuild reaches them. Patch the nodes on their own cycle and treat them like containers: roll a patched node pool and recycle the old nodes.

What should I turn on first for containers I’m already running in production?

Ask for the runtime’s own profiles rather than assuming you have them: on Kubernetes set seccomp.security.alpha.kubernetes.io/pod: runtime/default, because the kubelet runs pods unconfined otherwise, and on Docker don’t pass --security-opt seccomp=unconfined. Then take back the privileges you granted by hand: --privileged, host network, host mounts and anything running as root.

After that, add one detection with a response attached. Profiles are the expensive control, so write them for the workloads that would hurt most.

Related episodes

• #1 - Infrastructure as Code: rebuilding from code is what makes a VM replaceable, which is the argument Andrey leans on when he says containers didn’t invent short lifetimes.

Resources

• CVE-2019-5736 advisory, Aleksa Sarai: settles what a runc breakout required, including that the default AppArmor policy didn’t stop it.
• PCI DSS 2.0 summary of changes: says the 2010 release clarified the intent of the existing “one primary function per server” requirement rather than introducing it.
• Docker security documentation: what namespaces isolate, what cgroups limit, and which capabilities the daemon drops from container root.
• Docker seccomp documentation: what the default profile blocks, and the roughly 44 syscalls it disables out of 300-plus.
• KEP-135, seccomp in the Kubernetes API: the annotations that select a profile, the stable fields meant to replace them, and its statement that changing the default is out of scope.
• Docker user namespace remapping: how to stop container root being host root, and the features the mode rules out.
• OPA Gatekeeper: policy and governance for Kubernetes: how admission control enforces cluster-wide rules instead of per-team discipline.
• Sysdig 2019 container usage report: the lifespan and image vulnerability figures quoted above.

Since we recorded

PodSecurityPolicy was removed in v1.25 in 2022 and replaced by Pod Security Admission, which applies the same kind of rule against root and privileged pods through namespace labels. Seccomp moved too: KEP-2413 added a kubelet option that applies RuntimeDefault to workloads without an explicit seccomp profile on nodes where the option is enabled, stable in v1.27, which is the gap the annotation above was covering. Runtime detection grew up as well, with Falco graduating from the CNCF in February 2024. We came back to the state of Docker, and to what it ships for you now, in #109 - Docker Got Quiet. Did You Miss Anything?.