Every build ships an operating system you didn’t write

A Dockerfile is five lines and a large omission. You write FROM node:12, add your code, and what you push to production carries a Debian userland, a package manager, a shell and whatever CVEs were open against them the day that base image was built. None of it is in your repository. All of it ships.

That matters because the artifact is now the unit of patching. The OS and the application used to move separately: you upgraded NGINX on the host and the application never noticed. Package them together and an NGINX fix becomes a rebuild, a rescan and a redeploy of your application. The dependency chain you inherited is also the chain someone else can poison, and the running container is a thinner wall around it than most teams assume.

Security isn’t a feature you bolt on at the end. It’s a design decision, and it changes how people work. For container builds there are three decisions worth arguing about: who you trust to supply the bits, how little you can get away with shipping, and how you prove that what runs in production is what you built.

Signing proves origin, not contents

Start with the image, because an image is just a file on a disk.

What does a signature on a container image prove?

That the content carries a valid signature from a key you decided to trust, and that it hasn’t changed since. Nothing at all about what is inside.

The model is the one you already use for email, and the one language runtimes use for releases: a Node.js release is signed with the release key of the people who built it. Docker Content Trust does this for image tags. Publishers sign tags, and DOCKER_CONTENT_TRUST=1 makes the client refuse to pull or run a tag that carries no signature.

This is where we don’t agree. Mattias reads signing as proof that you know exactly what’s in the image, because you signed what you put in. Andrey won’t go that far: you get “a semi-guarantee that this image is what it’s pretending to be”, and that is the whole of it. Both halves are useful. A signature gives you someone to hold responsible for the contents. It says nothing about whether those contents are any good.

Provenance has a cheaper cousin: the digest. Anyone with push rights can repoint :latest, or :1.4.2, at new content, so pull by digest instead and you are pinned to exact bytes. Amazon ECR added immutable tags in July 2019. Then restrict the orchestrator to a registry you control. Do all three eventually, one at a time.

Deciding how much Docker Hub is worth to you

Signatures and digests only move the question along: whose key, and whose bytes?

Can you trust an official image from Docker Hub?

Only as far as you have checked who maintains it. The Official Images programme is a review and publishing process, not a promise that the software’s authors built the image.

Docker’s own documentation says so. A sponsored team reviews and publishes the content, and while “it’s preferable to have upstream software authors maintaining their Docker Official Images”, that isn’t a strict requirement. After acceptance, whoever proposed the image keeps it current. So read who the maintainers are, which source repository the Dockerfile lives in, and how the image gets built and published before you build on it.

One search box covers images from individuals nobody can identify, images from organizations like Red Hat or CloudBees, and official images most teams treat as a given. What you extend in each case is reputational trust, and reputation is not an audit. Pull the Jenkins image, run a vulnerability scan over it, and count the CVEs before you decide how far that reputation reaches.

Handovers are the other half of the problem. In 2018 the event-stream npm package gained a new maintainer who added a dependency called flatmap-stream; npm’s account of the incident says an attacker took over maintainership and the malicious package stayed up from September until the end of November. Container images inherit the same risk, and the insider version of it too.

The way out is to choose your suppliers, or become one. Larger organizations don’t pull jar files from Maven Central at all, as far as we’ve heard: they build from source so they know what they have. Most teams can’t maintain base images for four languages. Narrow the set instead: a handful of base images you pull once, scan, re-tag into your own registry and own from then on.

Isolation on a shared kernel has limits

Owning the image doesn’t help much if the thing that runs it leaks.

Is a container a security boundary?

Yes, and a bounded one. The isolation is deliberate; the kernel it rests on is shared.

A running container is a process with a private view of the machine: its own namespaces for processes, mounts and networking, plus cgroups that can enforce memory and CPU limits when configured. By default a container gets no resource constraints: the limits exist when you ask for them on docker run. Docker’s security documentation calls namespaces “the first and most straightforward form of isolation” and says a container “cannot see, and even less affect, processes running in another container, or in the host system”. It is just as plain about the edges: the default capability and mount settings “may provide incomplete isolation”, particularly alongside a kernel vulnerability, and cgroups bound resources rather than access.

What sits underneath is the part to keep in mind. Every container on a host shares one kernel, so the isolation holds as far as that kernel’s implementation does. Containers won the data center on density, by bin-packing more applications onto the same hardware. The isolation is real, and it is not a hypervisor.

Breakouts aren’t theoretical. The Kubernetes Product Security Committee wrote up CVE-2019-5736 in February 2019: a process running as root inside a container could “exploit a bug in runc to gain root privileges on the host running the container”, and from there reach every other container on it. Docker shipped the fix in 18.09.2, and the same write-up names two cheap mitigations: run container processes as a non-zero UID, and use images you have vetted and pinned by hash.

A breakout bug tells you how much weight the boundary carries, not that nobody meant it to carry any. Scanning adds another layer of assurance rather than a wall. The other lever is how much code you ship at all.

Ship the smallest thing that runs

That second lever is the one most teams leave alone.

How much of a base image does your application actually use?

A runtime, some libraries, and almost nothing else that’s in there.

DockerSlim, which lives at dockersl.im, takes a built image and removes everything the application doesn’t need, working that out by watching the application run as well as by reading the layers.

Two things improve at once. Packages you never call still count against you: ship a base image a few versions behind and it keeps known vulnerabilities that upstream patched months ago. Whether a scanner finding is reachable in your deployment is a second question, and the scan doesn’t answer it. Docker publishes a list of vulnerabilities its defaults mitigated, kernel bugs that the default seccomp profile and dropped capabilities made unexploitable from inside a container, and it names the ones those defaults missed as well. So treat a finding as triage rather than a confirmed route in, and carry fewer packages so there is less to triage. An attacker who lands in a container with no shell and no package manager has a much worse afternoon than one who finds bash, curl and apt waiting.

Google’s distroless images come at it from the other end: they “contain only your application and its runtime dependencies” and “do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution”. Alpine goes part of the way, with a far smaller userland than Debian.

The catch is that stripping needs evidence. Anything loaded at runtime, a config file, a locale, a certificate bundle, has to be kept on purpose, which is what DockerSlim’s --include-path and probe options are for. Test the slimmed image the way you’d test a release.

One image to build it, another to run it

Stripping an image afterwards is the retrofit. The structural fix is to keep the build tools out of the shipped image from the start.

Why shouldn’t the image that builds your code be the image that runs it?

Because a compiler, a package manager and your source tree have no job in production.

Multi-stage builds do this in a single Dockerfile: one stage with the JDK, Maven and the source, then a second stage from a minimal base that takes only the compiled artifact. Docker’s docs put it plainly: “You can selectively copy artifacts from one stage to another, leaving behind everything you don’t want in the final image.” So the image that reaches production holds a compiled artifact and little else.

Separation is not scanning. COPY --from moves files and checks nothing, so your pipeline still has to scan the dependencies before the build stage pulls them in, and scan the runtime image after it is built. What the split buys you is a smaller thing to scan and fewer packages to find CVEs in.

The build stage earns its keep twice over, because it is also your build environment, written down and versioned with the code. Nobody installs four language toolchains on a laptop to patch one microservice. The build runs the same way locally as in CI, because in both places the command is docker build.

Caching is what pays for the extra stage. We’ve watched a Maven build spend 20 minutes refetching artifacts because nothing cached the dependency tree, and resolving a big tree burns CPU as well as network. Pulling a layer that already holds them is closer to pulling a zip file.

Three layers to scan, and rarely one tool for all of them

Scanning is where teams believe they’re covered and usually are not.

What does a registry scan miss?

That depends on the scanner, which is the point. Find out what yours parses before you decide what it covers.

Registry scanning works on whatever it can unpack and recognise. You push, the registry expands the layers, lists what is installed and flags the entries with known CVEs. Amazon ECR added image scanning in October 2019, on push or through the API, at no extra charge, and what it reads is the operating system packages. That is a cheap floor under public images: pull from Docker Hub, push into ECR, read the findings, let the cluster pull from ECR only. Other scanners reach past the OS. Anchore, which we run and which as far as we could tell sits underneath more than one hosted scanner, added language package scanning for Node NPM, Ruby gem, Python and Java content back in 2018.

The second scan reads dependency manifests: package.json, Gemfile, requirements.txt, pom.xml. Snyk, Dependabot, WhiteSource and the rest resolve the versions you declared against a vulnerability database and flag the bad ones. Coverage goes language by language and package manager by package manager, and one product can span many: Snyk supported Java, .NET, Python, Ruby, Scala, PHP and Go alongside JavaScript by March 2018. Check the supported list against the manifests your repositories actually carry, package manager included, before you call a repository covered.

The third scan is your own source, and it is the one teams over-claim. Plenty of products sold as source scanning are really resolving your manifests again. Semantic analysis is a different thing and it finds real bugs: Microsoft’s Security Response Center modelled an Azure firmware component’s message handlers as untrusted input, ran Semmle QL taint tracking over it, and reported 33 vulnerable handlers across more than 400 message types in March 2019. A protocol of your own costs modelling time first: somebody has to describe its sources and sinks before taint tracking can follow anything through it. The patterns the shipped queries already describe need none of that work, and there are plenty of them. GitHub’s Semmle acquisition announcement in September 2019 describes Semmle QL as carrying “a library of thousands of queries, all open source”.

A green scan expires

Which leaves the part that catches people out: time.

Why doesn’t a passing scan stay passing, the way a passing test does?

Because the CVE list grows on its own, while a test only changes when the code, the dependencies or the environment under it change.

A unit test passes again tomorrow as long as nothing moved overnight: same code, same pinned dependency versions, same environment. Let a dependency float and the test can change its mind too. A scan is weaker still. It states what the world knew on the day you ran it, so an image that was green when you pushed it can be red next week with no commit to your repository.

Two habits follow. Scan what is running, not only what is being pushed, and then close the gap between the two: list the image digests your cluster reports, match each one against a digest you have scanned in the registry, and treat anything that doesn’t match as unscanned. ECR’s scanning API will scan images already sitting there, which makes that half cheap. An image scan also reads the image, not the running container, so a package installed at runtime, a mounted config map or anything written to a volume is outside its view.

Then shorten the window. Rebuild base images and everything inheriting from them on a cadence, weekly if you can stand it, and roll the result out rather than leaving it in the registry. It is more operational effort. What you get is that persistence planted inside a container goes when the container goes. What survives is everything outside it: credentials that leaked during the build, a persistent volume, a compromised host or build agent. Replacing images raises the cost of staying in. It is not an eviction.

Credentials in the build, not in the image

Building your own dependencies has a sharp edge: the build has to authenticate somewhere.

Where does a build credential actually leak?

Into the image you ship, or into the metadata, history and logs that travel alongside it.

A private Maven repository, a Git dependency over SSH, an artifact in S3: each one needs a secret inside docker build. A discarded build stage helps, since the image you ship isn’t the image that saw the credential, but “it gets thrown away” is a thin guarantee to hang a production key on.

Docker has been closing the gap. With BuildKit you can mount a secret into a RUN step with --secret, or forward your SSH agent socket with --ssh so the build can authenticate to an SSH host. Either way the value is available for the length of that one command and the build does not write it into a layer. That is temporary exposure rather than containment: what the command does with the secret is yours to check, and a script that copies it into the workspace or echoes it into the build log has put it straight back.

Build arguments and environment variables are the wrong tool, for two different reasons. An ENV value persists into containers started from the image, where anyone who can run it reads it back. An ARG is not present in those containers, but Docker’s documentation warns that build arguments “may persist in the image metadata” and “in the image history”, and build output is one more place they surface. Docker’s verdict on both is flat: they are “inappropriate for passing secrets”. The --secret syntax still wants BuildKit enabled and the experimental frontend, so check what your Docker version supports before you design around it.

Whatever route you pick, pass credentials that expire. An AWS STS session can be issued for as little as 15 minutes. Pick the shortest duration that still covers the steps of the build that need the credential. That shrinks the window in which a leaked value is worth anything, and does nothing inside it: until the session expires the credential works as well for an attacker as for your build. Treat a leak as a leak rather than waiting the clock out.

Runtime secrets are a different problem with a better answer: keep them out of the image. We encrypt ours with KMS, store them in a bucket, and let the container fetch and decrypt them at boot, which costs a few seconds of startup. The hard part isn’t the storage. It’s the access control, and IAM is the piece of a cloud provider you can’t take with you.

What this means for teams

None of this is a product you buy. It’s a set of defaults: a small number of base images you control, a runtime image carrying as little as possible, scans at three different layers, and a rebuild cadence that assumes today’s clean report has a shelf life.

The machine that builds those images deserves the same attention and usually gets less. It holds credentials for your registry, your cloud and your source control, and whatever it injects during a build is signed and trusted by everything downstream. Treat it as production. Create build agents and tear them down rather than keeping a long-lived box that quietly accumulates access, which is cheaper too, since nobody is building at three in the morning.

This week:

• Split one service’s Dockerfile into a build stage and a runtime stage, then compare docker images before and after.
• Turn on scan-on-push in your registry and read the findings for the three base images you use most.
• Schedule a recurring registry scan, then check the digests it covered against what your cluster is running right now.
• Replace a tag with a digest in whatever manifest production deploys, and see what breaks.

Common questions, answered

How do I pass a private registry token into a Docker build without it ending up in the image?

Mount it with BuildKit’s --secret flag and read it inside a RUN --mount=type=secret step, so the value is exposed for that one command instead of baked into a layer. For a Git dependency over SSH, --ssh forwards your agent socket instead. Check what the command does with the secret: a step that copies it into the workspace or prints it to the log undoes the mount. Don’t fall back on ARG or ENV: an ENV value persists into containers started from the image, and an ARG, absent from those containers, can persist in image metadata and build history.

How do I make sure my cluster runs exactly the image I built?

Reference the image by digest rather than by tag in whatever you deploy, and compare the digest your build produced with the one the cluster reports running. A tag is a pointer anyone with push rights can move; a digest is the content itself. Behind that, turn on immutable tags if your registry supports them, and let only the build pipeline push and only the cluster pull. Signature verification is worth adding on top: a registry you control still serves whatever somebody with push rights put there.

How do I find out whether an image already running in production has a new CVE?

Rescan the images in your registry on a schedule, then match what you scanned against the digests your cluster reports. Most registries expose scanning through an API, so the scheduled job is small work; the matching is the step teams skip. An image scan stops at the image, so anything installed at runtime or mounted in stays invisible to it. Decide in advance what a finding triggers: a base image rebuild and a rollout, not a ticket that ages.

Related episodes

• Infrastructure as Code: the argument for machines you re-create instead of repair, which is also the argument for throwaway build agents.
• GitOps: how the image you just built reaches a cluster, and who is allowed to put it there.

Resources

• Docker Content Trust: what signing a tag covers, and what DOCKER_CONTENT_TRUST=1 refuses to run.
• Docker security: namespaces as the first form of isolation, and where the defaults leave gaps.
• Docker security non-events: kernel CVEs the default profile blocked for containers, and the ones it did not.
• Docker Official Images: who reviews, publishes and maintains them, and why upstream authorship isn’t guaranteed.
• Multi-stage builds: the COPY --from pattern that keeps build tooling out of the runtime image.
• Build secrets with BuildKit: --secret and --ssh, and why build arguments are not a substitute.
• DockerSlim: what it strips, and what you have to keep on purpose.
• distroless: base images with no package manager and no shell.

Since we recorded

DockerSlim kept the job and changed the name: it now lives as Slim at slimtoolkit/slim. Docker announced the retirement of Docker Content Trust in July 2025, starting with Docker Official Images, and told publishers who sign with it to plan a move to something else, naming Sigstore and the Notary Project’s Notation; that post says the timelines for complete deprecation “are being finalized and will be published soon”. Docker Engine 23.0 set Buildx and BuildKit as the default builder on Linux in February 2023, so docker build reaches those secret mounts without DOCKER_BUILDKIT=1. We went back over what else changed in episode 109, “Docker Got Quiet. Did You Miss Anything?”.