The build that works on your laptop has to run somewhere else

Building an image on your own machine is one command. You type docker build, the daemon on your laptop does the work, and you get an image. We don’t ship what comes out of that. Our production images get built in CI, and CI itself runs in containers, so the build you tested by hand has to happen inside a container that has no Docker daemon of its own.

That single constraint decides more than it looks like it should. It decides whether you hand a build job access to the machine it runs on, whether the Dockerfile that works on your laptop works in the pipeline, and which Docker features you can use when a dependency needs a password. Get it wrong and the failure arrives as “it builds on my machine”, at the worst possible moment, with nothing in the logs to explain it.

Building a container from inside a container

Start with where the build runs, because that choice limits what the Dockerfile can do.

What does a build need when the container it runs in has no daemon?

Either a Docker engine it can reach and somebody keeps alive, or a builder that works in user space.

Three routes: point the build container at a remote Docker engine, mount the host’s /var/run/docker.sock so it talks to the daemon on the node, or drop the daemon entirely. The socket is the easy one and the dangerous one: whatever can talk to it sees every image and container on that node, and can start more.

We’ve gone both ways, because the environments differ. On Kubernetes you can mount the socket and keep builds on a node pool that runs nothing else, which takes your other workloads off the machines a build can reach. That cuts the exposure without drawing a boundary: Docker’s security documentation says “only trusted users should be allowed to control your Docker daemon”, because the daemon will start a container with your host’s / mounted inside it. On Fargate there’s no host to borrow a daemon from, and a remote engine is one more thing to keep patched, so a user space builder adds the least, if it runs unprivileged: privileged is not a valid parameter in a Fargate task definition, which rules out img while it still asks for one.

Run an engine for builds and Docker 19.03 changed the setup under you. The docker:dind image generates its own TLS certificates and requires them, a change that broke GitLab users’ jobs. A job that connected over plain TCP fails with Cannot connect to the Docker daemon at tcp://docker:2375, and the shared certificates fix half of that. The client also has to leave the plaintext port: Docker’s instructions are that “Docker over TLS should run on TCP port 2376”, with DOCKER_TLS_VERIFY on and DOCKER_CERT_PATH set to the client subdirectory of DOCKER_TLS_CERTDIR. Set those three, or let the official docker image’s entrypoint set them once the client certificates are there.

The user space builders are the other branch. img describes itself as a “standalone, daemon-less, unprivileged Dockerfile and OCI compatible container image builder”, though it still wants a privileged container. kaniko “doesn’t depend on a Docker daemon and executes each command within a Dockerfile completely in userspace”, and is meant to run as a pod in a cluster. Buildah runs no daemon and builds OCI images “while not requiring any root privileges”. We’ve run img in anger, and it does the job.

They all read a Dockerfile, and that’s where the similarity stops

Reading the same file is not the same as behaving the same way.

Why does a Dockerfile that builds locally fail in the pipeline?

Because the thing building it isn’t the same program. img uses BuildKit’s solver, kaniko runs the instructions itself in user space, and Buildah has its own stack and its own image store.

The feature gaps are the visible part. img doesn’t carry everything the BuildKit Dockerfile frontend offers, and the gaps we hit are the ones that matter here: secret mounts and SSH agent forwarding. We worked around them rather than switch builders.

The invisible part costs more. Build one project with Docker and another with Buildah on the same machine and you get two ecosystems: docker images won’t list what Buildah built, buildah images won’t list what Docker built, and each keeps its own cache. Every script that assumed one of them breaks. The other version is building locally under “it builds, so it builds”, pushing, and watching kaniko fail on the same Dockerfile.

So pick one, stay on it, and make the command you run locally the command CI runs.

Multi-stage builds put the build environment in the repository

With the builder settled, the biggest decision left in the file is how many stages it has.

Where should your build environment live?

In the first stage of the Dockerfile, next to the code it builds.

Before multi-stage, we watched teams package a JavaScript application outside Docker and copy the bundle in, or compile a binary on a CI agent and copy that in. Both work. Both mean a build environment you maintain separately from the Dockerfile, plus the copying of artifacts, plus the drift between the agent and your laptop.

Multi-stage builds fold all of that into docker build. The first stage carries the JDK, the Maven cache, the Node toolchain, whatever your build needs, and the source. The second stage starts FROM something small and takes only the artifact across with COPY --from. Dependencies installed for the build are dropped with the stage that installed them. Nobody installs four language toolchains on a laptop to patch one service, and the build runs the same way in CI as on your machine.

Credentials the build needs and the image must not keep

The stage boundary earns its keep again the moment your dependencies stop being public.

How do you keep a build credential out of the image you ship?

Mount it into the one RUN that needs it with BuildKit, and keep it out of ARG, ENV and COPY.

A private Maven repository, Go modules from a private Git host, an artifact in a bucket: each needs a credential inside docker build, and before BuildKit the common routes all left traces. You passed a build argument, set an environment variable from one, or copied in a file holding the credential and read it during the build. BuildKit changed that. Its Dockerfile frontend takes RUN --mount=type=secret with docker build --secret, so the build container can “access secure files such as private keys without baking them into the image”, and --mount=type=ssh with --ssh, which forwards your agent socket for the length of one command. That last one is what pulling from a private repository over SSH actually needs.

Both mounts are experimental, so you opt in twice: # syntax=docker/dockerfile:experimental on the first line of the Dockerfile, and DOCKER_BUILDKIT=1 in front of docker build. Miss either and the --mount line is a syntax error.

Doing it in a discarded stage helps, because the image you ship isn’t the image that saw the credential. It isn’t a guarantee you should take on trust.

So check before you push, and know what each check covers. docker history --no-trunc prints the build instructions in full rather than the truncated default, which is how you catch a credential passed as ARG or spelled out in a RUN. That is the history of the image, not its contents, so it will not find a key that a COPY dropped into a file: for that, export the image with docker save and read the layers, and read docker inspect for the environment that ships with the image.

Where secret mounts aren’t available, and on some builders they still aren’t, the fallback is credentials that expire: issue a session that covers the build and nothing more. Expiration limits how long a leaked credential remains usable, and where your issuer supports it, revoke it when the build finishes.

The rest of the image is distribution you didn’t ask for

Once the build tools are gone, look at what’s still in there.

How much of the operating system does your production container need?

A runtime and its libraries. The shell, the package manager and the rest arrived because the Dockerfile started FROM ubuntu.

Google’s distroless images are the other end of that: your application and its runtime dependencies, without the package manager, the shell, or the programs you’d expect to find in a Linux distribution. The way you use them is the multi-stage pattern again. First stage from a full base, build the artifact, then FROM a distroless image and copy the artifact across.

Two things improve at once. Storage, which stops being theoretical when every commit pushes an image somewhere. And the inventory an attacker finds after landing in the container: no shell, no package manager, and fewer shipped packages to hunt a known bug in. The application and the runtime it needs are still there, and a runtime is a way to execute code, so this takes the familiar tools away rather than the ability to use the container.

The compromise is tooling, and it’s a compromise rather than a wall. Nothing stops you copying a compatible binary in with the same COPY --from you already use, and distroless publishes :debug variants that “provide a busybox shell to enter”. What’s gone is the package manager, so there’s nothing in the image to install extra packages with at runtime. For a monitoring agent that doesn’t have to live in the application’s image, a sidecar is the option Kubernetes gives you: containers in a pod share the network namespace, so a tool in the same pod reaches the application over localhost.

For an application you can’t rebuild that way, DockerSlim takes an image you already have and strips what it doesn’t use. Either route, test the stripped image like a release: what loads at runtime is easy to remove by accident.

An image built in March is a March image

A lean image still carries the packages that were current on the day it was built.

What patches a container you deployed six months ago?

Another build, and then a redeploy. Nothing inside the container updates it in place.

The old shape of this was an ops team keeping a long-lived Apache server patched with Ansible, with the application dropped on top. Containers weld the server and the application into one artifact, so the patch gets produced upstream and shipped like a release.

Putting that rebuild on a schedule is necessary and not enough, because the build cache will hand you last month’s packages with today’s date on the tag. Docker’s cache rules start at the base image, and “once the cache is invalidated, all subsequent Dockerfile commands generate new images”. The flip side is that an unchanged RUN apt-get line matches its cached layer, and a FROM that resolves to the image already on the builder invalidates nothing. So ask for the refresh: Docker’s guidance is to rebuild your images often with --pull, which fetches a newer base instead of reusing the cached one, and --no-cache, which re-runs every step so the package install picks up current versions.

Those same rules decide what the cache can still save. When the base moves, every instruction after FROM in that stage is rebuilt, and no ordering prevents it. What survives is the work that doesn’t depend on the changed base, one more argument for keeping the build stage and the runtime stage on bases of their own.

The redeploy is where this usually stalls. imagePullPolicy: Always makes the runtime resolve the image reference against the registry every time a container starts, which helps when that reference is a tag somebody moved. It does nothing for the date-and-commit tags in the next section: the workload still names the old image, and the kubelet pulls exactly what you asked for. Something has to write the new tag into the pod template, because a Deployment’s rollout “is triggered if and only if the Deployment’s Pod template (that is, .spec.template) is changed”.

Pin the base, and make the bump a commit

Scheduled rebuilds have an obvious hazard. The thing you’re rebuilding against moves too.

What breaks when you leave FROM ubuntu untagged?

Whatever the next release renamed. We had an untagged ubuntu base move from 18.04 to 20.04 under a pipeline, where python-pip became python3-pip and every build failed until somebody edited the Dockerfile.

Nothing was wrong with the new release. Ubuntu’s announcement for 20.04 said plainly that it “includes Python 3.8 as the default version but Python 2.7 is still available in the Ubuntu archive”. The mistake was ours: we’d asked for whatever was newest and got it. Pin to ubuntu:18.04 and an LTS stays under you until you decide to move.

You can take that further, and we’d argue you should. Version the base image you build yourself, reference that version from the application Dockerfile, and let the weekly base build produce an automated commit that bumps the reference. The version bump becomes a commit like any other, so the history records which base each image was built on, and the commit is what triggers the application build. One pipeline, one trigger, no separate scheduler to maintain.

The objection is fair: that’s a stream of commits no human wrote, in a repository people read. What you get is traceability, a commit behind every image that says which base went in, which is the question you’ll be trying to answer later.

Traceability is not reproducibility. A tag can be repointed at a different image after you recorded it, and the packages a RUN apt-get installs move as well, so rebuilding the same commit next year need not produce the same image. If you want the base to be the same bytes every time, record its digest: Docker’s advice is that “you can pin the image version to a specific digest”, and pinning your dependencies and the rest of the build inputs is the other half of that job.

Tags are what you’ll have at 3am

By now you’re producing images faster than anyone can remember what went into them, and the tag is what you read off the screen when you list what’s running.

What should a tag tell you when something is broken?

Where the code came from and when it was built: a reverse date, then the short Git hash of the commit it was built from.

Start with what latest means, which is less than people assume. It’s the tag Docker uses when you don’t name one, a convention rather than a statement about what is newest. The copy cached on your laptop is whatever the registry served the last time you pulled, and the tag in the registry may point somewhere else by now. So latest on your laptop and latest in the cluster can be two different images wearing the same name, which is how “it works on my machine” happens to people who are telling the truth. We’ve watched engineers debug the wrong code for hours because the container wasn’t built from what they were reading.

Reverse date first, year then month then day, because that’s the format that sorts. Add hours and minutes: a few builds a day is normal, the Git hashes already tell those apart, and a build pushed at 03:40 that broke on arrival says something the hash doesn’t. A Unix timestamp sorts as well and reads worse, so pick your poison.

There is another way to identify what’s running. A pod reports the image digest, “a content-addressable identifier” that names those exact bytes whatever happens to the tags afterwards. Turning a digest back into a commit means a lookup somewhere else, unless the image answers for itself: the OCI image specification defines org.opencontainers.image.revision for the source revision and org.opencontainers.image.created for the build time, and a LABEL that sets them comes back out of docker inspect. Set them: one line in the Dockerfile, and they survive a retag.

The tag earns its place by being the field a pod listing already prints: read the dates off the tags, and anything still running from last quarter is visible without inspecting an image.

Where your images live is becoming a bill

Rebuilding on a schedule means pushing on a schedule, and in the past six weeks the economics of where those pushes land changed twice.

What happens to the images you pushed and forgot?

On Docker Hub’s free tier, they’re scheduled for deletion. Docker’s roadmap entry, filed on 31 August 2020, defines inactive images as “images that have not been pulled or pushed in 6 months”, says free accounts may keep them for up to six months, and sets the start date at 1 November 2020. Pro and Team subscriptions keep theirs indefinitely.

Rebuild and redeploy on a schedule and almost nothing you depend on is inactive by that definition. The exposure is elsewhere: the old public image a project pinned three years ago, the build you kept because a customer might come back. A company with a bandwidth bill charging for storage isn’t a scandal. It is, we suspect, another nail in the coffin for Docker Hub as the default.

A day later, GitHub announced its Container Registry in beta, “free for public images”, with anonymous pulls and registry permissions split from repository permissions. Storage on its own isn’t a product any more. Whoever scans what you push, and tells you what’s in it, takes this market.

Inside a company this mostly isn’t the question: you’re on ECR, Harbor or a registry you host yourself. It’s open source projects, and the public images the rest of us pull without thinking, that have to move.

What this means for teams

None of this is exotic. It’s the difference between a pipeline that produces images and a pipeline that produces images you can explain six months later.

Two mottos, both short. Keep your images lean, because small images are easier to deal with in every direction: storage, pull time, patching, and the amount of trouble available to anyone who gets inside one. Automate and stay secure, because every control here, the rebuild, the redeploy, the version bump, stops working the moment it depends on somebody remembering.

This week:

• Run docker history --no-trunc on the image you deployed most recently, then unpack a layer or two and read docker inspect.
• Check whether your laptop and your pipeline use the same builder. If they don’t, fix that, or write down what differs and where it’ll bite.
• Rebuild one image with --pull --no-cache, compare its package versions against the copy you’re running, then follow the new image to a pod. If nothing updates the image reference in the workload, the rebuild never reaches production.
• Retag one service with a reverse date and the short commit hash, set org.opencontainers.image.revision while you’re in the Dockerfile, then trace a running pod back to its commit in under a minute.

Common questions, answered

How do I give docker build access to a private Git repository over SSH?

Forward your SSH agent instead of putting a key in the build: make # syntax=docker/dockerfile:experimental the first line of the Dockerfile, add RUN --mount=type=ssh to the step that fetches the dependency, and run DOCKER_BUILDKIT=1 docker build --ssh default .. The key never enters the build context, and the socket is there for that one command only.

Don’t COPY a deploy key in and delete it later: the layer that added it keeps it, and anyone who pulls the image reads it back. Build arguments and environment variables are the same trap. If your builder has no SSH forwarding, use a credential with a short expiry.

How do I make my production image smaller without breaking the application?

Split the Dockerfile into stages and start the last one from a minimal base: build in a stage that carries the toolchain, then FROM a distroless image and COPY --from only the artifact. What ships is the application and its runtime dependencies, without the shell and the package manager.

If you can’t restructure the build, DockerSlim strips an image you already have down to what it uses. Either way, test the result like a release: a file that only loads at runtime is easy to remove by accident.

How do I tell which commit a running container was built from?

Read it out of the image tag, which is why the short commit hash belongs there. A tag like 20201012-1830-a1b2c3d answers it in one look, with no build system to log into.

If the tag is latest, run docker inspect: a build that set the org.opencontainers.image.revision label put the commit inside the image. Without it, you’re matching the pod’s digest against your build records, which answers only as far as those records go.

Related episodes

• Docker Secure Build: signing, scanning and where a credential leaks, which is the security half of the same build.
• Docker Runtime Security: what a small image buys once the container is running, and where container isolation stops.
• SemVer or Not to SemVer: the versioning argument behind the tag scheme, including what a commit hash in a tag is for.
• GitOps: the deployment half of automated patching, where an agent in the cluster notices the new image instead of a pipeline pushing it.

Resources

• BuildKit Dockerfile experimental syntax: the RUN --mount=type=secret and type=ssh syntax and the # syntax line that enables it, pinned to the revision current when we recorded.
• img: a standalone, daemonless builder, and what it asks of the container you run it in.
• kaniko: daemonless builds designed to run as a pod, and the README’s own warning that it doesn’t make untrusted builds safe by itself.
• Buildah: building OCI images without a daemon and without root, with its own image store.
• Docker daemon attack surface: what control of the Docker socket is worth to whoever has it, in Docker’s own words.
• distroless: what “only your application and its runtime dependencies” leaves out, per language, and the :debug variants.
• Introducing GitHub Container Registry: what the beta offers for public images, and how its permissions differ from repository permissions.
• Docker roadmap: inactive image retention policy: the exact definition of an inactive image, the six month window, and the 1 November 2020 start date.

Since we recorded

The deletions never arrived on the announced date: Docker delayed the retention policy to mid-2021, while the pull rate limits in the same announcement went ahead and are still how Docker Hub meters free use. GitHub’s registry went generally available in June 2021, and BuildKit stopped being the thing you opt into: Docker Engine 23.0 made it the default builder on Linux, so --secret and --ssh are ordinary flags and the # syntax line is no longer needed for them. The builder field moved too, with kaniko archived in June 2025 and DockerSlim renamed to SlimToolkit, a CNCF sandbox project since May 2023. We went back over what changed in Docker while nobody was looking in episode 109.