Day one, and nobody can tell you how to build the thing

You arrive at a new project. Before you can change a single line, you need the toolchain at the versions this code expects, the environment variables that get you into the right cloud account, and some way to run the thing without breaking your machine. Almost none of that is written down. It lives in one engineer’s head, in a README that stopped being true two refactors ago, or in a Slack thread from last March.

The cost shows up fast. Run terraform apply with a newer binary than your colleagues and the state snapshot can come back in a format their version refuses to read, which blocks everyone on the older release until they upgrade. HashiCorp says as much: later releases “may introduce new features that earlier versions cannot understand, including new storage formats for Terraform state snapshots”, and a downgrade isn’t guaranteed to work. That is a risk rather than a certainty, which is part of why it gets found late. Most of our customers are early stage startups, where development environment consistency is the last problem anyone has time for, so we bump into it unsolved on almost every engagement.

We went looking for the standard way companies solve this, and there isn’t one. There’s a short list of approaches, each with a bill attached: your bare laptop, a virtual machine, containers, somebody else’s computer, or a build system that owns the whole thing. Picking gets easier once you know what each one buys.

The repeatable build we had, and then gave up

Start with the strongest version of what good looks like, because it existed before most of our current tools did.

How did a five-year-old release stay buildable?

Because the compilers lived in version control next to the code, and the same selection that gave you the source gave you the toolchain. At Ericsson, before Git, that was ClearCase. A team prepared a view, you switched to it, and you got a specific version of the files plus the tools and scripts that went with them. Everything was in there, compiler toolchains included. You could go back five years, select that view, and recompile the whole thing from scratch.

That is still an impressive feat of engineering, and it took a lot of work from some smart people. A config spec is a list of rules evaluated in order, one per view, and a dynamic view resolves each file to a version as something reads it. Then we moved to Git and ruined all of it. Builds stopped being that repeatable.

The reason to care was commercial rather than aesthetic. In telecom you carry obligations to customers: you support a given version, and if it needs a patch you have to be able to compile that version. Your toolchain vendor might go out of business while you still owe someone a fix.

Step out of embedded into web development and the picture inverts. What works today breaks tomorrow because a dependency changed somewhere, or got deleted from GitHub. Bazel is the closest thing to a deliberate reconstruction of the old guarantee. Its hermeticity model treats tools as source and downloads them into file trees it manages. The same page names the usual leaks: host binaries in /usr/bin, absolute paths, and the system C++ compiler that native rules autoconfigure.

Pin the version, or the state file picks for you

ClearCase solved version selection for a whole organisation. Without it, you’re pinning versions one tool at a time, and the gaps are where the day goes.

What actually breaks when two people run different versions?

The person on the newer binary can write something the older binary refuses to read, and finding that out is somebody’s afternoon. Terraform state is the sharpest example, and the same shape turns up in a Go release a module needs or a runtime a project was written against.

The failure has a predictable script. A new developer joins, runs the setup script, and gets the latest of everything. The senior developer is two versions back. The new person says it doesn’t work, the senior says it works here, and the gap between those two statements is the afternoon. A build script can pin versions, and it can also just say latest, which is pleasant on the day you write it and wrong the next morning.

So write the version down somewhere the tooling reads. An install command that names an exact version does pin that installation, and a build file that lists its dependencies by version pins those. What neither one does is decide which binary runs six months later, on a machine that already has two of them on PATH. That takes a check at the point of use. Terraform’s required_version in the terraform block errors out before doing anything when the CLI doesn’t match. A .tool-versions file applies to the directory it sits in and everything below it, so the repository, not the laptop, decides which Go or Python runs.

The other half is environment rather than tools: cloud credentials, build parameters, which account you point at. For one of our more mature customers we built an environment-aware tool with another engineer. Say you want to be in dev, and it sets the variables for dev, downloads the tools, puts them where PATH finds them and keeps them updated. We built it in house because we hadn’t found an open source project doing that whole job for them, and we’re curious how other people approach it. The pieces are out there to assemble: direnv sets environment variables per directory, asdf pins tool versions per directory, and the asdf-direnv plugin joins the two, so entering a directory puts the pinned binaries on PATH without going through shims.

Three ways to keep your laptop out of it

Pinning tells you which versions to install. Where you install them is the argument.

Which isolation do you actually need?

The one that matches how often you throw the environment away. We split three ways on this, and all three work.

Mattias Dockerizes everything. No language runtimes on the laptop at all, not even pip, despite writing a lot of Python, and a template Dockerfile per project. Andrey runs an Ubuntu virtual machine in VMware instead, because macOS isn’t Linux and comes with its own shell quirks, and because customer tooling installed on the host stays there and gets hard to delete later. Julien installs straight onto the machine and automates nothing: a bin directory in the home directory, the version number in the binary name when two are needed, and no hesitation about resetting the lot. Java, Go, Python and Scala all turn up depending on the two weeks in front of them, so the brutally raw setup is the fast one. Most people program in one or two languages and should weigh it differently.

Automation earns its keep by frequency, which is why none of these are wrong. One of us rebuilds a machine maybe once or twice a year, and the estimate there is that the hours spent automating it would run past the hours it saves, so there’s no return on that investment. The substitute is a gist of commands and notes: Git config, generating a signing key, registering it with GitHub or GitLab, a line added whenever a new tool turns out to be useful. It gets updated every time it gets used. Where that estimate flips is a question of how long the automation takes to write, how many people and machines run it, and how much upkeep it needs as the tools move underneath it.

Virtual machines were the packaged answer to this, and Vagrant is still positioned as a development environment tool. It earned its place on projects nothing else could resurrect, an old Ubuntu with PHP 4.3 that will not install on a current Mac. The friction is size: images run to gigabytes, and every update to the base image means pulling it again.

When your laptop stops being the right computer

Isolation keeps the mess contained. It does nothing about the workload being bigger than the hardware.

What do you do when the stack won’t fit?

Move the environment to a machine that fits it. One of our projects needed Kafka, multiple consumers and producers, and Elasticsearch. Starting that Docker Compose stack overwhelmed the laptop. Put Ubuntu on a server with 40 cores and 30 GB of RAM and connect VS Code over SSH instead. The work happens there; your editor stays where it is.

The managed version of that idea is AWS Cloud9: a browser IDE and a terminal on a Linux machine, in one of two shapes. In an EC2 environment, Cloud9 creates the instance, runs it on Amazon Linux or Ubuntu Server, and owns start, stop and terminate. An SSH environment points at a server you already run, so a Packer-baked AMI carrying your tools belongs there, and its lifecycle stays yours.

The shutdown rules follow that split. Cloud9 stops an EC2 environment’s instance after the inactivity window you set, 30 minutes in AWS’s own pricing example, and terminates it when you delete the environment. Delete an SSH environment and the server stays, still yours to stop. Cloud9 adds no charge of its own: the bill is EC2 compute while the instance runs, plus the EBS volume for as long as it stays provisioned. We wrote scripts to preconfigure one, tried to force the habit, and went back to our own setup anyway.

GitHub Codespaces reached general availability for Team and Enterprise Cloud in August 2021, and pressing . on any repository opens the browser editor. GitHub’s engineering team published numbers when they moved: a 13 GB repository, about half a day for a new hire to get a local instance running, and codespace creation down from 45 minutes to five, or ten seconds with prebuilds. You pay while you use it, and the laptop stops deciding what you can run.

Local Kubernetes is the same story in miniature. minikube asks for 2 CPUs and 2 GB of free memory, and Mattias reaches for it for operator work. Julien’s answer: the cluster was never the problem. An operator needs a database and the services it talks to, and that stack is what borked the laptop. Julien doesn’t recommend trying it under 32 GB of RAM, and separately has watched people fill 16 GB with a browser. A small cluster in the cloud you can deploy to without disturbing anyone is the faster way.

Scripts, Makefiles, and the thing that pulls its own internals

Underneath every one of these setups is the same problem: dependencies, and not only the ones in your lock file. The database, the queue, the three services yours calls, and whatever ties them together when you type one command.

Some of those you cannot run at all, locally or in a cluster. An emulator is the stopgap, and it tells you its own limits. Google’s Pub/Sub emulator warns at startup that the implementation may be incomplete or differ from production. Messages it holds survive only for the lifetime of the session. It buys a fast loop, not a verdict.

Is a folder of Bash scripts enough?

Often, yes, and this is where we split. Julien adds a scripts folder to every project worth improving, holding build, run and an install step for dependencies, rarely more than 20 lines each, so a newcomer executes one script and doesn’t have to understand anything yet. Andrey’s view is that people put Makefiles everywhere, that the Makefile ends up managing environment variables and everything else, and that in 2021 it should be a solved choice: go to something with a book written about it, Gradle or Bazel. Those tools are genuinely complex, and Andrey dislikes them for it.

Two shapes hold a multi-service project together once scripts stop scaling. One is a main repository with the rest wired in as Git submodules and started with Docker Compose: pull the code to the correct version and build from there. The other is a monorepo built by Bazel, with infrastructure code, Kubernetes manifests and application code in one tree.

Both positions point at one property worth separating from the tooling fight. A build tool keeps the what in the repository and the how somewhere else: the repository holds a recipe, the tool brings its own internals and applies what your file asked for. Gradle’s wrapper is the concrete version. The version lives in gradle/wrapper/gradle-wrapper.properties, and the wrapper scripts are committed alongside it. You can check the downloaded distribution against a SHA-256 you set. A newcomer runs ./gradlew build without installing Gradle separately, and gets the version the project decided on. What the wrapper fetches is Gradle, not a JVM: Gradle’s installation documentation lists a Java Development Kit as a prerequisite, so the JDK is one more version somebody has to agree on.

One language for configuration you can validate

Build scripts are half the equation. The other half is the deployment and the manifests, and that half is mostly YAML nobody checks until something consumes it.

Can configuration be validated before it runs?

Yes, and CUE is the interesting attempt because it moves the constraints into the configuration language itself. The name stands for configure, unify, execute. Its own documentation calls it “an open-source data validation language and inference engine with its roots in logic programming”, and says plainly that “the language is not a general-purpose programming language”. You define configuration in it and it produces JSON or YAML. It can import a schema and check that what you pass matches, including conditions on values: this number must be above or below some amount.

Calling it protobuf for configuration is close enough to be useful. You can import a protobuf message schema and have it check against that, and you can generate OpenAPI documentation from the same definitions.

The contrast with YAML is narrower than it sounds. A YAML file carries no application schema of its own, so nothing in it says which fields your service requires or what range a number has to sit in. Checking ahead of time was already possible elsewhere: Kubernetes shipped server-side dry run and kubectl diff as a beta in 1.13, which push a manifest through defaulting, validation and the admission chain and hand back the result without persisting it. What CUE changes is where the constraints live, beside the configuration rather than in the system receiving it.

Dagger is where this shows up in a pipeline. It comes from people who built Docker and went off to start their own company. The idea is one tool for the whole run: you build with it, you run with it, you deploy with it. You describe all of that in CUE instead of a per-vendor YAML dialect. The syntax is short, less verbose than JSON, and what you pass through it gets validated rather than discovered at runtime.

We wouldn’t call it production ready, although some people already run it in production. It’s worth an afternoon of your time, and we see a lot of potential in that direction.

Write the scripts for yourself in six months

All of this lands on one question a project can be judged by, and it isn’t a question about tooling.

How easy should it be for a new person to run your project?

Easy enough that they get there without you, and the person to optimise for is yourself in six months. Write down how to build it, how to run it and how to deploy it, not as a favour to others but because you will have forgotten. Have empathy for the new user, because that new user can be you. Anyone who has inherited a project where nothing was written down knows what the alternative costs, and it isn’t a good use of anybody’s time. The same holds for an open source side project touched a few times a year: the build and test scripts are there to remind you how you did it last time.

That makes it measurable, in the way test coverage is.

Two honest caveats. Opening the pull request isn’t the slow part: clone, push, done. Build and test is the question, and CI is what answers it, since you push and the pipeline tells you how good it was. The second is that the measure assumes nobody games it, which is the wrong assumption to make about any metric.

There’s a cheaper habit underneath it. Delete your environment regularly, every Monday if you can stand it, and rebuild. After a while the script you rebuild from is well thought through, because you do it often enough to care. That’s frequency reducing difficulty, the argument continuous integration won with: pain grows with the gap between repetitions, so shorten the gap.

What this means for teams

We didn’t find a standard solution, and we don’t think one exists yet. What does exist is a clear split: your local environment, the way you build in CI and the way you deploy are three projects wearing one name, and dependency management isn’t solved in any language we work in. Expect hurdles. Everything has a solution once the problem is defined clearly. At the early stage startups we work with, it is quite often still unsolved when we arrive.

This week:

• Time a newcomer, or yourself on a borrowed machine, from git clone to a green build. Write the number in the README.
• Put every tool version in a file the tooling reads, a .tool-versions or a required_version, and delete the words “install the latest” from your setup docs.
• Add a scripts folder with build, run and an install step, under 20 lines each, before you argue about which build system to adopt.
• Find the one service your team can’t run locally, check whether an emulator exists for it, and write down what the emulator won’t tell you.

Common questions, answered

How do I stop a colleague’s Terraform upgrade from blocking everyone?

Add required_version to the terraform block and commit it, so a mismatched CLI errors out before it touches anything. The risk runs one way: a newer Terraform can write a state snapshot in a format older versions cannot read, and HashiCorp’s compatibility promises don’t guarantee a downgrade works. Pair it with a .tool-versions file or an equivalent, so the repository rather than each laptop decides which binary runs. Then upgrade deliberately, as one change everyone picks up.

How do I develop against a Kubernetes operator without killing my laptop?

Run the cluster somewhere else. minikube itself is modest, 2 CPUs and 2 GB of free memory, but an operator needs the database and services it integrates with, and in our experience that whole stack is what borked the laptop it ran on. We wouldn’t try it under 32 GB of RAM, and we’ve watched people fill 16 GB with a browser. A small cloud cluster you can deploy to without disturbing anyone usually beats fighting for memory locally. If you’d rather keep your own editor, move the compute: a remote host over SSH with the editor attached behaves like a local setup, sized for the job.

Is it worth automating my machine setup if I rebuild it twice a year?

Weigh the hours it takes to write and maintain against the hours it saves everyone who runs it. For one of us, rebuilding once or twice a year, that estimate comes out against automating: a gist of commands and notes, updated on every run, does the job. The arithmetic moves as the setup gets reused: more machines, more people, more rebuilds, and a script starts earning back, if somebody maintains it. Either way, write the steps down: the test is whether someone else could follow them without asking you a question.

Related episodes

• Best Practices for Building Docker Images: making the command you run locally the command CI runs, and multi-stage builds that carry the build environment with the code.
• Platform teams with Henrik: one declarative file per repository, shared build plans, and why every generated step needs an opt-out.
• Terraform in CI: the state file, why it’s the first thing to get right, and who gets to run an apply.
• How Do We Run Kubernetes In The Cloud?: what a cluster costs to operate, before you try to fit one on a laptop.

Resources

• CUE: an introduction: what CUE is in its own words, including that it is not a general-purpose programming language.
• APIServer dry-run and kubectl diff: what a server-side dry run checks, and what it deliberately does not persist.
• Bazel: hermeticity: what a hermetic build guarantees, and the host dependencies that quietly break it.
• The Gradle Wrapper: where the pinned version lives, what to commit, and how to verify the distribution.
• GitHub’s engineering team has moved to Codespaces: the August 2021 numbers for a 13 GB repository, from half-day local setups to ten-second prebuilt environments.
• Terraform v1.0 compatibility promises: what upgrading guarantees, and why downgrading across a state format change isn’t promised.
• asdf configuration: how .tool-versions applies versions to a directory and everything under it.
• EC2 environments compared with SSH environments: which Cloud9 workflow starts, stops and terminates the instance for you, and which leaves all of that to you.

Since we recorded

Dagger shipped SDKs for languages people already use, then ended support for its CUE SDK in December 2023, saying usage had dropped steeply and engineers would rather write pipelines in a language they already know. AWS closed new customer access to Cloud9 on 25 July 2024, pointing people at its IDE toolkits and CloudShell instead. We came back to the newcomer question in Backstage and Internal Development Platforms.