What your repository says and what your account runs

Production breaks at night. Someone with console access clicks the button that fixes it, the incident gets closed, and nobody writes the change down. Your templates still describe the account as it was before the outage, so the next time the pipeline applies them, the outage can come back with them.

That is the problem worth solving first, and it has nothing to do with which tool you picked. The thing most teams actually want from infrastructure as code isn’t automation. It’s an answer to “what am I running?” A repository you can read and diff gives you that: every change, the person who made it, and the pull request where someone asked why. After that, a database and an application go through the same review and the same pipeline, and the cloud stops being a separate place with separate rules.

The code also buys you less than the brochures suggest. It gives you no failover, makes nothing immutable on its own, and stops nobody from clicking in the console at 3am. What it does is make the gap between intent and reality visible, and that gap is the part you can work on.

Provisioning starts with an interface you can call

Before you decide where to run your first template, there’s a question about what you are able to ask for at all.

What decides whether you can provision infrastructure from code?

Whether the thing you want to create can be driven over a programmable interface. The tool is the last decision, not the first.

Think about what a sysadmin had before the cloud. You edited the Apache config, ran a graceful restart, and the service came back. That works for one server. It falls apart at fifty, because people are inconsistent, they make mistakes, and the drift never stops. Puppet, Chef and Ansible were built for exactly that: keep the servers you already have in a known state. That is server configuration management, and it isn’t the same thing as creating the server.

Public cloud and OpenStack changed what you could ask for. An interface that accepts “a server this big, with these parameters” is the prerequisite for creating one from code, the same way a software-defined network needs something to call. Without it you have a box, a network cable, SSH, and the habits you already had. You can still manage what runs on that box from code, because Puppet, Chef and Ansible configure machines that already exist. What you can’t do is declare the box itself.

The categories blurred afterwards. Ansible can call cloud APIs, so a playbook now does much of what a CloudFormation template does, and the vendors in that space sell configuration management as infrastructure as code. The line we’re drawing runs between provisioning and configuration, not between infrastructure as code and everything else. Kief Morris’s Infrastructure as Code puts server configuration inside the same discipline. The interface it asks for is a programmable one, and a scriptable command-line tool satisfies that as well as a REST API does. The distinction still matters when you are deciding what a tool was designed to do well.

Automate when the work needs automating

Having the interface doesn’t settle whether you should use it this quarter.

When is automating your infrastructure worth it?

When the work repeats often enough that doing it by hand costs more than building and keeping the automation.

If one cluster with a few nodes and one database is all you need, automating it is probably early. At two or three clusters, with a plan that ends in a thousand, start now, because retrofitting is the expensive version.

There is a counterweight, and it is the reason to write infrastructure down even when you won’t automate it. The code is the spec you check reality against: open the repository, open the console, compare. Writing it down is cheap. Owning the pipeline that applies it is not, and those are two separate decisions that often get made as one.

The cost lands on day two. These tools get you running fast, which is what they’re built for, and then upgrades, state files, drift and staff turnover arrive together. It is the gym problem: you watch someone lift 200 kilos, copy the movement, and hurt your back. Be honest about the arithmetic before you start. We’ve spent two days automating a task and saved about ten seconds of our own lives doing it. Count how often the task runs, how long it takes, and what the automation will cost to keep, and the number usually tells you more than the brochure does.

Console changes create drift

Declaring everything doesn’t remove hands from keyboards.

What do you do when someone fixes production by hand?

Put the fire out, then put the change into the code before the incident ticket closes.

Dogma loses in an outage. Go and do whatever it takes to stop the bleeding. The discipline is in the second half: if the manual fix never reaches the repository, a later rollout can put the broken state back, because the templates still describe the world as it was before the fix.

Access control helps, up to a point. Read access in production for everyone, write access for nobody, and a pipeline that applies the changes is a good default. It holds until the first on-call rotation, when read access turns out not to be enough to end an incident.

Two sets of hands are easy to confuse here. One is inside the machine: someone opens a shell, edits a config, and the server stops matching how it was built. The other is in the provider console, where a click changes the account itself.

The first set has a fix, and the code isn’t it. Infrastructure as code says nothing about immutability. You can run entirely stateful, mutable infrastructure from code, and what matters is how you manage it. Immutable images are the separate choice that takes hands off a running machine: bake the image with Packer, deploy a machine that comes up ready, roll a new image instead of patching a live one, and don’t enable SSH. If nobody can get in, nobody leaves traces you find three months later.

Neither of those reaches the console. A new image doesn’t un-click the button that resized the database or opened the security group, because that change was made against the provider, not inside a machine. The way back is reconciliation with the code: fold the change in, or put the account back to what the code says.

Then there are the resources that were never in the code at all. We’ve run accounts where developers had near-admin rights in the console and used them, which is how you end up with EC2 instances and S3 buckets that no template knows about. The fix was tags. A shared Terraform module generated a specific tag set, and a nightly Lambda function listed everything without it, posted the list to Slack, marked those resources for deletion, and deleted them the following night. AWS’s tagging guidance makes the same point about tags as the hook for automation and cost allocation.

Declare the intent, then keep it true

That sweep is a crude version of something with a name.

What’s the difference between a shell script and a declaration?

Scripts tell the machine what to do. A declaration states what should be true, and something else works out the difference.

Terraform does that comparison on every run: terraform plan reads what is actually there, compares it to your intent, and prints the changes it would make. CloudFormation works the other way round. It applies the template you hand it, and finding out whether the stack still matches that template is a separate operation, drift detection, added in November 2018 and started on demand against a stack or a single resource. Either way the comparison happens when you ask for it, so ask on a schedule: run the plan or the drift check against production nightly and read a non-empty result as a finding instead of a surprise during the next deploy.

Kubernetes does it continuously. You submit a manifest, and a controller keeps working until what you asked for exists, reporting back through its own channels when it can’t get there. The operator pattern is that loop with domain knowledge attached. Terraform itself doesn’t continuously reconcile your account between runs.

Security teams got to the reactive version first. An intruder stops CloudTrail; something notices and turns it back on. AWS Config ships a periodic managed rule, cloudtrail-enabled, which checks whether a trail is enabled in the account and reports NON_COMPLIANT when none is, rather than following each individual trail. Noticing isn’t repairing: turning the trail back on needs a remediation action configured against the rule, which since September 2019 can run without a human in the loop.

The same idea is reaching provisioning. Google made Config Connector generally available on 28 January 2020: a Kubernetes operator that “makes all GCP resources behave as if they were Kubernetes resources”, so a SQLInstance is a manifest and the operator sorts out the ordering. On the policy side, Forseti Security, open-sourced by Google and Spotify in 2017, scans the cloud APIs into a database and enforces rules against what it finds. Write the rule that no VM may have port 22 open, and it takes the access back when someone grants it.

You still own the state, and a loop that corrects things will correct the wrong thing as readily as the right one. Keep the rules small and readable.

Loops, generators and your own language

Readable is doing a lot of work in that sentence, because declarative syntax stays readable only while the setup is small.

What do you lose when infrastructure code grows loops and conditionals?

The readability you chose a declarative language for in the first place.

Terraform 0.12, released on 22 May 2019, allows expressions “to be used directly in any situation where a value is expected”, adds a for operator that builds one collection from another, and generates nested configuration blocks with dynamic. Useful features, all of them. They also mean that reading a module is no longer reading what will exist. You have to hold the logic in your head first, then hope the state agrees with you.

There’s a third option that gets discussed less, and it tends to show up in companies with the budget for it: code generators. Feed in a simplified YAML description, emit a plain declarative spec that anyone can read. It is more controversial than declarative against imperative, because you take decisions away from the people using it, and readable output is rarely what the generator’s authors optimise for. What you get back is the right level of abstraction. Most developers want a database and a connection string. They don’t want to be a DBA, and the business is paying for the features they aren’t writing while they learn.

Pulumi 1.0 arrived in September 2019 and takes the opposite route: you describe infrastructure in TypeScript, JavaScript or Python. Go was being added around the time we recorded, and we weren’t sure whether it had landed yet. The comfort is real if your team already writes one of those languages. The trick is that you still learn a DSL, so what transfers is the syntax and the loops, not the library.

Kelsey Hightower’s nocode is a joke that lands: “Write nothing; deploy nowhere.” For a team getting started, the safer bet is Terraform or whatever your cloud provider ships, because the tutorials, the blog posts and the meetups are where people actually learn. Then leave that tool the minute it gets in your way.

Legacy paid the bills

All of this assumes an empty account. Most of us walk into something that has been running for a decade.

Why doesn’t a rewrite work on a system that old?

Because that system is why the company still exists, and the people who built it are sitting in the room.

You can’t throw it away, because the systems have to keep running. There may not even be APIs to call: bare metal without virtualisation, a mainframe if it’s a bank, plus a cloud footprint, because almost everyone ends up hybrid. Arriving with a shiny idea and the message that everything they built is rubbish wins no friends, and in the end technology is people doing technology.

What works is smaller. Take a piece of new functionality, build it as code, and let it be the demonstration. Before that, find out what actually hurts. If production and development have drifted so far apart that every deploy blows up, the case makes itself. When the pain sits somewhere else, infrastructure as code is an investment competing with other investments, and the money could buy more developer productivity elsewhere. It pays off over time and costs you up front, so the question to answer before choosing any tool is whether you’re willing to live with the problem you have for another quarter.

Prove it by shifting traffic

Once it is all written down, someone will claim that integration and production are the same environment.

How do you find out whether two environments really match?

Move production traffic onto the other one and watch, including the faces in the room.

People who have done it answer calmly. A freeze tells you what the claim was worth. Environments can differ despite matching templates, and the way to find the difference is to practise failing over before an incident picks the date for you. Get into the habit of recreating things, shifting traffic and running failovers, and the confidence you gain is the real return on immutable infrastructure.

The pay-off shows up when you need it. We lost a production Kubernetes cluster at a customer, for reasons we’ll leave out. Everything was declared as code, so recreating the whole thing and getting back up took about one to two hours. Some of it wasn’t documented properly, and that is where the time went. Had it all been documented as it should be, we might have been back in 15 to 30 minutes, which is mostly AWS doing its thing: you call the APIs, wait for the resources to come up, and continue with the deployment.

Note what the code did not do there. It didn’t provide failover, and it didn’t build the inventory, the monitoring or the traffic shifting around it. Those are separate work, and recreating a cluster while services stay up is genuinely hard. Infrastructure as code is where that journey starts, and there is no silver bullet waiting further along.

What this means for teams

Start from the problem, not the tool. A shiny tool is nice, and using one for its own sake is a path to nowhere; what you’re paid for is business value delivered with the capabilities your team actually has. Educate yourself on the field rather than the product: Kief Morris’s Infrastructure as Code is still the best single source we know, and a second edition is in the works.

This week:

• Take the last console fix from an incident, write it into code, and deploy it to a lower environment to check it does what the manual change did.
• Schedule the comparison you currently run by hand: a nightly terraform plan against production, or drift detection on the stacks you care about.
• List every resource in one account with no owner tag, then work through the list. Some are declared and only missing the tag; the rest are the pile nobody owns.
• Ask the team that says integration matches production to shift traffic for an hour, and see what they say before they answer.

Common questions, answered

How do I check that my repository matches what is running in my AWS account?

Run the comparison on a schedule and treat a non-empty result as a finding instead of a surprise during the next deploy. terraform plan reports what it would change on every run; with CloudFormation the check is a separate on-demand operation, drift detection, which you can point at a whole stack or one resource. Neither sees resources that were never declared, so pair the check with an inventory pass over the account itself.

How do I find cloud resources that nobody declared in code?

Tag everything you declare, then treat whatever comes back untagged as a list to investigate. Generate the tag set from a shared module so declared resources carry it automatically, and have a nightly job list the ones without it and post that list to Slack. A missing tag means nobody claimed the resource, not that no template owns it, so read the list before you act on it. In one account where developers had near-admin console rights, we went further and let the job mark those resources and delete them the next night, which kept forgotten experiments off the bill without taking the console away.

Should I start with Terraform or Pulumi?

Start with Terraform or whatever your cloud provider ships, because the community around it is older and the tutorials, articles and meetups are where you’ll learn fastest. Pulumi fits a team that already writes TypeScript, JavaScript or Python and wants one language across application and infrastructure, as long as everyone understands they’re learning a new DSL rather than reusing a language they know. Whichever you pick, plan to leave it when it stops helping.

Resources

• Announcing Terraform 0.12: first-class expressions, for and dynamic blocks, and what they do to a configuration’s readability.
• Pulumi 1.0: which languages it supports and what “infrastructure in a real language” means in practice.
• New: CloudFormation drift detection: how CloudFormation compares a live stack to its template, and that you have to start the check yourself.
• Config Connector is generally available: GCP resources as Kubernetes resources, reconciled by an operator.
• Kubernetes operator pattern: the control loop that keeps trying, and where its knowledge of your system lives.
• Forseti Security: Inventory, Scanner, Enforcer and Explain, and how Spotify used them.
• Automatic remediation with AWS Config rules: what it takes to have a rule fix a resource without a human.
• Infrastructure as Code, Kief Morris: the book to read before the next tool tutorial, server configuration included.

Since we recorded

Terraform stopped being open source in the sense we meant it here: HashiCorp moved its products to the Business Source License in August 2023, and the OpenTofu fork landed at the Linux Foundation the following month. Forseti Security was retired, with the project announcing it would finish archiving by the end of June 2023, though GitHub records the repository as archived on 11 January 2025. The Kubernetes-as-control-plane idea spread well past GCP, with AWS shipping AWS Controllers for Kubernetes in developer preview in August 2020. Kief Morris’s second edition arrived later that year, and we came back to the whole topic in episode 101, “Infrastructure as Code in 2026: Still Essential or Already Changing?”.