Two people, one state file, and the keys to everything

One person running Terraform from a laptop doesn’t have a CI problem. You run terraform plan, read it, run terraform apply, and the state file sits on disk next to the code. The second engineer joins and that arrangement is finished. Two local state files are two versions of the truth, and runs that cross each other work from inconsistent state: a confusing diff if you’re lucky, conflicting changes to resources neither of you meant to touch if you’re not.

Two separate decisions follow, and it helps to keep them apart. The first is shared state: it moves to a backend both of you can reach, with locking, because there has to be one version of the truth. The second is automation, and that one is a judgement call. Once more than one person works in the same Terraform repository, it’s a good idea to bring CI in: terraform plan on branches, terraform apply when a change lands on the primary branch, and nobody provisions from a laptop. What you also built that afternoon is one unattended process holding credentials that can create and destroy every resource in the account.

Both halves are real and they pull against each other. Shared state and a pipeline are what let a team change the same infrastructure without stepping on each other. They also take the access that used to be spread across a few laptops and concentrate it into one service account that runs at three in the morning against whatever got merged. The pipeline you built to stop two people colliding is now the most valuable target you own.

The state file is the first thing to get right

Before any of the pipeline, the state has to live somewhere both of you can reach and neither of you can corrupt.

Where should Terraform state live once more than one person touches it?

In a remote backend that locks it, encrypts it and keeps old versions, because the state is both the coordination point for every run and a file that can hold your secrets in plain text.

Terraform ships a list of backends, and the choice matters less than two properties you need from whichever you pick. The first is locking, so two applies can’t run against the same state. On AWS this trips people up, because there is no DynamoDB backend: there’s an S3 backend whose dynamodb_table argument turns on “state locking and consistency checking”, using a table whose primary key is named LockID. One table can lock many state files. Leave the argument out and you have no locking at all. On Google Cloud there’s nothing to switch on, and the GCS backend docs say it in one line: “This backend supports state locking.”

Recovery is the second property. The S3 backend docs put it as a warning: “It is highly recommended that you enable Bucket Versioning on the S3 bucket to allow for state recovery in the case of accidental deletions and human error.” Turn it on. A truncated state file is the one failure where you want yesterday’s copy back in seconds, not an afternoon of terraform import.

If you’d rather not run any of it, HashiCorp has offered a free SaaS tier since October 2018 that takes state management and locking off your hands, free for teams of up to five since the full Terraform Cloud release in September 2019. It also puts the state outside the account it provisions: whoever gets into the account doesn’t get the state file thrown in.

HashiCorp’s own guidance on sensitive state says the same from the other end: marking a field sensitive hides it from CLI output and “will not encrypt or obscure the value in the state”. Provider credentials are the exception worth knowing: pass them as environment variables, as HashiCorp’s automation guide recommends, and Terraform doesn’t persist them. For what the resources carry, encryption at rest and an access policy you can defend are the minimum.

Credentials to the whole kingdom

The state is the data. The credentials that write it are the bigger problem.

How much damage can your Terraform pipeline do if someone takes it over?

All of it, which is why this part gets designed before the first webhook rather than after the first incident.

The same service account provisions the network, the clusters, the databases and the bucket the snapshots land in, so one credential covers both the estate and its restore path.

Start by giving each environment its own credential. The S3 backend docs sketch the shape in their multi-account pattern: an administrative account holding the state, and a separate account per environment that Terraform reaches by assuming a role. Separate accounts alone aren’t isolation, because the pattern as written lets one administrative identity assume into every environment. The isolation comes from the identity each runner uses: the staging pipeline gets credentials that can assume the staging role and read the staging state, and nothing beyond. A stolen staging token is then a staging incident.

Then keep credentials out of anything you store. The automation guide recommends passing provider credentials as environment variables, because Terraform doesn’t persist them into the plan or onto disk. The plan file is the other half of that: it carries a full copy of the configuration, the state and the variables you passed in, so a saved plan in a world-readable artifact store is a state file with extra steps.

Last, the people. Nobody proposing a change needs these credentials: contributors open pull requests, the runner holds the keys, and the number of laptops carrying production admin goes to zero. The credentials don’t disappear, though. They move to a machine that runs code written by whoever opened the pull request. Atlantis is blunt about it: you run it “on a server with credentials that allow access to your infrastructure”, and its exploit list runs from a local-exec provisioner that curls your keys to someone else’s host to custom commands in the atlantis.yaml that arrived with the branch. Planning is not an inert step either, since providers are binaries the runner downloads and executes to produce the plan.

So the code has to be trusted before a privileged runner executes it. Keep the runner off public repositories, leave fork pull requests switched off, allowlist the repositories whose webhooks it accepts, set a webhook secret, and treat merge rights as worth what the credentials are worth.

Plan on the branch, apply on the primary branch

With state and credentials settled, the pipeline itself is unremarkable, and that’s the point.

What should run on a branch, and how far can you trust the result?

terraform plan on the branch, a fresh plan against the merged code, a human approval of that plan, then an apply of exactly what was approved, because a branch plan goes stale the moment somebody else merges.

Continuous integration for infrastructure is just continuous integration: a change arrives on a branch, something runs, and you get feedback before it reaches anything that matters. Here the feedback is the plan, and what it answers is what the change does to what’s already running.

A one-line edit to a network block might whitelist an IP address or might recreate the cluster underneath it, and the plan is where you find out which. Replace is the word to look for.

Then somebody has to read it, and where it gets published decides whether they do. The automation guide keeps a person in the loop on purpose: “Have a human operator review that plan, to ensure it is acceptable.” A plan buried in CI logs is a chore people skip; posted as a comment on the pull request it’s a diff they read like any other. Atlantis has posted plans that way since before HashiCorp hired its maintainer in October 2018, and HashiCorp’s own GitHub Action for Terraform, out on 6 May 2020, feeds plan results back to the pull request.

Now the part that catches teams out. Two pull requests are open at once, both with green plans. Merge the first and the second describes a world that no longer exists: when a plan is applied, the automation guide says, “any other existing plans that were produced against the same state are invalidated”.

So re-plan on the merged code, and put the approval on that plan rather than on the branch plan somebody read three merges ago. The automation guide has the mechanics: save it with terraform plan -out=tfplan, have a person review that file, then terraform apply tfplan, so what runs is what was read. If the re-plan says something materially different, that difference is what needs approving. Serialize your applies, and let state locking catch the times you get it wrong. Even then, a plan proves only that Terraform could produce it, and some changes get no rehearsal anywhere, since a VPC that genuinely matches production isn’t trivial to stand up twice.

Split the code so one mistake stays small

Blast radius is a code layout question before it’s a permissions question.

Should all of your Terraform live in one repository?

Either way works, as long as you split it into layers: the account foundations, the shared platform, and the per-application resources.

This is where we disagree. Julien keeps the whole estate in one repository, from the account up to the application layer, with a folder per layer and change detection so a commit only plans the folder it touched. The payoff is debugging: one place to answer “what changed last, and who changed the access?” instead of chasing people across repositories. Andrey splits them, especially the account-wide security layer, because every change to IAM and access deserves review on its own, and because separate repositories stop people wiring one layer into another’s state.

The layers themselves are the same in both versions. Layer zero is the account: organizations, IAM, GuardDuty, the settings that change rarely and matter enormously when they do. Then the shared platform: networking, clusters, whatever everything else lands on. Then one state per application, owning its database, DNS name, load balancer, security groups and storage. Developers get free rein inside that last layer, and when a service is retired you run terraform destroy against its own configuration and state instead of hunting its leftovers through a monolith. Destroying is not the same as forgetting: terraform state rm drops resources from the state file and leaves them running, unmanaged and still billing.

One big state has a daily cost, too: every plan plans everything, and you wait for it.

The trap is coupling. Reach into another layer with data sources rather than copying identifiers around, and keep each lookup’s arguments independent of what the same run creates. Terraform queries data sources during the plan, but depending on the configuration it may defer reading until the apply phase. A lookup it resolves during the plan turns a missing role into a failure there, with a name you recognise; one waiting on a resource the same run is about to change doesn’t fail until apply.

Environments are a related question with a simpler answer. For applications, apply the same spec everywhere and vary it with a *.tfvars file per environment plus a small wrapper that picks the right one. Base layers keep separate directories, because one environment always has a VPC peering or an exception the others don’t. Terragrunt covers the orchestration part, with trade-offs we won’t settle in passing.

Nothing changed, so run it anyway

A pipeline that only runs when somebody pushes tells you nothing about the weeks when nobody does.

Why run Terraform when the code hasn’t changed?

Because the infrastructure changes without you, and the code quietly stops matching it until the day you need to ship.

Three jobs on a timer are worth having, and they answer different questions. The drift check is a nightly terraform plan that opens a pull request when the plan comes back non-empty against unchanged code. Read that diff as a lead rather than a verdict: terraform plan compares your configuration against the state and the resources it already manages, so a diff says something no longer matches. It doesn’t say who changed it, or whether a person was involved at all: a provider that reports an attribute differently produces one too, and anything outside Terraform’s management stays invisible either way.

The blunt version of the same job is a scheduled terraform apply, which doesn’t report drift so much as end it. “This is how it should be, make it so” is a defensible position if you’re willing to be there when it fails, which is why we run it on work days, Monday to Thursday. Nobody wants a Friday night apply breaking on Saturday.

The third job tests upgrades rather than drift: a scheduled run that bumps the Terraform version and refreshes the provider plugins against code you haven’t touched. Providers are binaries talking to cloud APIs that move underneath them, and you’d rather see the breakage on a Tuesday morning, with time to put it in the backlog, than on the day you have to ship through it.

GitHub Actions gives you the cron for free. The schedule event takes POSIX cron syntax, UTC by default, with a five minute floor, and “Scheduled workflows run on the latest commit on the default branch”, which for this job is what you want.

None of the three tells you the infrastructure works. A clean plan says reality matches the code, including when the code describes a broken load balancer. Checking that things still run after a change is monitoring, the same job you already do for an application deploy, pointed at the infrastructure. Answer who gets paged when a scheduled apply fails at 3am before you schedule one. Until everyone agrees how infrastructure gets changed, start with a plan and an alert, and promote it to an apply when people trust the result.

Read-only consoles, and a way to break the glass

Once the pipeline is trustworthy, the interesting question is what human hands are still allowed to do.

Who should be able to write to production?

The pipeline, plus a documented emergency path that expires on its own.

The setup worth copying: people get read-only access to the integration and production consoles, every write goes through CI, and a development account stays wide open so there’s somewhere to experiment. Two things follow. Hand edits stop, so most drift stops with them. And infrastructure change stops being an ops queue, because anyone who can open a pull request and get it merged can change infrastructure, whatever team they sit in.

Then the objection, which is a good one. Production is on fire, the fix is two clicks, and you want to be filling in a pull request template? No. So build the escape hatch while nothing is burning: a job that grants an elevated role for two or three hours, a second job that takes it back, and a log of both. Time-boxed, auditable and dull to use, which is what you want from a thing people reach for in a panic.

Rotation is the step after that, and only a team with a pipeline it trusts can take it. If the code really can rebuild everything, rebuild it on purpose and on a schedule. An intruder who got in and wasn’t noticed loses the host they were sitting on. AWS shipped a knob for exactly this in November 2019: maximum instance lifetime on an Auto Scaling group, anywhere from 7 to 365 days.

Kubernetes complicates the picture. Killing nodes doesn’t evict someone who has the API server, since the scheduler cheerfully moves their workload onto the replacement. Rotation is a control against persistence on hosts, not a substitute for knowing what’s running in your cluster.

Terraform behind a Kubernetes operator

One more option arrived while we were arguing about pipelines, and it comes at the problem from the GitOps side.

Google made its managed resources look like Kubernetes objects: Config Connector went generally available on 28 January 2020 as “a Kubernetes operator that makes all GCP resources behave as if they were Kubernetes resources”. HashiCorp followed on 17 March 2020 with an alpha Terraform Operator for Kubernetes, where a Workspace custom resource names a module and its variables, applying it creates the Terraform Cloud workspace and starts a run, and deleting it runs terraform destroy first.

The shape is the pull model from GitOps, a controller reconciling rather than a pipeline pushing, and the credential question comes along with it. It needs a Terraform Cloud team API token, and the announcement flags what that means: “Note that this is a broad-spectrum token”, sitting in a Kubernetes secret in your cluster.

What this means for teams

Most of this isn’t Terraform work. It’s deciding how much your automation is allowed to do, and arranging to find out when reality stops matching the code. A team of two with locked remote state, a plan on every pull request and a weekly scheduled apply is in better shape than a team of twenty with a wiki page describing which buttons to click in which order.

This week:

• Move state to a backend that locks and versions. On S3 that means a dynamodb_table with a LockID primary key, and bucket versioning switched on.
• Write down what your CI service account is allowed to do, then answer two questions: can those credentials delete your backups, and can the environment that runs least carefully reach the one that matters most? Either answer is the next ticket.
• Add one scheduled run on a work day. Start with plan and an alert on a non-empty diff, and only promote it to apply once you trust what it reports.
• Make the production console read-only for people, and agree the break-glass path before you need it, including who approves it and how it expires.

Common questions, answered

Do I need a DynamoDB table to lock Terraform state?

If your state lives in S3, yes: set dynamodb_table on the backend and create the table with a primary key named LockID, because without it the S3 backend does no locking at all. One table can lock many state files. The GCS backend locks with nothing extra to configure. When the bucket is yours, turn on versioning too, which the S3 backend docs recommend for recovery from accidental deletion and human error. Terraform Cloud handles state and locking for you and gives you no bucket to version, so recovery there is whatever state history it keeps.

Why does my Terraform apply fail when the pull request plan was clean?

Re-run the plan against the merged code, have someone approve that plan, and apply the saved file, because applying one plan invalidates every other plan produced against the same state. HashiCorp’s automation guidance recommends keeping one plan outstanding at a time: with two pull requests open, the second plan was computed against a state the first apply has already changed. The other cause is that a plan proves only that Terraform could produce one, and the cloud API may still reject the result at apply time. Serialize your applies and treat a branch plan as a review aid rather than a guarantee.

How do I get admin access when the console is read-only and production is down?

Build a time-boxed elevation path before the incident: one job that grants an elevated role for a couple of hours, another that revokes it, and a log of both. Read-only consoles stay survivable when there’s a way out that doesn’t need a policy rewritten under pressure, and a wide-open development account keeps experiments away from it. When the fire is out, put whatever you clicked back into the code, or the next scheduled run will quietly undo it.

Related episodes

• Infrastructure as Code: where the drift argument starts, and why changes made in a console are the thing automation has to outlast.
• GitOps: the pull model this topic keeps bumping into, and what a reconciling agent does that a pipeline doesn’t.
• What Are We Working On: credentials committed to repositories and who can read an S3 bucket, the two failures sitting underneath every state file.

Resources

• Atlantis: Terraform pull request automation you host yourself, so the runner holds the credentials instead of the contributors.
• Atlantis security: what a pull request can do to a server that holds infrastructure credentials, and the mitigations that keep untrusted code away from it.
• HashiCorp announces a Terraform SaaS free tier: the October 2018 announcement of free hosted state management and locking.
• S3 backend: the dynamodb_table locking argument, the LockID key, the bucket versioning warning, and the multi-account assume-role pattern.
• Running Terraform in Automation: human review of a saved plan, provider credentials via environment variables, what a plan file contains, and why applying one plan invalidates the others.
• Sensitive state: why marking a value sensitive does not encrypt or obscure it in the state file.
• Data source behavior: when Terraform reads a data source during the plan, and when it defers the read to apply.
• Creating workspaces with the Terraform Operator for Kubernetes: the March 2020 alpha, how a Workspace resource maps to a Terraform Cloud run, and the broad-spectrum token it needs.

Since we recorded

The stored-key half of the credential problem got an answer: GitHub Actions can assume a cloud role with a short-lived OIDC token since November 2021, so no long-lived cloud keys sit in the pipeline’s secrets. The authority behind them is unchanged, since the role still carries whatever you granted it, and the trust policy underneath has its own sharp edges, as we found out in episode 110. Locking changed too: Terraform 1.11 made S3 native state locking generally available behind use_lockfile and deprecated the DynamoDB arguments, so the lock table is now optional. Both promises we made here got kept, with Terragrunt two weeks later in episode 10 and staging environments in episode 16.