Two environments and a folder you copied

One person, one environment, Terraform from a laptop: nothing here needs fixing. You write the .tf files, read the plan, apply, move on. The second environment is where it starts costing you. You copy the folder, change the names, change the CIDR blocks, point the backend somewhere else. Now you own two copies of nearly the same infrastructure, and they have to stay nearly the same for as long as the project lives.

Everything after that is bookkeeping, and Terraform offers several ways to do each piece rather than one. Where do the per-environment values live? How does the application stack learn the VPC id the networking stack created? Which state file does this directory write to, and what stops a dev run from writing the production one? You can answer all of that yourself, and you probably will: a shell script here, a makefile there, a wiki page nobody updated. The person who joins in March will find none of them.

Anton Babenko maintains the terraform-aws-modules collection along with a set of tools around Terragrunt, and he’s an AWS Community Hero and a HashiCorp Ambassador. Terragrunt is his answer to that bookkeeping, and the choice it puts in front of you isn’t Terragrunt against Terraform, because Terragrunt doesn’t replace Terraform. It’s who decides your layout: you, over the next six months, or a tool that decided already.

What Terragrunt is actually for

Most of the suspicion about Terragrunt comes from assuming it replaces something.

Is Terragrunt a replacement for Terraform?

No. It’s a thin layer that invokes Terraform for you, and it splits what Terraform keeps in one place: the module, reusable and versioned, and the values you pass into it.

You don’t need it on day one, or in week one. It earns its place the first time you notice what your keyboard is doing.

Each directory gets a terragrunt.hcl naming one module, the inputs to send it, and where those inputs come from. The backend block is the clearest case. Gruntwork’s post announcing v0.19, from June 2019, states the problem plainly: “you have to copy/paste the same backend configuration into every one of your Terraform modules”. Terragrunt has you define it once in a root location and inherit that configuration in all child modules, so only the state key differs.

The folder layout follows the cloud rather than the tool. An AWS account holds regions, a region holds services, a service holds configuration, so the tree is account, then region, then service, with a global folder for what has no region: IAM roles, policies. Terragrunt treats one folder as one module invocation, so an S3 bucket’s configuration sits in a bucket folder inside its region folder.

The test for whether the layout worked: a person reading the repository shouldn’t need a second thought about what a folder is for, or which regions the Lambda functions run in. They read the tree and they know.

What the wrapper leaves out

The obvious objection is that you can build most of this yourself, and the objection is correct.

Can you get the same result with plain Terraform and a wrapper?

Yes, for the parts you’ve already thought about: one application spec applied everywhere, an environment variable that picks the right *.tfvars file, and a few lines of shell to tie them together.

That’s what we run for clients today. Foundational infrastructure, networking and IAM, lives in its own state. Each application owns a separate state holding its DynamoDB table, security groups and load balancer, and whether that’s a directory or a repository matters far less than the separate state file. An environment id arrives as an environment variable, say prod, a wrapper appends prod.tfvars to the terraform command when that file exists, and the same spec goes to dev, stage and production with different values.

Anton puts a number on what that leaves out: the wrapper is “one-fifth of what Terragrunt can do”. It’s his estimate rather than a measurement, and the missing four fifths are hooks, dependency wiring between state files, and generated backend configuration, which is what you start writing yourself in month three. Andrey’s position is that the wrapper is small, understood and already in place, and a second tool in the chain is a second thing to debug when a deploy goes strange.

We’ve run exactly the setup that estimate describes: two folders, int and prod, with the changes copied from one into the other as the project moves. That’s the duplication Terragrunt exists to end.

Both setups work. One of them is documented by somebody who isn’t you.

Hooks will eat your pipeline

The first of those four fifths is also the first one teams regret.

What belongs in a Terragrunt hook?

Close to nothing. Pulling a secret before terraform init is defensible; building your application there is how a tool stops being a tool.

Terragrunt has had before_hook and after_hook since v0.14.4, in March 2018, which “allow you to execute arbitrary shell commands before and/or after running Terraform”. Arbitrary is doing real work in that sentence. Build the application, bake an AMI, fetch secrets, run end to end tests afterwards, deploy: all of it can hang off one terragrunt apply.

We get to the same caution from two directions. One is what sustained use actually looks like: a hook that fetches a secret from SecretHub before Terraform runs and stays quiet enough that nothing lands in stdout or stderr, with every other before hook treated as a temporary fix until whatever it does moves somewhere better. The other is the Unix reason, one tool for one job, with Gradle as the counter-example: a tool that will do anything you like and nothing you can debug.

There’s one more use worth asking for, and it’s a fair ask. An after hook can run the application’s end to end tests once the apply finishes, so a scheduled terraform apply tells you the application still works rather than only that Terraform had nothing to change.

On the HashiCorp stack the secrets hook is unnecessary for the provider. The Vault provider’s vault_aws_access_credentials data source hands the AWS provider short-lived credentials, so nobody is carrying a static key that still works next month. That covers the provider and stops there. A backend block “may not contain interpolations”, as Terraform’s backend documentation puts it, so it cannot read a data source, and the backend needs credentials of its own by the time terraform init runs, resolved without the provider. The S3 backend takes a profile, a shared_credentials_file, a role_arn to assume or keys from the environment. What it can’t take is the value your Vault data source produces.

One hook earns its place: policy. Sentinel sits in HashiCorp’s paid tiers, since the Team and Governance upgrade is what buys “the ability to use Sentinel and Cost Estimation”. A pre-apply hook that runs Checkov or tfsec over the spec costs nothing and catches port 22 open to the world. Sometimes you do have to open 22. The point is that somebody chose to.

Passing the VPC id to the next module

Hooks are optional. Getting one state’s output into another module is not.

How do you get the VPC id out of one state file and into another module?

Terragrunt’s dependency block points at the folder that built the VPC, runs terraform output there, and hands the value over as an input.

Version 0.19.20, August 2019 added it, and the shape is four lines: dependency "vpc" { config_path = "../vpc" }, then vpc_id = dependency.vpc.outputs.vpc_id under inputs. No data source, no state reading of your own. Take element two out of the subnet id list and put the instance there.

There are other ways. Terraform’s own terraform_remote_state reads another configuration’s outputs directly, which the documentation concedes “creates a tighter coupling between configurations”, and which gets messy once several states read each other. What we run for clients is TF_VAR_ environment variables set by a setup script we hand to developers, which works because a VPC id rarely changes and we can redistribute the script when it does. Anton’s tfvars-annotations was another, and its README now opens with “This project has become redundant (yay!)”, because v0.19.20 did the same job natively.

You can point it across repositories, where it works and stops being easy to follow. Fine if you keep 20 to 50 module invocations in one place and want a bucket’s notification rules and access policy readable in one file. Less fine if your account-wide security layer sits in its own repository on purpose.

Workspaces are a renamed state file

There’s a Terraform feature that’s supposed to have solved all this, and it keeps not coming up.

Should you use Terraform workspaces for dev, stage and production?

Not as the boundary between them. Workspaces give one configuration several state files behind one backend block, and on their own they draw no line around who may read or write which state.

They’re the worst feature Terraform ever shipped, and the documentation is almost as blunt. It says organizations commonly want “a strong separation between multiple deployments of the same infrastructure serving different development stages (e.g. staging vs. production)”, where “the backend used for each deployment often belongs to that deployment, with different credentials and access controls”, then rules the feature out for exactly that: “Named workspaces are not a suitable isolation mechanism for this scenario.”

The separation is yours to build if you keep them. S3 does access control per object path, and its backend documentation carries an IAM policy “granting access to only a single state object within an S3 bucket”. Separate backend configurations stay the recommendation; the policy work is what skipping them costs.

What it recommends instead is worth reading twice if Terragrunt strikes you as exotic: re-usable modules for the common parts, with “each instance as a separate configuration that instantiates those common elements in the context of a different backend”, so that “the root module of each configuration will consist only of a backend configuration and a small number of module blocks”. A directory per deployment, a backend and a few module blocks: that is the Terragrunt layout, written down by HashiCorp.

The mechanics are smaller than the name suggests: “Workspaces are technically equivalent to renaming your state file. They aren’t any more complex than that.”

There’s a plainer route to one spec and several state files, the one we use: commit an empty backend "s3" {} and pass bucket, key and region at terraform init with -backend-config. Terraform keeps the merged result in the .terraform directory, “which should be ignored from version control”, so it sits in plain text on whatever machine ran init. It doesn’t stop there: a plan saved with -out encodes “the configuration, state, diff, and variables”, the plan documentation warns, unencrypted, so those settings travel with the plan. In February 2020, v0.22.0 went the other way: a generate block writes a real backend file into the working directory before Terraform runs, so the configuration is HCL beside the code instead of arguments reconstructed from a wrapper script.

Where Terragrunt gets expensive

An opinionated tool is worth having only where the opinion holds, and there are two places living with it gets expensive.

Where does Terragrunt get hard to maintain?

Two places: configuration that inherits through more than one level, which you end up assembling by hand, and apply-all over a deep dependency graph, which runs but pulls more into the apply than you meant to touch.

Take 20 Lambda functions. In this setup each one is more than the function itself: it also carries an alias, a dead letter queue, VPC settings and IAM policies. What you want is two lines of configuration per function, with everything else inherited from a Lambda level and an environment level above it. You can write that, and what you write is configuration that reads other terragrunt.hcl files from specific paths and merges them into maps, until the merging itself becomes impossible to work with. Change one setting for all 20 functions and you’re editing 20 places.

The second is apply-all. Point Terragrunt at a folder and it walks the dependencies and works out the order the modules have to run in, which usually goes fine. Several layers of dependencies is where it gets hard to live with: one Lambda function can pull a VPC reapply along with it, because the function needs access to the VPC, and the graph turns into “an interconnected, like big, big, big ball of mud”. The traversal is doing what it was asked to do. The estate is what got coupled.

That’s one Lambda function deployed with CodeDeploy, an alias and canary releases. Plain Terraform is no kinder, since wiring API Gateway to Lambda by hand is a week you won’t enjoy. And when Terragrunt costs you a full day on one problem, the instinct is to throw it out and write 20 lines of shell. For most cases what it does is set the config, the TF_VAR_ environment variables and the CLI arguments, then run terraform apply in a particular directory.

A real language, or no language at all

Which is where somebody always says: use a programming language and stop fighting the config format.

Would Pulumi or CDK solve this?

It depends which problem you think you have, and it’s the one question here where we don’t converge.

Andrey wants a general purpose language for the awkward parts. HCL is not short of expressions, and Terraform 0.12 brought first-class expressions, for expressions and dynamically generated nested blocks. The trouble starts where the thing you’re describing carries real branching: a Kubernetes deployment or an ECS task definition where this field exists only because that one was set. Terraform is declarative, and layering that kind of logic on top usually makes a mess, while the same logic is ordinary in Go or TypeScript. Pulumi also throws in policy as code for nothing, which counts for something when the alternative is a tier upgrade. CrossGuard, announced in December 2019, is “100% open source and available to all users of Pulumi, including the Individual Edition”, with organization-wide policy management kept for Enterprise.

Anton thinks the direction itself is wrong. Over-engineered out of the box, and nobody needs to be impressed by loops in an infrastructure definition. He’s unmoved by for_each on modules, which arrived in the Terraform 0.13 public beta on 3 June: in years of this work he’s wanted it about once. What application developers want is a black box with good defaults. Tell me the bucket name. Do you want it secure? Done.

Julien goes further: putting a Turing complete language back inside a declarative one is going in circles, and the layer worth working on is the one that generates the declaration. His warning holds whichever side you take: automate the wrong thing and you’re going to destroy yourself.

You are not the first person to build a VPC

The argument that survives all of this has nothing to do with which tool you picked.

Should you write your own VPC module?

Almost certainly not. In Anton’s experience, 90% of this work has been built, published and fixed by dozens of people before you, and it’s already in open source.

This is the structural case for Terragrunt, and it’s stronger than the DRY one. A terragrunt.hcl holds a module source, some inputs and little else, so the path of least resistance is to go and find a module instead of writing Terraform. Green field work where nobody has gone before is real, and it’s not where most of your week goes. It’s certainly not where your VPC comes from.

We’re taking that advice ourselves right now. The AWS Lambda module was tagged v1.0.0 on 4 June 2020, and we’re already moving our own Lambda implementation onto it. It ships a little Python, lambda.py, to build and package deployment artifacts, which isn’t ideal inside a Terraform module, but the archive_file data source doesn’t go far enough, and Python is already on the machine.

One last reason to look at Terragrunt even if you never run it: read the documentation for the opinions. It explains why things are laid out the way they are, which puts it closer to architecture documentation than to a manual. You may not adopt the answers, but you get a blueprint. Our reservation is that the reasoning got harder to find when the docs moved out of the GitHub repository and onto the website.

What this means for teams

The choice that matters is whether your layout decisions get made once, in public, by somebody who wrote them down, or twenty times over by whoever is on call. If you know Terraform well, you can build your own version and it’ll be smaller than Terragrunt. If you’re handing the repository to people who don’t, an opinionated tool stops them shooting themselves in the leg, and that beats elegance.

This week:

• Count your copies. If the same resource block lives in more than two environment folders and differs only in values, you have the problem Terragrunt was written for, whether or not you adopt it.
• List what your wrapper script doesn’t do: dependency wiring, backend generation, hooks. Those gaps are the four fifths, and they get filled in a hurry during an incident.
• Check what a CLI workspace is carrying. Where two environments want different credentials and access controls, a workspace gives you no boundary until you write the IAM policy yourself, so give each one its own configuration and backend, which is the layout Terraform’s documentation recommends.
• Take one module you maintain, a VPC, a Lambda, an S3 bucket, and compare it with the published one. If the published one is better, delete yours.

Common questions, answered

How do I stop copying the backend block into every Terraform module?

Define it once and let the tooling write it per directory. In Terragrunt that’s a root terragrunt.hcl the child configs inherit, where only the state key differs, and since v0.22.0 a generate block writes a real backend file into the working directory before Terraform runs. Without Terragrunt, commit an empty backend "s3" {} and pass the values at terraform init with -backend-config from a wrapper. Both give you one spec and one state file per deployment; either way the merged configuration lands in the .terraform directory Terraform tells you to keep out of version control, and in any plan you save with -out. The difference is that a generated file is HCL you can read and review beside the code, while init arguments live in whatever assembled them.

Do I need all my Terraform in one repository to use Terragrunt?

No, but its best feature assumes it. A dependency block reads outputs by running terraform output in another folder, so one tree is where it’s comfortable; across repositories it works and gets hard to follow, with submodules as the usual patch. If you keep a layer separate on purpose, an account-wide security layer for instance, pass the handful of values some other way: TF_VAR_ variables from a setup script, or a terraform_remote_state lookup with the coupling that brings.

Is it worth adding Terragrunt to a project that already works?

If your own wrapper already covers your environments and you understand it, not on its own merits. Adopt it when the people who inherit the repository don’t know Terraform well, or when you catch yourself writing the dependency wiring and backend generation by hand. The cost is one more layer to debug when something strange happens; the gain is that a stranger can read the folder tree and tell what’s deployed where.

Related episodes

• Infrastructure as Code: the first round of the loops and generators argument, and why declaring the intent is only half the job.
• Terraform in CI: shared state, the credentials a pipeline holds, and the per-environment *.tfvars wrapper this episode takes apart.

Resources

• Terragrunt: how to keep your Terraform code DRY and maintainable: Gruntwork’s June 2019 post on v0.19, covering the root terragrunt.hcl, inherited backend configuration and the folder per environment.
• Terragrunt v0.19.20 release notes: the dependency block that reads another module’s outputs, and the four lines it takes.
• Terragrunt v0.22.0 release notes: the generate block, which writes backend configuration into the working directory before Terraform runs.
• Terraform workspaces documentation, June 2020: why named workspaces are not an isolation mechanism for staging and production, and the layout HashiCorp recommends instead.
• Terraform backend configuration documentation, May 2020: that a backend block takes no interpolations, and where the merged configuration from -backend-config ends up on disk.
• terraform-aws-modules/terraform-aws-lambda v1.0.0: the Lambda module discussed here, including the Python packaging step it ships with.
• Terragrunt reference architecture: Anton Babenko’s worked example of the layout, worth reading before you decide you dislike it.
• serverless.tf: his opinionated approach to serverless on AWS with Terraform modules, where the seven modules per Lambda function come from.

Since we recorded

Terraform took back some of the duplication itself: count, for_each and depends_on for modules shipped in 0.13 in August 2020, so one module block can stand in for twenty near-identical ones. Terragrunt deprecated apply-all and its siblings in favour of run-all in v0.28.1, February 2021. The ground moved again in August 2023, when HashiCorp put Terraform under the Business Source License and the OpenTofu fork followed. We picked that story up in episode 64 and again in episode 83.