You already run Terraform, so what is Pulumi actually for?

Sooner or later someone on your team suggests Pulumi, and the comparisons you find online argue declarative against imperative and never answer the question you came with. The questions that decide anything are narrower than that. Where does the state live. What happens to the secrets inside it. Which providers exist, and what do you do when yours doesn’t. What breaks when you rename a resource. And whether any of it is worth touching infrastructure that already works.

Paul Stack is a software engineer at Pulumi, and before that he worked on Terraform from version 0.5 onwards. He wrote the original version of the AzureRM provider, and he contributed to Terraform’s modules, which he now thinks got used for something they weren’t designed for. People reach for modules as an encapsulation of resources rather than as a reusable resource, and the design has been bent to catch up: count, for_each and depends_on only reached module blocks in Terraform 0.13, in August 2020. That’s the person worth asking what the other tool does differently, because he has shipped both sides of the argument.

We came into this expecting Pulumi to be Terraform with a programming language bolted on top, and that turns out to be the wrong model. The engine is Pulumi’s own, the state has a different shape, and the secrets story is the part Terraform users will find most uncomfortable.

Declarative end state, imperative path

Start with the objection every Pulumi user hears first, because it’s the one that stops most people reading further.

Can infrastructure written in TypeScript still be declarative?

Yes, because the program describes the end state; what’s imperative is how you arrive at it. Your Pulumi application is end state by nature. Getting there means following the semantics of the language you picked, which has consequences you’ll notice on day one. You declare things before you reference them. Your program has one entry point, index.ts or __main__.py or main.go, and loads the rest from there. Point Terraform at a folder and it reads every .tf file in it, order and filenames irrelevant. Pulumi can’t do that and doesn’t pretend to.

The other half of the misconception is that Pulumi generates code for something else underneath. It doesn’t. Pulumi has an engine that coordinates the operations and drives them through provider plugins, and the language SDKs are the dumb part: types, argument classes, parameters with primitive types. The SDK and the language runtime make a gRPC call back to the engine, the engine decodes the object structures and hands each create, read, update or delete to the provider plugin, and the plugin is what calls the cloud API on your behalf. There’s no CloudFormation underneath and no Terraform CLI in the path.

That’s also why there’s no provisioner concept. In Go, TypeScript, .NET or Python there’s already an SSH library you can import, so you take the outputs of a resource, load the package you like and connect to the instance yourself. The runtime you’re already in is the feature.

Where Pulumi’s providers come from

If the engine is Pulumi’s own work, the provider coverage mostly isn’t, and that’s deliberate.

Does Pulumi ship its own provider for every cloud service?

No. Most of them are built from the matching Terraform provider, and a handful of native ones are generated from the vendor’s own API specs. Thousands of hours of work went into the Terraform provider ecosystem, and reinventing it would have put Pulumi six or seven years behind on day one. So the Terraform bridge points at a Terraform provider and code-generates an SDK from its schema at build time. At runtime the bridge connects the Pulumi engine to that Terraform provider’s own code over Pulumi’s RPC interfaces, which is the point of the exercise: the CRUD implementation is the part nobody wants to write twice. What you don’t need is the Terraform tooling around it. No Terraform CLI on your machine, no plugins in a Terraform plugin directory, no .tf files; you install one Pulumi provider and it brings what it needs. If a customer turns up missing a provider, scaffolding one on top of the Terraform equivalent is two or three hours of work.

The native providers are the more interesting half. The Kubernetes and Azure providers track upstream specs directly. When a new version of the Kubernetes OpenAPI spec is tagged on GitHub, a new SDK can be generated from it and a provider release cut within about fifteen minutes, which is how same-day coverage of a new Kubernetes API is possible at all. Pulumi’s next generation Azure provider, in beta since September 2020, generates from the Azure Resource Manager REST API specs Microsoft publishes. Terraform’s AzureRM provider still has a person mapping the SDK into provider code by hand.

This only works for clouds large enough to publish a usable spec, and plenty of smaller ones don’t.

State, backends, and the lock that isn’t there yet

The first question a Terraform user asks about any new tool is where the state goes, and the answer has a hole in it worth knowing before you start.

Do you have to use Pulumi’s hosted service to keep state?

No. Pulumi calls them backends, and a self-managed one is an S3 bucket, a Google Cloud Storage bucket or an Azure blob container, the same habit you already have from Terraform. Plenty of teams sit on each side. What the hosted service adds is SSO for login and RBAC over who can push to which project. The raw functionality, the CLI, the SDKs and the provider integrations are one to one with the self-managed backends.

The exception is locking. Self-managed backends don’t have it, so nothing stops two processes updating the same stack at the same time. A pull request to add it has been open since 2019 and is going through review right now. It creates a locks directory in the management directory, gives each backend implementation a UUID joined with the stack name, and sits behind the PULUMI_SELF_MANAGED_STATE_LOCKING=1 environment variable, because locking may turn out to be slower for people and an opt-in is the safer way to find out.

Values marked secret stay encrypted in state

Where the two state files really diverge is not where they’re stored, it’s what happens to the sensitive values inside them.

What stops a secret sitting in plaintext in your state file?

The secret mark on that value, and nothing else. Every Pulumi project gets a secrets manager at creation time, wanted or not, and it encrypts what carries the mark. On the hosted service the default is a key spun up per stack, so one stack’s key isn’t another’s. If you’d rather hold the key yourself, Pulumi’s secrets providers cover AWS KMS, Azure Key Vault, Google Cloud KMS and HashiCorp Vault, chosen when you create the stack. A secrets provider supplies encryption, not storage: the ciphertext stays in your state file, and the key store only encrypts and decrypts on the way past. If whoever runs the CLI can’t reach that key store, Pulumi stops the run there.

Any output of any resource can be marked secret. The mark travels to whatever you derive from it, so a value doesn’t come out clean by being interpolated into a string, and the state file holds the ciphertext, decrypted only when Pulumi needs the value. Without your KMS key or your Vault token, a marked value stays unreadable to whoever picks that file up.

Read that guarantee narrowly. Pulumi’s own account of secret tracking puts the encryption behind the mark: a config value set with pulumi config set --secret is encrypted, an unmarked one isn’t, and a generated password the provider doesn’t flag stays readable until you name it in additionalSecretOutputs. Everything unmarked is plain JSON: the bucket names and instance sizes that give away the shape of your infrastructure, and any password or token you passed in as a plain string. Grep a fresh state file for a value you know is sensitive before you trust the default.

The bridge carries the mark further than you might expect. Terraform provider schemas mark fields sensitive: true, and Pulumi reads those marks when it scaffolds the provider, so an RDS instance’s sensitive outputs land in state encrypted without anyone asking. You’d have to opt out.

Paul Stack hedges his own claim here, and the hedge is the accurate half: “I’m going to call it secure by default. Of course, it’s not secure by default. It’s we’re giving you the ability that you can be secure by default.” Terraform went the other way, the speculation on offer goes, because managed key services were young when it was designed and security sat below getting infrastructure into code at all.

Outputs are the real learning curve

If the secrets story is the pleasant surprise, the next question is what people actually get stuck on.

What’s the steepest part of the Pulumi learning curve?

Output resolution, by a distance. Both tools work out the order of operations themselves: Pulumi wraps resource properties in promise-like Inputs and Outputs, and the Outputs you pass from one resource into another are what the engine builds its dependency graph from, so nothing here is scheduled by hand. What’s new is where those values live. An Output is an unresolved value sitting in ordinary program code, and handing one straight to the next resource is fine. Looking at it yourself is not. Writing it to a file, printing it, feeding an IP address into user data: all of that goes through apply, and apply is where people stop.

The language itself is a much smaller obstacle than Pulumi’s own marketing made it sound, and the marketing is a fair thing to criticise. Telling people they’ll be “programming infrastructure in real languages” alienated a whole group of people who aren’t programmers and never planned to be. The honest version of the requirement is short. For Python: virtual environments, because that’s good practice in that community and gives each project its own dependencies; pip, so you can install and upgrade the Pulumi package; and the structure of a Python application, meaning __main__.py and loading other files from it. For Go: modules, and where packages come from. Know that much and you can start, refactoring later as you need to.

Renaming a resource shouldn’t mean deleting it

The second thing support hears about is names, and it starts with a bucket that isn’t called what the author called it.

Why does Pulumi append a random string to your resource names?

So that a replacement can be created before the original is deleted. A logical name of my-bucket becomes a physical name like my-bucket-d7c2fa0, and the docs give two reasons: two stacks of the same project can coexist without colliding, and updates can create the new resource, move the references and then delete the old one. AWS won’t let two launch configurations share a name, so without the suffix that order isn’t available. Pulumi is create before delete; Terraform is delete before create unless you ask for the opposite.

Hard code the physical name with name, or a type-specific property like bucket, and the replacement order becomes yours to set. Pulumi still creates first, so a replacement collides with the old resource that’s still holding that name. The naming docs give the fix: deleteBeforeReplace: true on any fixed-name resource that might be replaced. That option costs you what auto-naming was buying: the old resource goes first, whatever it served is down until the new one answers, and the replacement can cascade to resources that depend on it.

The URN is where this pays off. A Pulumi resource’s URN is made of the project, the stack, the type and the name. A stack is roughly Terraform’s workspace, holding the state plus the configuration for one program. The difference is how you get one: Terraform gives every configuration a default workspace you can’t delete, while Pulumi has no default, so every command runs against a stack you created or selected by name. No part of that URN is a file path, so moving a declaration between files, or three levels down a call chain, changes nothing about the resource’s identity. Compare terraform state mv and its hoops for pulling a resource out of a module.

Reparenting is the move that does bite, and it isn’t the same as relocating code. A child’s URN carries its parent’s type, so the URN changes when a resource acquires a parent, not when its declaration moves to another file. Wrap existing resources in a ComponentResource and pass each of them parent: this, as Ringo De Smet did when he refactored Cumundi’s stacks into components, and the next preview plans to recreate them. The fix is the one you’d use for a rename: aliases hand Pulumi the old name or the old parent, and it understands the resource is the same one it already has.

Pulumi as a library, not just a CLI

Everything so far assumes a person at a terminal, and that assumption is the one Pulumi is trying hardest to drop.

How do you give developers infrastructure without giving them the CLI?

Run Pulumi from your own code. The Automation API entered preview in October 2020, and Python support followed by February 2021. It exposes stacks and lifecycle operations as a programmatic interface, so the thing a developer touches can be your portal, your own CLI, or a form with four fields and a button. Behind it, Pulumi does the work and hands back the connection string or the kubeconfig. The argument for it is organisational, not aesthetic: getting the right CLI version and the right credentials onto 500 developers’ machines is a high barrier to entry, and a tool that only exists as a CLI loses that argument before it starts.

The examples are deliberately small. ploy is a Cobra CLI in Go: give it a KUBECONFIG and a directory with a Dockerfile, and ploy up builds the image, creates a private ECR repository, pushes to it, references the image in a Kubernetes deployment, creates the matching service and namespace on EKS, and returns a URL. Pulumitron is an Electron app from a hack day that provisions AWS infrastructure from a window. Pulumi also rebuilt its own GitHub Action this way: TypeScript, embedding the Node.js Automation API package, with up, preview, refresh and destroy as the four flows, released as v2.0.0 on 26 February 2021, and faster for it.

One component in four languages, and who gets to publish it

Shared components let teams reuse infrastructure code across languages. They are also where the uncomfortable security question lives.

Can a component written in Go be consumed from Python?

Yes. A multi-language component exposes a schema, and Pulumi generates the SDKs for the other languages from it. The component ships as a binary that gets installed as a dependency, and the runtime works out that a TypeScript call needs translating into the Go API underneath. The EKS package is the worked example: previously TypeScript only, it became usable from TypeScript, Python, .NET and Go in v0.22.0 in February 2021, with the authoring experience for your own components heading to public preview.

The use case is a centralised networking, infrastructure or SRE team that has to serve people who don’t write their language. Write once, version it with SemVer like any other package, and the database team consumes it from Python without learning Go. The trade-off is conformity. When everyone gets creative and then leaves, nobody knows what is going on in their corner. In an organisation too large for conformity, one team publishing for everyone is the closest you’ll get.

Then there’s the other edge of the same feature. A shared infrastructure library runs where the credentials are. A malicious PyPI package pulled into a backend service may be boxed in by restricted permissions and isolation, and the attacker still has work to do from there. A package inside your infrastructure tool starts with your cloud keys already in reach, and a small call home that ships them out is easy to miss in a diff. SolarWinds is the shape of that attack, and it is three months old. Pulumi’s answer is a SOC 2 Type 2 audit completed in January 2021, official SDKs that pull their binaries from get.pulumi.com through a release process with published checksums, and an acknowledgement that this needs continual checking. The rest is the hygiene you already owe yourself: don’t pull random libraries from random places, and put the shared ones on a registry you control.

What this means for teams

The honest answer to “why Pulumi over Terraform” starts with a question back: what are you trying to do?

Keep the condition in that sentence. Moving infrastructure that already works has to buy you something concrete, and being on the newer tool is not concrete. The case that stays open is the one where you want to build your own tooling on top of infrastructure as code, because that’s where the language runtime earns its keep: a Python class that polls Prometheus for sixty seconds and lets a canary deployment continue only if the standard deviation holds, or a dynamic provider that shells out to the Azure CLI for the one association the Azure SDK won’t do. terraform-exec gives you something similar from Go, and it’s a fair bet the HashiCorp ecosystem will grow more of this, but embedding a Python Prometheus SDK in it isn’t what it was designed for.

Four things worth doing this week:

• If Terraform works and nothing hurts, stay. Spend the time on something that pays.
• Try Pulumi on the next new thing instead of a migration, reading your existing outputs through the @pulumi/terraform package rather than moving any state.
• If you self-manage Pulumi state, decide now how you stop two runs colliding, because the backend won’t do it for you yet.
• Choose the secrets provider when you create the stack, not after. Switching later is work you don’t need.

Common questions, answered

How do I try Pulumi without migrating my existing Terraform?

Read your Terraform state from the Pulumi program instead of moving it. The @pulumi/terraform package exposes RemoteStateReference, which reads the outputs of a local .tfstate file or any of Terraform’s remote backends and behaves like a native Pulumi stack reference.

That makes the split team workable. A networking team that’s happy in HCL keeps its Terraform and exposes the right outputs; a team that wants TypeScript consumes those outputs and builds on top. Nothing is migrated, and either side can stop at any point.

How do I stop Pulumi adding a random suffix to a resource name?

Set the physical name yourself, through the resource’s name property or a type-specific one such as bucket, and set deleteBeforeReplace: true on it in the same edit. Pulumi creates a replacement before deleting the original, which a fixed name makes impossible, so that option keeps replacements working.

The cost is downtime: the old resource goes first, and the service it provided is gone until the replacement answers. If nothing outside your stack reads the physical name, the generated suffix is cheaper.

How do I move a Pulumi resource to a different file without recreating it?

Move it. A Pulumi resource’s URN is the project, the stack, the type and the name, and no part of it is a file path, so a declaration can move between files, or into a helper function, without the state noticing.

Two edits do change the URN, and both read as a brand new resource. Renaming is one. Setting a parent is the other: pass parent: this to a resource you have pulled inside a ComponentResource and its URN picks up the parent’s type. Set an alias carrying the old name or the old parent and Pulumi treats it as the resource it already has.

Related episodes

• Infrastructure as Code: where we set out what writing infrastructure down actually buys, which is the argument both tools inherit.
• GitOps: the pull-based alternative, and the one we’re running into edge cases with on a current project.
• Terraform in CI: shared state, pipeline credentials and concurrent runs, which is exactly the gap the Pulumi locking pull request is about to close.
• Are We Wrong About Terragrunt?: wrappers, module reuse and DRY Terraform, with Anton Babenko.
• All You Need to Know About Setting Up HashiCorp Vault: running the Vault instance you’d point a Pulumi stack’s secrets provider at.

Resources

• Pulumi state and backends: what a self-managed backend is and which object stores it supports.
• Support locking on filestate logins (pulumi/pulumi#2697): the open pull request that adds locking to the self-managed backends, and the environment variable it hides behind.
• Peace of mind with cloud secret providers: the four bring-your-own-key options and what the ciphertext looks like in state.
• Resource names and auto-naming: why the suffix exists, and what a fixed name obliges you to configure.
• How Pulumi works: the CLI, the engine, the language host and the provider plugins, and which of them talks to the cloud.
• The road to the Pulumi Go SDK 2.0: what an Input and an Output are, and how the engine gets a dependency graph out of them.
• The Pulumi Automation API: the October 2020 preview announcement, and what embedding Pulumi in your own program is for.
• Create Amazon EKS clusters in your favorite language: the first multi-language component in the wild, and what Pulumi says is coming next.

Since we recorded

Multi-language components stopped being a preview: Pulumi 3.0 shipped them as Pulumi Packages in April 2021, so a component written once is consumable from every supported language. The locking pull request merged on 16 March 2021, five days after this conversation, still gated behind PULUMI_SELF_MANAGED_STATE_LOCKING. The bigger change to the choice came from the other side: HashiCorp relicensed Terraform under the Business Source License in August 2023, OpenTofu forked it, and we covered the fallout in #54, HashiCorp’s BSL move and OpenTF. Paul Stack has since left Pulumi, and came back on the show in #92, From System Initiative to SWAMP to talk about what infrastructure tooling looks like when agents are the ones driving it.