Nobody can tell you what’s safe to delete

Ask around about a bucket nobody recognises, a security group attached to nothing, a CI job for a service with no users. You get shrugs, a “that was the migration, I think”, and nobody willing to say delete it. So it stays. What a leftover costs you depends on what it is. The bucket keeps billing. A secret nobody rotated still unlocks whatever it unlocked. An unattached security group charges nothing and exposes nothing, and still has to be explained to the next person reading the inventory. The CI job does its damage in search results, pretending to be live code.

Deleting is the easy part. Knowing that a deletion is safe is the work, and a cloud account is not built to tell you. The console shows what exists, but that inventory alone doesn’t establish that every consumer has stopped using it. We like deleting things enough to ask people to send their decommissions our way. Then we took production down ourselves, by running our own checklist correctly on a service somebody was quietly calling.

Every safe deletion comes down to two things: something that remembers what was created, and a way to put it back when you get it wrong.

A decommission is a checklist, not a destroy command

Start with the service you know about, because that’s the case where somebody has actually decided the thing is finished.

What does “deleted” mean for a service that’s been running two years?

Nothing is left that another engineer can find and wonder about.

terraform destroy handles what the code created, and that’s the smallest part of the job. The rest is what accumulated alongside: Kubernetes secrets created by hand during an incident, a service object nobody removed, a horizontal pod autoscaler for a deployment that’s already gone. Then there are the secrets manager entries somebody typed in rather than generated. We keep a running list of everything an application can touch, because it’s the only thing standing between a clean decommission and a leftover nobody recognises.

The repository goes last, after everything it deploys is gone. Code search reads one branch per repository, the default one you pick in the settings. GitHub’s documentation is blunt about it: “Only the default branch is indexed for code search." Create a branch, delete everything on it, and leave a README.md that says the repository is decommissioned and names the branch that still has the code. Push it, then set that branch as the default in the repository settings.

Deleting the Jenkinsfile along with everything else is what took the job out of our Jenkins, and how far that carries depends on your setup. A multibranch job picks up every branch that holds a Jenkinsfile, the branch you parked the code on included. By default a standalone multibranch job won’t re-index the repository for branch additions or deletions. It keeps showing what it last scanned until you configure periodic re-indexing or trigger a scan yourself. An Organization Folder is the exception. It watches a whole organisation and creates multibranch jobs for the repositories it finds, so it picks the change up without being asked. Dropping .circleci/config.yml doesn’t disconnect anything either: CircleCI follows the repository through a webhook and a deploy key that stay until somebody uses Stop Building in the project settings.

The code is still there for anyone who wants it, one branch away. What’s gone is its ability to look alive.

You can delete boldly if you can rebuild quickly

We ran that checklist end to end on a production service, finished it, and then watched the 500s arrive. Somebody was still using the thing.

What do you do when you delete something that’s still in use?

Put it back from code: an empty commit to the repository, and the pipeline rebuilds what it built in the first place.

Ours was back in about fifteen minutes, hand-typed secrets and all. That figure belongs to that afternoon rather than to the technique. It held because the deployment branch still built, and because the pipeline we pushed to was the same one that had created production. The infrastructure, the dependencies and the containers came back in order, and nobody had to remember what the service needed.

One piece didn’t come back on its own. A couple of secrets had been entered into the secrets manager by hand, so they went back in by hand.

How willing you are to delete is a function of how long your rebuild takes. At fifteen minutes, deleting something still in use is an incident you narrate in Slack. At two days of hand work, nobody deletes anything, and the account keeps filling up.

Let a nightly job delete what nobody claimed

A checklist handles the service somebody decided to decommission. The expensive leftovers are the ones nobody remembers creating.

How do you clean up resources nobody will admit to creating?

Tag what you mean to keep, and let a scheduled job delete the rest after it has announced the list.

Ours is a Lambda on a nightly schedule. It looks for resources missing the tags we require and tags those with a time to live of zero. Then it posts the list to Slack: here is what I found, I’m deleting it tomorrow. The next run deletes them. Anyone who wants to keep something sets a tag that exempts it, by hand, before the next run.

Some of the rules need real logic. Keep the last AMI and the last snapshot, whatever the tags say. Detach a security group from everything using it before you delete it. AWS enforces one of these for you: you can’t delete a snapshot of the root device of a registered AMI until the AMI is deregistered. That’s a decent hint that the others deserve the same care. Point the job at the resource types where this pays, not at everything you own.

The scheme drives behaviour through fear of losing your work, which isn’t the motivation we’d pick from scratch. It has run for a couple of years, it saves a lot of money, and developers tag their resources now.

Cloud Custodian has shipped the same pattern for years as mark-for-op and the marked-for-op filter. Its unattached Elastic IP example splits the work across three policies. One marks every unattached address for release in seven days. The next removes that mark from any address that got attached in the meantime. Only the third releases what is still marked when the date arrives. That middle policy is the one you would leave out. Ours is a human instead: somebody reads the Slack list and sets the exemption tag before the next run.

Something has to remember what it created

The cleaner works because it can tell intent from accident, the question underneath any tool that deletes.

Why does Terraform delete things that kubectl apply leaves behind?

Terraform keeps state, so a resource whose configuration block you removed has nothing to match and the next plan proposes destroying it. Applying that plan is what deletes it; plan alone changes nothing.

kubectl apply keeps a record too, of the wrong shape. It writes what it applied into a kubectl.kubernetes.io/last-applied-configuration annotation on each object: enough to patch that object next time, nothing about the set. That’s why kubectl apply --prune removes nothing until you name a scope, a label selector with -l, or --all.

Pruning isn’t a blanket sweep either: it looks at the prune list’s types under the selector you gave and skips any object without that annotation. What carries it and wasn’t in the manifests you just applied gets deleted. That spares a Helm release or a hand-made object from an incident, unless an earlier kubectl apply annotated it.

Scope is where this bites. Namespaced types are pruned in the namespaces you applied into. But the default list also carries Namespace and PersistentVolume, swept cluster-wide however you scoped the apply. A pruned Namespace takes its contents with it. The documentation calls pruning alpha and points at kubectl delete -f instead.

We disagreed about Argo CD on air. Andrey had it filed as a stateless kubectl apply that leaves orphans behind. Julien thought Argo CD deleted resources missing from the desired configuration, but hadn’t checked how. We were both half right. Argo CD labels what it applies with app.kubernetes.io/instance and uses that label to determine which resources form the app. That’s a record on the objects rather than in a file. The same FAQ warns that other tools write that label too, Kustomize among them, so it marks what Argo CD tracks rather than what it created. It just won’t act on it unless asked. The v1.7 docs say: “By default (and as a safety mechanism), automated sync will not delete resources when Argo CD detects the resource is no longer defined in Git.”

Flux is equally cautious. Garbage collection arrived in Flux 1.11.0, March 2019 behind the --sync-garbage-collection flag, labelled experimental. It labels what it applies from a repo, branch and path, then deletes anything with that label this sync didn’t apply. Its own documentation warns that changing the branch or path orphans resources you then delete by hand.

Tags are the contract, and the contract needs teeth

All of that depends on the label or the tag being there, which is a people problem wearing an API.

Should you force tags on people, or ask for them?

AWS gives you both levers, and the choice says more about your organisation than about the cloud.

The hard lever is a service control policy using the aws:RequestTag and aws:TagKeys condition keys, which refuses the create call when the required tags aren’t in the request. Next to it sit tag policies, which AWS launched across Organizations in November 2019. They define which keys and values are allowed, and can block non-compliant tag changes.

We don’t enforce either, and there are two camps about whether that’s a mistake. One says tagging should be tightly controlled. A tag is a writable string you can read back with read-only permissions, so it’s a channel for walking data out of an account. Those attacks exist. We’d say they’re unlikely in our environment. An attack like that needs more sloppiness around it than we think we have. We’d rather let developers tag their own resources and collect the savings a cleaner brings once it can tell what’s claimed.

There’s no recipe here that solves everything. Pick the compromise, then write down which one you picked and why, so the next person inherits a decision instead of an accident.

Manual changes belong in dev, and a rebuild finds the rest

A tag is how a resource states who owns it and how long it should live, whether a pipeline created it or somebody clicked it into existence. The clicked ones are worth a section of their own, because every account has them.

Is it ever all right to change production by hand?

In the middle of an incident, yes, and then the change goes into the code and rolls out properly afterwards.

Outside an incident it’s a bad idea, and the normal path costs less than it looks. Click around in dev until you know what you want. Write the Terraform. Roll it through the environments. Developers pick this up quickly once they’ve seen it from the other side. A resource the pipeline created is consistent across environments, and nobody spends the release scrambling because production is missing something dev had.

The enforcement we actually rely on is the cluster upgrade. We don’t migrate a Kubernetes cluster in place, we stand up a new one and run the pipelines. Everything created by a pipeline comes back. Everything else doesn’t.

The point of all of it is being able to say that whatever happens, this comes back from code. That’s what makes trying things out feel safe, and deleting them too.

Three layers, and the database that doesn’t belong in the cluster

Rebuilding from code is only a habit you can keep if everyone knows which parts they’re responsible for rebuilding. There are three.

Where do an application’s cloud dependencies belong?

Next to the application’s manifests, owned by the people who own the application.

The bottom layer is the bootstrap: networking, connectivity, IAM, the skeleton that has to exist before anything else does. It wants Terraform and nothing else, because all it needs is a file and an apply.

The middle layer is where it gets interesting. An application needs a database. You spin up a managed Postgres, MySQL, MongoDB or Redis for that one application. Its manifest can then live next to the definition of the thing it depends on. The developer owns what the cloud doesn’t manage, the schema and the data model. The infrastructure team keeps the platform secure, accessible and reliable. That’s a clean separation of concerns. It’s also the case where a Terraform controller inside Kubernetes earns its place, because the dependency ships with the thing that needs it.

The third layer is the application code itself, which is where developers spend their day.

Two caveats with this design. Not every dependency is one application’s. A managed database instance shared between services to save money gets complicated fast, and you don’t want a load balancer per application either. Kubernetes handles the second case well, since an ingress controller is deployed once and applications create Ingress objects against it. And while the Terraform Kubernetes provider will happily define your Deployments, that’s a hammer looking for a nail. Generate the templates with Helm and commit them where your GitOps agent reads from.

Trust, but check the logs

Every layer above assumes somebody knows whether the thing is still used, which is where our 500s came from.

How do you know a resource is really unused?

Ask the people, then go and check the logs anyway.

There’s a Russian proverb that translates as trust but verify, and decommissioning is where it earns its keep. Somebody will tell you nothing calls that service. Look at whether anything is calling it, whether connections are open, whether it’s consuming resources. Fifteen minutes of checking is cheaper than a restore.

The other half of this isn’t technical at all.

That record accumulates, and it makes the next decision faster. If you can see how much time goes into chasing people about deleting resources, you’ve learned something about a process that needs changing, without talking about tooling.

One deletion deserves special treatment: the disk. When you rotate servers constantly, as immutable infrastructure does, the forensic evidence rotates with them. A signal you react to an hour late has nothing left to read. That’s why we’re evaluating Auditbeat to stream audit events off the hosts instead of leaving them on a disk that goes away. It subscribes to the kernel’s audit framework directly and takes the same rule format as auditctl, so the rule sets we’ve been tuning come across. It’s also worth keeping the last snapshot of a rotated instance well past the rotation: maybe 90 days, maybe 180, depending on how long detection takes. Those numbers are off the top of our head rather than measured, so use your own. Snapshots are incremental and deleting an earlier one doesn’t break the later ones, so the bill is the only thing that grows.

What this means for teams

Cleaning up a cloud account is mostly a question of who remembers what. Terraform remembers through state, Argo CD and Flux through a label they write on everything they apply, the nightly cleaner through tags you require. A human with a checklist remembers the rest: the secrets manager, the stray autoscaler nobody wrote down. Where none of those is watching, things accumulate until somebody gets curious during a cost review.

The other half is recovery. A team that can rebuild an environment from code in minutes will delete things, find out it was wrong, and carry on. Teams without a reliable rebuild are more reluctant to delete anything, and they pay for what stays twice: once on the invoice and once in attack surface.

This week:

• Run your decommission from the other end. Pick a service that was deleted months ago, and search your account, your clusters and your secrets manager for anything still carrying its name.
• Replay one pipeline into an empty namespace or a scratch account and write down every step that needed a human. That’s your manual-step inventory.
• Check whether your GitOps agent is allowed to prune, and what it would take with it. If it isn’t allowed, decide on purpose, rather than finding out during an incident.
• Pick one resource type where untagged leftovers cost you money, and put a mark-then-delete job on it with the list going to a channel people read.

Common questions, answered

How do I stop a decommissioned repository from showing up in code search?

Move the code onto a branch and make an empty archive branch the default, because only the default branch is indexed for code search. Leave a README.md there saying the repository is decommissioned and naming the branch that still holds the code. Deleting the CI config doesn’t reliably stop builds. A Jenkins multibranch job still finds any branch carrying a Jenkinsfile, and CircleCI keeps its webhook until somebody uses Stop Building. Open the CI afterwards and confirm the job is gone. Do this after everything the repository deployed is gone.

How do I make Argo CD delete resources I removed from Git?

Enable automated sync and pruning together, with argocd app set <APPNAME> --sync-policy automated --auto-prune, or by setting prune: true under syncPolicy.automated in the Application. Auto-prune on its own does nothing while automated sync is off. If you’d rather not automate deletion at all, run a manual sync with pruning checked. Argo CD decides what belongs to an application from the app.kubernetes.io/instance label it writes on everything it applies. Pruning removes what it tracks as that application’s, not everything living in the namespace.

How long should I keep snapshots of instances that get rotated?

Set the retention from your own detection delay: the gap between something happening on a host and somebody looking into it. Instances that live a couple of weeks and rotate automatically take their disks with them. A security alert you follow up a month later needs a disk to read. The 90 and 180 day figures we float are guesses rather than a measured policy, so check them against how long your own alerts take to surface. EBS snapshots are incremental, and deleting an older one doesn’t affect restoring from a later one.

Related episodes

• Infrastructure as Code: where the drift comes from, and why a console fix disappears on the next deployment unless it goes back into code.
• GitOps: the agent in the cluster that pulls from Git, which is the thing we’re asking to prune here.
• Terraform in CI: state, plan on the branch, apply on the primary branch, and the pipeline that has to be able to rebuild what you deleted.
• AWS Security Maturity Roadmap: the inventory problem one level up, where the forgotten thing is a whole account rather than a bucket.

Resources

• Argo CD automated sync policy, v1.7.8: settles that pruning is off by default and gives the two ways to turn it on.
• Argo CD FAQ, v1.7.8: the app.kubernetes.io/instance label Argo CD uses to decide which resources form an application, the tools that collide with it, and how to change it.
• kubectl apply prune, v1.19.0: the conditions an object has to meet before pruning deletes it, and the default list, whose Namespace and PersistentVolume entries are swept cluster-wide.
• Flux garbage collection reference, 1.21.0: the two-phase label-and-sweep design, and the configuration changes that orphan resources it then misses.
• Searching code on GitHub: only the default branch is indexed, which is what makes the archive-branch trick work.
• Jenkins multibranch pipelines: why a standalone multibranch job needs periodic re-indexing configured, and what an Organization Folder discovers on its own.
• AWS launches tag policies: November 2019, organisation-wide rules for tag keys and values, with optional enforcement.
• Cloud Custodian EIP garbage collection example: mark, unmark anything that came back into use, then release, as three policies, if you’d rather not write the Lambda.

Since we recorded

Flux v1’s experimental flag is gone: Flux 2.0 went GA on 5 July 2023, and prune is now a field you set on every Kustomization. Argo CD’s safety default hasn’t moved, and automated sync still refuses to delete what you removed from Git until you ask it to. AWS added a Recycle Bin for EBS snapshots on 29 November 2021, but it only holds snapshots a retention rule covered before the delete. The database question came back in episode 90, on Kubernetes versus managed services.