Three environments, and nobody can say what the middle one is for

Dev, test, QA, integration, stage, prod. Teams keep inventing environments, and most of the places we’ve worked with, employed or consulting, settle on three. A dev or test environment where things are allowed to be broken and people experiment. A stage environment where changes wait before production so somebody can run a system test against something that isn’t moving. Then production. Ask what the middle one is for and the answer gets vague quickly.

Here’s how it goes wrong. Something sits in stage waiting on an external client to come back about a GUI change. Production then needs an emergency patch. You go to deploy it and a developer stops you: ship now and the half-tested thing in stage rides along with it, because the deploy takes whatever is on the branch. The environment bought to reduce risk is the thing holding up the fix.

So the question isn’t whether staging is good practice. It’s what the middle environment buys, what it costs to keep honest, and whether that money and those engineering weeks would do more somewhere else. The alternative on the table is deploying straight to production behind a canary, a feature flag, or both, and that alternative has a bill of its own.

What a staging environment has to be before it tests anything

Before arguing about replacing it, be clear what the condition for keeping it is.

When is a staging environment actually testing something?

When it behaves like production, and there’s a test for that: redeploy every service at its production version into staging, point real traffic at it, and it should work.

Few staging environments would pass, and in our experience that condition has never been met. The useful move then is to say which difference breaks which test, rather than to write the environment off. The data is different, so the query that drags in production is quick here, and performance work and data-shaped edge cases don’t transfer. Versions drift apart, so an integration test tells you how your change behaves against a set of services nobody has shipped. Plenty still survives: that the configuration applies, that the thing starts, that a human can click through the new screen or an external client can look at it before it goes out.

Imperfect isn’t the same as useless. A staging environment that only partly matches production is an acceptable risk when you can name the risk, and a bad deal when you can’t.

Here’s the version where nobody can. “I wouldn’t trust staging to pour milk on my cereal.” That line stuck with us because the pattern behind it is familiar. It works and you don’t know why, it breaks and you don’t know why, and sooner or later somebody finds that one component’s version doesn’t match another’s, or is simply old.

That’s the failure mode worth naming: a staging cluster kept because having one is what you do, with none of the process built to use it. It stops being a safety net and becomes a thing you maintain.

A branch environment isn’t a staging environment

One answer to the blocked release is to stop having a shared middle environment at all, and give each branch its own.

The model is simple enough to describe in a sentence. You branch everything you build, you can deploy any branch to its own environment so a designer or an external client can look at it, and you merge to master, or main as people were starting to call it, only once that’s done. Nothing unfinished ever reaches the branch that deploys, so nothing unfinished can block a release. Two environments, integration and production, and master is always what’s running.

This is where we split. Mattias runs it and finds the blocking problem disappears. Andrey thinks the testing is worth less than it feels.

Does testing on a branch tell you the merge will work?

Only when what you tested is the integrated code: either the branch already sits on top of master, so merging is a fast-forward and nothing moves underneath it, or the pull request builds the prospective merge result and tests that.

Test the branch alone and you’ve tested your subset of changes against a master that has moved on since. A branch left open for months is the loud version of this, and it may behave differently on merge from how it behaved on its own environment. The rule is the same either way: what gets tested has to be what will run.

Branch-per-environment also turns up inside GitOps, which is a different idea. There an agent in the cluster reconciles the declarative config in Git against what’s running, so deploying means merging rather than pushing. A change made in the environment but not in Git gets put back only for the resources that agent manages. With automated sync turned on, Argo CD applies a new commit when it sees one. Correcting drift against a commit it has already synced takes a second switch, selfHeal in the sync policy. One common arrangement gives every environment a long-lived branch and moves a change forward by passing it from branch to branch, usually tagged with a Jira ticket or a GitHub issue. Those are not the short-lived feature branches above; they outlive every change that passes through them. We worked through the mechanics in episode 2. The sharp edge is the passing: moving specific commits between long-lived branches is a Git skill, and not everyone on your team is comfortable with it.

If you drop staging, the work moves into production

Delete the middle environment and the testing doesn’t vanish. It relocates.

What replaces staging if you delete it?

The candidate version runs in production itself, on the same cluster and taking a slice of real traffic, while you watch whether it behaves.

That’s a canary release, and the pattern is old enough to have a name and a short list of prerequisites. You need a way to send a fraction of the traffic to the new version. Weights you can set precisely need something doing the routing, a mesh or an ingress controller: Istio expresses it as weights on a VirtualService. Without one, the split can be approximated by replica counts behind an ordinary Service, which is what Argo Rollouts falls back to when no traffic provider is configured. You also need the new version to be safe beside the old one, because it reaches the same database. And you need something that decides whether the 10% is healthy, which has to be more reliable than the thing it’s watching.

Those two routing answers put the candidate version in different places. A Service builds its endpoints by listing pods in its own namespace, so a replica-count split needs both versions side by side in one namespace under one Service. Give the candidate a namespace of its own and something has to route across the boundary, which is the mesh or the ingress controller again.

Our own answer has been that nothing off the shelf ties the traffic shifting to the observability, so every canary ends up bespoke to the place it runs, and the hedge on that belongs in it: there might be such a thing and we just aren’t aware of it. There is, and it had already shipped. Flagger 1.0 arrived in June 2020 and does the whole loop: it shifts traffic in steps, queries Prometheus, Datadog or CloudWatch for success rate and latency, and rolls back after a set number of failed checks. Argo Rollouts v0.9, from August 2020, runs the same analysis from an AnalysisTemplate and added SMI TrafficSplit support for canary steps.

What they don’t settle is the hard part. Neither tool knows which metric means healthy for your service, or what threshold to abort on. They also want different things underneath: Flagger routes through a service mesh or an ingress controller, in its own words, while Argo Rollouts will use either and manage replica counts when you have neither. They move the work from “build a canary system” to “define what good looks like”, which is a real saving and not the same as free.

The infrastructure that ships with the container

What those controllers promote is what lives in the cluster, and that is more than an image: Flagger has tracked ConfigMap and Secret changes since January 2019, promoting them with the canary. Neither controller builds the rest of your change, or keeps the candidate’s copy of it clear of production’s.

Can you canary the infrastructure too?

Not for free, because a namespace scopes Kubernetes objects and stops at the cluster edge, while the queue, the bucket and the IAM role that came with your change sit outside it.

Kubernetes is precise about that limit. Namespace-based scoping covers namespaced objects, “e.g. Deployments, Services, etc.”, and not cluster-wide ones such as Nodes or PersistentVolumes. Nothing in it reaches into your cloud account.

A deploy usually runs Terraform first, creating the resources the container needs, then rolls out the container. Put the candidate version in a namespace beside production and the Terraform has to follow it: build a parallel copy of those resources, name them to match the namespace, and under no circumstances touch the production ones. That’s a spec with conditionals threaded through it, in a language that was declarative from the day it was an idea and is not pleasant when you fight that.

HashiCorp’s workspaces documentation draws the line in a useful place. A temporary copy for a feature branch is what named workspaces are for: a developer “might create a corresponding workspace and deploy into it a temporary ‘copy’ of the main infrastructure”, destroyed once the change merges. What they rule out is the case where the backend for each deployment “belongs to that deployment, with different credentials and access controls”, which is how staging and production usually get their separation. “Named workspaces are not a suitable isolation mechanism for this scenario.” That one means separate configurations, separate state, separate credentials.

Staging only tests against the baseline it happens to have

Even a staging environment that works tests one thing, and it’s worth being exact about what.

Why does staging drift away from production?

Because staging’s baseline is every change anyone has promoted and not yet released, and production’s baseline is only the released ones.

Picture ten services, all at version 0 in production. You push one of them to version 1. In the clean case staging now runs nine at version 0 and one at version 1, which is exactly what production will look like after the release, and the test means something. Now add four teams promoting in parallel. Staging has nine services at versions nobody has shipped, your change sits on top of that, and you learn how your version 1 behaves in a world that doesn’t exist.

The fix we’ve seen presented at conferences is to stop sharing. Each team gets an environment created on demand from the production baseline, with only their own container swapped in, used, then destroyed. That answers the baseline question and takes most of the rot with it, because an environment that lives for an afternoon can’t accumulate a year of version drift.

It doesn’t make the test exact. The baseline you copied is production as it stood when the environment was built, so another team’s release landing while you test leaves you holding a result about a production that has moved on. Recreating per test rather than per quarter shrinks that window without closing it.

That assumes someone can create environments on demand and make them match. In the cloud with everything defined as code, that’s a platform team’s job and a solvable one. On-prem we’ve never seen it work: building it means rebuilding the elastic part of the cloud in your own racks, which is not a side project.

Staging or better monitoring: you only get to build one

Both of the good answers cost engineering weeks, and your team has a fixed number of those.

Should we build a staging environment or improve production monitoring?

Improve the monitoring, because the canary strategy depends on it completely and the staging strategy is worth less without it anyway.

This is a different question from the one everyone argues about. Having staging versus not having it is a preference. Building a second environment versus spending the same weeks making production observable is a choice with an opportunity cost, and one of those two investments also pays off at three in the morning.

Push the canary argument hard and it reverses. If you deploy straight to production, your monitoring has to be good and resilient, or a bad release leaves you blind instead of merely wrong, which is worse than having no canary at all.

How fast you can undo a release changes the sum too. Take the case where you ship, production starts failing, you revert to the previous version, and the restore takes five minutes. It’s fair then to ask what the separate environment was earning, because what it was bought to prevent is what you just fixed in five minutes. That isn’t a free pass. Five minutes of a broken checkout is still a broken checkout. The 2019 Accelerate State of DevOps report puts elite performers under an hour to restore service, against a week to a month for low performers. Those bands measure how fast teams recover, not what the failure cost while it lasted.

Everything you ship has to survive two versions at once

Canary releases and feature flags have the same consequence, and it lands on the developers rather than on the platform.

How do you change a database schema when two versions are live?

In three deployments, not one: ship code that handles both the old and the new schema, migrate the data, then ship code that only handles the new one.

That’s parallel change, expand and then contract, and the temptation is always to skip to the last step because the new pod is the only one that matters. It isn’t, while the old pods are still serving. The same rule covers any migration a container runs on startup: it has to leave the previous version working.

Feature flags are the gentler version of the same idea. The new path ships dark, an environment variable or a managed service like LaunchDarkly decides who sees it, and 10% of requests take the new branch for an A/B test. Pete Hodgson’s survey is the one to read on the kinds of flags and the cost of leaving them in forever.

Where it stops being gentle is a public API. Once something is in use you can deprecate it, but breaking it is a project of its own, and we’d put it at ten times the work of writing it, spent entirely on making sure everyone migrated rather than on anything new. Which raises the uncomfortable question of whether you make things backward compatible forever, and whether that’s a design burden anyone can carry.

Not everyone carries it. Google retired Reader on 1 July 2013 because usage had declined, and the same reflex is the subject of Steve Yegge’s August 2020 post on Google Cloud’s deprecation policy. Yegge searched for “deprecated” on both clouds’ developer sites and found that Google’s docs “mention deprecation around 7x as often”, even though AWS has hundreds more services. He offers that as a rough yardstick, not as proof.

What this means for teams

There’s no universal answer here, and the honest version of the recommendation is that it depends on what your team is already good at.

A strong platform team should go for an environment per team, created on demand from the production baseline and thrown away after. A strong product team with weak platform support should go straight to production with feature flags and very good monitoring. If you’re resource constrained, or the canary pipeline would be the most complicated thing your team owns, keep the simple multi-environment setup and spend the saved weeks on production. The worst outcome is a bespoke progressive delivery pipeline that only its author understands, where the next person to join can’t predict what their change will do.

One thing holds either way: define the environments as code. A console-built environment can still tell you that the service starts and that a human can use the new screen, but not what it differs from production in, so a failure there is hard to attribute and a pass is hard to reproduce. Two environments kept matched by clicking carefully do not stay matched.

This week:

• Write down the condition your staging environment would have to meet to be a real rehearsal of production, then check which of your tests still hold where it doesn’t.
• Count the changes sitting in staging that aren’t going to production in the next release. That number is your drift.
• Pick one service and define what healthy looks like as a query: success rate over what window, latency at which percentile, aborting at which threshold.
• Take the next schema change through expand, migrate and contract as three deployments, and see what it costs you.

Common questions, answered

How do I stop a half-finished change in staging from blocking a production release?

Keep unfinished work off the branch that deploys. If something is waiting on an external review or a client’s answer, it belongs on its own branch with its own environment, and merges once the answer arrives. The release gets blocked because the branch feeding staging has become a holding pen for work in several states, so a deploy from it carries all of them. Once merging means finished, the fix can go out alone.

How many environments does a small team actually need?

It depends on what you have to test and on who can keep the environments honest, so start there rather than from a number. Where work waiting on a GUI review or a client’s answer can sit on its own branch environment, two is enough: one place where things can break while people integrate, and production. A third earns its keep when something there can’t be done in either of the others: a system test against a frozen baseline, or a sign-off a human has to perform. If you can’t name that something, you’re paying for a cluster and for keeping it close to production.

How do I run a canary release on Kubernetes without building the tooling myself?

Use an operator that already does traffic shifting and metric analysis: Flagger reached 1.0 in June 2020 and Argo Rollouts was at v0.9 by August. Both shift traffic in steps, query a metrics backend between steps, and roll back when the checks fail. No tool can guess what healthy means for your service, so that stays yours. The other choice is how traffic gets split: Flagger routes through a service mesh or a supported ingress controller, while Argo Rollouts will use one if you have it and otherwise manages replica counts behind an ordinary Service.

Related episodes

• GitOps: the branch-to-environment model in full, and what changes when an agent in the cluster pulls instead of your pipeline pushing.
• SemVer or Not to SemVer: why breaking a published interface is a project rather than a commit, which is the constraint behind every canary.
• Terraform in CI: plan on the branch, apply on the primary branch, and giving each environment its own credential.
• Theory of Constraint: the general form of the staging-or-monitoring question, which is where one team’s limited weeks actually go.

Resources

• Canary Release: Danilo Sato, 2014. What the pattern requires of your routing layer, and why rollback is a routing change rather than a deploy.
• Parallel Change: the expand, migrate, contract sequence, and the warning that skipping the contract phase leaves you worse off than when you started.
• Feature Toggles: Pete Hodgson’s taxonomy of flags, how long each kind should live, and the carrying cost of the ones you forget.
• Flagger 1.0: June 2020. Settles that off-the-shelf canary analysis existed, and which meshes and ingress controllers it routes through.
• Argo Rollouts FAQ: settles that a canary does not require a mesh, and what the controller does with replica counts when there is no traffic provider.
• Terraform workspaces documentation, June 2020: the temporary feature-branch copy workspaces are meant for, and the separate-credentials case they are not.
• Accelerate State of DevOps 2019: the time-to-restore bands behind the claim that a fast rollback changes the calculation, and the limits of what they measure.
• Dear Google Cloud: Your Deprecation Policy is Killing You: Steve Yegge, August 2020, and his informal count of how often each cloud’s docs say “deprecated”.

Since we recorded

The off-the-shelf tooling arrived and stayed: Flagger joined the Flux family, which graduated in the CNCF in November 2022. Argo CD 2.3 bundled ApplicationSet in March 2022, and its pull request generator turns each open pull request into a generated Application, which its docs describe as fitting “the style of building a test environment when you create a pull request”. What that environment contains, and whether it matches the production baseline, is still whatever template you write. Andrey made good on the cliffhanger in episode 20, on everything wrong with how we monitor, and the branching half of this argument came back with Johan Abildskov in episode 26.