Your improvements land everywhere except the queue

Ask a team that has spent a year on DevOps what got faster. Say the answers are a quicker build, a new dashboard, a test suite someone parallelised. Then ask how long it takes to get a feature in front of a customer. If that number hasn’t moved, check whether any of those improvements touched the step that sets the pace.

The reason is usually in plain sight. One step in the chain has everybody waiting on it: a VM request that takes three months, a change advisory board that sits on Tuesdays, one reviewer who has to see every database migration. Work piles up in front of that step, and everything downstream of it idles. The hours you spent on build time went into a step that was never the problem, so delivery never noticed them.

Henrik Høegh works on exactly this problem. He is a senior DevOps and cloud native consultant at Eficode Praqma, and his background is systems administration. He and a colleague installed Kubernetes on bare metal in 2016, when there were barely any scripts for it. He later got the Atlassian Data Center stack onto Kubernetes as Atlassian Software in Kubernetes, a StatefulSet and an operator in place of scaling Jira by hand on VMs. The book that reshaped how he reads an IT organisation is a novel from the 1980s about a factory.

A bottleneck and a constraint aren’t the same thing

Start with the vocabulary, because the distinction gets flattened in casual use and the rest of the argument rests on it.

What’s the difference between a bottleneck and a constraint?

A bottleneck is any step slower than the ones around it, and on a production line the constraint is the step that limits overall throughput below customer demand.

Picture a factory floor, with real workbenches rather than servers. A workstation that does four units a second while the customer wants five is a constraint, and it sets your output at four. A workstation that does four a second while the customer wants three is a local bottleneck, worth optimising one day, and it isn’t deciding anything today. You usually have only one constraint.

Finding it is easier than people expect, because it leaves two marks. Inventory piles up right in front of it, and the station right after it starves, because that station can process faster than the constraint can feed it. Swap inventory for tickets, open branches and half-finished epics, and you’re looking at your own value stream.

That’s why theory of constraints is a focusing mechanism rather than an improvement programme. Lean maps the value stream and goes after waste wherever it finds it. Theory of constraints asks one question first: where do we put effort to get the most value right now? It comes out of Eliyahu Goldratt’s The Goal, a business novel from the 1980s in which a plant manager has 90 days to save his factory. The Phoenix Project is the same skeleton retold in an IT department, and Beyond the Phoenix Project goes back to Goldratt, Deming and Taiichi Ohno directly.

The constraint moves, and it starts at infrastructure

Naming the constraint is the first step. The second is knowing where it tends to sit, and in traditional IT shops the answer is fairly consistent.

Where is the constraint in a traditional IT organisation?

Typically in infrastructure, in the gap between asking for a server and getting one.

We’ve heard of waits of three or four months, and possibly six. However long it runs, it doesn’t matter how much faster your developers get: the work queues up waiting for a VM either way. Hiring more developers only makes the queue longer.

Cloud is the obvious way out, and infrastructure as code is what makes it repeatable: capacity on demand, and the request is a merge rather than a ticket. Kubernetes earns its place here for the same unglamorous reason. When you run out of resources you add worker nodes, and a worker node is more of a thing you already run.

Solve that and the constraint doesn’t disappear, it moves. The next stop is usually the people who deploy, which is the argument for continuous integration and delivery. From there it tends to land on security and test, where a change advisory board evaluates every change and the security review arrives at the end. Building security in and automating the tests is how that one gets answered. Development typically comes last in that sequence, and making development faster buys you something once it does.

The sequence is written down in The DevOps Handbook, and it describes where constraints have tended to sit, not a schedule your organisation has to follow. Find where work actually waits in your own pipeline before you pick a department to improve. Teams that skip that step and start at the end, hiring developers while provisioning still takes a quarter, get a bigger pile of unreleased work and a team that feels slower than it did before.

Your backlog is a buffer, so put a limit on it

If the constraint sets the pace, everything else has to be arranged around it. Goldratt’s mechanism for that is drum, buffer, rope.

Why does a huge backlog mean something’s wrong?

Because the backlog is a buffer in front of your constraint, and a buffer that only grows is telling you the constraint can’t keep up.

The drum is the constraint, beating the pace the rest of the system works to. The buffer is a small amount of work parked in front of it so it’s never idle: if the constraint is an oven, an idle oven wastes the whole factory’s time. The rope is what stops the front of the line pushing in more than the drum can take. In software the backlog does the buffer’s job, and almost nobody treats it like one.

We’ve seen a team carrying a backlog three years deep. That’s the constraint announcing itself in the only language it has, and nothing in that team’s process changes until the backlog does.

A work in progress limit on the backlog deals with it honestly. Pick a number, twenty or thirty, and don’t create a new issue until an old one is closed or dropped. The objection arrives immediately: this new thing is very important. Then it’s more important than something already in there, because everything in there was important when it went in. If it isn’t, delete it and make room for the work you want done.

Watch for the workaround, which is a second list. A pre-backlog of ideas is the same pile under a different name. The real fix sits one workstation upstream, with the business, which works at epic level and should stop breaking epics into atoms until developers are ready to take them. Fifty epics broken into a million ordered pieces isn’t agile, it’s a plan. Agile is about making decisions as late as you responsibly can, which is impossible when everything is already decided.

Kanban carries the same lineage, which surprises people who assume it came from lean. The card comes from lean. The method came from David J. Anderson, whose earlier book applied theory of constraints to software management years before the Kanban book existed.

Borrowed answers skip the learning

The same mistake repeats one level up, where an organisation adopts someone else’s framework because it worked for them.

Is reading the Agile Manifesto worth anything if you already run stand-ups?

Yes, and this is where we disagree about how much it buys you.

Henrik’s workshop version is two questions. Are you agile? Yes. Have you read the manifesto? They haven’t, so the room reads it together and then talks about it, which is the healthy part. If he had to pick one piece of it, it would be the values. The talk he’s preparing runs the same exercise as a show of hands: who’s agile, hands up, then the manifesto goes on screen and anyone who hasn’t read it takes their hand down. Andrey’s objection is that the four values and twelve principles are compressed experience: they were written by people who had been through the pain, and reading them without that pain leaves you with text you nod at. He wants a third level of detail underneath, the way Martin Fowler writes up a snowflake server, where you get the term, the problem it came from, and what to do instead. People in the stone age drew the tiger on the rock. “Responding to change over following a plan” doesn’t draw the tiger.

SAFe is where we split. Andrey’s advice to anyone who hasn’t met it is to stay away. Henrik works inside a company that uses it and finds real parts, PI planning among them. The reports he collects from companies that rolled it out are mixed: the very good teams got better or worse, depending on who is telling it, while the weakest teams improved. That is what forced structure does. His position is that SAFe is a way to start when you can’t design your own process, and nothing more. Take it apart afterwards, keep the pieces that work, throw the rest away and don’t care that what’s left isn’t SAFe. Treat it as the destination and you have repeated the Spotify model, copied from videos meant as inspiration, by which time Spotify had moved on from it.

The lean version is the andon cord. The story goes that a plant pulling it 1,500 times was asked whether that was a problem, and answered that the problem was it only got pulled 1,500 times, because each pull was a chance to learn. Copy the practice and you get a cord on the wall. The learning stays where it happened.

Cloud native is not a synonym for Kubernetes

Borrowed vocabulary fails the same way borrowed process does, and the term taking the damage right now is cloud native.

What separates a cloud native application from one that merely runs on Kubernetes?

The test we’d apply is whether the application talks to the platform’s API, rather than depending only on an address the platform hands it.

Here’s a quiz worth putting to your own team. To reach another service in Kubernetes, do you use the DNS name of a ClusterIP Service, which resolves to the virtual IP that kube-proxy routes onwards? The virtual IP directly? A headless Service, whose records point straight at the pods behind it? Or none of the above? The answer we’re arguing for is none of the above.

None of those three needs to know it is running on Kubernetes, and that’s the point. DNS and virtual IPs are older machinery, in Kubernetes partly so an application that knows nothing about the cluster keeps working. The documentation draws the same line, under the heading cloud-native service discovery, the Kubernetes 1.19 Service page says “If you’re able to use Kubernetes APIs for service discovery in your application, you can query the API server for Endpoints, that get updated whenever the set of Pods in a Service changes”. For non-native applications it offers “ways to place a network port or load balancer in between your application and the backend Pods”. It gives the API route that name; it does not rule the other routes out. Operators are what the API route looks like in practice, because they watch the API server and react to what changed.

Which leaves the definition itself. The CNCF’s v1.0 definition opens with “Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments”, then says containers, service meshes, microservices, immutable infrastructure and declarative APIs “exemplify this approach”. Examples and benefits. Read it looking for a test you could apply to your own service and you won’t find one, which is how a term ends up meaning “we run Kubernetes”. We did this to DevOps, got a clean start with cloud native, and spent it from day one. DevSecOps is going the same way.

Finance is a workstation too

If the constraint can sit in a provisioning queue or a review board, there’s no reason it stops at the edge of engineering.

What does finance have to do with delivery speed?

It owns the buy half of build or buy, so bring it into that decision before you commit engineering time.

The decision is genuinely hard from the engineering side, because the build estimate is a guess. A service costs a known number per month. Building it ourselves costs an unknown number of weeks. Put finance inside the development process rather than at the end of it, and that comparison gets made by people who can price both halves.

What happens instead is funnier and worse. An executive tells you the company isn’t in the cloud. Then you sit in a meeting with a cloud provider and hear that you’re already a customer, because several teams are paying by credit card. Separate cards don’t roll up into one account, so nobody above those teams sees the spend or the dependency, and the pipelines are already running there.

Accounting is ahead of us on one axis and nowhere near us on another. We talk about getting to data-driven. Accounting has been money-driven for years: if you aren’t making money, you’re out the door. The stories that travel with it are about people rather than line items: the large firm said to rank its accountants yearly, fire the worst performers and hire replacements to see what happens. That is the opposite of what Deming argued for, and it is not the part to copy.

Site reliability engineering is the professional version of putting numbers on reliability, with error budgets saying how much unavailability you can spend and service level agreements putting money behind it. We’d say the same thing about SRE that we say about agile: read the discipline’s source material before you assume what the title means. The book is free. “I haven’t read it” is a fine answer. Reading the two words in the name and deciding from there isn’t.

Pick tools for problems you can name

All of this arrives at the table as a tool request, which is the easiest place to apply the same test.

How do you decide whether you need a service mesh?

Name the problem it solves for you this quarter. If nobody can, you don’t need it yet.

Ask what Istio solves and you get a list of capabilities: tracing, mutual TLS, traffic shifting. All real, and none of them an answer to which of your problems it solves this quarter. Istio is good engineering aimed at genuinely hard problems, and the complexity it carries is the complexity of those problems. The project knows it. Istio 1.5, in March 2020, consolidated the control plane into a single binary called istiod on the grounds that fewer moving parts are easier to install, run and upgrade. We’ve watched companies run it in production on alpha features. Linkerd takes the other route, where every step is a command you can read first. linkerd install prints the control plane manifests instead of applying them. The install is that output piped onwards: linkerd install | kubectl apply -f -. Your own workloads join separately. Annotate their namespace with linkerd.io/inject=enabled, then restart the deployments there. Linkerd’s Knative walkthrough from March 2020 is those lines end to end.

Microservices get the same question in a different shape, and the one that ends most conversations is backups. You have a database per service. How do you take a consistent backup across all of them, and restore it? We’ve asked that in a kickoff meeting where the plan was event-driven microservices, and the person who wanted them had no answer. Taking a backup isn’t an advanced requirement.

Orchestrators got decided while people were still arguing about them. Mesos keeps an edge in a shrinking area around Spark-style workloads. Kubernetes stopped being something you install the hard way and became a button in a console. The prediction worth repeating is that in five years nobody talks about Kubernetes, the way nobody talks about the Linux kernel, and what matters is what gets built on top. Knative and Cloud Run already point that way from two directions. Knative abstracts Kubernetes rather than replacing it: you ship a container and some YAML, and somebody still runs the cluster underneath, which is why that Linkerd walkthrough starts by installing Knative Serving into one. Fully managed Cloud Run is the version where the cluster genuinely isn’t yours, because Google runs the infrastructure under it.

What this means for teams

The hard part of all this is the conversation afterwards. We’ve stood in front of a team and said that months of their work was good and had to be thrown away, then stayed late that evening and built something that did more than the thing it replaced. The room goes quiet. Changing what people believe is slower than changing what your cluster runs, and it’s the work that decides whether any of the rest lands.

So start where the evidence is rather than where the enthusiasm is. This week:

• Walk the path from idea to production and write down the waiting time at each handoff, not the working time. The longest wait is your candidate constraint.
• Count your backlog. Over a few dozen items, agree a cap, and start deleting to make room instead of adding.
• Before the next tool goes in, write one sentence naming the problem it solves for you, with a number in it. If the sentence needs a conference talk to make sense, wait.
• Ask finance how many cloud accounts and corporate cards your engineering teams are using, then ask your cloud provider the same question.

Common questions, answered

How do I stop developers waiting months for a server?

Move provisioning onto a self-service path and take the hand-offs out of it, which in practice means a cloud account and infrastructure as code rather than a ticket queue. A wait of months comes from one team handling every request by hand for twenty development teams at once. Improving downstream capacity alone won’t increase throughput while provisioning remains the constraint. If you already run Kubernetes, the same logic covers capacity: when you’re out of room you add worker nodes, which is a decision about a number rather than a procurement project.

How big should our backlog be?

Cap it at a number you pick in advance, something like twenty or thirty items, rather than letting it grow and then deleting it in a panic. The backlog is a buffer in front of your constraint, so a backlog that only grows is a signal rather than a resource. When something new arrives described as very important, it has to displace something already in the list, because everything in the list was important when it went in. Anything that can’t displace anything isn’t worth keeping, and dropping it makes room for the work you want done.

How do I tell whether a team is agile or just running the rituals?

Ask how late the team is still allowed to change its mind, and whether anyone has read the Agile Manifesto. A team working through fifty epics that were broken into ordered tasks months ago is executing a plan, whatever the stand-ups look like. The values take ten minutes to read, and reading them alone doesn’t transmit the experience behind them, so go through them as a group and talk about which ones you’re failing.

Related episodes

• DevOps What: where DevOps came from and how it lines up against SRE, the vocabulary this episode keeps testing.
• Scale vs Scaling: what to do when the thing that needs scaling is the organisation rather than the software.
• What Are We Working On: picking an orchestrator, and the case for not running Kubernetes.

Resources

• Henrik Høegh on LinkedIn: our guest, and where his posts on backlogs and SRE turn up.
• The Goal: Goldratt’s business novel from the publisher, the origin of theory of constraints and of drum, buffer, rope.
• The Phoenix Project: the same story in an IT department, and how most engineers meet constraints at all.
• Beyond the Phoenix Project: Gene Kim and John Willis on Deming, Goldratt and Ohno, for the lineage rather than the story.
• Manifesto for Agile Software Development: four values and twelve principles. Settles what agile says, not what any team does with it.
• CNCF Cloud Native Definition v1.0: the official wording, and the evidence for the complaint that it lists benefits and examples where a definition should be.
• Kubernetes 1.19 Service documentation: names querying the API server cloud-native service discovery, and offers DNS and virtual IPs for applications that can’t.
• Failed #SquadGoals: Jeremiah Lee, April 2020, on why Spotify had left the Spotify model behind while everyone else was adopting it.

Since we recorded

The CNCF replaced v1.0 with a rewritten v1.1 in February 2024: it opens on practices rather than technologies, and calls cloud native systems loosely coupled, secure, resilient, manageable, sustainable and observable. Declarative APIs are still one example among several, so the revision changes the wording rather than settling what the term rules out. The Kubernetes documentation still calls querying the API server cloud-native service discovery, though what you query has changed, because Kubernetes 1.33 officially deprecated the Endpoints API in favour of EndpointSlices. Henrik came back in episode 38 to talk about platform teams and Team Topologies, this argument with the constraint one workstation further on.