#69 - Who Is Paulina?
Three-hour test runs and two-month approvals reveal where delivery gets stuck.
17 min read
Join Andrey and Mattias as they sit down with Paulina Dubas, an independent DevOps consultant and public speaker. Who is Paulina, and what experiences does she bring to the table? What topics particularly resonate with her? Tune in to learn more about Paulina since we have a feeling that she is here to stay
The delivery problem nobody sells a tool for
You have the pipeline, the infrastructure as code, the cloud accounts and a review process everyone follows. A one line fix still takes weeks to reach production, and nobody in the chain can tell you where the weeks went. They aren’t on the architecture diagram, because they aren’t in the architecture.
Paulina Dubas is the new voice on the show, and that gap is her working life. She’s an independent lead DevOps engineer and architect, with clients mostly in Denmark and a life split between Poland, Belgium and Denmark. The route in was mechatronics, then automation and robotics, then a thesis spent crashing one meter drones in a basement under the university buildings because the construction tent was too cold to keep them connected. Her first IT job was DevOps at Maersk. The work now is DevOps maturity assessments, solution design and the implementation that follows: where the teams are, where they want to get to, and what has to change in between. She co-founded Code Club Copenhagen and spoke at DevOpsDays Antwerp in September 2024 on sustainability in cloud and DevOps.
What she finds walking into a new client is rarely a tooling problem. It’s communication, feedback loops measured in hours, approval gates measured in months, and an ownership line drawn at the merge button. Every item on that list is fixable. None of the fixes is a purchase.
Two thousand tests before you can merge
Start with the clock, because it’s the part of this you can measure today.
Why does a one line commit wait three hours for feedback?
Because the whole suite runs before anything is allowed to merge. We’ve seen infrastructures running 2,000 tests on every commit, with no merge until they finish. You send one commit, you have feedback in three hours.
Those tests usually aren’t wrong. When you support several versions of your software, or customers on different frameworks and devices, backwards compatibility is a real requirement and it needs real tests. The mistake is where they sit. Teams overuse testing when they want to make things secure, and the structure is what suffers: shift left means fail as soon as possible, and a compatibility matrix that takes hours fails about as late as a pre merge check can. Put the fast checks in front of the merge, and run the long matrix on a schedule or before a release.
If you measure one number this quarter, measure the time from push to first red or green, then work out which part of it is protecting something.
Test volume is the bottleneck you can time. The one we name first isn’t in the pipeline at all.
That one shows up on no dashboard, so a review of the toolchain can run straight past it.
The gate where the business sits in your pipeline
Slow tests at least finish on their own. The other delay waits for a person who has other things on.
What happens when a human approval sits in front of the merge?
Everything behind it waits for that person. We’ve seen a commit wait two months before merge because somebody on the business side didn’t agree it could go to production, and the automated process behind that gate counted for nothing while it sat there.
Gates like that often exist for a reason. Public sector rules, industry regulation and contractual commitments all produce approvals that somebody has to give. What’s worth examining is what each gate approves, and whether that approval has to happen where it currently sits.
The same question is worth asking about the approvals nobody counts as gates. People love pull requests and approvals, and rubber stamping, which turns review into a queue rather than a control. For each one, ask what it would catch and who would act on it. If there’s no answer, you have found time you can give back.
What a merge queue protects
There’s a fix for slow tests that doesn’t involve deleting any of them.
How does a merge queue help when the tests take hours?
It merges for you, once the required checks pass on top of everything already ahead of your change. You say you’re done, your pull request joins the queue, and it waits its turn instead of your attention.
GitHub’s merge queue went generally available in July 2023. The queue builds a temporary branch for each entry, prefixed gh-readonly-queue/{base_branch}, holding the target branch plus the changes of every pull request ahead of yours, and fires a merge_group event your CI has to report on. Entries go first in, first out. One that fails its required checks is removed, and the branches behind it are rebuilt without it.
What you get is narrower than a green mainline. The queue protects you from the failures your required checks actually catch, on the combination of changes about to land. A check you don’t run, or a bug no check looks for, lands the way it always would have. The protection also depends on a setting: with Only merge non-failing pull requests off, a group merges when its last entry passes, so a pull request that failed its own required checks can merge inside that group. The setting exists for teams living with intermittent test failures, and it’s a reasonable trade as long as you know you made it.
Inside that boundary the payoff is real. Nobody layers work on top of a change that failed its checks, so you don’t spend a morning picking apart a ball of commits to find who broke what, and for that class of failure there’s nothing to revert, because the change never landed.
The Friday objection is fair: you queue your change, go home, and find out on Monday whether it merged. That’s the trade, and it beats the failure mode at the other end of the automation. One story involved automated reverts and what we think was a faulty test environment: tests that should have passed failed, so the automation reverted, retested, reverted further, retested. By Monday the CI system had walked back about a year of history. It’s Git, so the repair was a force push from a local copy, but nobody enjoyed that morning. Automate the merge decision before you automate the revert decision.
Merging is not shipping
A queue settles what lands on the mainline. It says nothing about what happens to the change afterwards.
Where does a developer’s ownership end?
Further along than the merge button. “I merged to the default branch, I’m done” is where it usually stops, and at that point the change hasn’t done anything for anyone.
How far it goes depends on the organization. Some teams do development and deployment both, some have a dedicated team for the second half. The part that doesn’t vary: follow up on what you merged, either by watching it yourself or by getting the feedback from whoever runs it. We meet developers who write the code, push it, and never look at what the pipeline said.
Compare the release branch against the default branch and count the commits sitting between them. We’re considering a script that would walk an organization’s GitHub repositories and do exactly that, with a small web page listing the top three repositories by clogged pipe.
The bill arrives the day you need a high priority fix out. You can’t ship it on its own: it drags fifteen unreleased commits to production with it, and some of them carry problems nobody has seen yet. So you either accept a high risk delivery, or you revert everything that’s queued up, ship the fix, and put the reverted work back afterwards. Both options are expensive, and both were created weeks earlier by people who thought they were finished.
A patch is done when production is patched
The same gap between merged and running turns a security fix into a story you tell yourself.
When is a vulnerability actually fixed?
When the fixed version is running in production. We’ve watched someone whose entire job was fixing security issues merge each patch into the default branch and stop there. Development was secure. Production was the part nobody was looking at.
The tooling makes that mistake easy. Dependabot alerts track your repository’s default branch, and fire when the advisory database or your dependency graph changes. Merge the bump there and the alert clears, whatever is deployed. If your default branch isn’t on the same stream of environments that reaches production, the cleared alert and the running code have nothing to do with each other. There are no CVEs, nothing is screaming, and a developer with a couple of years of experience reasonably concludes the job is done. The system misled them.
Three things close that gap. Check the deployed artifact and the dependency version inside it, rather than the repository: on Kubernetes that means the image digest the pod is running, since a tag can be moved to different content. Know which branch each scanner is pointed at, and say so out loud when the answer is the default branch. And track the age of the oldest merged but unreleased fix. That measures how long production has remained unpatched since the fix merged. Count the fixes awaiting release too, which is the clogged pipe above seen from the security side.
Partial promotion needs a team that can run it
The honest objection to all of this is that you can’t always ship everything that’s queued, and sometimes you shouldn’t.
Can’t feature flags solve the stuck commit problem?
They can, if your team can carry them. Promote only the layer that needs promoting, push the bug fix without the rest, or merge the change dark behind a toggle and switch it on later.
We rarely see that done well. More often we see toggles misused, which is its own kind of debt. The technique is documented in detail: Pete Hodgson’s catalogue of feature toggles, published on Martin Fowler’s site, separates the short lived release toggles from the ones that live in the code for years, with the testing cost of each. Blue-green and shadow deployment ask for more again, each in a different way. The blue-green switch is a router you flip from the live environment to the idle one running the new release. Shadow deployment needs request mirroring instead: the proxy in front copies each live request to the new path and throws the shadow response away. Both need tests and observability good enough to tell you the new path is working. That takes a strong technical team, and you don’t always have one.
What a missing rollout mechanism costs on one update path is a matter of public record. CrowdStrike shipped sensor releases in stages, and customers controlled which sensor version they took. Rapid Response Content travelled a different path, as a configuration update delivered from the cloud rather than part of a sensor release, and that path had no staging. The root cause analysis traces the July 2024 Windows outage to a content validator that accepted a template instance referencing 21 input fields while the integration code supplied only 20 inputs to the interpreter. The read past the end of the array crashed the machine. What CrowdStrike committed to is the staging that path lacked: canary deployment first, then successive rings with bake time between them, rollback when the telemetry says so, and customer control over when rapid response content reaches their fleet. Its president accepted the Pwnie for Most Epic Fail at DEF CON a few weeks later, on stage and on camera, which is roughly how we’d want our own worst week handled.
You don’t pick the tool, the people who keep it do
Rollout mechanics run into the same wall as everything else on this list: what the client’s own engineers can operate once you’re gone.
Which infrastructure tool should a large client standardize on?
The one the internal team can maintain after the consultants leave. At a big organization you don’t have much say anyway, and reusing what people already know is usually the right call rather than a compromise.
In practice that means Terraform almost everywhere, with Python around it and sometimes Go. Pulumi turns up, though not in Denmark: the broad usage we’ve seen was at a Swiss customer. OpenTofu went GA with 1.6 in January 2024, and we’ve seen some organizations migrating, while others are testing proofs of concept first. That caution is the right instinct. Before you move a large organization to a new tool you test it on something small and find out whether it does what you need.
Is the field stuck, then? Terraform’s first tagged release, v0.1.0, went out in July 2014, which makes it ten years old as we record. Git has been the default for longer than that. If your job is building tools, that looks like stagnation. From inside a client with hundreds of modules and a team that has to run them next year, it’s the reason any of it works at all.
Out of your own bubble
Every fix above assumes somebody in the room has seen a second way of working.
Why do teams keep doing the thing that doesn’t work?
Because plenty of engineers have only ever seen one setup. Sit in the same team for ten, fifteen or twenty years and that setup stops being a choice, which is the hardest thing to work against as a consultant.
The evidence shows up in odd places. A few years ago people were still fixed on SVN rather than Git, and last year we met developers who hadn’t heard of Lambda. Most people never get to a conference and don’t follow what’s shipping, so they have nothing to compare their own setup against. We’re in a bubble of people who do, and it’s easy to mistake our own agreement for a settled industry view.
The other half is adapting what you find. “I read it, let’s put it in exactly the same way” is how a pattern that works in a startup ends up installed at an enterprise, where it blows up or quietly solves nothing. Knowing how to learn a thing, then fit it to your own constraints, is the skill that’s actually short.
Code Club Copenhagen runs on that premise: a community of practice in Copenhagen, roughly monthly outside summer and Christmas, hosted by a different company each time so the input changes with it. Two talks of 40 to 45 minutes, because attention doesn’t stretch further, then a break, food and drink, and the part people come back for. Quality of talks was the thing to hold onto from the start, since there are always more people who want to speak than there are talks worth sitting through. People have found jobs and collaborators in that room.
Conferences reward the same deliberateness, and which ones pay off is personal. The events with a few thousand attendees are good for finding out what the vendors have shipped, and we’ve come away from them overstimulated and connected to nobody. Events of 150 to 400 people have worked better for us, DevOpsDays Antwerp among them, where the people around us had come to connect. Speaking is the cheat code: you get the organizers, the speaker dinner and the other speakers. If you’re attending, go to the expo area while everyone else is in the talks. The vendor staff work the crowd between sessions and have nothing to do during them, they’re good company, and they know who is doing what.
What this means for teams
If you lead delivery, the uncomfortable part of this list is how little of it is technical. The three hour pipeline, the two month approval, the commits parked on the default branch and the security alert that closed without a deployment are all decisions somebody made once and nobody revisited. An outside pair of eyes picks them out because none of it is familiar yet, which is a good reason to invite one in. You can see them yourself too, but you have to go looking.
If you write the code, the line worth moving is the one between merged and running. That’s where ownership gets vague, and it’s the cheapest place to be the person who checks.
Four things worth doing this week:
- Measure push to first signal and merge to production as two separate numbers, then say which of them you’d defend to a new joiner.
- Diff your release branch against your default branch and count the commits in between. That count is work in progress, and it’s the risk in your next urgent fix.
- Take your longest manual gate, write down what it approves and what depends on it, and move it in front of production promotion if it would do the same job there.
- Pick a security alert that closed last month and check the dependency version in the artifact production is running, by image digest where you have one.
Common questions, answered
How do I turn on a merge queue for my GitHub repository?
Enable the merge queue requirement for the target branch in its branch protection rule or ruleset, then make your CI report on the queue’s temporary branches. GitHub Actions workflows need merge_group added to their triggers, and third party CI has to run on pushes to branches prefixed gh-readonly-queue/{base_branch}. Miss that step and the required check never reports, so nothing merges. Leave Only merge non-failing pull requests on unless flaky tests force your hand: with it off, an entry that failed its own checks can merge inside a group whose last entry passes. The GitHub documentation covers both.
Why did my Dependabot alert close while production is still vulnerable?
Verify that the fixed dependency version is running in production, and read the alert as a statement about one branch. Dependabot alerts track your default branch and react to changes in the advisory database or your dependency graph, so merging the version bump there clears the alert even when nothing has been released. If your default branch isn’t on the stream of environments that reaches production, the cleared alert says nothing about what’s deployed. Check the artifact itself, by image digest where you have one.
How do I get value out of a conference when I’m not speaking?
Spend the talk slots in the expo area instead, where the vendor staff are idle and will tell you what they’re seeing across their customers. The huge events are useful for finding out what the vendors shipped, less so for meeting anyone, at least in our experience. Events of 150 to 400 people have worked better for us. Pick one or two sessions you genuinely need and watch the rest on YouTube afterwards. A local community of practice with a monthly meetup is the cheaper version of the same thing.
Related episodes
- #68 Julien’s last episode?: the handover that made room for a third chair on the show.
- #41 Great communication FTW: our earlier take on the bottleneck that tops this list.
- #14 Theory of constraint: bottlenecks, work in progress and why faster development alone doesn’t ship more.
- #26 Git branching strategies: keeping the mainline releasable, the problem merge queues automate.
- #64 From Terraform to OpenTofu: what migrating a large estate to the fork actually took.
- #7 How do we learn: staying current when your own setup is all you see.
Resources
- Managing a merge queue: settles the mechanics, including the
merge_groupevent, thegh-readonly-queue/branches your CI has to handle, and whatOnly merge non-failing pull requestschanges. - About Dependabot alerts: settles that the scan covers your default branch, which is why a cleared alert says nothing about production.
- Falcon content update preliminary post incident report: settles which update path was staged and which was not, and commits to canary deployment for Rapid Response Content.
- Channel File 291 incident root cause analysis: CrowdStrike’s own account of the July 2024 outage and the 21 fields against 20 inputs mismatch behind it.
- Feature toggles: Pete Hodgson on the categories and the carrying cost, worth reading before you promise your team can run them.
- OpenTofu is going GA: dates the 1.6 release to January 2024.
- Terraform
v0.1.0: the July 2014 tag that settles how long the default tool has been the default. - Paulina Dubas on LinkedIn: where the assessment and community work gets posted.
Since we recorded
Paulina stayed. She has co-hosted since, through 100 episodes later and well past it. Merge queues, test volume and the hunt for a feedback loop you can live with came back in continuous integration in 2026, with the newer twist of an AI reviewer arguing with a coding agent on the same pull request. The patching argument got sharper rather than softer: the 90 day patch window is dead covers what happens to “you’re done when production is patched” once an attacker can turn a published patch into a working exploit in hours.
