Nobody pays you to keep the servers warm

Your company doesn’t pay you because servers are hard. It pays you because the business has to change, and change is the thing most likely to break what already runs. A new feature, a new price list, a new regulation, a customer with a security questionnaire: each one arrives as a request to alter a system that people are using right now. Turning those requests into infrastructure that stays up is the job.

It’s a strange job to hold, because the tooling to make a change has been good for years. We have APIs for everything, we have infrastructure as code, and we still can’t make a change safely without someone who knows what this particular system does when you touch it. That gap is where the salary lives, and it decides what you should get better at next.

Change is where the outages come from

Start with the part of the work that refuses to automate itself.

If the tools are this good, why isn’t applying a change routine by now?

Because running the change is the easy half, and the hard half is knowing what it does to a system nobody holds in their head completely.

Google’s SRE book puts a number on it. In the Change Management section of its tenets, it says “SRE has found that roughly 70% of outages are due to changes in a live system”. That figure gets quoted as a study, and it isn’t one: it’s Google’s own experience of its own systems, with no breakdown by change type and no claim that your ratio matches. Take it as a strong prior rather than a measurement of your pipeline.

The flip side is the tell. Leaving a system unchanged avoids the failures that changes trigger, and every other cause of outage stays where it was. Machines out there still read punched cards and do their one job correctly, and that is the whole of what standing still buys. The 30% isn’t a plan you can choose, though. A business that stops changing has a shorter problem than its outage budget.

The system gets more complex than the people who run it

If change is the risk, the next question is why the risk doesn’t shrink as a team gets more experienced.

What happens when a system outgrows the people who understand it?

The reasons behind it stop transferring, and the next team relearns them by breaking things.

Jonathan Blow made that argument for an entire civilisation in Preventing the Collapse of Civilization, his 2019 DevGAMM talk. Knowledge doesn’t persist on its own. Every generation has to be taught it, teaching is slow and lossy, and a technology can stay in daily use long after the reasoning that produced it has left living memory. The same mechanism runs at the scale of a CPU design or a service. Complexity grows past the point where any one person holds it, one veteran ends up as the only complete copy, and people fresh out of school can’t see the full weight of a decision they weren’t there for.

You’ve met the small version of this. You join a team, you read the code, and the question forms on its own.

This is also the honest answer to why the work never runs out. The same problems keep arriving at teams that have no record of how the last one was solved.

Teaching is the part of the job that compounds

Knowledge that doesn’t transfer gets paid for twice, which puts teaching on the list of infrastructure skills.

Why does teaching belong in an operations job at all?

Because the person best placed to explain a problem is whoever just finished solving it, and that window closes fast.

The best time to plant a tree was twenty years ago, the second best time is now, and teaching works the same way. There’s no qualifying level. If you had a problem, that’s a talk: here was the problem, here’s how I solved it, here’s why. A blog post does the job. A five-minute explanation at a meetup does the job. Write for the person two years behind you, who is about to make the decision you already paid for.

The side effects are larger than they look. The friendships that produced this show started at talks given to strangers: a KTH lecture on Istio service mesh, a hallway at KubeCon. It opens doors as well. Google’s Developer Expert program recognises people who already share what they know with other developers, and once you’re in, Google helps you teach Google Cloud to more of them.

Then pick the format you will actually use. Writing is genuinely hard for some of us, and a recorded conversation with people who will argue with you is an easier way to get the same material out. That’s the origin of this show: a standing wish to start teaching, a blog that kept not getting written, and a yes to a conversation instead.

Who the work is actually for

Teaching points outward, and so does the next question, which is who benefits when you do this job well.

Who is the platform you’re building actually for?

The developers who have to use it, which is a different goal from having a cluster.

Moving a hosting platform onto AWS and Kubernetes is the current in-house job at Booli, the Swedish property search site, and the cluster is the means. The deliverable is a platform the developers love, so they stay happy and ship what the company needs. That reframing changes what you measure: whether the people on your platform are waiting on you.

Ask the same question one level out and the answer gets less comfortable. A lot of the strongest engineers of this generation work on financial systems. Walk into a hospital, look at the IT that carries people’s health records, and you want to cry. 2020 spent a year showing everyone what public health data reporting looks like under load. The games industry is a defensible compromise: it cures nothing, and it does add some joy to people’s days.

There’s a blunter version of the same measurement, and it’s worth knowing about your own system. An outage at a property search site has different consequences from an outage in a hospital system. Which kind of system you operate tells you how much of your life it’s entitled to.

The lonely part of going independent

That question sharpens when you work for yourself, because then you also pick the clients.

What do you actually lose when you go independent?

You can lose the colleagues you’d normally turn to when you’re stuck.

You decide what you work on and who you work with. You may collaborate with a team inside the customer, sometimes a team doing work close to yours, and you’re still slightly an outsider in it. Nobody has your back by default.

FivexL is one answer to that: five independents on one stack, infrastructure as code, immutable infrastructure, mostly HashiCorp tools and containers, all pointed at application delivery platforms for startups and small companies on AWS. Shared tools mean they can reuse each other’s work and scrum together on a stuck problem instead of grinding at it alone. Each of them keeps the independence and gains a fallback.

What a pooled team ships

The first visible output of pooling people like that is the small code everyone needed anyway.

What does a group of five independents produce besides invoices?

The pieces every client turns out to need, which is why the first two things out the door were a spot price lookup and a tag generator.

fivexl/terraform-aws-ec2-spot-price reads current spot prices across the availability zones you name and returns spot_price_min, spot_price_max and spot_price_avg. The one worth knowing about is spot_price_over: the maximum price multiplied by custom_max_price_modifier, which defaults to 1.05. Read that five percent as price headroom. AWS took the bidding out of Spot on 28 November 2017, so your maximum price competes with nobody. It’s the ceiling at which EC2 interrupts your instance once the spot price climbs past it. What the interruption does to the instance is decided by the interruption behaviour you set: terminated unless you say otherwise, or stopped if you ask for stop on a persistent request. A little headroom keeps an ordinary price move from taking your capacity away.

The other kind of interruption is untouched by any of this. When EC2 wants the capacity back it issues an interruption notice two minutes ahead, whatever maximum you set, and AWS says those notices are emitted on a best effort basis. Set hibernation as the interruption behaviour and the two minutes go away: the notice still arrives, but hibernation begins at once. Plan for the capacity to leave, not for the warning to reach you.

fivexl/terraform-aws-tag-generator encodes the conventions from AWS’s tagging whitepaper: lowercase keys, hyphens between words, a prefix of your own. It fills default:git:remote by running git config --get remote.origin.url, so a resource carries a pointer to the repository that created it. The rest covers the cases that bite later: default:data:pii, default:data:phi, default:data:pci and default:data:classification for data, plus default:garbage-collector:enable and default:garbage-collector:time-to-live for the cleanup program that hunts untagged resources, so a resource can say whether it may be collected and how long it should live.

The tag module is honestly a work in progress. The keys are there; the written reasons for each value aren’t yet, and that gap is the thing we just spent two sections warning about.

Why the same advice lands when it comes from outside

There’s a reason a group of outsiders gets hired to say things a company’s own staff have been saying for a year.

Why does a consultant get heard saying what your team already said?

Because a signal from outside the relationship is easier to act on than the same signal from inside it.

Employees say the infrastructure should be code, or that this cluster needs an owner, and nothing moves. A consultant says it, and suddenly these people know what they’re talking about. What changed between the two meetings is where the message came from.

The analogy we keep coming back to is a couple who have had the same argument for years, where words from a third party land differently than the same words across the kitchen table. There’s an app sketched out of that idea: let a couple log the invisible work and flag an emotion with one button, the way an agile board makes work visible, so the conversation has something to point at instead of two people’s memories. It started as a joke about husband points, one for every changed diaper. Monetising it would wreck the purpose, so it would have to be built as a non-profit.

The transferable part has two halves. Make the work and the risk visible in writing, so the argument stops being about who remembers what. Then, when you’ve been repeating yourself for a year, spend money on one outside read instead of a louder version of the same meeting.

Fit the tool to the problem you have

Outside reads are also how teams find out they bought the wrong tool for the scale they run at.

How do you know whether you need the architecture everyone is talking about?

Start by writing down the scale you actually operate at.

The analogy that sticks: you can build your own house and get the same comfort as a hotel room, and the hotel is faster and more convenient. Both are correct answers to different trips. The failure mode is picking microservices because they scale, when nothing you run needs that scale and you’ve bought the operational cost of a system designed for someone else’s traffic.

What this means for teams

The job is delivering change into systems that outlive everyone’s memory of why they’re built that way. Tooling moves that line a bit each year; the reasoning and the handover don’t automate. If you want to be worth more next year, that’s where to spend the effort, in-house or independent.

Four things worth doing this week:

• Write the reason next to the next weird thing you touch, in the code or the pull request, so the next person doesn’t pay for the lesson twice.
• Tag resources as you provision them with an owner, an environment, a data classification and the Git repository that creates them.
• Publish one problem you solved this month. Three hundred words counts.
• Write down the scale you actually run at, in requests, data size and team count, before the next architecture decision.

Common questions, answered

How do I start teaching if I have never given a talk?

Write up the last problem you solved, in about 300 words, wherever you already post. Keep the shape simple: here was the problem, here’s what I tried, here’s why the fix works. You don’t need a topic nobody has covered, because the reader who needs it is two years behind you, not ahead. Talks come later from the same material. If sitting down to write is the part that stalls you, say it out loud to someone who’ll argue back and record that instead.

What tags should I put on AWS resources before I need them?

Four: an owner, an environment, a data classification, and the Git repository that created the resource. AWS’s tagging whitepaper asks for lowercase keys, hyphens between words and a prefix of your own, so acme:data:pii rather than PII. Add a lifecycle pair if anything in your account deletes resources automatically, so a resource can declare whether it may be collected and how long it should live. Agree all four values as part of provisioning and record them in the tags then, because working them out six months later means asking around.

How do I get management to act on advice my team already gave?

Put the risk and its cost in writing where people can point at it, then buy one outside read of that document. The same message carries differently when it comes from outside the relationship, which is why companies hire consultants to repeat what their staff said. Use that rather than resent it: the written version is what the outsider will anchor on, and it means the decision stops depending on who remembers which meeting.

Related episodes

• #1 Infrastructure as Code sets up the baseline these Terraform modules assume.
• #7 How Do We Learn covers the input side of the teaching argument: where the knowledge comes from before you pass it on.
• #12 Scale vs Scaling is the long version of knowing what scale you actually operate at.
• #19 Deleting Resources in the Cloud is the cleanup side of tagging, including the functions that go looking for orphans.

Resources

• Site Reliability Engineering, Change Management: the source of the 70% figure, and the place to see that it’s Google’s own experience rather than a published study.
• Preventing the Collapse of Civilization, Jonathan Blow, DevGAMM 2019: the argument that knowledge has to be re-taught every generation or it goes.
• Streamlined access to Spot capacity, AWS, 28 November 2017: the announcement that ended bidding and turned the maximum price into an optional ceiling.
• Spot Instance interruption notices, AWS documentation: settles that the notice is best effort, that hibernation gets no two-minute warning, and which interruption behaviours you can pick.
• AWS Tagging Best Practices whitepaper: the tag key conventions, and its guidance on applying tags as part of provisioning.
• fivexl/terraform-aws-tag-generator: the full tag list, including the data classification and garbage collector keys.
• fivexl/terraform-aws-ec2-spot-price: the price outputs and the custom_max_price_modifier default.
• Where to find us: bisconti.cloud, lifeandshell.com, andreydevyatkin.com and fivexl.io.

Since we recorded

Julien co-hosted until #68 in October 2024, and Paulina Dubas joined from #69. “Mostly HashiCorp tools” reads differently since 10 August 2023, when HashiCorp put future releases of its products under the Business Source License. That licence change prompted the OpenTofu fork of Terraform, which we covered in #54 and returned to with Timur Bublik’s migration experience in #64. The career question came back around in #112, with a different answer about what’s worth learning first.