#8 - DevOps What
Buying Ansible or hiring a DevOps engineer does little without changing how teams work together.
15 min read
Andrey tells us the story of how DevOps came into existence and took over the market. We discuss the marketing around it, its relationship with DevSecOps. We tried to shed a light on what is marketing strategy versus implementing DevOps in an organization. We also compared DevOps to SRE (Site Reliability Engineering).
A job title that nobody can define
Someone asks you what the difference is between a DevOps engineer and a site reliability engineer. You’ve worked beside both. What comes out of your mouth is a list of tools, and you can hear that it’s the wrong answer while you’re saying it. We got that question from a student after a university lecture, gave an answer we weren’t happy with, and went back to the source material to find a better one.
The honest answer is awkward, because the two aren’t the same kind of thing. DevOps is a way of working with no single definition everyone agrees on. SRE is a job, with a published discipline and a free book behind it. Asking which one is better is close to asking whether Scrum beats senior developer.
That distinction pays for itself past the pedantry. It decides what your manager thinks they’re buying when they say the company needs DevOps, who the recruiter sends, what you study this year, and what you put in the headline of your own profile. Hire a DevOps engineer, send them on an Ansible course, and nothing about how your organisation ships software has changed.
DevOps started as a name for a conference
Begin with where the word came from, because the hole it left explains most of what happened next.
Why is there no single agreed definition of DevOps?
The word was coined to name an event rather than to label an idea, and the definitions came later, written by whoever needed one.
Two weeks before the Agile 2008 conference in Toronto, Patrick Debois wrote that he would be speaking on agile infrastructure and operations: “It often strikes me that the Agile community is only known in development circles, whereas the Agile idea also applies to other sections of the company.” In March 2009 he counted twelve submissions to Agile 2009 in Chicago about pulling sysadmins, infrastructure and service management closer to agile development, and called it “frontier stuff, an emerging field of new ideas”. Andrew Clay Shafer and Paul Nasrat had two of them, on agile infrastructure and on building infrastructure with Puppet.
The original idea was that plain. Take the practices development had been sharpening for a decade, continuous integration, short feedback loops, iterative planning, and apply them to operations so that delivery stops dying at the wall between the two groups. It had a name that said what it was: agile system administration.
That name didn’t survive its own length. Later in 2009 Debois ran the first DevOpsDays in Ghent, and in a 2012 InfoQ interview he described the naming as an accident: agile system administration was too long for a conference title, and too narrow, because it put sysadmins alone in the frame. The giveaway sits in that first program. Six sessions: Kanban in operations, continuous integration and build pipelines, agile infrastructure with Puppet, monitoring rebuilt for the cloud, non-functional requirements, a private cloud for developers. Not one of them has DevOps in the title.
A word with no definition gets filled by whoever is selling
A term that means something good and nothing specific is a gift to a marketing department, and two groups unwrapped it.
Does buying a configuration management tool make you a DevOps team?
No, and the question is backwards: the tool is the small part of the change, which is exactly why it’s the part that got sold.
Infrastructure tooling vendors worked out that DevOps sold, so their products became DevOps tools. We won’t point fingers at Chef, Puppet and Ansible, except that we just did. The pitch answered the only question buyers were asking. How do we do DevOps? You run our tool, and then you’re doing DevOps. Tools are the tangible bit, the thing you can install on a Tuesday, so the forest disappeared behind the trees.
The other group was managers who wanted an easy fix, with recruiters happy to supply one. If DevOps is good, the reasoning went, we should do DevOps, and doing DevOps means hiring a DevOps engineer. On the other end of the phone sat a pile of system administrators nobody was placing. Learn Ansible, get relabelled, and now you’re a DevOps engineer who configures servers. Buying the gym membership and gaining muscle are different events, and the card in your wallet does nothing on its own.
This isn’t only our complaint. Google’s own SRE workbook says the same thing from the other side: you can’t acquire this by hiring people with the title on their badge, because what has to change is how the organisation works.
DevSecOps arrived the same way
The pattern repeated with security, close enough that you can watch it happen in slower motion.
Who coined DevSecOps?
We haven’t established who coined it.
Our recollection, hedged: we heard DevSecOps from conference stages in 2014 and 2015, mostly from Sonatype, while almost nobody else was using it. Sonatype sells Nexus, the artifact repository, and was building component analysis around it. Its 2013 launch of Sonatype CLM covers the open source components an application pulls in, where “security, licensing and quality intelligence are key to assessing risk and fixing flaws”. A product like that needs a word, and the word has to sound bigger than the competition’s. We didn’t do the digging on this one that we did on DevOps, so treat it as a hunch.
What the record shows is messier. devsecops.org carries an early organised statement of the idea, a manifesto arguing for security as code and for “leaning in over always saying no”, and Shannon Lietz is the person behind it. A 2015 post there sets out the thinking: everyone is responsible for security, and security decisions belong with the people who have the most context. Sonatype launched All Day DevOps in 2016 and has sponsored it since, which is the loudspeaker we remember. None of that settles who said the word first.
The origin matters less than the test you can run at home. Ask whether anything about how security work reaches your pipeline changed, or whether a slide got a new title.
What DORA counts instead of tools
If the tools aren’t the thing, something has to replace them as the thing you look at, and survey research has been doing exactly that for years.
If DevOps isn’t the tools, what is there to measure?
Four delivery metrics and a catalogue of capabilities, published yearly by the DevOps Research and Assessment team, DORA.
The metrics are deployment frequency, lead time for changes, change failure rate and time to restore service. Two of them measure speed, two measure stability, and the finding that made the reports famous is that teams don’t trade one for the other. The 2019 State of DevOps report put 20% of respondents in the elite group, up from 7% the year before, deploying 208 times more often than the low performers. That research carries weight because of who stands behind it, Nicole Forsgren, Jez Humble and Gene Kim, and because of where the answers come from. Almost 1,000 working professionals answered the 2019 survey, one year’s slice of the over 31,000 responses the research has collected across its six years, and that sample sits well outside any one vendor’s customer list.
What the research actually sells you is the capability list: continuous integration, continuous delivery, trunk-based development, test automation, loosely coupled architecture, monitoring and observability, shifting left on security, generative culture, transformational leadership. Tools show up in that list once, as empowering teams to choose their own tools, which says who chooses and nothing at all about what gets installed.
One practical note if you go hunting for the older reports. DORA was acquired by Google in December 2018, so the original research site redirects into Google Cloud and the PDFs live there.
SRE is a job with numbers attached
Now the other half of the student’s question, which is a different kind of answer because SRE is a different kind of thing.
What is an SRE actually responsible for?
The reliability of a service, stated as numbers agreed in advance with the people who depend on it, and defended with code instead of heroics.
Google published the discipline as a free book, and the part that reframes the job is the arithmetic of failure. “100% is probably never the right reliability target,” the book says, because “not only is it impossible to achieve, it’s typically more reliability than a service’s users want or notice.” You pick a service level objective, and the gap between that number and perfection is a quarterly error budget you’re allowed to spend on shipping.
That turns an argument into a conversation. Do you need to be up all year, or do you need to survive Black Friday? An emergency service answers differently from an internal reporting tool, and the answer sets how much ceremony a change needs. Without the number, every deploy is a judgement call made by whoever is most nervous.
The engineering half is solving operational problems with code. Take request draining, which bites most teams running their own web servers: a deploy swaps in a new version while requests are in flight on the old one, and if nothing drains them, customers collect errors from a release that looked clean. At ten deploys a day that becomes a line on a graph. Fixing it means reading code you didn’t write, which is why SRE tends to be a senior role and why the book expects SREs to advise product teams on resilient design.
How the two fit together is a question of emphasis. One reading makes SRE a specialisation: apply the delivery practices to infrastructure, couple that with software engineering, and SRE is what comes out, DevOps centred on feedback and SRE on SLIs, SLOs and SLAs. There’s an order to it, since continuous delivery fails without robust continuous integration underneath it. That coupling is loose, though: a team can do serious reliability work and still deploy by hand. The workbook puts it in a line, “class SRE implements interface DevOps”, and adds that SRE is closer to a concrete job than DevOps engineer ever was. Debois backs the feedback half: his 2009 map of the ideas centres on feedback loops between business, delivery and operations.
The work most of us do is platform work
Both labels leave out what fills our actual week, and a third term describes it better.
What do you call the team that runs Kubernetes, Vault and the pipelines?
A platform team, building an internal product whose users are the product teams.
Look at why each piece is there. Kubernetes isn’t on the cluster because it’s interesting; it’s there because we’re assembling an application delivery platform. Vault joins it so secrets stop living in repositories. Jenkins, GitLab or whatever runs your pipeline takes an application through its steps and puts it somewhere. The result is that product teams stop answering the same three questions every quarter: how do I route traffic, how do I expose my service, how do I get my secrets. We also go and coach them. If a team doesn’t know Docker, we teach it, because packaging with dependencies is the handover we need from them.
Team Topologies splits that work in a way worth borrowing. A platform team supplies “a compelling internal product to accelerate delivery by Stream-aligned teams”. An enabling team “helps a Stream-aligned team to overcome obstacles” and “detects missing capabilities”. We’d been calling ourselves the enabling team, and the book says the delivery platform is the platform team’s product, with the coaching sitting in the other box.
Putting platform engineer on a public profile is a different question. We weren’t sure recruiters would search for that title.
Marketing yourself when the titles are broken
Which leaves the question the student was really asking, one job interview away.
Why is it so hard to describe what you do?
The string that gets you found and the sentence that describes your value aren’t the same string, and most of us try to make one do both jobs.
Ask us and you get two answers. One is software engineer: that’s what the degree says, that’s what a decade of the work was, and the infrastructure and cloud part is the specialty on top. The other is whatever the LinkedIn profile currently says, probably DevSecOps engineer, and that’s a guess rather than a memory. The searchable label is one option, written because it’s the term recruiters type, and a profile nobody finds helps nobody. Underneath whichever label you pick, the description should be about what you changed, which is the argument in Patrick McKenzie’s Don’t Call Yourself A Programmer: “Engineers are hired to create business value, not to program things.” His own example is a small Ruby program written at a Java shop that made the company $30,000, after which nobody asked what had produced the deliverable.
Certifications sit in the same bucket, and nobody has ever asked us in an interview whether we hold one. Every time one has come up, it came up because we mentioned it. That doesn’t make it worthless. Six AWS exams in 2018 took real study and real guessing on the day, and the hands-on formats prove more than multiple choice does, because the CKA drops you at a command line in front of a live cluster and you either fix the thing or you don’t. The plan for the Terraform Associate is one pass through the study guide to catch anything basic that daily work skips, then sit it once and not think about it for two years.
From the hiring side, the filter we keep coming back to is simple. Does the candidate treat something they don’t know as a challenge, or as not their problem? Both answers are legitimate depending on whether you want a deep specialist or a team that figures things out, and the interview you design says which culture you have. One warning from the other direction: value delivered beats technology chosen while you’re inventing and investing, and when a product moves into harvest you standardise anyway, because maintenance that depends on one person’s unusual stack is a bus factor of one.
What this means for teams
Nobody is going to hand you a definition that settles the argument, so stop waiting for one and pick the parts that do work. The capability list is real, the four metrics are real, error budgets are real, and the job title is marketing in both directions.
This week:
- Write the definition of DevOps you’d give a customer in one paragraph, then check whether your team agrees with it. Misaligned vocabulary costs more hours than misaligned tooling.
- Measure the four delivery metrics for one service: deployment frequency, lead time, change failure rate, time to restore. Guessing doesn’t count.
- Pick one capability from DORA’s list you don’t have, trunk-based development or test automation or monitoring and observability, and put it on this quarter’s plan with a name against it.
- Agree an availability number with the business for one service, subtract it from 100%, and spend that error budget on purpose.
Common questions, answered
How do I start using error budgets when nothing has an SLO?
Pick one service, agree an availability target with the people who pay for it, and treat the difference between that target and 100% as the budget you’re allowed to spend. The number matters less than the conversation it forces: asking whether the business needs the service up all year, or needs it to survive a Black Friday, matches the target to what that service actually has to do. Google’s SRE book says 100% is “probably never the right reliability target”, because it’s typically more reliability than a service’s users want or notice. Start where deploys make people nervous, since that’s where a budget changes behaviour fastest.
Do cloud certifications help you get a DevOps job?
Pick one for what preparing for it teaches you, because that part is yours whatever a hiring process asks for. No interviewer has ever asked us whether we hold any, so in our own experience it has only ever come up because we raised it. AWS exams are wide enough that you have to know the services to pass, and performance-based formats like the Certified Kubernetes Administrator exam, where you fix a live cluster from a command line, test more than multiple choice can. Our own plan for the Terraform Associate is one pass through the study guide, then sit it once.
Should we hire a DevOps engineer to fix our delivery problems?
Start with the capability you’re missing, because one job title doesn’t change the handoffs between your teams. Google’s SRE workbook says you can’t acquire this by hiring people with the title on their badge, since what has to change is how the organisation works. Relabelling system administrators doesn’t change how teams work. Pick a DORA capability you can’t honestly claim, put a name against it, then hire for the gap that leaves.
Related episodes
- Infrastructure as Code: the same tools-versus-practice argument, in the one place where tools genuinely do carry the practice.
- What Are We Working On: whether to run Kubernetes at all, which is the decision sitting under every delivery platform.
- How Do We Learn: where the study plans and certification habits in this one started.
Resources
- Agile Infrastructure and Operations: How Infragile are you?: Patrick Debois, July 2008, the idea stated plainly before the word existed.
- Patrick Debois on the State of DevOps: the 2012 InfoQ interview where he explains the naming and what the tool noise did to it.
- What is DevSecOps?: Shannon Lietz, 2015, an early written account of what the term was meant to carry.
- Accelerate State of DevOps 2019: the report itself, including who answered it and how many.
- DevOps capabilities: DORA’s catalogue, which is the closest thing to a checklist this field has.
- Embracing Risk: the SRE book chapter on why 100% is probably never the right reliability target, and where the error budget comes from.
- How SRE Relates to DevOps: Google’s own answer to the student’s question, including the line about hiring not being a strategy.
- Team Topologies key concepts: the four team types, and the difference between a platform team and an enabling team.
Since we recorded
Platform engineering got the vocabulary it was missing: the CNCF’s platforms white paper defines a platform as “an integrated collection of capabilities defined and presented according to the needs of the platform’s users”, which is the job description we couldn’t put on a profile in 2020. DORA’s research kept running and now lives at dora.dev, though the four metrics didn’t hold still: its dated history of the metrics records time to restore service redefined as failed deployment recovery time in 2023, and deployment rework rate added as a fifth metric in 2024. We came back to what has held up across a hundred episodes in episode 100, and to what to learn and what to call yourself now in episode 112.
