Podcast about DevSecOps

#31 - Hiring an infrastructure automation person

A small Terraform assignment reveals how candidates search, troubleshoot and handle credentials.

15 min read

Have you ever conducted an interview to hire an infrastructure automation person? What would you ask? How do you check their skills? And what skills are essential? Tune in for our tips on hiring and finding the right person for your team!

Discuss the episode or ask us anything on LinkedIn


The job title tells you almost nothing about the role

You’re about to hand someone production credentials, write access to the pipeline and a place in the on-call rotation. The CV in front of you lists tools. It doesn’t tell you whether that person reads a Terraform error message or changes something at random until the red text goes away. Nothing in it covers how they look things up, how they keep their own secrets, or what they do when they hit a wall and nobody can help.

It also doesn’t help that “infrastructure automation person” is several different jobs. In one team it’s the person who keeps Terraform honest, in another it’s whoever runs the cluster, in a third it’s a developer who inherited the pipeline. Define which one you’re hiring for before you write a single question, because the questions change with the role. We’ve been doing a lot of this lately: a summer internship intake at a small consultancy, plus a senior position to fill. That means interviews most weeks, from both sides of the table.

Our short version of what works: hire for attitude, train for skill, keep the technical exercise small. Spend the interview watching someone work in their own terminal instead of asking them to recite. Where the three of us don’t agree is how much of a candidate’s life outside work is any of your business.

What the paperwork round is for

Nothing technical happens first. There’s a round of asynchronous work, and it isn’t there to filter on pedigree.

What do a CV, a personality test and a short essay actually tell you?

How someone writes, and how they think about themselves, which is the part a bullet list of tools leaves out.

We advertise on public job portals and ask for a CV. On top of that comes a personality test from any public site that runs one, usually the free MBTI-style questionnaire at 16personalities.com, plus an English assessment from any public service. The test result is not a score we rank people by, and we don’t put anyone in a box with it. It’s a direction to dig in the conversation later. That’s our practice, not a vote of confidence in the instrument. A 2019 critique of MBTI theory finds the evidence behind the four letters thin, which is reason enough to treat them as a conversation opener rather than a measurement.

The piece we read most carefully is the short essay that goes with it: what do you think about your own result? That answer tells you how a person thinks. The way they speak comes through in how they write, and the sophistication of a page of prose about themselves carries more signal than the questionnaire that prompted it.

Then a screening call, fifteen to thirty minutes. It’s enough to put a face to the person and get a first feeling in both directions: do we like them, do they like us. Candidates who get past that get the technical assignment.

One small Terraform assignment, as deep as you want to go

That assignment is one sentence long, and one sentence turns out to be enough.

What should a take-home for an infrastructure role look like?

Small enough to state in a line, open enough that five candidates solve it five different ways.

It sounds easy. The answers are not the same. Some people boot an EC2 instance straight away, install Apache or Nginx and serve the file. Others go to S3 and turn on static website hosting. One candidate built an API Gateway with a default response that returns the HTML, which was clever and not something we’d have thought of. You can do it with a Lightsail instance, with a container on Fargate, or with an ALB default response. If you want to raise the difficulty, ask for three different services, three different ways, plus an argument for which one fits which situation.

The EC2 answers are where the follow-up questions live. How do you make sure this server doesn’t change its address? You can see immediately whether an Elastic IP is attached. Is port 22 open to the whole internet? Is port 443 open with no TLS behind it, because they saw that in an example somewhere? Some candidates get further and add create_before_destroy to the lifecycle block, so a replacement instance exists before the current one goes away and the Elastic IP moves over. You also find out whether credentials got committed to GitHub along with the code.

None of that is in the assignment text. All of it comes out of one line about an HTML file.

Watch them work with their own tools

The repository isn’t what we grade. Candidates demo what they built and explain it, and the demo is where the information is.

Why a screen share instead of a whiteboard?

Because a whiteboard takes away the tools people use on the job, and how they use those tools is most of what you’re hiring.

Here’s the pattern that shows up every single time with the EC2 route: the AMI ID is hard-coded. All of them do it. So we ask how you’d make it dynamic, how you’d look the ID up instead of pasting it, and the usual answer is no idea. Then we say use Google, figure it out, go ahead. That’s the actual exercise. We watch the search queries they build, which page they land on, and whether they understand what’s written there. Some candidates open the right documentation page and still can’t tell what it’s saying.

The same goes for failures. Terraform prints a message. Does the candidate read it, work out what the tool is telling them and fix that, or do they change something and run it again? You see which operating system they’re on, how they move around a terminal, whether they type out a whole path or use tab completion. The browser is on screen too during a share, bookmark folders and all, which people forget about.

So we tell candidates the opposite of the usual interview instruction. Use whatever you want: a translator, Google, your own notes. Work the way you normally work and ignore us, we’re just watching over your shoulder a little.

Hire for attitude, then check how they learn

A demo shows you what somebody can do now. The technical part can be trained; the attitude for learning is the harder thing to help anyone with, so it’s worth building questions around it.

How do you interview for attitude without asking “are you a fast learner”?

Ask what someone reads, watches and listens to, and where they put what they learn.

What do you read? Any podcast you’d recommend? Which YouTube channels? How do you process information, and do you have a system for storing it? These are simple questions and it’s easy to work out why they’re being asked, which doesn’t stop people from answering with nothing at all. The answers sketch where a person is and what they keep in their head.

The surprising one is about passwords. How do you deal with your passwords? Everybody has a pile of them, so everybody has an answer. The sophistication of that answer tells you a lot about whether this is a person you can trust with the company’s secrets. In our last batch, one candidate used KeePass with the database kept in Dropbox, which is a solid setup. Another used Chrome’s built-in store with MFA on the account, and we’d call that good enough for everyday life. There’s something worth knowing before you accept “the browser handles it” as an answer. Chromium’s own documentation says that on Linux the browser falls back to a plain text store when no keyring is available. That one sentence covers a wide range of outcomes.

Questions candidates rarely see coming

Habits are one half. The other half is self-awareness, and that needs questions with no correct answer.

Which questions have caught candidates off guard?

The ones about them rather than the job: what are you good at, and what counts as a lot of money.

Tell me one thing you’re good at. Something that, whether you like it or not, you’ll solve one way or another. Something I can trust you with. It’s a favourite because the answer measures self-knowledge, which is worth a lot in a colleague. People who understand their own strengths and weaknesses will tell you what to expect from them. The ones who don’t, can’t, and you find out on the job.

The other route in is money and time. What is a lot of money? Then: imagine you have that, what would you do? Last: imagine ten times that and you never have to work again, describe your perfect day. The answers map ambition and plans. Some people name a figure barely above what they live on now. Others name ten times an average salary, which tells you they have ambitions. Most are caught off guard, start thinking out loud, and the thinking out loud is the point. What a good answer looks like depends entirely on what you’re hiring for.

Hobbies, culture and the line around personal life

Those questions push toward the edge of what an interview is for, and this is where the three of us stop agreeing.

Should you ask about life outside work?

We don’t have one answer, and it’s worth settling as a panel before you walk into the room.

Mattias asks. What do you do in your spare time, how many Raspberry Pis do you have, what are you building with them? That kind of fiddling says something about genuine interest in the craft. Andrey looks for the opposite signal: hobbies that have nothing to do with IT. The company he’s hiring for is trying to diversify its backgrounds, and he wants colleagues you can talk to about something other than computers. Julien wouldn’t ask about personal life at all and keeps the conversation on what you can provide at work. His argument is about culture at scale. A five-person company that grows to fifty has the founders down to ten percent of the room, so decide early what culture you want.

Code is the easier case. One of us spent time publishing code, only to face interview questions those repositories already answered. As far as we can tell, nobody looked at them before making an offer. Public code is professionally relevant and fair game.

Social media is the grey area. Twitter, Instagram, Facebook. We generally don’t look, and we wouldn’t raise what we found there in an interview anyway. Someone posting racist content in public might well stop you hiring them, and people did publish it for everyone to see. Still, there’s a line, and the far side of it is visible. Clearview AI scraped social media at scale to build face recognition, then sold it to US police departments and companies. Looking up one candidate isn’t that. It’s a step in that direction, and hiring is supposed to be the start of a trust relationship.

Specialist or generalist

Culture questions decide who fits. The shape of the skills decides what the person can take on, and that’s the other thing to settle before the interview.

Is it better to go deep on one thing or wide across many?

Specialists can generally find work and get paid more for it, but a specialist who knows only their one thing is hard to work with.

The adjacent areas come with the job. Work with databases and you still need monitoring and logging, because that’s part of running a database. Build on AWS with containers and ECS and you still need networking and logging. Be very good at one thing, and know enough to get by in the areas around it. There won’t always be someone in the team to hand that part to, and bringing in an outside specialist isn’t always an option.

What changed the calculation is DevOps. When ops was its own department you could be a DBA and nothing else. Now the surface each person owns is wider: the database, the logging around it, how it all fits together. You also help developers get started with the database early rather than at the end. The classical administrator who works with hardware has fewer employers in the West than they used to. Managed services are taking that work over, and running a database isn’t most companies’ core business. Cloud adoption varies by market, so this reads differently in different places. For a DBA who wants to stay a DBA, the safest career bet is probably a cloud provider.

Leave fifteen minutes for their questions

One part of the interview gets treated as a formality and shouldn’t be: the time at the end when the candidate gets to ask.

What do you learn from the questions a candidate asks?

How they frame a problem, and whether they’ve thought about what they want from the job.

We leave time and we make the offer explicit, because “any questions?” usually produces a question about logistics. Right now you have fifteen minutes with people who do the work you say you want to do. Ask anything: the work, the money, the team, personal. What do you need to clear up to know you’re heading in the right direction? Then we listen to what comes back, and to how it’s put together. Formulating a good question is a professional skill, and it’s one you’ll use every day in the job.

If you’re the one being interviewed, ask as many questions as you can, because it shows you’re interested and good ones will set you apart from the other candidates. The one to ask: what is the work I’m going to be doing, and how does that work contribute to the bottom line of the company? If the person interviewing you can’t answer that, either you’re in the wrong place or you should ask to be interviewed by someone who can. Not knowing how a role connects to the business tells you something about the culture and the politics you’d be joining.

There’s a line none of us could place the source of, and it’s the right note to end on. You don’t hire smart people to tell them what to do. You hire smart people so they can tell you what to do. Plenty of job ads describe a manual worker who doesn’t think too much, and then the team expects autonomy from whoever answers the ad.

What this means for teams

Most of this costs nothing to try. It’s a reallocation of interview time from questions that produce rehearsed answers toward evidence you can see: a small exercise, a shared screen, a few questions people haven’t prepared for.

The one thing that does need a decision in advance is the personal-life question. A panel that disagrees in the room looks disorganised to the candidate, and you get inconsistent answers to compare.

Four things to do this week:

  • Write the role down in one paragraph, naming what the person will own, before anyone drafts questions.
  • Replace one trivia round with a one-line take-home and a thirty-minute demo where the candidate drives, with their own tools, Google included.
  • Add “how do you deal with your passwords?” and “what are you good at?” to the screening call.
  • Block the last fifteen minutes for the candidate’s questions and say out loud what the time is for.

Common questions, answered

How do I stop candidates from Googling the answers to my take-home?

Don’t stop them. Watch how they do it instead, because searching is the job. Tell the candidate to use whatever they normally use, then watch which queries they build, which page they open and whether they can tell what that page says. A candidate who finds the right documentation and still can’t read it has told you something that no closed-book test would have.

Should I look at a candidate’s GitHub profile before the interview?

Yes for code they published, and skip the rest of their internet. Public repositories are professionally relevant, and it’s worth reading them before the call so you don’t spend the interview asking questions the code already answers. Social media is a different question. We generally don’t look, and we wouldn’t bring it up in the interview if we did. Agree on that as a panel before the first call, rather than candidate by candidate.

What should I ask my interviewer before accepting an infrastructure job?

Ask what work you’ll be doing and how that work contributes to the bottom line of the company. If your interviewer can’t answer, ask to be interviewed by someone who can, and treat the gap as information about how clear things are inside that organisation. Ask plenty of other questions too, since showing interest and formulating questions well is part of how you stand out from other candidates.

  • #8, DevOps What: why buying a tool or hiring a “DevOps engineer” doesn’t change how a team works, which is the argument for defining the role before you advertise it.
  • #7, How Do We Learn: our own answers to the learning questions we put to candidates, including what we read and how we keep track of it.
  • #22, Who are Mattias, Julien and Andrey?: the three career paths behind the opinions above, and how each of us ended up in infrastructure work.

Resources

Since we recorded

About three weeks after this episode’s release, we published the candidate’s side in #32, Getting hired as an infrastructure automation person. GitHub Copilot became generally available in June 2022, so ask candidates whether they used an assistant on the take-home and have them explain the code it produced. HashiCorp relicensed Terraform under the Business Source License in August 2023, and the Linux Foundation launched OpenTofu in September 2023, so the assignment now runs on either tool. We returned to the migration question with Timur Bublik in #64, From Terraform to OpenTofu.