Podcast about DevSecOps

#56 - Backstage and Internal Development Platforms (IDP)

A developer portal can replace a ticket or just change who files it.

18 min read

In this episode of DevSecOps Talks, join Andrey, Julien, and Mattias as they dive into the world of Backstage, the notable internal development platform. Mattias is keen to peel back the layers and understand what makes people think of Backstage as a must-have in modern DevOps toolchains. Andrey highlights the platform’s core feature: a comprehensive registry that keeps track of every software service running within a company. Could this signify a revival of IT change management, but with a twist? We’ve moved on from the days of cumbersome ticketing systems to streamlined internal development platforms. The team also ponders the future role of infrastructure engineers as they navigate the rising tides of AI—will AI become the new face behind these developer portals? Tune in to find out!

Discuss the episode or ask us anything on LinkedIn


Nobody has the list

Your company runs more than one compute platform. There’s a Kubernetes cluster, so kubectl get svc across the namespaces gives you a picture of that part. Then there are a few EC2 instances, some Lambda functions, something on App Runner, and all of it spread over several AWS accounts. Ask what’s running and who owns it, and the honest answer is that nobody has the list.

That isn’t an abstract governance problem. It’s the thing you built, forgot, and never switched off, still billing as a zombie on an invoice nobody reads line by line. It’s the 2am Slack message asking who owns this, sent to a channel of three hundred people. It’s the service in dev that someone may or may not have promoted to production, and you keep discovering more of them.

Backstage is the best-known answer to that, and the word around it, internal developer portal, now covers a lot of ground that has nothing to do with lists. It’s worth separating the two before you commit an engineer to running one. A portal does solve the list problem. It also quietly inherits a different one: the queue.

What Backstage actually gives you

Start with the thing itself, because the category name promises more than any single tool delivers.

What do you get when you deploy Backstage?

A catalog and a socket for plugins. Almost everything else you configure or build yourself.

Spotify open-sourced Backstage on 16 March 2020 and donated it to the CNCF, where it moved from Sandbox to Incubating in March 2022. The Software Catalog is the part that earns its keep: metadata YAML files kept next to the code, harvested into one place that tracks ownership and metadata for every service, website, library and data pipeline. Write documentation in your repository, follow the convention, and it appears on that component’s page. One UI shows the metrics, who created the repository, and which team to reach when something needs changing. In a big company that replaces people shouting on Slack asking who is responsible for this.

Templates are the other half. Ready-made scaffolding for a new project, so a new Java backend or a new marketing site starts from the shape the organisation already agreed on.

Then the catch, which matters more than the feature list.

The portal in your head, with deploys and dashboards and database provisioning, is a set of plugins you pick, configure, or write. Spotify’s internal build has plugins the open-source one doesn’t, including data management where a field is traceable back to its origin, which is genuinely useful and not something you get by running yarn dev.

The cheap version of the catalog is tags

If the list is what you need, you can have it without operating a portal at all.

What does a tag-based registry actually tell you?

Where each resource came from, who to ask about it, and what it costs. Not whether it’s safe to switch off.

Tag every resource with the repository that deployed it and the team to ask. Then activate those keys as cost allocation tags, in the Billing console or with the API AWS added in June 2022: until a key is activated, the tag sits on the resource and the bill knows nothing about it. Once it’s on, Cost Explorer groups spend by those keys, and Resource Explorer searches your account by tag, name or ID across regions once you turn the indexes on. Nothing to operate, nothing to patch, nothing to upgrade on a Friday.

Consistency is the whole trick, and you can enforce it rather than ask for it. A service control policy with a Deny statement and a Null condition on aws:RequestTag/<key> refuses the create call when the tag is missing. AWS’s own tagging walkthrough pairs that with a tag policy for the allowed keys and values. A second Deny covers the tag-removal actions, scoped by aws:TagKeys to the keys you care about. Nothing in a create-time condition stops someone stripping the tag a week later. Budget for the long tail. Check each creation action. The Action element takes a list, so actions that share a condition and resource scope sit in one statement; the ones whose authorization requirements differ get their own. Leave out the actions that can’t pass tags on create. Null with a value of true matches an absent key, so an action that cannot supply the tag matches every time and the Deny blocks it outright. The services don’t behave identically either, so read the list before you widen the resource element.

What tags won’t tell you is the part you most want to know. Is this production ready. Did anyone set up alarms. Is it still needed. On the teams we work with, five or ten developers with people who already know how to do the work, the list has been the real need and tags have covered it. The portal is a second system to maintain, and the question that decides it is whether you want to become an expert in that tool rather than spend the time elsewhere.

Where you draw the self-service line

A list tells you what exists. Who’s allowed to create the next thing is the harder question, and the portal doesn’t answer it for you.

How much infrastructure should a developer’s repository own?

Anywhere between all of it and none of it, and where you land is a bigger decision than which tool you run.

At one end, the repository holds the application and its infrastructure. Load balancers, security groups, IAM, databases, the lot. Developers commit, the pipeline applies, and they can change any of it. At the other end, everything is provided for them: the pipeline builds a container, gets an image ID, and changes one image version from one value to another. Nothing else moves. Any other change belongs to someone else.

Almost nobody lives at either extreme. Every company picks a point between them, and a portal is one way to deliver whatever was picked, letting developers provision what they need through an interface instead of a ticket. The other way is Terraform modules written by the platform team with predefined defaults the caller can’t override. That one runs into its own wall: hand-written HCL is error-prone, and not everyone has the Terraform skills, or wants them.

The ticket didn’t go away, it changed hands

Wherever your line sits, watch what happens to the request itself.

Does a developer portal remove the ticket?

Only where it automates the work behind the request. Where it doesn’t, the ticket just changes hands.

Before DevOps you had dev and ops, and they had to talk to each other to ship faster. Now you have dev, platform engineer, and the networking people behind them. The developer who used to file the ticket isn’t filing it any more. The platform engineer is, asking someone to please open that port on the firewall.

That’s the uncomfortable reading of platform engineering, and it lands close enough to the truth to sting. Everyone was DevOps, now everyone is a platform engineer. We had Jira and change management systems before, and a portal is still a change management system in a nutshell. The difference is real where the automation is real: in a good one you make the change yourself instead of waiting for someone to make it for you. A self-service change beats a queued change. It remains a change.

Set one up and it will show you exactly how your organisation works, because the templates are the company’s opinions written down. That’s culture as code, and you can read off what the organisation blesses and where it intends to go. Sometimes that’s the most valuable output of the whole exercise.

Golden paths are cheap on the path

Templates are where those opinions live, and where they start charging you.

What does it cost to do something the golden path doesn’t cover?

Everything the path was giving you for free, paid out of your own time.

Golden path is Spotify’s name for the templated tutorial through the blessed tools, written in TechDocs and read inside Backstage. On the path, the experience is good. You’re a Go developer joining a company that does Go, you take the repository template, run the configuration pipelines, and spend your time on business logic instead of boilerplate.

One step to the left or right of it and that reverses. Metrics, logging, SLOs and SLIs become yours to arrange. Platform engineers support the platform, which is what the name says, and outside it there’s nobody waiting to help. The bar for adding something new is higher than it looks, too. We remember a list going round from someone at Google on what it takes to bring a new language into the mix, and it was weeks of work before anyone writes a line of business logic. The enterprise version of this conversation was Oracle: you want a different database, so we need to buy the licences and provision the servers, look how much work that is. Swap in Rust and the shape is identical.

If we skip that second half, the catalog can become a ceiling. When nothing new ever enters it, the way you did things three years ago is the way you’ll do them in three more. There’s no point running a portal while people quietly do their own thing beside it either, so the route back in has to exist.

The template is also where “production ready” stops being folklore. A developer arriving doesn’t know what the platform wants from them: the alarms, the logs, the metrics, whatever your definition of done includes. Put the repeatable checks in the scaffolding so they arrive by default, and write the rest down where someone will find it before launch. A requirement that lives only in a reviewer’s head shows up after the service is already live.

The 200% knowledge problem

Abstractions have the same shape as golden paths, and they fail in the same place.

Why does an abstraction sometimes make the work slower?

Because when it doesn’t fit, you have to know the abstraction and the layer underneath it.

The case that taught us this: a third-party finance application, no source, just a JAR and good luck. Documentation existed and didn’t help. The CI still had to be built, and the CI template meant learning the language of the abstraction the platform team had written, plus how the underlying CI actually worked, because the abstraction didn’t cover this. Two complete models held at once, for one deployment. It took roughly ten times longer than it should have.

The alternative was obvious and ugly. Put it on a VM, SSH in, audit-log everything. Much faster. What you give up is consistency, and you bypass a lot of the security checks the platform was quietly performing on everything else.

There’s no clean answer, which is why this is a culture question rather than a tooling one. Who’s responsible for the thing on the VM. Who supports it when it breaks, and it’s when rather than if. What happens to it during the migration that’s already scheduled. Decide that before the exception exists, because afterwards the decision gets made by whoever is on call.

Who the portal is actually for

The 200% problem gets heavier the less you already know, which points at who a portal actually helps.

Who gains most from a developer portal?

The person who just arrived and doesn’t have your context yet.

Doing everything yourself means networking, CI/CD, the deployment path, the security checks, all of it. Someone with ten or fifteen years behind them gets productive quickly because they’ve seen most of this before and only need your specifics. Someone fresh out of university hasn’t seen any of it, and three months of learning your stack before they ship anything is not what you set out to buy.

Hand that person building blocks and they spend their time on the code, which is the expertise they actually have.

The analogy that fits is a SaaS business you founded. A customer asks for a change, and you go straight into the database and make it, because you know everything. Your customer experience colleague can’t do that and shouldn’t have to learn how. So you build them a portal where they change the thing without knowing how it’s implemented. A developer portal is the same move, pointed inward.

The thing behind the portal

Which raises a question about whoever is on the other side of that interface, and for how long.

Is AI going to replace the people behind the portal?

That’s the obvious direction, and it’s a good reason not to take the job of hiding back there.

The end state is easy to picture. You talk to the portal: deploy this piece of code, do the thing. We’re not far off. The platform engineers who were on the far side of the interface are then not on the far side of the interface at all.

The broader effect of this wave is a question rather than a tool: do we really need to build that. A full-stack application used to mean front end, back end and a data model, and weeks of work to get there. Plenty of people now pay ten or twenty dollars a month for something that just works and buy back the time they’d have spent figuring it out. Most companies still need infrastructure, and that hasn’t changed. The threshold for needing a custom thing has moved.

The limits are still easy to find. Google launched Bard with a waitlist in March 2023 and dropped that waitlist in May. In one test, asked for a mission statement for a startup, it drafted something that read better than what the founders had. When we tried it on blunt 360-review feedback, bullet points handed over to be rephrased so they could be sent, it polished the good points and answered the negative ones with advice to go and talk to that person directly. Reasonable advice. Not what was asked for.

What else covers part of the job

Whichever way that goes, the interface doesn’t have to be Backstage. The world doesn’t revolve around it, even if the conference talks suggest otherwise.

What else covers part of this job?

Pieces of it, from HashiCorp, Argo and AWS. No single one of them is the whole thing.

HashiCorp’s Waypoint is a self-hosted build, deploy and release workflow, and it shipped in October 2020 with plugins for Kubernetes, Nomad, Amazon ECS, Google Cloud Run, Azure Container Instances and Docker. It needs no HashiCorp-provided service to run, so its limit is scope rather than vendor lock: it handles the deploy and leaves the catalog alone, which is one slice of what a portal does. Version 0.11 reworked the ECS and Lambda plugins, letting ECS deployments bring their own application load balancer. Argo CD gives you a UI over what’s running in Kubernetes, and that UI creates as well as shows. The getting started guide has you make an Application with the + New App button. Sync it, and Argo CD applies the manifests for resources the cluster listed as Missing a moment earlier. The manifests come from a Git repository you point it at. What it doesn’t carry is the rest of a portal: no ownership catalog, and no template that scaffolds a service nobody has written yet. AWS Service Catalog is closer in intent, and SCPs let you bound what someone can ask for, so an RDS instance is allowed and that instance size isn’t. On the Backstage side, AWS shipped a Proton plugin in June 2022 with an entity card for a service’s status and a scaffolder action to create one. When we went looking for an ECS plugin, there wasn’t one.

The gap worth noticing is the cloud providers themselves.

Instead everyone builds it and integrates it with whatever they already run, which is why operating one needs people doing it full time. That’s the honest size of the commitment, and it’s why this is mostly a tool for organisations that can hand a few engineers to it and let them work out the use cases.

Leaning on a provider’s own version has its own price, of course. On-prem you’re building it regardless. And most companies are multi-cloud whether they’ve admitted it or not, because somewhere there’s a manager who put a credit card on a second provider.

What this means for teams

Two decisions sit underneath all of this, and you can make both before evaluating a single tool. What do you want to be able to see, and what do you want a developer to be able to change without asking. A portal is a delivery mechanism for both answers, and it is paid for in engineer-years. If you can’t name the people who’ll run it, you’re buying the second problem without the first solution.

This week:

  • Tag every resource with the repository it came from and the team to ask, then activate those keys as cost allocation tags so Cost Explorer groups by them. Enforce the keys with an SCP that denies creation when aws:RequestTag/<key> is absent, and denies the tag-removal actions on those keys so the tag survives the week.
  • Write one sentence placing your self-service line between “the application repository owns its infrastructure” and “the pipeline changes an image tag”, then check whether your pipelines match it.
  • Count the requests a portal would absorb. If each one only changes author, from the developer to a platform engineer, you’ve rebranded the queue.
  • Move the repeatable production-ready checks into the template: alarms, logs, metrics. Document the rest where a developer looks before launch, not where a reviewer finds it after.

Common questions, answered

Do I need Backstage to know what my company is running?

No. On AWS, one mandatory tag naming the repository each resource came from and one naming the owner gets you most of the way. Activate those keys as cost allocation tags first, in the Billing console or with the API AWS added in June 2022, or the bill will not group by them. Cost Explorer then splits spend by owner, and Resource Explorer searches resources by tag. Enforce the keys with a service control policy that denies creation when the tag is absent, listing only the actions that support tagging on create, and deny the tag-removal actions on those keys so the tag survives. What none of that tells you is whether a service is production ready or still needed.

How many people does it take to run an internal developer portal?

Budget at least one engineer on it full time, more if you need plugins that nobody has published. Backstage ships the core that runs plugins, so the catalog, the templates and the integrations with everything you already run are yours to configure and often to write. Someone has to work out the use cases and keep it current as the platform changes. A team of ten is usually better served by consistent tags and a few Terraform modules with locked defaults.

How do we tell developers what production ready means?

Put the repeatable checks in the template a new service starts from: alarms, logs, metrics, whatever your platform expects by default. Write the rest down where a developer will look before launch, because the requirements that can’t be scaffolded still apply. A developer arriving has no way to guess what the platform wants from them, and finding out during a review after launch is the expensive way to learn it.

Resources

Since we recorded

HashiCorp archived the Waypoint repository on 8 January 2024, with the README stating that the Community Edition is no longer actively maintained and pointing at HCP Waypoint instead. AWS ended support for Proton on 7 October 2026, and its deprecation and migration guide names Harmonix on AWS, a partner solution built on Backstage, among the routes out. The argument itself hasn’t moved much, which is why we returned to it with Steve Wade in From Platform Theater to Golden Guardrails and with Joachim Hill-Grannec in Keeping Platforms Simple and Fast.