Your cloud is a business decision, and it keeps changing

You probably didn’t pick your cloud provider, and you won’t pick the next one either. We have moved the same kind of workload from OpenShift to Kubernetes, from on-prem to Google Cloud, from Google Cloud to AWS, and then back to on-prem again. One move happened because the company rented out cars and Google builds self-driving ones, which made Google a competitor. Another happened because EU rules about sensitive data ruled out US regions, and the European ones weren’t accepted either.

Neither decision came out of engineering, and both landed on the people who own the pipelines. That makes “should we go multi-cloud” the wrong question. The useful one is narrower: which parts of your delivery setup can you carry across a move, and what does the rest cost when you try to carry it too? The cheap part is your pipeline. The expensive parts are identity, data and every managed service you built on.

Henrik Hoegh joined the Nordic digital bank Lunar four months ago, after fourteen years as a consultant, and works in the platform team there. Lunar runs its own Kubernetes clusters, a multi-cloud strategy written by a squad of about five people, and an open source CLI called Shuttle that developers use instead of touching any of it. Henrik was among the first in Denmark to work with and teach cloud native technology, and he co-organizes the Cloud Native Aarhus meetup.

Portability lives in your pipeline, not in your cloud

Start with the cheap end, because that is where most of the migration pain actually shows up.

Why does Bash keep winning the portability argument?

Because every CI system can run a shell script, so the script survives the move your pipeline file doesn’t. GitHub Actions, Jenkins, GitLab and Tekton disagree about almost everything else, and all of them run bash.

The answer has a limit. The State of DevOps Russia survey, run in October 2021 with 2,145 respondents, found that Ansible had lost first place as a configuration management tool to Bash and PowerShell. As a configuration management result that’s bad news, and there are tools built for the job. As a pipeline result it just describes what people do. We’ve written far more Bash than we’d recommend, and it worked: zero dependencies, manageable idempotency, quirky error messages.

Past a hundred lines or so, or as soon as there are real data structures and branching logic, move to a programming language. Henrik and a former colleague built a tool called Pandora exactly that way: 1.0 in Bash, 2.0 in Go, once depending on jq and yq and their incompatible versions made it slow to install and painful to debug. The Bash version was still the fast way to find out the idea worked at all.

The stable version has a better shape. Take the logic of a CI step, wrap it in a Docker image with a shell script as the entry point, and pass parameters as environment variables. Your pipeline then holds the orchestration and the decisions about where things run, and nothing else.

You also get to run the step locally, which is where small differences bite: sed -i wants an empty string argument on a Mac and refuses it on Linux. Run the container, not the host.

Multi-cloud has tiers, and you pay for the one you pick

Pipelines move cheaply. Workloads are where the bill shows up, and the word “multi-cloud” hides three very different bills.

What does multi-cloud actually cost you?

It depends which tier you mean, and the operational work grows with each one. You can hold accounts with two providers and call yourself multi-cloud. You can run different workloads in different providers. Or you can have things floating between providers dynamically, which is the harder one, because every workload then has to be able to run in either place.

Price that third tier honestly. Count what’s in the repository of one microservice you’d migrate: application code is a small share, and the rest is configuration, manifests, pipeline files, tests and the containers they need. Abstract all of it away from the provider and you’ve built your own cloud, staffed by people maintaining what the provider already offers. Add the transfer cost of a terabyte between clouds, and the rewrite when the other security model doesn’t match. Teams price that and leave the data where it is.

So the joke answer, stay on one cloud and be happy, is close to the real advice. If a service already runs in one cloud, leave it there. If you need the other cloud, build the new thing there and connect the two.

Lunar’s own multi-cloud strategy wasn’t an architecture decision. They bought a company that ran in Azure, and a team in Copenhagen had decided Google was a good idea and opened an account, so three providers existed before anyone wrote a strategy for them. That’s the usual way it arrives: an acquisition, a regulator, a competitor on the other side of the contract. One reason teams do pick it deliberately is commercial: being able to move is what you bring to a price negotiation.

Link the clusters, don’t abstract the clouds

If workloads stay where they landed, the problem shrinks to one question: how do they call each other?

How do you run in two clouds without rebuilding for both?

Connect the clusters at the network layer and leave the workloads alone. At Lunar, that’s Linkerd, whose multicluster support links clusters with mTLS on both ends and makes a remote service look local to the caller. A service in Azure calls what it thinks is a local name, and the mesh carries the call to the right pod in the other cluster. Linkerd shipped that in 2.8 in June 2020 and graduated in the CNCF in July 2021, so it wasn’t a bet on a preview.

The mesh carries that call through a gateway, and the gateway is where the caller’s identity stops. A cross-cluster call goes from the source pod to the target cluster’s gateway and from there to the target pod, so the destination sees the gateway as its client, not the service that called it. An authorization policy on the destination can’t separate one remote caller from another, or one remote cluster from another. What you can authorize is which identities may reach the gateway, and which local services the gateway may call. For anything finer, Linkerd’s maintainers suggest several gateways with different policies, which is manual setup beyond the linkerd-multicluster chart.

The same wiring buys failover. Run everything in two clusters for a short period, shift traffic, then close the old one: Lunar moves from one AWS cluster to a new one within an hour with no downtime. ExternalDNS handles the records in all three providers, and the team wrote an operator that gives a cluster an identity, so routing weights shift gradually through a pull request. Once that works, the new cluster doesn’t have to be in the same provider.

There’s a tell for how well the abstraction holds. The acquired team had been told mutual TLS is good, so they implemented it in their first service on the cluster. They didn’t need to: the mesh already did it.

Consul and Cilium solve the same shape of problem, and that layer can prefer a local service and fall back to the remote cluster. Pick on operational taste: you’re buying a network to run.

A platform is the cut just below your domain

Linking clusters is plumbing. The question developers care about is what they have to know about any of it.

Where does the platform stop?

It stops where the stack starts to smell like your business. Go up from bare metal and at some point you hit the bank domain. Cut right below that line: what’s under it is the platform, generic enough to sell to someone else, a useful test even if you never do.

Under that line at Lunar, developers get two tools. Shuttle handles build and deploy: every project has a shuttle.yaml saying what it needs and pointing at a shared plan that holds the actual scripts. Go services get a default plan, and any single step of it can be opted out and replaced by a script in the developer’s own repository. Shuttle generates the pipeline too, so the Jenkins job is one task: run Shuttle. Everything Jenkins does runs locally as well, including the database or queue the service declared.

Escape hatches are easy to claim and hard to use. A platform where you don’t have to do much is pleasant while your case is one it handles; step outside it and you’re wrestling the platform team instead of your problem, and third-party open source is usually where that starts. The documentation is the tell. How an in-house tool works lives in the heads of the people who built it, and what’s written down is sparse at best and sometimes plain wrong, which is how a developer ends up back in Bash and jq.

The second tool is hamctl, named after Ham, the first chimp in space. CI puts the built manifest in an S3 bucket, and the release manager releases it into an environment, triggers GitOps and tells you in Slack. With Backstage in front, a new service is a button. Developers don’t need to know about ingresses, service meshes, routing or mutual TLS, and some new hires don’t know it’s Kubernetes underneath: the platform team could translate the same shuttle.yaml to Nomad without telling them.

A lighter version also works: templates plus Terraform, where anyone can add a resource or apply their own manifests, and standardization comes from the tools being good enough to use. At Lunar, the team that opts out most is the platform team itself.

A platform pays at one stage and costs at the next

Before you build any of that, check where your company is, because the same platform is an accelerator and a brake at different sizes.

When is it too early to build a platform?

It may be too early while you’re still building things and throwing them away, because you’d be standardizing on an architecture you haven’t found yet. A startup in that phase can spend real effort for very little back.

Once you’ve found something that works and the job becomes scaling it, the platform starts paying. You’ll create a lot of new repositories and you want structure in them, and people who move between teams stay productive because the tools are the same.

Grow further and the platform becomes a legacy software product like any other. It evolves, it gets more complex, it gets harder to change, and at some point the honest option is to start over. That’s when it starts to hinder the innovation it was built to speed up.

Watch for the slow version of that decay. Suppose you built a scheduler back when Kubernetes had no CronJob. Kubernetes then ships one and keeps improving the API while your tool stands still. A developer asks for the upstream feature, the migration is a project nobody budgeted, and the platform team carries it unfinished. Code is an expensive dependency: what you write, you maintain, upgrade and secure.

Enabler or gatekeeper

Every platform eventually tells someone no. What happens next decides which kind of team you are.

What turns a platform team into a gatekeeper?

“Your case doesn’t fit” becoming the answer instead of the start of the conversation. The pressure is structural: no tool covers every use case, so the first instinct is to ask whether the round thing fits the square hole after all.

We hit this with a client this week. They run Jenkins shared libraries we built: a Go repository that follows the specification gets built and deployed with almost no parameters. Then someone needed a library build that also ships a font file in the package, which the Dockerized packaging step doesn’t do. Rather than change the step for one case, we delivered the file as a Lambda layer. Good outcome, same resistance every developer meets at the edge of a platform.

We split on how to answer that at the cluster level. Mattias says yes and runs it side by side: when a developer asks for Kong rather than building an auth service in-house, the answer is let’s try it, and if more developers move there the ingress controller gets replaced. Henrik’s reply is that a developer shouldn’t be thinking about ingress controllers at all, and that swapping something like cert-manager needs a strong reason, because everyone depends on it.

The framing comes from Lean and Team Topologies. In a value stream the next step is your customer, which in a handover-shaped org leaves developers treating operations as the customer. Flip it: the platform team’s customers are the developers, and if they can walk away the platform has competition. Where everyone is forced onto the IT department’s thing, nobody pushes it to get better. One caveat: a choice between the platform and no support at all is the choice between a bed and the street. Aim for a platform that blocks nobody, the kind people use because it’s the easiest thing in reach.

From one-shot runs to control loops

The same gatekeeping question applies to the infrastructure the platform team runs for itself, and there the tooling is moving.

Why replace Terraform runs with a control loop?

Because an apply tells you what the world looked like when someone ran it, and a controller keeps checking. The blunt version: Terraform is the best of the tools that you shouldn’t use. Lunar’s platform team runs Atlantis on top of it, has typed atlantis plan and atlantis apply into pull request comments more times than anyone should have to, and is moving off where it can.

What they’re looking into is Crossplane, a CNCF incubating project since September 2021. The developer-facing end wouldn’t change: a shuttle.yaml says the service needs a bucket, and the platform generates a custom resource from it. The appeal is what happens after that, with credentials landing in a Kubernetes secret nobody commits to Git, and a fresh set whenever the bucket is recreated.

That’s the intended workflow rather than a result, and the seam is where the credentials come from. A bucket and the keys to it are separate managed resources in Crossplane’s AWS provider, and an IAMAccessKey is what emits an access key id and secret as connection details. In a September 2021 issue a claim’s secret kept coming back empty, and the cause was configuration rather than a missing feature: writeConnectionSecretToRef has to be set on every managed resource in the composition, with the keys named under connectionDetails.

Terraform fixed the problem of 2015, which was how to call the cloud API at all. What people want now is infrastructure that converges on the declared state with nobody driving the loop, which is also why Lunar wants away from kops: you run it, then you hope. Google already exposes the pattern through Config Connector, where a bucket is a Kubernetes object, and AWS has the same shape in flight: AWS Controllers for Kubernetes went out as a developer preview in August 2020, covering S3, DynamoDB, SNS, SQS, ECR and API Gateway V2. The preview exists already, so the open question is when it becomes something you would run a bank on. You still need your way around Kubernetes and the cloud either way, so it buys convergence rather than an easier life.

What portability costs you on AWS

Running your own clusters to stay portable is defensible, as long as you can name what it costs.

Which features do you give up?

Less than the lock-in argument suggests, and the real cost is setup. IAM roles for service accounts, from September 2019, let a pod assume a role through its service account instead of the node’s instance profile. A mutating admission webhook injects the role and the projected token, and AWS’s announcement offers it to do-it-yourself clusters like kops. So running your own costs you neither pod-level IAM nor hand-edited pod specs. What it costs is the identity wiring: register the cluster’s issuer as an OIDC provider in IAM, write the role trust policy, run the webhook. Skip only the webhook and you add AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and a projected token volume to every pod spec yourself; the OIDC provider and role trust policy are still required. Without the OIDC provider and role trust policy, the pod can’t assume the intended role.

The old objection to EKS was the network model: pods get real VPC addresses, and the count per node was capped by the instance’s interfaces. Prefix delegation in VPC CNI 1.9 raised that ceiling in 2021 by assigning /28 prefixes to ENIs rather than single addresses, behind ENABLE_PREFIX_DELEGATION. The limits move rather than vanish: it needs Nitro instances, and pods per node still depend on instance type and max-pods. A different CNI was always an option, at the price of the VPC-native addressing that made the AWS one attractive.

The list cuts both ways: the managed service you lean on may have no equivalent next door. You pick managed RabbitMQ or RDS, someone says Aurora is better, then multi-cloud arrives. Build that workload in the cloud that has the service and connect it through the mesh. mTLS proves which service is calling; it doesn’t decide which services may call. Linkerd added that half in 2.11, September 2021: inside a cluster, Server and ServerAuthorization resources name which identities may reach a port.

For data, the nearest thing to a portable standard is the Postgres wire protocol. CockroachDB speaks pgwire, though its docs list what Postgres does that it doesn’t. Plain Postgres shifts under you too, across versions and features. The twelve-factor app says treat backing services as attached resources, and that holds until it doesn’t: Keycloak supports MySQL and still broke for us on a cluster with multiple writers, where the database had to exist first.

What this means for teams

The platform work that survives a cloud migration is the work that was never about the cloud: a pipeline that holds orchestration only, a declarative file developers own, an escape hatch from every default. The abstraction layer built to make two providers look the same is the part that doesn’t. How your platform answers a request it didn’t plan for tells you which one you’re building.

This week:

• Take your noisiest CI step, move its logic into a container with a shell entry point and environment variables for parameters, and run it locally once.
• Write down which multi-cloud tier you’re in, two accounts, split workloads or floating, and what the next tier would cost in egress and rewrites.
• Count the opt-outs in your golden path, then have a developer outside the platform team use one from the documentation alone. If they can’t, or they need a ticket to you, that’s your gatekeeper line.
• List the provider features you’ve ruled out to stay portable, and put the name of whoever decided next to each one.

Common questions, answered

How do I fix sed -i working in CI but failing on my Mac?

Pass an empty string as the backup suffix: BSD sed on macOS needs sed -i '' 's/old/new/' file, where GNU sed takes sed -i 's/old/new/' file. The portable fix isn’t to branch on the operating system, it’s to stop running the script on your host: put the step in the container your pipeline uses and run that container locally. Aliasing sed to the GNU one fixes a laptop; the container fixes everyone’s.

How do I let developers ship to Kubernetes without teaching them Kubernetes?

Give them one declarative file in their own repository and translate it on the platform side. A developer says what the service needs, a database or a queue or a bucket, and the platform generates the ingress, the manifests and the credentials. What keeps that from becoming a cage is per-step opt-out: any generated step can be overridden by a script the developer commits next to their code. Write that override path down: an escape hatch only its builders know how to open isn’t one.

Which database should we pick if we might change cloud later?

Standardize on the Postgres wire protocol, then check what your application uses beyond it. Distributed databases such as CockroachDB speak pgwire, so the client side of a move is mostly unchanged, and CockroachDB’s docs list the Postgres behavior it doesn’t cover. Plain Postgres shifts under you too, across versions and as you adopt features. Write down the extensions and behaviors you depend on, because the work of a move is whichever of them the target doesn’t support and what replacing each one takes. Support for an engine is not the same as support for your setup: Keycloak supports MySQL and still broke for us on a cluster with multiple writers.

Related episodes

• Theory of Constraint: Henrik’s earlier visit, on applying the theory of constraints to IT transformation, which is where the Lean thread here starts.
• How Do We Run Kubernetes In The Cloud?: kops against managed clusters, the CNI address ceiling and pod-level IAM, in more detail.
• Do I Need a Service Mesh?: what a mesh costs to operate, before you use one to link clusters across providers.
• Infrastructure as Code revisited 2021: where Terraform sat for us a couple of months before this conversation.

Resources

• Shuttle on GitHub: Lunar’s build and deploy CLI, and how plans are shared between projects and overridden per repository.
• Announcing Linkerd 2.8: how multicluster links work, what the gateway does and where the mTLS terminates.
• Announcing Linkerd 2.11: the Server and ServerAuthorization resources, and why mTLS identity alone is not an access decision.
• Crossplane moves to the CNCF incubator: the September 2021 status change, and what the control plane model claims over one-shot applies.
• Amazon VPC CNI increases pods per node limits: prefix delegation, the instance types it needs and the constraints it leaves in place.
• Introducing fine-grained IAM roles for service accounts: how pod-level IAM works on EKS, and what a self-managed cluster has to wire up itself.
• What is a Thinnest Viable Platform?: the Team Topologies case for keeping a platform as small as it usefully can be, and treating its users as customers.
• State of DevOps Russia 2021: the October 2021 survey of 2,145 respondents, where Ansible lost first place to Bash and PowerShell.

Since we recorded

The control loop direction held, and the August 2020 ACK preview matured rather than AWS starting over. Its Kubernetes controllers for RDS, Lambda, Step Functions, Managed Prometheus and KMS reached general availability in September 2022. Crossplane graduated in the CNCF in November 2025, and Shuttle is still developed in the open. We came back to the platform question several times since, in Backstage and Internal Development Platforms, From Platform Theater to Golden Guardrails and Keeping Platforms Simple and Fast, and the argument has barely moved: the hard part is still whether developers can opt out.