#90 - K8s vs Managed Services: Cost, Lock-In, and Reality
Cheaper servers leave someone owning storage, networking and upgrades.
18 min read
We get into K8s vs native orchestrators. Do you still need Kubernetes when managed services cover most needs? How do cost, lock-in, and team skills change the choice? Expect a heated debate.
Three places to put the workload, and an argument that never ends
You have a service to run. You can rent a server and install things on it, you can run Kubernetes somewhere (your own hardware, EKS, GKE), or you can hand the whole thing to native managed services and let the provider do the scheduling. The decision looks reversible while it’s fresh. Then the dependencies accumulate: the identity wiring, the storage classes, the deploy tooling, the runbooks, the habits of whoever carries the pager. Reversing it stops being a decision and turns into a project.
The argument gets fought on cost and lock-in, and both sides usually bring the wrong numbers. “Kubernetes means no vendor lock-in” doesn’t survive a look at what you configure by hand on EKS. “The same hardware is twenty times cheaper at Hetzner” doesn’t survive a look at what else is inside an AWS bill. Underneath both sits the question nobody puts on the slide: what does your team already know, and can the business afford for them to learn something else right now?
Mark Shine, co-founder and CTO of Ankra, a Stockholm platform that provisions Kubernetes clusters and handles the day-two work on top of them across the hyperscalers and European providers like Hetzner, OVH and UpCloud, works the on-prem end of this question every week.
Kubernetes got smaller while you weren’t looking
Start with the thing that changed, because a lot of the complaints are about a version of Kubernetes that no longer exists.
Is Kubernetes still as heavy as it was the last time you tried it?
No. The core has been deliberately stripped, and the distributions underneath it got much lighter.
Kubernetes shipped v1.0 in July 2015, so it turned ten this year. It arrived with the bells and whistles built in, and the project has spent years pulling them back out: v1.31 completed the removal of every in-tree cloud provider integration. That’s the direction, a skeleton you add your dashboard, your security module and your load balancer to, and nothing you didn’t ask for.
The distros moved the same way. Rancher’s K3s put a real cluster on a single small machine: the K3s requirements are 2 cores and 2 GB for a server node, 1 core and 512 MB for an agent, which is why a Raspberry Pi on a desk works. kind runs a multi-node cluster on a laptop so you can simulate node loss. Talos Linux removes the parts you used to have to secure: no SSH, no shell, no package manager, every node operation through an mTLS API.
What you add has its own lifecycle, and one of the oldest pieces just got an end date. The community Ingress NGINX controller has been the default answer for most of a decade. On 11 November 2025 the Kubernetes project announced its retirement: best-effort maintenance until March 2026, then no further releases, no bugfixes and no security patches. SIG Network and the Security Response Committee recommend moving to Gateway API or another Ingress controller immediately. Running clusters keep working, which is how this stays off the roadmap until it’s urgent.
The certificate clock in a self-managed cluster
The other thing you take on with the install is a PKI, and it comes with a deadline nobody puts in a calendar.
Which certificates does kubeadm certs check-expiration actually cover?
The ones in the PKI folder and in kubeadm’s own kubeconfigs: admin.conf, super-admin.conf where the cluster has one, controller-manager.conf and scheduler.conf. The copy at ~/.kube/config is not among them.
That distinction decides how you read a clean report. kubeadm issues client certificates with a one-year lifetime by default and renews them during a control plane upgrade. The lifetime is configurable, but at the default they expire after a year unless an upgrade or a separate kubeadm certs renew replaces them. The old setups are where the scars came from: Flatcar plus Ansible plus PXE booting, with that clock running underneath. kubectl reads the copy in your home directory, which can hold an expired certificate while every line of the report comes back in date.
Recovery is three steps and a limit. Run kubeadm certs renew all on every control plane node when the control plane is replicated, not only the one you are logged into. Restart the control plane pods afterwards, because the components do not reload certificates while running. Then deal with your own kubeconfig: copy the renewed /etc/kubernetes/admin.conf over ~/.kube/config only when your file is a copy of it. An overwrite wipes every other context in that file and leaves you connecting as cluster-admin, so merge the single entry instead when the file holds more than this cluster. The limit: kubeadm cannot manage certificates signed by an external CA, so those you reissue through new CSRs yourself.
Kubespray is what’s different about running this yourself today: Ansible roles with the hard-won fixes already in them, so you inherit years of other people’s work.
The gap is your laptop to production, not install to cluster
Creating a cluster in ten minutes tells you nothing about the week after, which is where the pain actually lives.
Why does the thing that worked on Minikube fall over in production?
Because the parts that make it production, spread, storage, ingress, identity, are exactly the parts local dev never makes you write.
An nginx page, a database and an ingress is a handful of YAML, close enough to a Docker Compose file that it feels solved. Then you move it into a real organisation and you have three replicas of the web server sitting on one node, because nothing on your laptop forced you to think about topology. The node goes. All three go with it. You had three replicas and one failure domain, and anti-affinity on a single-node Minikube has nowhere to put the second pod.
The inner loop is good now. Skaffold redeploys to Minikube when you save, which is genuinely how you want to build an operator. The outer loop is where the hoops are: Dockerise, build, push to a registry, then write the manifest, then the Helm chart, then the values for each environment. Developers want what Vercel gives them, a push and a URL, and when it breaks they want to know what broke. The thing Kubernetes abstracted is the generic work, not the specific knowledge.
YAML itself got a small answer this year. KYAML landed as an alpha kubectl output format in Kubernetes 1.34, behind KUBECTL_KYAML=true and kubectl get -o kyaml: a strict subset of YAML that closes the indentation and quoting traps. It isn’t the foundation.
The alternative is worse than the YAML. Copying the application onto the box works, needs no pipeline, and quietly accumulates config drift until the server nobody has rebuilt in two years is the only one that runs the app. A pipeline that reconciles is what stops that.
Managed Kubernetes is not an exit from lock-in
If that’s the cost, the usual justification is portability. It’s weaker than it sounds.
Does running Kubernetes keep you off the hyperscaler hook?
Not on its own. The scheduler is portable. The identity, networking and storage you wire around it are not.
EKS is close to vanilla Kubernetes wrapped in AWS, and the wrapping is configuration you own. You don’t have to bring a CNI: the Amazon VPC CNI is deployed on every EC2 node, and it is the only CNI plugin AWS supports on its own infrastructure. What you own is how it gets its permissions: AmazonEKS_CNI_Policy can hang off the node IAM role, or off a separate role bound to the aws-node service account through IRSA or EKS Pod Identity. Then the same exercise again for every workload that calls an AWS API.
A provider-neutral Deployment moves to another cloud unchanged, once what it depends on exists on the other side. Many manifests aren’t provider-neutral: node selectors and topology constraints naming instance types or zone labels, storage classes, ingress annotations. The OIDC provider or Pod Identity association behind the workload, the roles and the trust policies, you rebuild by hand.
The usual comeback is that the managed services are open source underneath, so you can run them yourself. That holds for particular capabilities, not as a rule. MinIO provides an S3-compatible API, and the limits doc in the MinIO server repository names the S3 APIs it deliberately leaves out, bucket and object ACLs, CORS and website hosting among them, as “redundant or less useful outside of AWS S3”. That list rules those calls out rather than guaranteeing the rest, so test the operations your application actually makes. The disks behind it are yours either way: capacity, failure domains, upgrades, and whatever durability you can promise. You can bring MinIO in this afternoon and spend the next two months learning it properly.
So the honest question isn’t whether you’re locked in. It’s which lock you prefer. Restricting yourself to the primitives every cloud has is a tax you pay every sprint for an option you may never exercise. Taking the managed services moves a named set of work instead: AWS draws the line at security of the cloud against security in it, so the facility, hardware and hypervisor are theirs, while your data, IAM policies and network configuration stay yours.
A Hetzner box at a twentieth of the price is a different product
Which drags the argument back to the bill, where it usually ends up.
Is the hardware really twenty times cheaper outside the hyperscalers?
The hardware is much cheaper. It isn’t the same product, which is where the comparison falls apart.
This is where we split. Mark’s figure: roughly 90 cores and about a terabyte of RAM, your own hypervisor on top, for around $500 a month at Hetzner, against twenty times that for the same spec at a hyperscaler. Andrey’s objection isn’t the arithmetic, it’s the units. You’re comparing apples to pineapples: at Hetzner you’re buying bare hardware, at AWS a VM plus everything wrapped around it. CloudTrail records management events with nothing switched on, and the first copy of them is free in the region; data events, such as S3 object reads, are opt-in and billed per event. None of it exists on a dedicated server until you build it.
The WAF comparison is worth checking. AWS WAF’s price list is $5 per web ACL per month, $1 per rule per month, and $0.60 per million requests inspected: one ACL, ten rules and 50 million requests a month comes to $45. The pricing page can’t diagnose a bill near $2,000, and neither can we. It does name what stacks on top: request volume, capacity above the 1,500 WCU a web ACL includes, and paid rule groups such as Bot Control.
Run the same honesty at the cheaper end, where Hetzner sells two different things. On Hetzner Cloud there’s a thin layer to build on: the hcloud CSI driver attaches Cloud Volumes to cloud servers, and there’s a load balancer an ingress controller can point at. A dedicated server is the machine. Hetzner’s own FAQ is blunt: Cloud Volumes “only work with cloud servers”, so persistent storage on bare metal is local disks and whatever you run across them to replicate. Neither product hands you a NAT gateway. Multiply those hours by what you pay the person doing them, and put that in the comparison.
Compliance decides this more often than preference does
Cost is an argument you can win with a spreadsheet. The next one isn’t.
When does the choice stop being yours?
When a customer or a regulator names where the system runs and who is allowed to operate it.
There are municipalities in Sweden that require a Swedish provider before you can sell them anything, and regulated sectors in Germany with their own bar. Whether a hyperscaler could technically meet it is beside the point: the requirement is political. You have an answer or you’re out of the tender.
PCI DSS is the version most engineers meet first, remembered as a staffing rule: whoever writes the code isn’t allowed to deploy it. PCI DSS v4.0.1, active since v4.0 retired at the end of 2024, splits that across two requirements, and neither counts heads.
Deployment is 6.5.3 and 6.5.4: pre-production separated from production with access controls, then roles and functions separated so only reviewed and approved changes get deployed. A pipeline carries most of it: no human holds standing write access to production, the deploy role does, and only the production job can assume it. Where individuals perform multiple roles, the applicability note on 6.5.4 allows “additional procedural controls that provide accountability”.
Code review is 6.2.3, and its note allows “either manual or automated processes, or a combination of both”. The independent reviewer sits in 6.2.3.1, behind its opening condition: “If manual code reviews are performed”. Review by hand and the change goes to someone other than the author, with management approval before release. A scanner beside the human doesn’t get you out of it: the condition is whether manual review happens at all, so only an entirely automated review falls outside 6.2.3.1. Either way 6.2.3 stands, and the code is reviewed for vulnerabilities before release.
DORA is worth reading rather than paraphrasing, ours included. The version in our heads was that critical financial systems must be cloud-agnostic and prove an annual failover to another provider. The regulation, applying since 17 January 2025, is narrower: exit strategies for ICT services supporting critical or important functions under Article 28(8), and yearly testing of ICT business continuity and response and recovery plans under Article 11. It doesn’t name a second cloud provider.
Then the reminder arrived on its own. Cloudflare’s outage on 18 November 2025 started with a bad Bot Management feature file and ended with core CDN and security services returning 5xx, Swedish sites included. Everything in front of your compute is a dependency too.
Multi-cloud will not fix your reliability
That outage is usually read as an argument for spreading across providers. It’s the wrong lesson.
Will running on two clouds keep you up?
Not when the failure comes from your own software, and that’s the failure mode to rule out first.
If your service can’t already run across two regions of one cloud, multi-cloud is that same unsolved problem with a second vendor bolted to it, and the thing you build to switch between them is a third system to run. One problem you could have optimised becomes three you can’t.
We would put the ceiling on a second provider lower than the pitch does. It won’t keep you from going down. The most it buys is not being down for the full day, and the conditions on that carry the whole argument: the software has to be built to fail over, and the recovery procedure has to be one you have tested. Without both, the second provider is somewhere else to be broken, and you doubled the complexity to get there.
Leaving the cloud is a symptom, and there’s a middle path
Which leaves the motive. Most cloud exits start with a number, not an architecture.
When do teams actually start caring about the cloud bill?
When the growth stops. In the startups we’ve worked with, nobody read the bill closely while the numbers went up; it got read when growth stalled, or when the runway had to stretch to the next raise.
Andrey’s verdict is blunt: if the plan is to save millions by going on-prem, the business is already irrelevant and the move is somewhere to die. Mark won’t take it that far. The economy is doing this to everyone right now, profit per employee is a metric companies are genuinely measured on, and a company with no resilience in the hard years never gets to the good ones.
Skills decide more of it than either number does. A team that knows Kubernetes will see Kubernetes everywhere, and you’ll live with that. Mark’s one objection to picking for the skills you have is hiring, and Chef is his example: his read is that willingness to learn it ran out, and a Chef estate gets hard to staff once that happens. The tool itself is still shipping, Chef Infra Client 19 arrived in April 2025, so it’s a hiring worry rather than a verdict on the tool. Weigh the technical debt you take on against whether you can hire people who want to work on it.
With ECS Anywhere AWS keeps the control plane and you register your own servers, wherever they are, for $0.01025 per managed instance-hour on top of whatever the hardware costs. The agent pulls images and reports back. The hardware failures, the storage failures, the network outages and the physical security are yours now, and that’s the trade.
The connectivity argument behind it is the strongest reason to care. Under Cloud Interconnect, Google sells Dedicated and Partner Interconnect to join an on-premises or colocated network to a VPC, and Cross-Cloud Interconnect to reach another cloud provider; AWS and Azure have equivalents. This is what made Equinix matter long before the cloud did. We couldn’t find Hetzner’s own page describing a direct line into GCP, so check that before planning around it. The principle holds either way: a team that can move has a negotiation. A team that can’t just takes the price.
What this means for teams
The choice is less about Kubernetes than about which bill you’d rather pay and who’s on the team to pay it. Four things worth doing this week:
- Price the thing you’d actually buy. Take one real workload and write out both bills: the hyperscaler one including the managed services you use, the alternative including the hours someone spends building storage, ingress and identity. Cores and RAM alone will lie to you in whichever direction you lean.
- Run
kubeadm certs check-expiration, or your distribution’s equivalent, on every cluster you own, then check the kubeconfig your team actually uses. Find the one that expires before your next upgrade window, now rather than on the morning it stops answering. - Count your actual lock-in. Go through your manifests and tally the IAM role mappings, storage classes, node selectors and ingress annotations that are provider-specific. That number, not the fact that you run Kubernetes, is what a move would cost.
- Check whether you are running the community Ingress NGINX controller, with
kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx, and put the migration on a roadmap that ends before March 2026.
Common questions, answered
How do I compare a Hetzner bill to an AWS bill without fooling myself?
Price the bundle, not the cores. List what the hyperscaler gives you on top of the VM: audit logging, load balancing, managed storage, identity, and the NAT gateway neither Hetzner product includes. Then price each of those as hours of work on bare hardware, at the salary of whoever would do them.
The raw hardware really is several times cheaper. The comparison only misleads when it stops at CPU, RAM and disk. Do it on one workload you already run.
Do we need two cloud providers to satisfy DORA?
No. DORA requires an exit strategy for ICT services supporting critical or important functions under Article 28(8), and testing of your ICT business continuity and response and recovery plans at least yearly under Article 11. Neither names a second cloud provider.
What it does close is the untested playbook. If you’re in scope, plan for a test you actually run and can evidence. That is different work from standing up a second provider, and usually more useful.
Why did my Kubernetes cluster stop accepting kubectl after about a year?
Read the error kubectl returned and open the kubeconfig it actually used before you renew anything. kubeadm certs check-expiration covers the PKI folder and kubeadm’s own kubeconfigs, not the copy at ~/.kube/config, so a clean report says those certificates are in date and nothing about why your connection failed. Check the expiry on your own client certificate, and refresh that credential only once you have confirmed it is the expired one.
When the report does show the cluster’s own certificates expired, run kubeadm certs renew all on every control plane node and restart the control plane pods, which do not reload certificates while running.
Related episodes
- #79 Going Local: What’s Driving the Move?: the European provider landscape underneath this argument, and why the migration bill lands in the managed services rather than the compute.
- #59 Migration Off The Cloud: To Leave or Not to Leave?: the earlier round of the cost fight, and why a hyperscaler quote and a VPS quote are not the same line item.
- #67 Building MVP On AWS: what the native managed services route looks like from the first account, the option Kubernetes is compared against here.
- #51 Provisioning bare-metal servers: everything below the operating system that becomes yours when the hardware is.
Resources
- DORA, Regulation (EU) 2022/2554, EUR-Lex settles what is required: exit strategies for critical ICT services in Article 28(8), yearly testing of continuity and recovery plans in Article 11, no second provider named.
- PCI DSS v4.0.1, PCI Security Standards Council carries the requirement text: 6.5.4’s limited-personnel note, and the “If manual code reviews are performed” condition that scopes 6.2.3.1. Active version since v4.0 retired on 31 December 2024.
- AWS shared responsibility model settles where the handover sits: AWS takes the facility, hardware and hypervisor; you keep data, identity, configuration and encryption.
- Certificate Management with kubeadm, Kubernetes docs settles the one-year client certificate default, what
check-expirationreads, and that kubeadm cannot renew certificates signed by an external CA. - MinIO server limits,
docs/minio-limits.mdatRELEASE.2025-04-22T22-12-26Zsettles which S3 APIs the MinIO server deliberately does not implement: bucket and object ACLs, CORS, website hosting and bucket logging. - Ingress NGINX retirement, Kubernetes blog settles the end date, March 2026, and what stops then: releases, bugfixes and security patches.
- Amazon ECS Anywhere pricing settles the hybrid rate, $0.01025 per managed on-premises instance-hour, billed from registration to deregistration, on top of your own hardware.
- Mark Shine on LinkedIn and Ankra for the platform side of the Kubernetes day-two argument above.
Since we recorded
The AWS European Sovereign Cloud opened on 14 January 2026, with a first region in Brandenburg under its own aws-eusc partition. KYAML reached beta in Kubernetes 1.35, enabled by default but still only via -o kyaml. Mark came back in #103 European Cloud Sovereignty with the number this argument was missing: one team cut operating spend by about 60% after moving infrastructure off GCP.
