Your VPN is carrying the whole security model

Most of us have lost an evening to a VPN instead of to the work the VPN was supposed to make possible. Traffic that may or may not be going through the tunnel, so you open Wireshark to find out. An internal resolver you have to switch to by hand, because the DNS server your laptop picked up at home can’t see anything inside. We still have the scars from debugging that stack. The average person shouldn’t need a packet capture to get through a working day.

The flakiness isn’t the real problem. Underneath it sits an assumption that stopped being true when everyone went home: that there’s a trusted network, that your machine joins it, and that production is reachable from inside it. That trusted network now ends at a kitchen table. In the PCI environment we worked in, it ended on laptops the developers owned themselves, with WhatsApp running next to the work. If joining a network is what grants access, your perimeter includes every home router your colleagues bought on sale.

So the useful question isn’t how to make the VPN stable. It’s what the VPN was buying you, and whether anything else buys the same thing cheaper. The replacement is fewer places a human has to reach at all, and a check on every request instead of on where it came from. Then enough detection to notice when one of those checks was wrong.

There’s no trusted network when the office is a kitchen table

The first thing to give up is the idea that a device can be trusted because of where it’s plugged in.

Why doesn’t the trusted network survive remote work?

Because where a device connects from tells you nothing about whether that device can be trusted. In the PCI environment we worked in, developers used their own laptops.

One machine for work and everything else is what people want, and telling them to drop WhatsApp was never going to hold. What worked was treating every laptop and developer environment as untrusted, with the cardholder environment the only thing inside. The old model is a castle: a moat, a wall, and once you’re past them you can walk into any room you like. Physical security gave that up a long time ago. Any office building lets you into reception and asks for a card on the floors that matter.

The other half of the border model is being taken away whether you like it or not. TLS 1.3 encrypts the server’s certificate after the key exchange (RFC 8446), so a middlebox can no longer read the hostname off a certificate in passing. Terminating the session on a proxy your fleet already trusts still works; reading it off the wire doesn’t. Ordinary SNI still carries the requested hostname in the clear for anything watching the link, and Encrypted SNI was the proposal to close that gap. DNS over HTTPS hides the lookups too. If your control depends on reading names at the border, keeping it working means refusing the traffic that hides them, which is what the Great Firewall did in July. Researchers measured the blocking from 29 July 2020: the 0xffce extension triggers it, packets are dropped rather than reset, and the source, destination and port stay blocked for 120 to 180 seconds afterwards. That’s where passive hostname inspection ends up once the hostname is encrypted.

Pick the fight on ground you control

If no laptop can be trusted, the tempting next move is to lock every laptop down. That’s a fight across Windows, macOS and Linux, on machines you may not even own.

Which ground should you defend?

The path to production, because everything that ships has to cross it and you own it outright.

For us that was the build server. Jenkins was the one place every change passed through, so the controls went there: scan what comes in, know whose commit it is, and fail the build rather than argue about operating systems. The things people log into, portals and dashboards, stayed reachable from the internet with no IP restrictions, and the effort went into the login instead, with sessions and token refresh set up properly and the traffic watched for behavior. ModSecurity rules describe how the application is actually used. Somebody lists their objects, then opens one of them. A client that opens an object without ever listing anything skipped a step, and that’s a question worth asking.

That line comes from an IT security class about ten years ago, and we haven’t found a better test for a control since. A developer who notices your security is a developer spending the afternoon on your problem instead of theirs.

Identity is the gate, the address is a clue

Leaving services reachable from the internet only works if something decides who gets in, request by request.

What replaces the IP allow list?

A check on who is asking and what device they’re asking from, applied every time they ask.

Google wrote this up as BeyondCorp. Their 2014 paper puts the problem in one line: with perimeter security, “when that perimeter is breached, an attacker has relatively easy access to a company’s privileged intranet”. What the same paper puts in its place is access that depends on device and user credentials, “regardless of a user’s network location”, whether that location is an office, a home network or a coffee shop.

Allow lists also rot in ways that have nothing to do with attackers. We’ve seen a Kubernetes cluster where every node carried a public IP and each of those addresses was allow-listed on the API endpoint. That cluster couldn’t scale, because a new node arrives with an address nobody listed, and it couldn’t roll a node for the same reason. The fix was to build it again.

Cilium’s documentation makes the same case from the data plane: iptables filters on addresses and ports, and in a cluster where addresses change constantly the rule set grows while the thing it’s trying to name keeps moving. Cilium attaches a security identity to groups of containers instead.

Where an address still earns its keep is as a signal. An SSH login from Sweden and another from China fifty minutes later is worth an alert, even though neither address is forbidden. Use it to notice, not to decide.

Servers nobody can log into

The cheapest access to secure is the access that isn’t there.

What do SSH certificates fix that keys don’t?

Distribution, and expiry when you ask for it: a certificate authority signs your public key, so nobody installs it on the host, and a signature issued with a short validity interval stops working on its own.

First ask how long the server should exist at all: treat nodes as cattle. The plan on the cluster we’re rebuilding is a cron job that kills the oldest node and lets the pool replace it, so nothing lives more than about seven days. That’s a proposal, not a result we can report on. A shorter-lived node is a shorter window for anything that lands on it.

Don’t take that too far. Taking interactive access away removes an access-management job, not the accounts and not the administrative work. A node still runs system and service accounts, and what people did over SSH moves to the Kubernetes API, the cloud API and your deployment pipeline. PCI DSS still wants a unique ID for everyone who reaches a system component and an audit trail of “all actions taken by any individual with root or administrative privileges” (requirements 8.1.1 and 10.2.2). Weekly replacement doesn’t buy an exemption from either.

When people do need a shell, certificates beat keys. OpenSSH has shipped them since version 5.4 in 2010: the CA signs a public key with identity and validity constraints, and sshd trusts that CA through TrustedUserCAKeys, so nobody logs into each host to add a key by hand. Set that validity interval yourself. ssh-keygen defaults it to always, so sign with -V and a short lifetime, or the CA has issued a credential that never expires (ssh-keygen(1)).

A short validity interval covers the ordinary leaver, once you stop the CA issuing to them. When removal can’t wait, revocation is a file on every host that trusts your CA. RevokedKeys in sshd_config names it, and keys listed in it “will be refused for public key authentication”. It does nothing until it reaches each affected host, so distribution falls to your configuration management, and “if this file is not readable, then public key authentication will be refused for all users” on that host. None of this switches off authorized_keys. TrustedUserCAKeys adds the CA beside the keys a host already reads rather than replacing them (sshd_config(5), OpenBSD 6.7), and an entry nobody tracks there stays valid until a human deletes it, which fails open. Delete or revoke those too. Sessions already open keep running until you end them.

Push the encryption out of the application

Inside the cluster the same argument shows up one layer down.

Why isn’t server-side TLS enough inside a cluster?

Because it authenticates the server to the client, and inside a cluster the client is anything that has a route.

Take Fluentd shipping logs to Elasticsearch in plain text across the cluster network. Elastic made TLS and role-based access free in 6.8 and 7.1, so switching it on is an option. It proves one end though, and mutual TLS proves both. The alternative is to leave both components as they are and let the infrastructure do it. Linkerd’s automatic mTLS covers “most HTTP-based communication between meshed pods”, with each proxy holding a certificate scoped to its pod’s ServiceAccount, rotated every 24 hours without anybody watching. Fluentd uses Elasticsearch’s HTTP API, so that hop qualifies. Both pods have to be in the mesh, and the request has to address Elasticsearch by name. A Host header holding an IP address is one of the cases the docs list as not upgraded. Meet those conditions and neither end learns a thing about encryption. Developers keep testing in plain text on a laptop, deploy the same thing to int, and production supplies the encryption.

The bill arrives when something breaks. Your application shows no error, so debugging moves into the infrastructure, which is unfamiliar ground for most developers. The injected init container rewrites iptables inside the pod to route traffic through the proxy (architecture docs), one more moving part between a request and where it was going. Read the gaps before you count on the coverage. Non-HTTP traffic isn’t encrypted automatically yet, and neither is an HTTP request whose Host header is an IP address rather than a name, which is why Linkerd’s own Prometheus scrapes aren’t. Linkerd doesn’t enforce mTLS either: an unencrypted request inside the mesh is upgraded where it can be, and one from outside the mesh arrives as it is. Encrypting everything is good practice. How much of your traffic needs it badly enough to pay the debugging tax is a local question. Your own public key infrastructure has the same shape. A few years back it was the answer to every security question at conferences. It still leaves you a certificate authority to run, secure and keep available.

The credential you never type

All of this shifts weight onto the credential that proves who is asking.

Why is a hardware key easier than a private key file?

Because a credential generated on the chip stays on the chip, so there’s no file to copy to a second laptop, paste into a chat window or commit by accident.

A password manager is the floor, and we assumed everybody had one until we started asking. Above it sit the cryptographic keys. Browsers talk to the chip directly through WebAuthn, a W3C Recommendation since March 2019, which is what GitHub and Google accounts use for hardware second factors. The browser hashes client data containing the challenge and origin; the key signs that hash together with authenticator data without exposing its private key.

The other mode is a different animal. A YubiKey can also act as a keyboard and type a long secret you never learn, which makes it usable as the passphrase on an encrypted SSH key: touch the key, get a session for the next fifty minutes, close it when you’re done. Nothing cryptographic happens there: the device types characters exactly as you would, and your SSH private key is still a file on the laptop, where a copy plus that typed secret is as good as the original. It spares you a passphrase you’d otherwise have to invent and remember. It doesn’t move the key onto the hardware.

We wouldn’t say the hardware is strictly more secure. We’d say it’s a more usable security primitive, and the difference shows in how each one fails. A key in a file gets copied to a second laptop, shared when somebody is in a hurry, and eventually committed: we’ve pushed service account JSON keys to GitHub ourselves and found out from an automated mail telling us to revoke them.

Trust your developers, verify the request

Credentials are half of it. The other half is what you assume about the people holding them.

What does trusting developers actually buy you?

Speed, and an honest picture of what’s running, because people stop routing around you.

The alternative is the cage, and the cage is always the easier answer. At a large company we had to write a document for every open source tool, explaining why the commercial product everyone already used wasn’t good enough, then wait for a committee. Approval could take six months, and we needed ModSecurity immediately.

A thread on LinkedIn asked how to protect a company against internal hacking, and the replies were all DMZs and firewalls. Our answer was to ask whether the developers are happy. Treat them well and they’re unlikely to turn on you. Treat them like dirt and they’ll return the favor.

Then verify anyway, because the insider path doesn’t need an ill-intentioned insider. Twitter lost control of accounts in July, and the company’s update of 30 July says the social engineering on 15 July “targeted a small number of employees through a phone spear phishing attack”, after which those employees’ credentials opened internal support tools. An attacker can exploit an employee’s access without that employee intending to help.

Picture the same trick where you work. Somebody turns up in your private Slack, asks for a hand with something plausible, and in a company big enough that you don’t know everyone you have no way to place them. That’s an invented example rather than the path into Twitter, and it’s the shape of the gap. Trust the people, and keep a way to confirm that a person is who they say they are.

Assume they get in, then be faster than they are

Everything above lowers the odds. None of it takes them to zero.

What’s the goal if you can’t stop the intrusion?

Getting them out and back to a known state faster than they can come back in.

A forensics team at a ransomware talk walked us through one investigation. The trail ran to a Tor exit address in another country, the victim was a private company with nobody to call, and that was where it stopped. We took the lesson that chasing the person was the wrong place to put the budget. Detect the behavior, evict, rotate the nodes, redeploy from source, and look at the way in while you’re there: we may be able to identify and fix the entry point during recovery.

Behavior is the thing to watch. Falco, which the CNCF took into its incubator in January 2020, reads syscalls and flags what’s ordinary on a VM and odd in a container, an interactive shell being the obvious case. The same idea works higher up: run the application in a test environment, record how it behaves, then treat that recording as the rule set in production. An application that starts doing something it never did in test has earned a page.

If you can replace every node and redeploy every service on demand, your recovery is a pipeline run rather than a project, and that can put you back to a known state considerably faster. Monitoring is the part that tells you to start, which is why it’s worth more than it gets credit for.

What this means for teams

Remote work didn’t create any of this. It removed the cover. The office network was never the control people treated it as, and the VPN inherited a job it was never designed to do, which is why it hurts now that every request runs through it.

This week:

• Write down what still requires the VPN. For each one, ask what would have to be true to put it on the internet behind an identity check instead.
• Find the IP allow lists in your infrastructure, starting with anything in front of your Kubernetes API endpoint, and check whether adding a node would break them.
• Turn on mutual TLS between two services in a non-production cluster, and write down what you had to learn to debug it.
• Order hardware keys for whoever holds production access, and move one SSH key behind one of them.

Common questions, answered

How do I fix a VPN that breaks internal DNS for remote developers?

Check which resolver your laptop uses inside the tunnel and outside it; an internal name usually resolves on only one. Point it at the DNS server inside the VPN network and the name tends to appear. The other option is to stop the client needing the VPN’s resolver at all: an endpoint reachable from the internet, with a publicly resolvable name and an identity check in front, resolves the same from a kitchen table as from the office. That’s an architecture change, not a DNS setting.

How do I remove a leaver’s SSH access across hundreds of hosts?

Stop the certificate authority issuing to that identity, then get the revocation to the hosts. A short validity interval only blocks new authentications, so when removal can’t wait, use RevokedKeys: each host reads its own copy, your configuration management has to deliver it everywhere, and a host where it isn’t readable refuses public key authentication for everybody. End the sessions they already have open. You skip editing authorized_keys on hundreds of machines only if those hosts take certificates alone: TrustedUserCAKeys adds the CA beside any keys they already read, so delete or revoke those too.

How do I encrypt traffic between services in Kubernetes without changing the applications?

Put a service mesh in front of them and let the proxies do mutual TLS, which is what Linkerd’s automatic mTLS does for most HTTP communication between meshed pods. Each proxy gets a certificate scoped to its pod’s ServiceAccount and rotated every 24 hours, and the applications keep speaking plain text. Know the gaps first: non-HTTP traffic isn’t covered, nor is an HTTP request addressed to an IP rather than a name, mTLS isn’t enforced so unencrypted requests still get through, and when something breaks you debug the infrastructure, not your application’s logs.

Related episodes

• AWS Security Maturity Roadmap: the order to do this in, and why permanent access keys are the first credential to retire.
• All You Need to Know About Setting Up HashiCorp Vault: short-lived credentials issued on demand, the same argument as SSH certificates one layer over.
• Docker Runtime Security: what a container runtime can tell you about behavior, and where Falco fits.

Resources

• BeyondCorp: A New Approach to Enterprise Security: the 2014 paper from Rory Ward and Betsy Beyer, what a breached perimeter hands an attacker, and the model that grants access on device and user credentials regardless of network location.
• Exposing the Great Firewall’s dynamic blocking of ESNI: the measurements behind the August 2020 reporting, including what triggers the block and how long it lasts.
• Linkerd automatic mTLS, as documented in 2020: the HTTP traffic the mesh encrypts by default, the 24 hour ServiceAccount certificates, and the gaps the project lists itself.
• ssh-keygen(1), OpenBSD 6.7: how a CA signs a certificate, and why -V is the difference between a short-lived credential and one valid forever.
• sshd_config(5), OpenBSD 6.7: what TrustedUserCAKeys, AuthorizedKeysFile and RevokedKeys do, and what happens on a host that cannot read the revocation file.
• Cilium 1.8 introduction: why address and port filtering breaks down when addresses churn, and what identity-based policy replaces it with.
• If you’re not using SSH certificates you’re doing SSH wrong: the case against authorized_keys at scale, and why unmanaged keys fail open.
• WireGuard: if you keep a VPN, the small one that’s now in the Linux kernel is the one to look at.

Since we recorded

Linkerd 2.9 extended automatic mTLS to all TCP connections in November 2020; Istio’s ambient mode brought per-node mTLS to GA in November 2024. Whether you want a mesh at all got an episode of its own, Do I need a service mesh?. ESNI didn’t survive either, and its successor Encrypted Client Hello is what Cloudflare switched on for free zones in September 2023. Support for passkeys expanded after Apple, Google and Microsoft committed to FIDO passwordless sign-in in May 2022, and we covered what they fix in Understanding Passkeys.