When the control costs more than the thing it protects

Compliance says you have to show what a person typed on the jump box. Your developers have read access to an S3 object and still can’t download it. Someone committed every credential a service needs into the root of its repository in clear text, and you can’t pull them back out without stopping people from working. One engineer is being asked to run Kubernetes on bare metal on top of everything else they already own.

Four problems out of two weeks of work, and they share a shape. The control that’s correct on paper, whether that’s session recording, customer managed keys, a secrets manager or an orchestrator with every feature, costs more to operate than the thing it protects, or more than the team that has to operate it. What you get then is the worst option available: the control is half installed, people work around it, and the compliance document says it’s there.

So the useful question isn’t which control is strongest. It’s which one still works in six months, when whoever set it up has moved on. We’re not experts, and in a year we may disagree with half of this. Here’s what we tried and what we walked away from.

Recording what people do on the jump box

The audit requirement is cheap to state and expensive to meet: after a breach, show what people did on the hosts they reached.

Can AWS Session Manager do the recording for you?

It records commands and output and gives you a shell with no VPN and no open port, once you configure logging and settle who each person lands as.

Install the SSM Agent with an instance profile that can call the Systems Manager API. The agent dials out, AWS proxies your connection back, and on-prem machines appear beside your EC2 instances. Recording is a separate step: turn session output logging on in preferences and name an S3 bucket or a CloudWatch log group the instance profile can write to. Miss it and you have shells and no audit trail.

Configuration only takes you so far: logging isn’t available for sessions that connect through port forwarding or SSH, because SSH encrypts the data inside the tunnel Session Manager is carrying. The trail covers interactive shells only.

The first concern was reach. Credentials that may start a session get a shell from anywhere; IAM keys are all anyone needs. We could live with that, because the environment we work in keeps no static credentials.

Short credential lifetimes bound less of that than you would think. AWS documents that after a session is established, it is not affected by IAM role session duration, so expiry governs who can open a shell, not how long an open one runs.

Identity was harder. By default you land as ssm-user in /usr/bin, not a home directory, and that account can sudo. Run As answers that: tag an IAM user or role with SSMSessionRunAs and the tag’s value becomes the OS account. A role can carry the tag, so dynamic credentials aren’t the obstacle. Per-person identity is the obstacle: the jump box issues dynamic SSH keys.

We went back to AWS’s older bastion-host recording recipe: sshd forces every login through a wrapper that records with script and uploads to S3. A log the recorded user owns is a log they can edit, so we changed that. tlog writes the session as JSON into the journal and runs its recorder SUID/SGID as a user the recorded user can’t touch. Root still can. Getting it off the box is unfinished: the CloudWatch agent reads files, not journals.

Know the ceiling. curl https://example.com/install.sh | sh records as one line, and suppressed output isn’t output. auditd in a PCI environment buried us; Elastic barely kept up.

Why an open bucket can still refuse to serve the file

Session recording fails quietly open. Encryption on S3 fails the other way, and it fails on your own people.

What changes when you encrypt a bucket with your own KMS key?

The reader needs permission on the key as well as on the object, so read access to the object stops being enough on its own.

The two options we’re comparing are SSE-S3 and SSE-KMS, and the difference lands on the reader. With SSE-S3, S3 holds the key and read access on the object gets you the plaintext. With SSE-KMS and a customer managed key, S3 still does the decryption, but it calls KMS using the caller’s permissions, and AWS is explicit: “To download an object encrypted with an AWS KMS key, you need kms:Decrypt permissions for the key.”

So a bucket left open to the world, with objects under a customer managed key, hands anonymous readers a denial rather than data, because there’s no anonymous caller you could grant the key to. The blunter fix for public buckets shipped in November 2018: S3 Block Public Access, account-wide or per bucket, and aws_s3_bucket_public_access_block in Terraform. Use that first. The key is a second lock, not the first one.

The cost of the second lock lands on your own people. A Lambda deployment package has to sit in a bucket in the same Region as the function, though the bucket can belong to another account. That’s a property of the Code block rather than a quirk of the Go runtime. We build in one Region and run in several, so the binary gets copied cross-account into a bucket beside each function. The S3 grants were right and the download still failed.

Two ways out. Grant the other account kms:Decrypt on the key, which across accounts takes both halves: the key policy allowing that account, and an IAM policy on the caller allowing the call. That route needs a customer managed key anyway, since objects under the AWS managed aws/s3 key can’t be shared across accounts. Or ask what the key is protecting. We moved those artifacts to SSE-S3, because a Lambda binary we publish into our own accounts wasn’t the thing we were trying to keep from anyone. The cross-account documentation we found covers the bucket policy and says little about the key, so take that as our reading.

Keys in the repository, and where they belong instead

Encryption arguments assume the keys are somewhere defensible. A lot of them are in Git.

What do you fix first when a repository is full of live credentials?

Rank the keys by what they reach and invalidate the dangerous ones on that basis, then make local development work without the rest.

Here’s the shape of a bad one. Google and AWS service account keys as files in the root folder, clear text, helpfully named. One configuration file carrying dev and prod together: usernames, passwords, API credentials, mail and auth logins. The part that should worry you most isn’t any of that. To mock the database locally, the setup opens an SSH tunnel to production and maps local port 3307 to 3306 on the production SQL server, so every laptop holds a key with a route into production data. That behaviour is why we want to disable port forwarding on production SSH servers.

Order the work by exposure rather than convenience. AWS’s guidance for an exposed access key starts there: work out what the credential can reach, then invalidate it on that basis. The key that tunnels into the production database doesn’t wait for a Dockerfile. What can wait is the long tail, because pulling all of it at once breaks the local setup and people can’t work. So dockerize the stack until it runs locally against local services and needs no keys, then deal with what’s left.

What’s left moves into the deploy, with each environment handed its own set: test credentials for the test cluster, production ones for production. “Keep the secrets in the build chain” reads two ways, so be precise: what the deploy hands over is the cluster’s copy, not a layer in the image. An image with secrets baked in is a secret held by anyone who pulls the image. Ship one with nothing in it and let the workload fetch what it needs at startup: with Vault’s Kubernetes auth method the pod presents its service account token, receives a Vault token, and pulls its secrets. Steal that image and run it outside the cluster and you don’t have the service account that makes it work.

Whether that means Vault depends on where you run. Hybrid or multi-cloud, it gives you one interface across the lot. On one cloud with no plans to leave, much of what Vault does has a native equivalent: STS for dynamic credentials, RDS IAM authentication for database logins, Secrets Manager, KMS. Either way it’s a journey.

Why the keys were there in the first place

Nobody in that repository set out to leak anything, so the question is what the alternative would have cost them.

Why does a developer hard code a secret they know is wrong?

Because the approved path costs an afternoon of learning and the wrong one costs ten seconds, and nobody is measuring them on the first.

You have a feature to land. Security tooling isn’t where you spend your days, your manager doesn’t understand why you’re blocked, the pressure is on. Hard code it, move on, and nothing looks at it again until someone reads the repository two years later. Try an honest test on yourself: can you generate a TLS certificate with openssl on the command line right now, without looking it up? Most of us can’t. It’s a twice-a-year job, and the flags don’t stay in your head any more than the right tar invocation does. Keeping those commands in a Bash script you run next time is a fix for one person, not for the tooling.

Some of that gap is who builds the tooling. Those of us who build infrastructure, automation and security tools have freedom developers don’t get: outside the team nobody understands what we do, and everyone is content as long as it works. The cost is that we build for people like us and call the result usable.

Tooling is only half of it. Vault, installed across availability zones with redundancy and ready to go, and then the project stopped dead on who should have access to what. That turned political, and the whole installation came out again.

The thing worth building is a fast sanctioned path. One we’ve sketched and not built: a bot you ask for one specific privilege, say a login on one server. It messages the people who approve that privilege, their decision and the grant are logged, and the access expires on its own. The attraction is that you write rules instead of a UI. Before trusting it we’d want to know how a request gets spoofed, and we’d keep in mind that a chat workspace isn’t an authorization system. Once someone is in, there’s little fine-grained role control standing in their way, and a private channel is a perimeter inside a perimeter.

The network you’re on shouldn’t decide your access

A perimeter inside a perimeter is still a perimeter, which is the part that keeps failing.

Is the VPN the problem, or the model behind it?

The model. A VPN answers one question, are you inside, and on a flat network that answer is the whole access decision. You can narrow it: VPN filters can restrict each user’s network access, and traffic the filter doesn’t permit is dropped. The services they do permit still need appropriate authentication and authorization, and that work happens at the service rather than at the tunnel.

The tunnel’s own limits showed up this spring. CISA’s Enterprise VPN Security alert, 13 March 2020, says organizations “may have a limited number of VPN connections, after which point no other employee can telework”. It tells teams to test those limits, and to consider rate limiting so the users who need bandwidth get it. We’ve heard that even Cisco had to ration its own staff, which we can’t confirm. Split tunnelling is a lever here rather than a cop-out. Cisco’s guidance of 16 March 2020 suggests looking at it, and AnyConnect can split selectively by allowing only trusted IP ranges. The same post says that change has security ramifications your security team has to weigh. That’s capacity bought with less inspection, deliberately.

Google wrote down the alternative in 2014. BeyondCorp drops the privileged internal network, so where you connect from doesn’t determine what you can reach and every request is authenticated and authorized on its own. A castle with walls around it, against a guard in each room checking ID. It’s harder to implement, mostly because it’s less mainstream and the learning curve is real.

For services rather than people, the same argument is happening inside the service mesh war. The orchestrator war ended with most of the industry marching behind Kubernetes and singing, and the mesh one hasn’t settled, possibly because execution environments differ too much for one answer. On the hybrid project we run Consul for service discovery and configuration, with Nomad above it for scheduling. Zero trust networking between those services, through Consul Connect, is the intention rather than something we’ve finished. The teams who get furthest with this aren’t the ones with the strictest rules. They’re the ones who can see what happened, which is why good monitoring makes people relaxed about production. If you run A/B tests, you already test in production.

Pick the orchestrator you can staff

Something still has to start the containers, and that choice is where most of the operating cost gets decided.

How do you choose between Nomad, ECS and Kubernetes?

Start from what you need to achieve and who you have to run it, then take the simplest thing that gets you there.

On the hybrid project, Kubernetes on bare metal would have been a networking project before it was a platform, so we went with Consul and Nomad. The pieces map onto concepts you already know, and the engineer on the ground, who hadn’t used either before, had it up in a couple of days. Documentation thins out at the predictable place: the proof of concept where everyone can do anything is easy, and ACLs and agent tokens are where it gets blurry. One assumption didn’t survive contact with HashiCorp’s own people. Consul agents health check each other across the cluster, so a single Consul cluster covering everything wasn’t going to work, and we segmented.

Without the on-prem half we’d have started with ECS. There’s no charge for the orchestration itself on the EC2 launch type, and the per-node overhead is one agent container. It’s less mature than Kubernetes, and that’s the trade you’re making.

The free control plane argument is narrowing from both ends. EKS halved its price to $0.10 per cluster per hour in January 2020. GKE, where masters used to be free, will charge $0.10 per cluster per hour from 6 June 2020, with one free zonal cluster per billing account. That’s about $73 a month before a single node, which is still a rounding error next to the person who runs the thing.

The honest argument for the crowded choice is insurance. Pick what everyone else picked and someone else hits your problem first, writes it down, and teams up with you on the fix. That’s worth real money, and it’s a different claim from “Kubernetes is the right tool for this”, which is the claim people usually make instead. Start from what you want to achieve rather than from the tool. The best battle is the one you don’t fight.

What Kubernetes costs after the first deploy

Counting the people cost matters because the platform keeps asking for it long after the first deployment works.

What is the real work of running Kubernetes?

Upgrades and the configuration around a deployment, not the deployments.

You will upgrade the cluster, on EKS as much as anywhere. From a security standpoint we’d rather create a new cluster and migrate the workloads than upgrade in place, because an in-place upgrade keeps things backwards compatible, which usually means newer safer defaults stay off. That makes every upgrade a migration, and a migration needs you to know every workload in the cluster, every configuration applied to it and everything attached from outside, with ingresses coming and going and load balancers wired to them. Manifests in Git that the cluster reconciles from solve part of this, which is the strongest argument for GitOps we have.

Kubernetes arrives as a platform with nothing configured: cluster autoscaler and the cloud’s own scaling, resource requests and limits, horizontal pod autoscalers, pod disruption budgets, and a list of objects you haven’t met yet. Template a normal Helm chart to disk and count the files. Your Deployment is one of them. The platform also moves fast enough that somebody has to follow it release by release, which on a one-person team means that is now the person’s job.

Mattias doesn’t buy the depth argument. You deploy a pod and an ingress and it works. cert-manager, ExternalDNS and the NGINX ingress controller are a Helm install away, with close to no configuration. Namespaces give you separate environments cheaply, and splitting traffic for an A/B test is a YAML file. Andrey’s answer is that deploying was never the hard part, since you can run containers on an EC2 instance with docker-compose in an afternoon. They split on state too. Julien would keep databases out of the cluster and run one on a VM where the tooling for databases lives. Mattias labels a node, pins a StatefulSet to it, and lets Kubernetes handle the rest.

Nomad is the counter-example and the trade in one. Its autoscaler arrived in March 2020 as a technical preview with 0.11, horizontal application scaling only, long after Kubernetes had the feature. Fewer features is a genuine cost, and so is a platform changing faster than your team can read.

What this means for teams

None of this argues for less security. It argues for pricing the operating cost in the same breath as the control, because an incomplete control can leave gaps the compliance document doesn’t show.

This week:

• Try to read an object from one encrypted bucket using the role that actually consumes it, in the account that owns it. When it fails, go to CloudTrail for the decrypt denial that S3 didn’t show the caller.
• Grep your repositories for key material, then read what the local development setup connects to. A tunnel from a laptop into the production database is worse than the key sitting beside it.
• For the next control you add, write down who operates it in six months and what happens while they’re on holiday. If the answer is the person who installed it, you’ve bought fragility with a budget line.
• Price your orchestrator in people before dollars. The cluster fee is roughly $73 a month; an engineer tracking releases to keep your defaults safe costs more than that before lunch.

Common questions, answered

How do I fix Access Denied when I can see an S3 object but can’t download it?

Read CloudTrail before you change a policy, because an S3 denial doesn’t by itself mean KMS: look for a kms:Decrypt denial naming the key the object is under. Then grant the caller kms:Decrypt on that key. Inside one account the key policy alone can do it; across accounts you need the key policy and an IAM policy on the caller, and a customer managed key, because the AWS managed aws/s3 key can’t be shared across accounts.

How do I record SSH sessions on a bastion host for an audit?

Force every login through a recorder in sshd and ship the recording off the host as it is written. Session Manager is an alternative to SSH rather than an addition: turn session output logging on in its preferences and let the instance profile write to the destination. Its recording covers the interactive shell, and sessions that connect through port forwarding or SSH aren’t logged. Either way, run the recorder under an identity the recorded user doesn’t control. Expect gaps: a piped installer records as one line that says nothing.

How do I give someone production access for an hour instead of forever?

Put a request and approval step in front of a short-lived grant, so the access arrives scoped to one privilege and expires on its own. A chat bot is a cheap way to run it: you ask, the approvers decide, the decision and the grant are logged. Don’t make the chat workspace the security boundary: once someone is in it there’s little fine-grained role control to stop them. On AWS a short STS session puts a clock on API access only. A Session Manager shell that is already open keeps running, so closing that is a separate step.

Related episodes

• GitOps: the pull-based model and keeping desired state in Git, which is what makes rebuilding a cluster survivable.
• Docker Secure Build: keeping build credentials out of the image you ship.
• Docker Runtime Security: containers against VMs, and why monitoring decides how much access people need.

Resources

• How to Record SSH Sessions Established Through a Bastion Host: AWS’s forced-command recipe, and the part it leaves fragile.
• tlog: records terminal sessions as JSON into journald or syslog, and its install notes cover the SUID/SGID setup that keeps the recorded user away from the recording process.
• Logging session activity: the S3 and CloudWatch Logs options, the permissions the instance profile needs, and the sessions they don’t cover.
• Using server-side encryption with AWS KMS keys: states the kms:Decrypt requirement for downloads and why cross-account sharing needs a customer managed key.
• Introducing Amazon S3 Block Public Access: the account-level and bucket-level switch that overrides permissions allowing public access.
• Cisco PIX/ASA VPN filters: how per-user and per-group ACLs restrict traffic once the tunnel is up.
• CISA alert AA20-073A, Enterprise VPN Security: connection limits under mass telework, and the advice to rate limit.
• BeyondCorp: A New Approach to Enterprise Security: the 2014 paper that drops the privileged internal network.

Since we recorded

New buckets were already private, and from April 2023 AWS also turns on Block Public Access and disables ACLs for every new bucket, so the settings that keep a bucket from being made public are on unless you switch them off. The journal problem went away: the CloudWatch agent reads journald entries natively from August 2026, which until then needed the export-to-disk step we were investigating. The Nomad Autoscaler grew past the preview and added horizontal cluster autoscaling. We came back to Vault in detail in episode 13, and to VPN alternatives, SSH certificates and hardware keys in episode 15.