Security never makes the scaling list

Ask how a system scales and you get three answers: compute, storage, networking. Each one has a dashboard, a budget line and someone who owns it. Security gets a sentence instead, usually “security is a journey, we work on it continuously.” We have yet to meet a team that says it does security well and means it. What we hear is closer to “we make it work.”

The bill arrives in two places. One is access nobody can account for: the service account holding a role it has not used since the day it was created, the contractor whose SSO login was switched off while their database password kept working. The other is a dependency nobody looked at. On 15 April, Codecov disclosed that its Bash Uploader had been altered repeatedly between 31 January and 1 April, exporting whatever sat in a CI job’s environment to a third party server. The fallout reached far enough that HashiCorp rotated a release signing key.

Both of those are scaling problems rather than freak events. Security is usually left out of scaling conversations because it looks unmeasurable, and that’s the part we think is a myth. Count the steps it takes to add, change or remove one access right, then count the people involved in each step. Write the number down and you have something to work on, the same way a p99 gives you something to work on.

Scaling security means counting steps

That count is worth taking seriously because every step in it is one somebody built, which makes every step one you can shorten.

How do you measure security the way you measure compute?

Count the steps needed to grant, change and revoke a single access right, and count how many people have to touch each one.

Think about a village of a hundred people. A few roads, more animals than residents, basic infrastructure. A city of ten thousand serves the same purpose, people live there, and it’s a completely different thing: different layout, different utilities, different failure modes. Then there’s Tokyo or New York, where the city goes vertical because there’s no space left, and you stop measuring distance in metres and start measuring it in walking minutes because a car is useless. None of the three is a bigger copy of the one before it.

Systems go the same way. You start with one product and a monolith, and for a while you can buy your way out of load. Then you break it up, and microservices ship your organisation chart so two teams can release without queueing behind each other. If you’re ten people running microservices, we feel sorry for you, because it’s a lot of work for what you get back.

Costs move on the same curve, and logging is the clearest example. In a monolith everything logs to one place. With microservices a single transaction can generate logs across several services, the one calling and the one receiving, plus the networking in between and every error either side can raise. Storage and query costs climb with it, and they tend to hurt long before any technical limit does.

Access rights are the same kind of problem. Somebody asks for a grant, so the grant path tends to get built. Nobody asks to lose access, so the revocation path is the one we find missing, and it’s the one worth practising.

A coverage uploader, and somebody else’s signing key

Counting steps assumes you know what runs in your pipeline, and April has been unkind to that assumption.

Why does a dependency you don’t ship still matter?

Because build-time tools run inside the job that holds your deployment credentials, and they see everything in its environment.

Codecov’s security update, published on 15 April, describes periodic unauthorised alterations of its Bash Uploader script beginning 31 January 2021. Codecov learned of them on 1 April, after a customer compared the SHA256 published on GitHub against the hash of the script they had downloaded. The modified script exported environment variables and the git remote -v output from the CI job to a third party server, and the same uploader sits behind the Codecov GitHub Action, the CircleCI Orb and the Bitrise Step.

A week later HashiCorp published HCSEC-2021-12. The GPG private key used to sign the SHA256SUMS files for product downloads was exposed through that incident and has been rotated. HashiCorp found no evidence the key was used, which is the good part. The scope is narrower than “HashiCorp downloads”: the advisory limits the exposure to SHA256SUM signing, and to the provider verification Terraform performs against that key. macOS notarisation, Windows Authenticode and the Debian and RPM package signing on releases.hashicorp.com were not affected. If something in your pipeline checks the signature on a SHA256SUMS file, it needs the new fingerprint, and the Terraform patch releases carrying the new key are v0.11.15, v0.12.31, v0.13.7, v0.14.11 and v0.15.1.

The same economics apply to the vulnerabilities you inherit. A CVE is a public description of a problem, and responsible disclosure gives the vendor time to patch before the details go out. Once the number exists, take it to GitHub and there’s a fair chance you will find a working proof of concept, which is why the gap between disclosure and your patch is the window that matters. If you run open source, follow its releases, and keep a channel open to every vendor whose code runs in your build.

Certificates stopped being a project

Signatures and identity are the same problem in two places, which brings us to the control most teams still treat as exotic.

What is a certificate?

A public key with a name attached, signed together by a party both sides trust, so an identity can be proved instead of claimed.

That’s the whole idea. Once a service can prove which service it is, you can authorise on that identity on the server side, which is what mutual TLS between services buys you. Public key infrastructure used to be the reason nobody bothered: issuing, distributing and renewing certificates by hand was a job in itself. Let’s Encrypt changed the economics by making automated issuance free and pairing it with a short lifetime, for the reasons it set out in Why ninety-day lifetimes for certificates: limit the damage from a stolen key, and encourage automated renewal.

Inside a cluster, a service mesh issues and rotates certificates for the workloads that join it, and it handles the mutual TLS handshake without the application knowing. The qualifier matters. A workload gets a certificate only once it is in the mesh, so anything running without a sidecar stays outside the scheme. Issued certificates are also not the same thing as enforced mutual TLS: Istio configures destination workloads in permissive mode by default, which accepts plaintext as well as mutual TLS, and Istio’s security best practices recommend moving to strict mode where you can, because permissive weakens your security stance. Outside a cluster, Vault’s PKI secrets engine will act as a CA and hand out short-lived certificates on request. Both replace the spreadsheet of expiry dates, which was the part that failed as teams grew.

None of this has retired the old stack. Most companies have an SFTP server moving files somewhere, and financial systems in particular still run on it: it’s old, convenient, battle tested and it has audit logs. Moving that traffic higher up the stack would be an improvement, and it keeps losing to “it works.”

Cloud IAM is the lock-in nobody budgets for

Certificates are portable. The thing you build around them isn’t.

Which part of your stack is hardest to move to another cloud?

The security model, because it’s the one layer the cloud doesn’t abstract for you.

You set it up out of cloud primitives: service accounts, IAM bindings, the resource hierarchy, firewall rules, key management.

Each provider has its own levels of granularity. Google Cloud attaches policy down a hierarchy of organisation, folders, projects and resources, and an inherited allow at the top is an allow at the bottom. AWS splits the same concerns across accounts, organisational units and per-service policy. Sometimes one cloud gives you a level of granularity the other has no equivalent for, and that’s where a migration stops being a lift and starts being a redesign. A cross-cloud single pane of glass for access is hard for the same reason: the clouds do more or less the same thing, and the more or less is the difference between a village and a megacity.

You cannot secure what you cannot list

Granularity is only useful if you can see what is currently set, which is where most teams find out how much they own.

Where do you start when you inherit someone else’s cloud account?

With a list of what is in it.

Google Cloud’s Asset Inventory enumerates resources and IAM policies across an organisation, and you can export that snapshot into BigQuery and query it like any other table. Build that list on purpose, because no credential arrives with a map of its own reach attached. What an account can discover depends on what it was granted: Asset Inventory’s access control puts the search and export calls behind permissions such as cloudasset.assets.searchAllResources, held at the organisation, folder or project the call names. Which role carries them, roles/cloudasset.viewer, another predefined role or a custom one you wrote, makes no difference to the API. Holding the relevant permissions at the scope being queried is what decides whether the call returns an inventory or an error. Permission testing answers a narrower question: hand testIamPermissions a named resource and a list of permissions, and it returns the ones that caller holds. A key leaked into a public repository will tell a patient attacker plenty that way, one resource at a time, and considerably more when the account behind it can search assets too. Either way the map is yours to maintain, so tracking who has access to what stays a job someone on your side has to own.

Usage data narrows it further. Role recommendations look at which permissions a principal actually exercised and propose removing the rest, so you get told that a service account has not written to a bucket in three months rather than inferring it from a design document.

Access patterns are their own signal. A critical resource read at 2am by an account that never works at 2am, from an IP in a country you don’t operate in, is worth an alert even when every permission involved is legitimate. Runtime behaviour counts too: an application writing to the container’s root filesystem with no volume mounted is usually a misconfiguration, since logs belong on stdout and that disk disappears on restart. Set spec.containers[].securityContext.readOnlyRootFilesystem on any workload that has no reason to write to its own filesystem, and give the ones that do a volume instead.

Move the feedback, not the gate

Visibility changes what you can enforce. Timing changes whether anyone acts on it.

When should a security finding reach a developer?

While their editor is still open on the branch that caused it.

One pattern we run: every merge request gets built and deployed into the cluster so the developer can test the change for real. The side effect is that the artefact exists before anything reaches master, so the image can be scanned for known vulnerabilities and the code can go through static analysis at the same time. The developer gets pinged while the context is still loaded, branch checked out and config files open, so nobody has to reconstruct the change before deciding what to do about the finding.

Shifting left doesn’t mean shifting every control onto the laptop. A local Elasticsearch with no TLS and no auth is simple to work against, which is the point of running it locally. Deployed to the cluster, the mesh takes over the network half of the problem at no cost to the developer: mutual TLS proves which workload is calling and encrypts what passes between the two. It does not decide what that workload may then do. Istio’s security best practices are blunt about the split, that mutual TLS “provides only authentication, not authorization” and that “anyone with a valid certificate can still access a service”, so which workloads may call which is a separate authorization policy you write. Permissions expressed in your data stay with the application: in Elasticsearch that’s its own users and roles, down to which indices, documents and fields a role can read. Where a later layer genuinely does the job better, skip the control early rather than paying the friction twice.

Outside requirements decide what gets funded

Good timing still needs budget, and budget usually arrives from outside engineering.

Why does security suddenly become a priority?

Because somebody outside the company made it a condition of operating, and an external requirement moves work up the list in a way no internal argument does.

GDPR and HIPAA are law. Governments picked spaces where failure leaks somebody’s personal information and attached penalties, which raised the priority of security inside the companies working there. PCI DSS arrives by another route. The card brands maintain it through the PCI Security Standards Council, and as the council’s own quick reference guide sets out, each payment brand runs its own compliance programme, so what binds you is the contract with your acquirer rather than a statute. The effect on the roadmap looks much the same. Who you argue scope with does not. Where neither applies, security sits low in the backlog, and that isn’t cynicism, that’s what we observe.

The same lever can be pulled too hard. Europe is discussing rules for artificial intelligence, proposed on 21 April, that sort uses by risk and ban some outright. Our worry, and we aren’t lawyers, is that European startups carry a compliance cost their competitors in the US and China don’t, which makes it harder to build here. Innovation and restriction have to be balanced, in AI rules and in your own controls. A completely secure system is one you switched off, and it earns nothing.

Do it often and it stops being hard

Which leaves the part that no statute and no contract will do for you.

How do you get good at revoking access?

Revoke access often. The DevOps answer applies here: if you’re not good at something, do it often, and you get better at it.

Visibility is what makes that practice cheap. If working out who holds what takes an afternoon of digging, nobody repeats it monthly, and stale access is usually removal that nobody got around to. Change companies and two years later you can often still open the monitoring dashboard you used, sometimes the production database, because the password is the password and nobody rotated it.

Single sign-on is what turns removal from six steps into one. Point everything at one source of truth, switch the person off there, and new logins to the cloud console, the servers and the dashboards stop together.

A session issued before the switch keeps working until it expires, and AWS wants an explicit revocation of the role’s older sessions to end one early. A local database password, a personal access token or an SSH key left on a host survives the account being disabled at all. So offboarding has two tests: try to log in, and try to use something that was already open. The tooling for the first test has got better, with Boundary released in October 2020 for identity-based access to hosts and services without VPN credentials or bastion keys, and Cloudflare Access putting internal applications behind the same identity.

Then drill what you hope not to need. Fake phishing campaigns are common and they work for the same reason fire drills do. The one more teams skip is restoring a database backup, which is a practical way to test whether you can recover encrypted data without an attacker’s decryptor: paying a few Bitcoin for a key you may not get isn’t a recovery plan. A restore isn’t a security control by itself, and it does nothing about data copied out before anything was encrypted. What a timed restore gives you is the recovery number, before the day you need it.

What this means for your team

Security that scales isn’t a purchase. It’s a set of steps you can count, shorten and repeat, which means treating access the way you treat deploys: frequent, scripted, boring. That isn’t a question of owning more tools. Aim for a system where asking for access, seeing who has it and giving it back all take about the same small effort, and where someone looks at the list on a schedule.

Four things worth doing this week:

• Rotate the secrets exposed to any CI job that ran the Codecov Bash Uploader, GitHub Action, CircleCI Orb or Bitrise Step since 31 January, and update anything that verifies a HashiCorp SHA256SUMS signature to the new GPG key.
• Export your cloud asset inventory somewhere queryable, then rank what it lists by what a compromise would cost: a leak that brings a regulator in, a system whose loss stops revenue, a database nobody would miss for a week. Answer one question from the top of that list, such as who can read the production data store, service accounts included, and put the heavier controls there.
• Take the first role recommendation that touches something near the top, remove the unused permission, and write down how many steps the removal took. That count is your baseline.
• Restore last night’s database backup into a scratch environment and time it.

Common questions, answered

How do I check whether the Codecov incident touched my pipeline?

Look for any repository in your organisation that used the Bash Uploader, the Codecov GitHub Action, the CircleCI Orb or the Bitrise Step between 31 January and 1 April 2021, the uploaders and window Codecov names. Codecov emailed affected users on 15 April and put a banner in the application after login; on 29 April it added detail in-app on which environment variables may have been obtained and updated its indicators of compromise, so check both. If you compared the uploader’s checksum before running it, Codecov says the issue may not affect you.

Then rotate what the job could see. Run env in that pipeline, and treat everything private in the output as exposed: registry credentials, cloud keys, signing keys. The altered script sent that environment and the git remote -v output to a third party server, so regenerate each one and audit where it was used.

How do I find permissions nobody uses in Google Cloud?

Open role recommendations for the project in the IAM console, where Google proposes removing or replacing roles based on the permissions a principal actually used. Apply them somewhere low risk first and read each one before accepting it, because a permission exercised once a quarter, by a billing job or a disaster recovery runbook, looks unused inside a shorter observation window. Pair the cleanup with an asset inventory export so you can prove afterwards what changed.

What’s the cheapest security habit to start with?

Restore a backup into a scratch environment and time how long it takes. Write down the number and every step that failed on the way, because that record is what tells you whether your answer to ransomware is a restore or a negotiation. Run it again next quarter: a restore time is only useful once you know which way it’s moving.

Related episodes

• Episode 20, Monitoring Done Wrong or Dreaming for a Better Monitoring is the monitoring wish list this post’s visibility argument starts from.
• Episode 11, AWS Security Maturity Roadmap is the same enumerate, audit and prioritise argument written out for AWS accounts.
• Episode 13, All You Need to Know About Setting Up HashiCorp Vault is the background on dynamic credentials and what running Vault actually involves.
• Episode 24, Ways To Protect Yourself From Data Breaches is the other half of supply chain risk: what to do when the breach happens at a service you use.
• Episode 12, Scale vs Scaling is where we worked out what scaling means before applying it to anything.

Resources

• Codecov security update (15 April 2021) settles the timeline: alterations from 31 January, discovery on 1 April through a customer’s checksum comparison, and what the script exported.
• HCSEC-2021-12, Codecov security event and HashiCorp GPG key exposure (22 April 2021) settles which key was rotated, which signing mechanisms it covers and which it does not, and the Terraform versions that ship the new one.
• Resource hierarchy on Google Cloud settles where policy can be attached and how inheritance flows down to resources.
• Overview of role recommendations settles how usage-based recommendations are produced and what they will and won’t propose.
• Asset Inventory access control settles which permissions an account needs, and at which scope, before it can search or export an inventory at all.
• Istio security best practices settles what the mesh gives you by default, that mutual TLS authenticates without authorising, and what strict mode and authorization policies add.
• Shifting left on security is Google Cloud’s write-up of moving controls into development rather than running them as a release gate.
• Why ninety-day lifetimes for certificates (9 November 2015) settles the reasoning behind short certificate lifetimes and why they encourage automated renewal.

Since we recorded

Google published SLSA in June 2021, a levelled framework for build integrity aimed at the kind of tampering the Codecov uploader showed. The US executive order of 12 May 2021 told NIST to write secure software development guidance and NTIA to define the minimum elements of a software bill of materials. Forseti Security is no longer maintained, and its repository was archived in January 2025. We returned to the subject twice, in episode 46, Software supply chain attacks and episode 53, Open Software Supply Chain Attack Reference Framework.