Podcast about DevSecOps

#81 - Keeping Secrets Safe

One shared database password can erase accountability and turn rotation into guesswork.

17 min read

Still pasting tokens into Slack? What types of secrets are at risk, and which tools fit which consumer—humans, CI/CD, or workloads? Where do most teams stumble, and how do you fix it fast? Hear our no-nonsense checklist.

Discuss the episode or ask us anything on LinkedIn


The secret is still in Slack, and it still works

Search your company chat for a few characters of a password you already know. Not the whole string, just enough to match, and key or token while you’re there. A hit is not proof of a breach, it’s a lead: find out which channel it sits in, who can read that channel today, and whether the credential still authenticates. Often enough the answer is that it went in the day someone needed a colleague to get into staging, everyone who joined the channel since can read it, and nobody ever revoked it.

That’s the shape of secrets management in 2025. Not exotic key management, not quantum anything: hard-coded credentials and passwords pasted into chat. GitGuardian counted 23.8 million secrets leaked on public GitHub repositories during 2024. Of the ones leaked in 2022, 70% were still valid two years later. Verizon’s 2025 Data Breach Investigations Report puts the median time to remediate a secret leaked to a public GitHub repository at 94 days. The chat window you just searched is in the same research. Secrets turned up in 2.4% of corporate Slack channels and 6.1% of Jira tickets. GitGuardian rated 38% of the incidents it found in Slack, Jira and Confluence as highly critical or urgent, against 31% in source control.

So those places do get scanned. What they lack is an owner. The irritating part is that the tools are all here, most of them cheap or free: what’s missing is a decision about which secrets you control, who or what consumes each one, and where each one is allowed to live. Make those three calls and most of the tooling questions answer themselves.

Sort secrets by who consumes them

Before picking a vault, split the problem, because a login page and a partner’s API key have almost nothing in common.

Which tool fits which secret?

Humans get a password manager, applications get a secrets manager, and after that the line that matters is whether you control the secret’s lifecycle.

Human secrets are the solved category. Bitwarden is open source, macOS ships a Passwords app, and there are paid options like 1Password, so there isn’t a good excuse left. The part teams underuse is sharing. 1Password can share a copy of an item by link, with an expiry you choose and access limited to named email addresses, and the person receiving it needs no 1Password account. Bitwarden’s equivalent is Send, also link-based, also fine for people without an account, deleted automatically after a lifespan you set: seven days by default, 31 at most. Either way nobody has to paste a credential into a messenger.

Sharing outward is where this gets ragged. PGP works when both sides already hold keys, a share link works when they don’t, and both end with plaintext on the recipient’s screen and no say in what happens to it next. Agree that it goes straight into their own manager, and agree it before you send, because otherwise every exchange gets improvised from scratch.

Machine secrets split in two. There are the ones you control: the database password you generated, the IAM credential you can replace, the token you can shorten the life of. Then there are the ones handed to you from outside. A partner sends an API key, and you can’t generate it, can’t set its TTL, and rotating it means an email and a human on their side doing work on their schedule.

Then ask where the value is consumed, because the delivery mechanism differs: a Docker build, a Terraform run, a pipeline step, a running container, or a command somebody types on a laptop.

One shared password costs you the audit trail

The category that does the most damage is the one teams treat as temporary: a single credential several people and several applications all use.

What does a shared database password actually cost you?

Traceability first, and then your ability to rotate without breaking things you can’t name.

It starts with a database that only has root credentials. The password gets shared, people log in with it, and every session in the audit log is the same user. Tracing who ran the query that dropped the table becomes guesswork. Applications end up using the same credential, so your logs show one busy user doing everything.

When you can’t remove the shared account yet, put something in between: people sign in with their own identity to a gateway, and the gateway uses the shared credential on their behalf. You get a record of who went through. It does nothing for the applications, which still show up as one anonymous client, and it’s a step on the way out rather than a destination.

Short-lived credentials beat rotating static ones

Rotation is the thing everyone agrees you should do and few people do, which is a hint that the goal should be credentials that expire by themselves.

Can you get short-lived database credentials outside a managed cloud?

Yes. HashiCorp Vault’s database secrets engine creates a user on demand and deletes it when the lease ends, and the price is the SQL you write to clean up after a user who won’t exist tomorrow.

On AWS some stores skip the password entirely. DynamoDB is reached over HTTPS and authorizes the caller’s IAM role, so there’s nothing to share. For PostgreSQL and MySQL on RDS, IAM database authentication generates a token bound to your role, and each token has a lifetime of 15 minutes. Those 15 minutes bound when a token can open a connection, not how long a session lasts: AWS says the token “is only used for authentication and doesn’t affect the session after it is established”. The work for whoever owns the application is to hold a valid token every time it opens a connection, so a pool that reconnects has to mint tokens as it goes instead of reading a string out of config. Developers are busy with business logic, which is why this gets postponed. Infrastructure engineers can do that instrumentation themselves with a coding assistant and a few iterations, without waiting for the application team.

The cloud version is a bit of a cheat, because the platform is doing the hard part. Plenty of teams run their own MySQL on their own hardware, and Vault is the answer there: it logs into the database with its own credentials, mints a user and a temporary password for you, and drops the user when you’re done. The catch we hit in practice is what the temporary user leaves behind. Everything it created had to be handed to something permanent, and our revocation statement turned into advanced SQL.

How much of that you have to write depends on the engine. MySQL’s DROP USER leaves the databases and objects the old user created in place. Since 8.0.22 it fails by default when the account is named as the DEFINER of a stored object, unless the caller holds SET_USER_ID, which turns the error into a warning and leaves the stored object orphaned. PostgreSQL won’t drop a role until the objects it owns are dropped or reassigned and its privileges revoked, in each database that holds them. REASSIGN OWNED then DROP OWNED is the usual way to clear that in bulk, with ALTER ... OWNER TO to move objects one at a time. It’s possible either way, it just takes time that nobody budgeted.

Every copy of a secret is another thing to rotate

Short-lived or not, the next failure is distribution: the secret gets copied somewhere convenient and stays there.

Why does copying a secret into CI make rotation harder?

Because each copy is another place to update and another audit log to correlate.

Keep one source of truth and the trail stays readable. Read a value from AWS Secrets Manager and CloudTrail records the GetSecretValue call, so you can see which role read which secret when. Copy that value into GitHub repository secrets and you now need GitHub’s audit log too, plus the patience to line up timestamps. The more systems hold a copy, the less you know about what breaks when you rotate.

One correction worth making: the integration does exist. AWS publishes an action, aws-actions/aws-secretsmanager-get-secrets, that pulls secrets at job runtime and adds them as masked environment variables. The job still needs AWS credentials, and those don’t have to be permanent either: GitHub Actions can exchange its OIDC token for a role, so the repository holds no long-lived key at all. Terraform or the AWS CLI inside the pipeline does the same job.

Infrastructure state is the copy people forget. Marking a variable sensitive = true keeps the value out of CLI output, not out of the file: HashiCorp’s own docs say values marked sensitive are stored in state and plan files, and anyone who can read those can read the values. Encrypt the backend, restrict who can read it, and pass ARNs rather than values where a resource accepts them.

If your secrets manager can’t integrate with the systems that need its secrets, that’s a reason to replace the manager, not a reason to start copying.

When a secret lands in Git, revoke it

Copies get committed, which brings the question every team argues about for longer than the fix would take.

Is a secret in a private repository really leaked?

Treat it as leaked. Revoke it, replace it, move on.

You can have the “it depends” conversation about who has access to the repository today, who should have, whether it’s self-hosted, and whether contractors are in the org. It’s a legitimate conversation and it’s the wrong one to have while a valid API key is sitting in a commit that’s already in forty clones. If there’s the slightest doubt that someone else has seen it, revoke. When it’s a client’s key, tell them, and ask for a new one.

An exposed credential needs revoking whatever replacing it costs. The cost decides how fast that happens. Where rotation means a day of coordination across three teams, people negotiate with themselves and the key stays. Make generating and distributing a replacement easy, and automate the distribution where your tooling allows. Revocation still takes someone’s time. What shrinks is the delay and the scheduling that leave a live key sitting there in the meantime.

Stopping the commit is better than cleaning up after it. GitHub’s push protection blocks a push that contains a recognized secret, and since February 2024 it has been on by default for pushes to public repositories. That default lives on the user account, and any user can switch it off in their own settings. Enforcement is a separate control: turning push protection on for a repository, organization or enterprise is free for public repositories and part of the paid tier for private ones. Push protection stops the commit on its way out. Repository scanning finds what already landed.

Coverage reaches further than vendor token formats and is still not complete. GitHub’s secret scanning matches over 300 patterns and uses Copilot to detect generic passwords, so an unstructured credential is not automatically invisible to it, but one stored under a bland name like config_value can be missed. Add a scanner in pre-commit, keep local config in .gitignore, keep secrets out of logs, and keep a plan for the leak that none of them caught.

Keep production secrets in production

Prevention also means limiting how far a production credential can travel in the first place.

Should CI/CD hold permanent credentials?

No. Workers should get access scoped to the job, on a session short enough that expiry does the cleaning up.

The pressure is familiar. Someone needs production data to count how many users did a thing, the shortest path is a production password on a developer laptop, and once it’s there it stays. Keep production credentials inside production environments and that path closes. It also reduces the risk of a command meant for staging hitting production from a developer workstation, because that workstation no longer holds a credential that reaches production at all.

CI/CD is where the rule quietly gets dropped, because the CI tool is the one system that reaches everything. A Jenkins that can assume roles in every account is one compromise away from being the whole company’s problem, and it usually holds static credentials because that was faster to set up. Hand workers temporary credentials instead, keep the session duration short, and destroy the worker when the job ends.

Be precise about what expiry gives you. AWS temporary credentials stay usable until the interval they were issued for runs out, and a finished job does not shorten that interval, so a twelve-hour session on a three-minute build is a twelve-hour credential sitting in whatever the build left behind. Short sessions and disposable workers are what close the window, not the job exiting.

Time windows help on top of that. Roles that only grant deployment access during a service window mean a pipeline fired at the wrong time fails on permissions, which is the behavior you wanted.

Fetch the secret for one command, then let it go

Laptops are the last place a static copy hides, usually because someone needed a secret for a one-off script.

What’s wrong with a .env file?

It’s an unencrypted copy sitting on a disk, readable by anything running as you.

Wrap the command instead. A five-line Bash script that calls aws secretsmanager get-secret-value, exports the result into the environment of that one process and runs your tool keeps the value out of files on disk and leaves a GetSecretValue entry in CloudTrail. 1Password’s CLI does the same from a vault. op run resolves the secret references in your environment at runtime and hands the values to the subprocess. They are available for as long as that process runs, and no longer. If the subprocess prints one, op run conceals it in the output. The config file in your repo then holds an op:// reference instead of a password.

Be clear about what that buys. Fetching a static secret does not invalidate it, so the value is exactly as valid after the command exits as it was before. What you shortened is how long a copy sat on a laptop. A static secret ends when you rotate or revoke it, which is the argument for credentials that expire on their own.

Watch the command line itself. MySQL’s manual calls a password typed as mysql -u app -psecret convenient but insecure. Clients usually overwrite the argument with zeros while they initialize, which still leaves a brief interval where ps shows it to anyone on the box. On some systems the overwrite doesn’t take, the manual says, and the password stays visible for the whole run. It also lands in your shell history. The manual’s remedy there is a restrictive access mode on ~/.bash_history and the MySQL history file, which is precisely the permission nobody checks on a shared box. Its own preference: let the client prompt, or read the password from a protected option file.

The gap isn’t the tooling

We have managers that share by link, vaults that mint credentials on demand, scanners that block the push, and we’re still finding passwords in clear text in repositories.

Why do teams keep doing this when the tools exist?

Because the sanctioned path is slower than the shortcut, and because nobody showed the person who joined last month what the sanctioned path is.

Cognitive load decides this. Where getting a secret into the tool takes more steps than pasting it into a channel, the channel wins, and “I’ll fix it later” is the comment attached to the commit. Teams often do have a process; it lives in a document written two years ago that onboarding never mentions, so everyone improvises their own.

Ownership is the other half, and it bites hardest with external keys. The API key arrives by email to whoever owns the commercial relationship, a developer puts it in the code, a platform team stores it in the manager. When it has to be revoked at 2am, it isn’t clear whose job that is, so it doesn’t happen. We’ve seen a credential sit untouched for seven years and surface only because another team changed something near it. Name an owner per secret, and when one turns up in the open, revoke it the same day and say so publicly. That teaches more than a policy page.

What this means for teams

None of this needs a program or a new platform team. It needs a few decisions about where secrets live, who owns them, and how fast you can replace one, and then the discipline to stop making copies.

Four things worth doing this week:

  • Search your chat tool for fragments of passwords you know, plus key and token. For each hit, check who can read that channel and whether the credential still works, then revoke it and tell the team what you revoked and why.
  • Write down every place a secret lives today: password manager, Secrets Manager, Parameter Store, CI secrets, Terraform state, Docker build args, chat, tickets, laptops. Give each store a named owner.
  • Take the worst shared database password and replace it with per-identity access: RDS IAM authentication, a Vault role, or at minimum one account per person and per application.
  • Time one rotation end to end. If replacing a single third-party API key takes more than an afternoon, fix that before buying anything new.

Common questions, answered

How do I send a password to someone at another company?

Use your password manager’s share link rather than a messenger: 1Password and Bitwarden both produce a link that works for someone with no account of their own, with an expiry you set, and 1Password can restrict it to named email addresses. PGP is a reasonable alternative when both sides already hold keys. Either route ends with plaintext on the recipient’s screen, so agree before you send that it goes straight into their own manager instead of living in their inbox.

How do I stop secrets from reaching Git in the first place?

Turn on push protection, then check it is on where you think it is. GitHub enables it by default on user accounts for pushes to public repositories, and a user can switch that off for themselves, so enable it on the repository or organization as well: free for public repositories, part of the paid tier for private ones. Add a secret scanner to pre-commit so the block happens on the developer’s machine, keep local config files in .gitignore, and keep secrets out of application logs. Detection coverage is not complete, so none of this removes the need to revoke.

How do I find out where our secrets are already sprawled?

List the stores before you count the secrets: password manager, Secrets Manager, Parameter Store, CI/CD secrets, Terraform state, container images, chat history, tickets, wikis, laptops. For each one, answer who can read it, who owns it, and when it was last rotated. The last answer is usually “never”, and that’s the finding you act on.

Resources

Since we recorded

The numbers got worse rather than better. GitGuardian’s 2026 edition counts 28.65 million new secrets in public GitHub commits during 2025, a 34% rise, with leaks tied to AI services up 81% and commits written with a coding assistant leaking at roughly twice the baseline rate. The advice above didn’t change, but the consumer list did: coding agents now read your environment variables and config files, which we picked up in episode 89. The OpenTofu and Terraform comparison teased at the end of this one, state encryption included, became episode 83 with Cole Bittel.