The database password everyone has and nobody can change

One password for the production database, copied into twenty containers and a couple of laptops. Then the monitoring shows rows leaving that database at three in the morning, and all you have is a source IP address. You can’t say which workload it was, or whether it was a workload at all. Revoking the password isn’t an option either, because that takes down the nineteen containers doing legitimate work.

Both halves of that are credential design rather than bad luck. A shared static credential buys you no audit trail worth the name and no way to cut off one consumer without cutting off everyone. What you get instead is a rotation project that keeps slipping, and an incident that takes days to narrow down because the first question, who holds this password, has forty answers.

HashiCorp Vault is what we reach for here. We’ve run it since 2017, back before 1.0, and 1.5 shipped on 21 July 2020. Calling it a secrets manager undersells it, because storing static secrets is the small half of what it does. We’re not security researchers, we’re practitioners who use the tool and recommend it.

Dynamic credentials, and the audit trail that comes with them

Start with what you get for the trouble, because a place to park passwords isn’t it.

What does Vault do that a password manager doesn’t?

It creates the credential when you ask for it, hands it to one workload, and takes it back when the lease runs out.

Static storage is useful: a certificate from a CA like DigiCert lives in a key-value path. The interesting half is the database secrets engines. You ask for credentials, Vault logs into the database with its own privileged account, creates a user and a password, and returns them with a timer attached. Vault 1.5 supports Cassandra, Elasticsearch, MongoDB, MySQL, Oracle, PostgreSQL and Redshift, though Oracle’s plugin “is not bundled in the core Vault code tree”.

The timer is a lease. Vault’s lease documentation says that “When a lease is expired, Vault will automatically revoke that lease”, a revocation Vault schedules rather than one the database guarantees. Expiry stops the next login with that credential. Whether an open session dies with it is up to the database: MySQL documents that DROP USER “does not automatically close any open user sessions”, and that the drop “does not take effect until that user’s session is closed”. Test both before you call a TTL containment.

Per-workload credentials change incident response. Twenty replicas get twenty sets of credentials, so “which one of these is exfiltrating data” becomes a question with an answer, and killing that lease doesn’t touch the other nineteen.

Compare that to RDS, where identity, lifetime and audit trail come apart. IAM database authentication covers the first two: tokens last 15 minutes, and the policy’s resource ARN names one database account, so each principal can be pinned to its own database user.

The trail splits in two. The database does record the login, and AWS’s December 2019 walkthrough of IAM authentication says how: set log_connections to 1 in the parameter group and PostgreSQL writes connection authorized: user=pgmonitor database=pgbench SSL enabled for each one. That line names the database user, not the IAM principal, and AWS is explicit that “CloudWatch and CloudTrail don’t log IAM authentication”. We asked AWS support and were told they’re working on it, no timeline. A database secrets engine ties the two together for you, one user per lease, and Vault’s audit log records who asked.

One caveat. If you’ve picked one cloud for good, most of this has a native equivalent: STS for temporary credentials, Secrets Manager rotation for the rest. Hybrid or on-premise is where that stops.

Run Vault separately from application workloads

Where you put it decides how much that audit trail is worth.

Can Vault live in the same Kubernetes cluster as the applications?

It can, and we’d still put it somewhere else, because an application compromise in that cluster leaves an attacker one hop from every secret you own.

Separation here is about the infrastructure rather than the network path. Your applications go on calling the Vault API, authenticating and reading what their policy allows, and that traffic is the whole point of running Vault. What you take away is the shared host, the shared cluster and the shared account, so a compromised application is left with the API and has to say who it is like everyone else.

A separate cluster is the straightforward answer, and when that’s too much cost or hassle, ECS or plain virtual machines do. The production hardening guide goes further than we would: “Vault should be the only main process running on a machine”, with bare metal preferred to a VM and a VM preferred to a container.

A separate AWS account is the version we like, with the API reached over VPC peering. The payoff is that the account gets quiet enough to alarm on. Nothing but Vault belongs in there, so an EC2 instance starting up there is a high severity event rather than something somebody might have had a reason to do.

Storage is the other placement decision. DynamoDB has been fine for us, and the docs are clear that it’s community supported, with high availability, with a warning that clock skew across Vault nodes causes contention on the locks. Consul is the one HashiCorp supports itself. S3 says “the S3 storage backend does not support high availability”, which reads worse than it is: that line is about the backend coordinating leader election rather than about your deployment’s availability. Vault 1.5 made it possible “to use Integrated Storage as HA Storage by using a different storage backend as regular storage”, so a backend with no leader election of its own can borrow Raft for it. Integrated Storage holding the data as well went generally available in Vault 1.4 on 7 April 2020 and removes a whole system from your diagram. It also moves the backup problem onto you: lose all three machines at once and you’ve deleted your secrets.

Unsealing without three people in a room

Wherever it runs, Vault starts sealed and can’t read its own storage until you fix that.

Do you still need a quorum of humans to start Vault?

Not on a cloud with a key management service: vault operator init splits the key into five shares needing three by default, and auto unseal hands that job to a KMS key instead.

The chain of keys is worth getting right, because the shares aren’t the master key. Vault’s seal documentation describes shards reconstructing the unseal key, which decrypts the master key, which protects the keyring that encrypts your data. Auto unseal “delegates the responsibility of securing the unseal key from users to a trusted device or service”, and at startup Vault asks that service to “decrypt the master key Vault read from storage”. So the master key doesn’t go away, it just gets protected by something other than people.

That mode isn’t a paid feature. HashiCorp open sourced Cloud Auto Unseal in Vault 1.0 in December 2018, for AWS KMS, Azure Key Vault and GCP CKMS. Configuring it on AWS is a seal "awskms" stanza with a kms_key_id, and Vault needs kms:Encrypt, kms:Decrypt and kms:DescribeKey on that key.

Use a customer managed key rather than an AWS managed one. With your own key you write a key policy that lets Vault use it and nobody else, which is the actual control here.

Then comes the question everyone asks: what if the key goes away? We’ve never heard of AWS losing one, and we haven’t gone looking for cases either. If that risk keeps you up, importing your own key material is what AWS offers, and it is worth being exact about what it buys. You keep the original copy while AWS KMS uses a copy of it, so material that expires or that you delete can be imported into the same KMS key again. It does nothing for the KMS key itself: scheduling that for deletion “requires a waiting period of 7 to 30 days, after which you cannot recover the deleted KMS key”, imported material or not. The material gets protected by somebody either way. Choosing KMS is choosing who.

Use the root token once, then revoke it

Unsealed and running, Vault hands you exactly one credential, and it’s the most dangerous one in the building.

What do you do with the root token after you initialise Vault?

Set up the auth methods and policies with it, then revoke it.

The root policy “can not be modified or removed” and its holder can do anything, which is why Vault’s own policy documentation says “it is highly recommended that you revoke any root tokens before running Vault in production”. HashiCorp’s hardening guide puts the sequence plainly: use it to set up auth methods, treat Vault configuration as code with policies in version control, and “once setup, the root token should be revoked to eliminate the risk of exposure”.

Getting one back is deliberately awkward, which is the feature. vault operator generate-root starts with -init and one of two output modes, and they are not interchangeable. Pass -otp and “The resulting token is XORed with this value when it is returned”, so you run generate-root -decode with that same OTP to read it. Pass -pgp-key instead and the token “will be encrypted and base64-encoded with the given public key”, which you open with the matching private key; -decode has no part in that one. Either way, a quorum of key holders then submit their shares with the -nonce value from the init step. Several commands and several people, for something that should happen roughly never.

Which shares, though, depends on how you seal. On a Shamir seal they’re the unseal keys from vault operator init. Under auto unseal there are none, because initialisation produces recovery keys instead: the seal documentation says “a set of Shamir keys are generated that are called recovery keys”, for the operations that “still require a quorum of users”, generating a root token among them. They also authorise rekeying, and the unseal API asks for them after a manual seal. What they are not is a spare KMS key, since the seal is still the thing that decrypts the master key at startup. Store them accordingly, and expect to use them about once.

One path per application, one policy per path

Everything after the root token runs on two objects: a token and a policy.

How do you stop one application reading another application’s secrets?

Give each application its own path, and a policy that allows reading that path and nothing else.

Vault is path based all the way down, and policies are deny by default, so “an empty policy grants no permission in the system”. The capabilities you attach to a path are create, read, update, delete, list, sudo and deny. A key-value path per application, secret/web1 for the web1 service, with a policy granting read on it, is the pattern. On version 2 of the key-value engine the policy path is secret/data/web1 rather than secret/web1, which is the mistake everyone makes once. Listing lives on secret/metadata/, and the docs warn that listed key names aren’t filtered by policy.

Tokens come from auth methods, one per kind of consumer. Developers authenticate through LDAP and get policies based on their group. Workloads use the Kubernetes auth method: the pod presents its service account token, Vault “accesses the Kubernetes TokenReview API to validate the provided JWT is still valid”, and returns a Vault token with the role’s policies attached.

Here the two of us don’t agree. Mattias wants a write-only policy for developers, create and update without read, so somebody can add a third party API key without browsing everyone else’s. Andrey’s objection is that without read you can’t check what’s there, so you overwrite the path every time instead of looking first. Read-only for applications is the half nobody argues about.

Generate the policies rather than writing them. A Terraform module that creates the path and the policy together, for every application, keeps the shape identical and the diff readable. The same trick works for people: a script that hands a developer temporary cloud credentials in the morning, revoked eight hours later, and a wider set of policies on request when somebody needs write access for an hour.

Hand the secret to the process, not to the container

Policies decide what an application may read. Something still has to do the reading.

How does a container get its secrets without building Vault into the application?

Put a helper in front of your program: it authenticates, reads the paths, and starts the program with the values in its environment.

Linking the Vault SDK into the application costs more than it looks: it then needs Vault anywhere it runs, a laptop and an end-to-end environment included. envconsul takes the other route. It launches “a subprocess with environment variables populated from HashiCorp Consul and Vault”, renews the token at half its lease duration with renew_token on by default, and restarts the child when values change.

The subprocess part is a security property, not a detail: those variables belong to one process rather than to the container. /proc/<pid>/environ is governed by a PTRACE_MODE_READ_FSCREDS check that turns on the reader’s own credentials: the target’s real, effective and saved user IDs matching the reader’s, or CAP_SYS_PTRACE. A shell running as the application’s own user reads that environment, no escalation needed. The subprocess model helps against an intruder who lands as a different unprivileged user, who does need one first.

Our scar tissue is on the token handover. We ran Vault Agent as a sidecar, writing the token to a shared volume. The agent lost connectivity, authenticated again later and got a fresh token, and envconsul never learned about it. The leases from the old token went with it, because “when a token is revoked, Vault will revoke all leases that were created using that token”, so the application ran on credentials that no longer worked. An init container made the handover deterministic, and a side process watched for those errors and killed the container. Developers don’t expect a password to change underneath them, so that handling is rarely there.

Two things save work. HashiCorp’s agent injector, a mutating admission webhook from December 2019, adds the init container and the agent sidecar to any pod with the right annotations. We built our own first; no reason you should. On EKS, IAM roles for service accounts have given each pod its own IAM role since September 2019, so you may not need Vault for AWS credentials. EKS hosts the OIDC issuer and runs the pod identity webhook; the IAM OIDC provider and the role’s trust policy stay your job. AWS says it “is also available in a DIY Kubernetes setup on AWS”, kops included, with both of those on you. Nomad has no equivalent.

Lock the box and stream the audit log

The last question is the one that arrives after everything works: how do you keep people out of the thing that holds all your secrets?

What do you monitor on a Vault server?

Its own telemetry and its audit log, both shipped off the box, so nobody has a reason to log in there.

Start by closing it down. The hardening guide says to disable SSH and remote desktop outright; if you keep SSH for emergencies, lock the security group so reaching the box takes a privileged change first. That change lands in CloudTrail, so someone opening port 22 to the world becomes an alert with a name on it rather than a quarterly review finding.

Then make sure nobody needs to go there. Vault’s telemetry exposes Prometheus metrics once you set prometheus_retention_time. The scrape has two sharp edges. The endpoint is /v1/sys/metrics and the format goes in the job’s params, because “?format=prometheus in the path won’t work”. The policy path is sys/metrics with read, since ACL paths drop the /v1/ prefix that metrics_path needs. Only active nodes serve the endpoint, so standbys look dead unless you enable unauthenticated metrics access.

Auditing is the one you can’t skip, and it isn’t on when you start. “When a Vault server is first initialized, no auditing is enabled”, and the audit device documentation explains what you’re switching on: every authenticated interaction, errors included, as one JSON object per line, with strings HMAC-SHA256 hashed under a salt. Ship it somewhere the Vault operators can’t edit, and restrict who reads it, because hashing isn’t the same as redaction.

Vault 1.5 added rate limit quotas to the open source build in July 2020, with lease count quotas in Enterprise, which caps a runaway client. For the rest, start from the reference architecture and the hardening guide, take the Helm chart and the Terraform modules, and expect to adjust them: they can’t know your context, so they assume one.

What this means for teams

Vault pays off in the moment you have to answer who held this credential and then take it away from them.

This week:

• Pick one service and one database and move it to dynamic credentials. Let a lease expire in a test environment and check both halves: that a fresh login with those credentials fails, and what happens to the connection that was already open.
• Find out who holds the root token. If anyone does, use it to finish configuring your auth methods and policies, then revoke it and write down how to run vault operator generate-root if you need another.
• Enable at least two audit devices and send them somewhere the people who run Vault can’t edit.
• If Vault runs in the same cluster as your applications, price a separate cluster or account, and put that number next to what those secrets are worth.

Common questions, answered

How do I find out who connected to my RDS database?

Turn on the database’s own connection logging, then give each workload its own database user. On RDS PostgreSQL that is log_connections set to 1 in the parameter group, which AWS’s December 2019 IAM authentication walkthrough recommends; a login shows up as connection authorized: user=pgmonitor database=pgbench SSL enabled. That names the database user, and AWS documents that CloudWatch and CloudTrail don’t log IAM authentication, so nothing maps it back to an IAM principal unless you keep the two in step. A database secrets engine does that for you, one user per lease, and Vault’s audit log records who asked.

Which storage backend should I pick for Vault on AWS?

Integrated Storage, unless you have a reason not to: generally available since Vault 1.4 in April 2020, supported by HashiCorp, one fewer system to run. The cost is that backups become your job, and losing the nodes together means losing the data. DynamoDB is the alternative we’ve run for years, community supported with high availability, and sensitive to clock skew because the locks use each node’s clock. S3 is documented as not supporting high availability, which is about the backend’s leader election: since Vault 1.5 you can pair it with Integrated Storage as HA storage and still run a cluster.

I revoked the root token. How do I get a new one?

Run vault operator generate-root -init with either -otp or -pgp-key, then have a quorum submit their shares with the -nonce from that step: unseal keys on a Shamir seal, recovery keys if you auto unseal. With -otp the token comes back XORed with your OTP, so generate-root -decode and that OTP reveal it. With -pgp-key it comes back encrypted to that key, and the matching private key opens it. Revoke the new token once the task that needed it is done.

Related episodes

• What Are We Working On: keys in Git repositories, and a Vault installation that got torn out again over who should have access to what.
• AWS Security Maturity Roadmap: where dynamic credentials sit on the path from root account keys to short-lived access, with Vault’s AWS secrets engine as one route.
• Docker Secure Build: why secrets don’t belong in an image, and what you do at container startup instead.

Resources

• Vault production hardening guide: single tenancy, firewalling, disabling SSH, revoking root tokens and enabling auditing, in HashiCorp’s order of priority.
• Vault reference architecture: the deployment shapes HashiCorp recommends, down to what belongs in each availability zone.
• Seal/Unseal: the key chain from unseal shards to the master key to the keyring, and what recovery keys are for once auto unseal is on.
• Audit devices: what the audit log contains, how sensitive strings are hashed, and why one blocked device stops Vault answering requests.
• Securing Amazon RDS and Aurora PostgreSQL database access with IAM authentication: AWS’s December 2019 walkthrough, and the one that settles how to log IAM logins on the database side with log_connections.
• Introducing fine-grained IAM roles for service accounts: settles what EKS provides for pod-level IAM roles and what you create yourself, the IAM OIDC provider included.
• envconsul: the subprocess model, the token renewal defaults and what happens when rendered values change.
• Vault Configuration as Code via Terraform: Stories From the Trenches: Andrey’s talk from HashiConf Digital, which ran 22 to 24 June 2020, on managing auth methods and policies as code.

Since we recorded

HashiCorp moved Vault from MPL 2.0 to the Business Source License in August 2023, and IBM completed its acquisition of the company in February 2025. The fork, OpenBao, now sits under the Linux Foundation’s OpenSSF as “an open source, community-driven secrets manager and fork of Vault”. What hasn’t closed is the RDS gap that pushed us toward dynamic credentials: AWS still documents that CloudTrail doesn’t log IAM database authentication. We worked through the licence move in HashiCorp’s BSL Move and OpenTF, and went back over the whole secrets question in Keeping Secrets Safe.