Day 4 - Secrets Manager and Parameter Store: Lifecycle and Rotation

Hardcoding a single password into code is easy. One line, DB_PASSWORD = "supersecret", and it works. But the moment that one line stays in Git history, gets cloned by dozens of developers, is printed in CI logs, and is replicated onto backup disks, that password is no longer a secret. Moreover, once a password is set, it stays the same forever. If someone has seen it even once, the password stays alive even after that person leaves the company. The entire field of secrets management exists to deal with these two problems: "where do we store it safely?" and "how do we change it periodically?"

Today we look at AWS's two secret stores, Parameter Store and Secrets Manager. Not merely "which of the two do I use," but why secrets need to be rotated, how a password is changed without downtime during rotation, how the two labels AWSCURRENT/AWSPENDING guarantee an atomic swap, and why the fundamental KMS design of envelope encryption becomes the key to cross-account sharing. Secrets management appears on the DOP exam almost every time, and typical scenarios include "zero-downtime rotation," "cross-account secrets," "cost-optimized choice," and "the rotation failed."

Why Rotate — The Secret Lifecycle and the Window of Exposure

First, the fundamental question: why should a password be changed periodically? The answer is to shrink the window of exposure. For any password, exposure risk accumulates as time passes. A developer may have shown it on screen where someone saw it, it may have been printed once in a log, or it may be inside a leaked backup. If you change a password every 30 days, then even if it leaks today, its value disappears after 30 days at the latest. Rotation is not a defense that "prevents a leak from happening" but one that "limits the period of damage even if a leak happens."

This is part of the larger security topic of credential lifecycle management. Interestingly, NIST SP 800-63B (Digital Identity Guidelines) actually discourages forced periodic changes of passwords that humans memorize (because when people are forced to change periodically, they switch to weak patterns like Password1, Password2). But the same document strongly recommends periodic rotation of system and machine credentials. A machine does not create weak patterns, and generating a fresh random 64-character password every time costs nothing. The rotation target of Secrets Manager is exactly this machine credential. Since no human needs to memorize it, you can make it as long and random as you like, and change it often.

The effect of rotation is explained, from an information theory perspective, as "a decay in the temporal value of the information the attacker obtained." It is the same intuition as why you must not encrypt too much data with the same key in cryptography (the key reuse limit). Because the exposure risk of keys and credentials grows in proportion to usage and time, they are replaced periodically. A TLS session key being negotiated anew on every connection, Kerberos tickets having a short lifetime, and AWS STS temporary credentials defaulting to 1 hour all follow the same principle — the short-lived credential philosophy that the shorter the lifetime, the less useful it is even if stolen.

A more radical approach, in contrast to rotation, is "eliminating the password itself." RDS IAM authentication is an example. Instead of a password, an IAM Role with the rds-db:connect permission obtains a 15-minute authentication token on the spot and connects with it. Since there is no password to store, there is nothing to rotate and nothing to leak. This is the ideal of a secret-less architecture. There are limits, however — token generation carries IAM call overhead, which is a burden for large workloads that open thousands of new connections per second, and some engines/tools do not support the token method. So in reality, "rotated passwords (Secrets Manager)" and "passwordless IAM authentication" coexist depending on workload characteristics. If "we don't want to store a password at all" appears on the exam, IAM authentication is the answer; if it is "keep the existing password approach but rotate it safely," it is Secrets Manager.

Two Stores — Parameter Store and Secrets Manager, Similar but Different

AWS has two places to put secrets, and this is confusing at first. Both encrypt with KMS, both control access with IAM, and both manage versions. But the two have different origins. Parameter Store started out as a "configuration store." It was meant for putting non-secret settings such as DB hostnames, log levels, and feature flags under hierarchical paths, and was extended with the SecureString type so it can hold secrets too. Secrets Manager was "secrets-only" from the start, with automatic rotation at its center. The built-in rotation engine is the biggest difference between the two services.

The key decision criterion is simple. If you need rotation, use Secrets Manager; otherwise, use Parameter Store. And the cost differs by nearly 10x ($0.40 vs free). So the practical pattern is a mixed setup: "100 simple settings in Parameter Store Standard for free, and only the 5 passwords that need rotation in Secrets Manager." Putting everything into Secrets Manager costs 105 × $0.40 = $42 per month, but mixing ends at 5 × $0.40 = $2.

⚠️ Pitfall: The thought "it's a secret, so always Secrets Manager" is the most common cost pitfall. A static secret that is not rotated (for example, a third-party API key with no rotation policy) is safe enough with KMS encryption and free even when put in a Parameter Store SecureString. Conversely, "putting even rotating passwords in Parameter Store to save cost" is also a pitfall — Parameter Store has no rotation engine, so you must build and schedule the rotation Lambda by hand, and you end up reimplementing expensively what Secrets Manager gives you for free. The axis of judgment is not "is it a secret or not" but "does it need rotation or not."

The distinction between the two stores is where separation of concerns in software architecture meets the "Config" principle of the 12-Factor App. 12-Factor says "separate code from configuration, and inject configuration from the environment." But among configuration, secrets (credentials) have fundamentally different security requirements from ordinary settings — encryption, rotation, auditing, least privilege. So mature systems physically divide an "ordinary configuration store (Parameter Store)" from a "secret store (Secrets Manager/Vault)." HashiCorp Vault, Azure Key Vault, and GCP Secret Manager all implement the same separation.

AWSCURRENT and AWSPENDING — The Secret of Atomic Swap

To understand the heart of Secrets Manager rotation, you need to know three staging labels: AWSCURRENT, AWSPENDING, and AWSPREVIOUS. How these labels make a "no-downtime swap" possible is the real substance of rotation.

Changing a password is inherently risky. If the password changes in the very instant an application reads it and connects to the DB, it fails trying to connect with the old password. If you do it simply in the order "change the DB password → change the store's value," all connections break in the short gap between the two operations. Secrets Manager solves this problem with indirection by attaching labels to versions.

The application always reads the version that the AWSCURRENT label points to. Rotation proceeds like this.

The 4 rotation steps (standard implementation of the RDS rotation Lambda)
==================================================
1. createSecret  : Create a new random password and store a new version with the AWSPENDING label
                   (at this point AWSCURRENT is still the old password)
2. setSecret     : Connect to the DB and change the actual password to the AWSPENDING value
3. testSecret    : Test a DB connection with the AWSPENDING password (verification)
4. finishSecret  : Move the AWSPENDING label to AWSCURRENT (atomic)
                   → the old version automatically becomes AWSPREVIOUS

Step 4, finishSecret, is the magic. The act of moving the label is atomic. When the application calls GetSecretValue, it gets the old value if the label hasn't moved yet and the new value after it has. There is no broken state in between. And the old password pushed to AWSPREVIOUS stays alive for one rotation cycle, leaving room for connections that read and cached the password just before rotation to keep working a little longer.

Indirection through labels is a pattern all too familiar in computer science. The way transactions see a consistent snapshot in database MVCC (multi-version concurrency control), Git's branch pointers (a branch is a moving label that points to a commit), and the way a router switches the environment it points to all at once in Blue/Green deployment — all share the same idea of "leave the actual objects alone and move only the pointer atomically." David Wheeler's maxim "all problems in computer science can be solved by another level of indirection" applies exactly here. AWSCURRENT is one level of indirection layered over the password.

Single User vs Multi User — How to Eliminate the Gap During Rotation

The four steps above have one subtle gap. From the moment the actual DB password is changed in step 2, setSecret, until the label moves in step 4, finishSecret, the DB's actual password (the new one) and the value AWSCURRENT points to (the old one) are briefly inconsistent. If a new connection is attempted with the old password in this short window, it can fail. Single User rotation accepts this gap. In environments where rotation is infrequent and the connection pool caches passwords for a long time, it is practically almost never a problem, but the theoretical gap exists.

Multi User (Alternating Users) rotation eliminates this gap at the source. The idea is simple and elegant. You create two users with equivalent privileges in the DB (alpha, beta), and at each rotation you change the password of the one not currently in use and then switch to it.

Alternating Users rotation
==================================================
Current AWSCURRENT = alpha (the app is connected as alpha)
Rotation occurs:
  1. Change the password of beta (currently unused) + beta credentials in AWSPENDING
  2. Test connection as beta
  3. finishSecret → AWSCURRENT = beta
  → alpha is still valid with the old password (untouched)
Next rotation:
  → this time change alpha and switch to alpha

The key is that the password of the user currently used by traffic (alpha) is never touched. Throughout rotation, every connection made as alpha stays intact, and new connections move gradually to beta. It is true zero-downtime. The price is a bit of complexity: operating two users in the DB and keeping their permissions synchronized.

AWS provides standard rotation Lambda templates for RDS PostgreSQL/MySQL/MariaDB, Redshift, and DocumentDB (SecretsManagerRDSPostgreSQLRotationSingleUser, ...MultiUser, and so on). You don't need to implement the four steps yourself; you just attach one of these templates.

aws secretsmanager rotate-secret \
  --secret-id prod/myapp-rds \
  --rotation-lambda-arn arn:aws:lambda:...:function:SecretsManagerRDSPostgreSQLRotationMultiUser \
  --rotation-rules AutomaticallyAfterDays=30

🎯 Scenario: "Rotate the RDS password of a payment service every 30 days, but not a single transaction may fail even at the moment of rotation. At peak, thousands of new connections open per second." — The answer is Multi User (Alternating) rotation. In an environment where new connections open at high frequency, even Single User's short inconsistent window can lead to dozens of connection failures. Multi User does not touch the password of the user currently in use throughout rotation, so neither in-flight nor new connections are affected. With Single User, a symptom of "some connections fail right after rotation" would appear.

The most common failure of rotation automation is the rotation Lambda's network disconnection. If RDS is in a private subnet but the rotation Lambda is placed outside the VPC (the default), the Lambda stalls at step 2 of the 4, setSecret (DB connection). Then AWSPENDING has been created with the new password but the DB still has the old password and the label hasn't moved, so the rotation fails in an awkward state. If a human doesn't intervene manually to clean up AWSPENDING in this state, the next rotation gets tangled too. The lesson is clear — if the DB is private, put the rotation Lambda in a private subnet of the same VPC as well, and make sure there is a path to the Secrets Manager endpoint (a VPC endpoint or NAT).

RDS Integration — Entrusting the Password Entirely to AWS

If even attaching a rotation Lambda is a hassle, you can let RDS manage its own password. A single --manage-master-user-password flag is enough.

aws rds create-db-instance \
  --db-instance-identifier prod-db \
  --engine postgres \
  --manage-master-user-password \
  --master-user-secret-kms-key-id alias/rds-secrets

With this flag on, RDS (1) automatically creates a master password secret in Secrets Manager, (2) automatically rotates it every 7 days by default, and (3) configures the necessary IAM Role and rotation Lambda by itself. The operator never sees the password even once and never builds a rotation Lambda. Because it is a native integration between RDS and Secrets Manager, RDS also manages the rotation gap internally. It is the cleanest pattern of "handling the password of a managed database in a managed way," and if "rotate the RDS master password with the least operational burden" appears on the exam, this is the answer.

KMS and Envelope Encryption — The Foundation of Cross-Account Sharing

Now we move to the trickiest topic, cross-account secret sharing. To understand it properly, you first need to know the fundamental KMS design, envelope encryption.

A KMS master key (CMK) in fact does not encrypt data directly. Instead it works like this. When storing a secret, KMS creates a one-time data key and encrypts the secret with that data key. Then it encrypts the data key itself with the master key and stores it attached next to the encrypted secret. The secret (data) sits inside an "envelope" called the data key, and that envelope is sealed again with the master key, hence envelope encryption.

Envelope Encryption
==================================================
Store:  secret ──[encrypt with data key]──> ciphertext
        data key ──[encrypt with master key (CMK)]──> encrypted data key
        What is stored = ciphertext + encrypted data key (the master key stays only inside KMS)

Decrypt: encrypted data key ──[send to KMS, decrypt with master key]──> plaintext data key
         plaintext data key ──[decrypt ciphertext locally]──> secret

Why does this design matter? First, it is fast and cheap because large data is not round-tripped through KMS directly (KMS handles only up to 4KB directly). Second, the master key never leaves the KMS hardware security module (HSM). Third, and this is the key to cross-account, to decrypt the data you must always go to KMS and unwrap the data key with the master key. That is, "the permission to read the secret" and "the permission to use the encryption key" are separated.

Envelope encryption is the same idea as hybrid encryption in PGP/GPG. PGP also encrypts the message body with a symmetric session key and encrypts only that session key with the recipient's public key before sending them together. Symmetric ciphers are fast but key distribution is hard, while asymmetric ciphers make key distribution easy but are slow — hybrid/envelope encryption takes only the strengths of both. The TLS handshake has the same structure (exchange a symmetric key using asymmetric cryptography, then communicate symmetrically). KMS turns this classic pattern into a managed service.

Now it becomes self-evident why cross-account requires two layers of permissions. For account B to read a secret in account A, B needs both (1) permission to fetch the secret and (2) KMS permission to unwrap the data key that encrypted that secret. So there are three places to configure.

// Owner account (A) — the secret's Resource Policy: "B may fetch this secret"
{
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::CONSUMER-ACCT:role/AppRole"},
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "*"
}

// Owner account (A) — KMS CMK Key Policy: "B may decrypt with this key"
{
  "Sid": "AllowConsumer",
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::CONSUMER-ACCT:role/AppRole"},
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

// Consumer account (B) — IAM Policy: "my Role fetches and decrypts A's secret"
{
  "Effect": "Allow",
  "Action": ["secretsmanager:GetSecretValue", "kms:Decrypt"],
  "Resource": ["arn:aws:secretsmanager:...:OWNER:secret:shared/api-key-*",
               "arn:aws:kms:...:OWNER:key/CMK-ID"]
}

⚠️ Pitfall: By far the most common cause of failure in cross-account sharing is a secret encrypted with the AWS managed key (alias/aws/secretsmanager). You cannot modify the Key Policy of a managed key, so there is no way to allow kms:Decrypt to another account. So access to the secret is allowed by the Resource Policy but blocked at the decryption step, producing "AccessDenied (KMS)". There is only one fix — you must encrypt the secret with a customer managed key (CMK) from the start. If you already created it with the managed key, you must re-encrypt it with a CMK or create the secret anew. If "cross-account but a KMS error" appears on the exam, nine times out of ten it is this managed key problem.

KMS permissions have, besides the Key Policy and IAM Policy, a third mechanism called a Grant. A Grant gives a specific principal temporary, fine-grained permission to use a key, and lets you give and revoke permission at runtime without touching the Policy. Secrets Manager sometimes uses Grants internally when giving a rotation Lambda permission to use a key. Think of the Key Policy as the constitution of "who can use this key," the IAM Policy as the civil law of "which keys can my principal use," and the Grant as a temporary delegation that is "lent out briefly at runtime." The most confusing point is that a KMS key is usable only when the Key Policy and IAM Policy both allow it — one side alone is not enough.

Cross-Region Replication and Caching — Wrapping Up Availability and Cost

If a secret exists in only one Region, it is a problem when that Region fails or when a multi-Region app has to read the secret in another Region. Secrets Manager replicates secrets to other Regions.

aws secretsmanager replicate-secret-to-regions \
  --secret-id prod/myapp \
  --add-replica-regions Region=us-east-1,KmsKeyId=alias/secrets-use1

A replica is updated when the original is rotated, and is re-encrypted with each Region's KMS key. It enables local reads per Region in DR scenarios and global apps.

Finally, a wrap-up on cost and performance. If a Lambda calls GetSecretValue through the SDK on every invocation, API call cost and throttle risk pile up. The AWS Parameters and Secrets Lambda Extension solves this. When you attach the Extension as a layer, a local cache proxy (localhost:2773) starts inside the function instance, and only the first call hits the real API while the rest are served from the cache.

# Inside Lambda: the first call hits the API, then responses come from local cache for the TTL
curl "http://localhost:2773/secretsmanager/get?secretId=prod/db"

While the same function instance is reused (warm), it reads the secret from the in-memory cache, greatly reducing API calls and cost. However, this reduces call frequency and cost, not cold starts — if anything, the Extension layer itself adds a little initialization time to the cold start.

It is common for a team to call GetSecretValue directly every time from a Lambda invoked thousands of times per second, run into Secrets Manager's API throttle (the per-account, per-Region calls-per-second limit), and see the function fail intermittently with ThrottlingException. The secret value changes once every 30 days, yet reading it thousands of times per second is plainly wasteful. If you add a TTL-based cache with the Lambda Extension (or an SDK caching library), actual API calls drop to once per TTL per function instance, the throttling disappears, and cost falls sharply. Exploiting the property that "a secret is read often but rarely changes" with a cache is the standard approach.

Summary

Today's picture has four parts. First, rotation is a defense that shrinks the window of exposure and is especially suited to machine credentials — more radically, there is also IAM authentication, which eliminates the password itself. Second, the indirection of the AWSCURRENT/PENDING/PREVIOUS labels creates an atomic, no-downtime swap, and Multi User (Alternating) rotation achieves true zero-downtime by not touching the current user. Third, because of envelope encryption, decrypting a secret always goes through KMS, and so cross-account requires two layers, secret permission + KMS permission, and is impossible with a managed key (a CMK is mandatory). Fourth, mixed storage (free Parameter Store + rotating Secrets Manager) and Lambda Extension caching wrap up cost and performance.

In the next post we review the whole of Week 9 with scenarios. We'll see how SSM, State Manager, AppConfig, and secret rotation interlock into a single picture in real operating situations.

📌 Today's Summary

Secrets Manager was designed around rotation capability and label-based version management. The AWSCURRENT/AWSPENDING duality enables zero-downtime replacement. Parameter Store is cheap and simple but has no rotation and weak secret management features. At the scale of 1000+ secrets, the automatic rotation of Secrets Manager shows its cost-effectiveness for audit compliance (SOC 2, PCI).