Day 2 - Secrets Manager, Parameter Store, and CloudHSM: Three Tools for Handling Secrets and Configuration

Look at the statistics on software security incidents and you'll almost never find a case where "the encryption algorithm was broken." What you will find, every single week somewhere, is "a password / API key / DB credential got committed to a code repo in plaintext." By GitHub's own numbers, more than 12.5 million secrets were leaked to public repos over 2023, and 91% of them were automatically harvested by bots within 24 hours. In other words, the realistic assumption is that a secret is "stolen the instant it's exposed."

The cloud-era answer to this problem is a central secrets manager. Don't bake secrets into code, environment variables, or config files — store them in a dedicated service and have the application fetch them at startup using IAM permissions. That way the secret never lingers in plaintext anywhere in your code repo, CI logs, or container images, and rotation, auditing, and access control all become possible from one place. AWS split this problem across two services — Secrets Manager (launched April 2018, automatic rotation and highly available secret management) and Systems Manager Parameter Store (launched 2016, free storage for configuration and plaintext parameters). And beneath those sits CloudHSM, which satisfies the strictest compliance demands. In this article we'll look at what problem each of the three services answers, how to choose between them, and the scenarios you'll run into most often.

The Core Value of Secrets Manager: Automatic Rotation and RDS Integration

Secrets Manager's biggest differentiator is automatic rotation. If all you need is "encrypt a secret with KMS and store it," Parameter Store SecureString is enough. But the requirement "automatically change the RDS password every 30 days, and at that moment have the application seamlessly pick up and use the new password" is something only Secrets Manager handles cleanly.

Automatic rotation works internally by triggering a Lambda function. AWS provides pre-built rotation Lambda templates for major databases like RDS / Aurora / Redshift / DocumentDB, and once the user just clicks "enable rotation," a 4-step lifecycle runs automatically.

[ Secrets Manager rotation: 4 steps ]

1) createSecret
   ├─ Generate a new password (random)
   ├─ Store it as the AWSPENDING version
   │  (AWSCURRENT is still the old password)

2) setSecret
   ├─ Apply the new password to the DB via the RDS API
   │  (change the RDS user's password)
   ├─ At this point the DB accepts both old + new passwords (brief window)

3) testSecret
   ├─ Test-connect to the DB with the new password
   │  Roll back on failure

4) finishSecret
   ├─ Promote AWSPENDING → AWSCURRENT
   ├─ Old password → AWSPREVIOUS (for rollback)

The key insight of this lifecycle is that AWSCURRENT does not switch immediately — it's only promoted after testSecret succeeds. If the new password is generated but applying it to the DB fails, AWSCURRENT stays on the old password, so the application doesn't grind to a halt. And because AWSPREVIOUS stays alive for a period, a client that happened to be holding an active connection at rotation time keeps working with the old password until its next reconnect. This is exactly why "zero-downtime rotation" is possible.

💡 Related theory: Secrets Manager's rotation model is called the dual-credential pattern, or blue-green credentials. You open a short window where the database or service accepts two credentials simultaneously, and during that window you gradually cut over to the new credential. This works because an RDS user can essentially change its password with ALTER USER, and existing connections aren't dropped when the password changes. You'll see the same pattern in Kubernetes Secrets + cert-manager, HashiCorp Vault's dynamic credentials, and elsewhere.

🔍 Going deeper: The RDS Proxy + Secrets Manager combination makes automatic rotation one step safer. RDS Proxy sits between the client and RDS doing connection pooling, and it can pull the password directly from Secrets Manager and operate with different credentials on the client side and the DB side. So even when rotation happens, the client only needs to know RDS Proxy's credentials, and RDS Proxy absorbs the backend password change. In environments like Aurora Serverless v2 where the connection count fluctuates frequently, this is a near-mandatory pattern.

📚 Case study: In 2020 a mobile gaming company switched its operational procedure of manually rotating the Aurora MySQL password every 90 days over to Secrets Manager automatic rotation. Before that, at every rotation an SRE would push the new password into a ConfigMap and redeploy the Pods, and a single failure caused a large-scale outage. After adopting Secrets Manager the rotation procedure was fully automated, and on a rotation failure it auto-rolled back at the testSecret step, so operational incidents dropped to zero for 18 months. The one troubleshooting headache they hit was "the rotation Lambda was in a different VPC from RDS, so setSecret failed due to a network issue" — placing the Lambda in the same VPC subnet as RDS became the operational standard.

Parameter Store: The Zero-Cost Configuration Store

Systems Manager Parameter Store launched two years before Secrets Manager, and its basic usage is free. It's the best fit for storing "configuration that isn't secret" (e.g., DB hostname, S3 bucket name, environment variables), and it also supports KMS encryption via the SecureString type. So the most common pattern is a division of labor: "put only the secrets that truly need automatic rotation in Secrets Manager, and keep the rest of the configuration in Parameter Store."

The fact that the Standard tier is free is decisive from an operating-cost standpoint. If 100 microservices each hold 20 configs per environment, you need 2,000 parameters — storing all of them in Secrets Manager costs $0.40/secret/month × 2,000 = $800/month. With Parameter Store Standard it's $0. So it makes sense to separate "real secrets that need rotation (DB passwords, API keys)" from "simple configuration (DB host, queue name)" when storing them.

[ Common division of labor ]

Secrets Manager (secrets that need rotation)
   ├─ /prod/rds/admin-password
   ├─ /prod/stripe/api-key
   └─ /prod/oauth/client-secret

Parameter Store (configuration + lightweight secrets)
   ├─ /prod/app/db-host
   ├─ /prod/app/s3-bucket-name
   ├─ /prod/app/feature-flags/new-checkout (value: true/false)
   └─ /prod/app/log-level (value: INFO)

Another advantage Parameter Store has is a hierarchical namespace. You can use slash-structured names like /prod/app/db-host, and pull an entire subtree under a given path at once with the GetParametersByPath API. That makes it possible for an application to load all its configuration in one shot at startup with aws ssm get-parameters-by-path --path /prod/app --recursive --with-decryption. Secrets Manager is weak at this kind of hierarchical lookup and requires fetching each secret individually.

⚠️ Pitfall: A common misconception is "I stored the password in Parameter Store as a SecureString, so rotation happens automatically too." SecureString provides only KMS encryption — it has no rotation feature. If you need rotation, you have to build a separate Lambda and trigger it with an EventBridge cron, and in the process you have to implement the dual-credential pattern yourself. One mistake means downtime, so for secrets that genuinely need rotation, going to Secrets Manager is far less operational burden.

🔍 Going deeper: Parameter Store and Secrets Manager actually overlap in some features. After Secrets Manager launched in 2018, AWS evolved both services for a while, then from 2020 clarified the guidance as "secrets = Secrets Manager, configuration = Parameter Store." So Parameter Store gained the ability to reference Secrets Manager secrets ({{resolve:secretsmanager:...}}), and conversely Secrets Manager came to support databases other than RDS (via custom rotation Lambdas) and user-defined rotation. As a result, the pattern where both services point at the same secret from two places (storing a Secrets Manager ARN under a Parameter Store name) is also common.

Secrets Manager's Advanced Features: Multi-Region Replication and Resource Policies

The two features that give Secrets Manager a decisive edge over Parameter Store in enterprise environments are multi-region replication and resource policies.

Multi-region replication was added in March 2021. Create a secret in one region and turn on the "replicate to ap-northeast-2 and us-west-2" option, and whenever the source changes (including rotation) the replicas are automatically updated too. It's usable for both active-passive and active-active DR scenarios, and you can call GetSecretValue directly in the replica region, so latency is low too. Parameter Store has no such automatic replication and forces you to sync it yourself with scripts.

A resource policy is a feature that attaches a policy to the secret itself to grant access to a Principal in another account. For example, when a SaaS company wants to share part of a secret with a customer account, it allows the customer account root as a Principal in the secret's resource policy. Then the customer only has to grant its own account's IAM Role the secretsmanager:GetSecretValue permission to fetch the SaaS company's secret directly.

// Secrets Manager resource policy example (allowing another account)
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowCustomerAccess",
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::222222222222:root"},
    "Action": "secretsmanager:GetSecretValue",
    "Resource": "*",
    "Condition": {
      "StringEquals": {"secretsmanager:VersionStage": "AWSCURRENT"}
    }
  }]
}

The catch is that you have to share the secret's KMS key along with it. Secrets Manager encrypts the secret with KMS, so for another account to fetch the secret, both (1) the Secrets Manager resource policy and (2) the KMS key policy must allow that account. Miss either of these two steps and you get an AccessDenied — but the message doesn't tell you which one is blocking, which makes troubleshooting tricky.

📚 Case study: In 2022 a payments SaaS used the Secrets Manager cross-account pattern to share a "test secret key" with a customer. At first they set only the resource policy and used the AWS Managed Key (aws/secretsmanager) for KMS — but AWS Managed Keys can't be shared cross-account, so every customer got a KMSAccessDeniedException. After switching to a CMK and adding the customer account root to the key policy, it worked correctly. Afterward, "any secret that needs cross-account sharing must use a CMK" went into the company's standard guidelines.

CloudHSM: Only When Compliance Demands It

CloudHSM is a dedicated, customer-only hardware security module operated by AWS. KMS also runs on HSMs internally, but it's multi-tenant, whereas CloudHSM is single-tenant — only one customer's workload runs on that HSM cluster. You get FIPS 140-2 Level 3 certification under your own name, and its biggest defining trait is that not even AWS employees can access the key material.

CloudHSM's operational model is completely different from KMS. You create a CloudHSM cluster inside a VPC, distribute HSM instances across multiple AZs, and design high availability yourself. You manage client certificates and communicate directly through standard interfaces like PKCS#11 / JCE / OpenSSL Dynamic Engine. Cost is also billed per instance-hour (over $2,000/month for a 2-AZ setup), so unless it's a "genuinely required" case, KMS is overwhelmingly more favorable.

The scenarios where CloudHSM is the answer are usually three. ① When a regulation explicitly requires that "not even AWS employees may access the keys" (some finance / government / healthcare). ② When migrating from an existing on-premises HSM (IBM 4768, Thales nShield) to the cloud. ③ When you need to use SSL/TLS keys inside the HSM so that the web server's private key is never exposed in plaintext in memory. Otherwise, a KMS CMK + Customer Managed Key policy can satisfy nearly every compliance requirement.

💡 Related theory: FIPS 140-2 Levels 1-4 are the cryptographic-module security grades defined by NIST. Level 1 is software, Level 2 is hardware that shows tamper evidence, Level 3 is tamper-resistant hardware + user authentication, and Level 4 zeroizes the key material the instant tampering is detected. Most commercial HSMs are Level 3, and Level 4 is military / national-secret grade. Card-industry regulations like PCI DSS require Level 2 or higher, and some government regulations (e.g., parts of Korea's ISMS-P requirements) require Level 3 or higher. CloudHSM being Level 3 is usually the reason it's sufficient.

🔍 Going deeper: The KMS External Key Store (XKS), launched in 2022, acts as a bridge connecting KMS to CloudHSM (or an external KMIP server). The key material lives in the external HSM and KMS operates as a proxy, so you can keep your keys in an external HSM while still using KMS's rich service integrations (S3, EBS, RDS, etc.) unchanged. It's the pattern that answers the regulatory demand of "we manage the keys ourselves" + "we still want to use AWS services as-is" — but there's a risk that if the external HSM goes down, KMS can't operate with those keys either, so it needs additional availability design.

Standard Usage Patterns on EC2, Lambda, and ECS

How you fetch these three services from compute is the part you deal with most often in practice.

Lambda most commonly puts just the ARN in an environment variable and fetches it directly from code via the SDK. It fetches once at cold start and caches it in memory.

import boto3, json, os
from functools import lru_cache
 
@lru_cache(maxsize=1)
def get_db_password():
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId=os.environ["DB_SECRET_ARN"])
    return json.loads(resp["SecretString"])["password"]

The catch is that even after a secret rotates, if the Lambda doesn't cold-start again it can keep holding the old value. That's why the AWS Parameters and Secrets Lambda Extension, launched in 2022, has effectively become the standard. The extension provides a local HTTP cache, and once the TTL (5 minutes by default) passes it automatically re-fetches. Your code just calls http://localhost:2773/secretsmanager/get?secretId=..., and the extension handles IAM permissions and caching for you.

ECS Task injects the secret as an environment variable at container startup when you put the ARN in the secrets section of the task definition. You get the secret with no code changes.

{
  "containerDefinitions": [{
    "name": "app",
    "secrets": [
      {"name": "DB_PASSWORD", "valueFrom": "arn:aws:secretsmanager:..."}
    ]
  }]
}

EC2 standardly receives permissions via an IAM Instance Profile and fetches from UserData. Or, if the EC2 has strong security requirements, connecting without a password via EC2 Instance Connect or SSM Session Manager is safer.

⚠️ Pitfall: The anti-pattern of putting the secret itself in plaintext in a Lambda environment variable is extremely common. Lambda environment variables are encrypted with KMS, but they can appear in plaintext in the console and in CloudTrail, and if you deploy with IaC (Terraform/CDK) they also remain in plaintext in the state file. The correct answer is always "only the ARN in the environment variable, fetch via the SDK in code."

Hands-On with the CLI

# Create a Secrets Manager secret (RDS password format)
aws secretsmanager create-secret \
  --name prod/rds/admin \
  --secret-string '{"username":"admin","password":"InitialPass!"}' \
  --kms-key-id alias/saa-app \
  --tags Key=Environment,Value=production
 
# Automatic RDS rotation (30 days, using the AWS-provided Lambda)
aws secretsmanager rotate-secret \
  --secret-id prod/rds/admin \
  --rotation-lambda-arn arn:aws:lambda:ap-northeast-2:111:function:SecretsManagerRDSPostgreSQLRotationSingleUser \
  --rotation-rules AutomaticallyAfterDays=30
 
# Multi-region replication
aws secretsmanager replicate-secret-to-regions \
  --secret-id prod/rds/admin \
  --add-replica-regions Region=us-west-2,KmsKeyId=alias/saa-app-uswest2
 
# Get a secret (AWSCURRENT automatically)
aws secretsmanager get-secret-value --secret-id prod/rds/admin
 
# Get a previous version (for rollback)
aws secretsmanager get-secret-value --secret-id prod/rds/admin --version-stage AWSPREVIOUS
 
# Parameter Store Standard SecureString
aws ssm put-parameter \
  --name /prod/app/api-key \
  --value "secret-value" \
  --type SecureString \
  --key-id alias/saa-app
 
# Fetch everything at once with a hierarchical lookup
aws ssm get-parameters-by-path \
  --path /prod/app \
  --recursive \
  --with-decryption
 
# Reference a Secrets Manager secret from Parameter Store
aws ssm put-parameter \
  --name /prod/app/db-secret-ref \
  --value "{{resolve:secretsmanager:prod/rds/admin:SecretString:password}}" \
  --type String
 
# Create a CloudHSM cluster
aws cloudhsmv2 create-cluster \
  --hsm-type hsm1.medium \
  --subnet-ids subnet-aaa subnet-bbb
 
# Add a CloudHSM instance (per AZ)
aws cloudhsmv2 create-hsm \
  --cluster-id cluster-1234 \
  --availability-zone ap-northeast-2a

Wrapping Up

AWS's three tools for handling secrets and configuration have a clear division of labor. Secrets Manager is for real secrets that need automatic rotation and RDS/DocumentDB integration, Parameter Store is for configuration and lightweight secrets stored for free, and CloudHSM is for when compliance demands a standalone HSM. On the exam it's quickly solved by keyword matching (automatic rotation → Secrets Manager, free → Parameter Store, FIPS L3 + dedicated → CloudHSM), but in practice two pitfalls are the most common — ① the anti-pattern of putting secrets in plaintext in code/environment variables, and ② the mistake of assuming a SecureString is rotatable. For any secret that needs rotation, going straight to Secrets Manager is the least operational burden.

In the next article we'll look at the tools that manage the users themselves — Cognito User Pools and Identity Pools. Where KMS and Secrets Manager handle "the secrets of internal systems," Cognito handles "authentication and authorization of end users," and full-stack security is only complete when the tools from both domains come together.