Automating Database Credential Rotation with AWS Secrets Manager and Lambda

Desired outcome

Secrets Manager automatically rotates the credentials for an Amazon RDS/MySQL database on a schedule you define. After each rotation the secret’s AWSCURRENT version holds the new username/password, dependent applications can retrieve the updated credentials, and the previous version is retained as AWSPREVIOUS for rollback.

Prerequisites

1. An existing Secrets Manager secret that stores the RDS username and password (e.g., prod/mysql/credentials). The secret must be created with the SecretString format {"username":"admin","password":"****"}.
2. Network access from the rotation Lambda to the RDS instance. The Lambda must be placed in the same VPC (or a VPC with peering) and have routes to the DB subnet.
3. An IAM role for the Lambda that includes at least the following permissions:
   • secretsmanager:GetSecretValue and secretsmanager:PutSecretValue on the secret ARN.
   • rds:ModifyDBInstance (to change the password).
   • If the secret is encrypted with a customer‑managed KMS key: kms:Decrypt and kms:GenerateDataKey on that key.
   • Basic Lambda execution permissions: logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents.
4. The AWS CLI (v2) installed and configured with credentials that can call Secrets Manager, Lambda, and RDS APIs.

Procedure

1. Create the rotation Lambda function

Use the AWS‑provided rotation template for MySQL/MariaDB. This template knows how to generate a new password, call ModifyDBInstance, and store the new credentials back into the secret.

aws lambda create-function \
  --function-name  \
  --runtime python3.9 \
  --role arn:aws:iam:::role/ \
  --handler lambda_function.lambda_handler \
  --zip-file fileb://aws-secretsmanager-rotation-mysql.zip \
  --timeout 300 \
  --vpc-config SubnetIds=,,SecurityGroupIds=

Replace placeholders:

• – e.g., rds-mysql-rotation
• – your AWS account number
• – the IAM role created in the prerequisites
• , – subnets that have routing to the RDS instance
• – a security group that allows inbound access to the RDS port (usually 3306)

The ZIP file aws-secretsmanager-rotation-mysql.zip can be downloaded from the AWS Samples repo.

2. Store the secret (if not already present)

If you need to create the secret, run:

aws secretsmanager create-secret \
  --name prod/mysql/credentials \
  --secret-string '{"username":"admin","password":"InitPassw0rd!"}' \
  --kms-key-id   # optional; omit to use the default AWS managed key

3. Configure rotation on the secret

Attach the Lambda ARN and set the rotation interval (e.g., 30 days). You can also request an immediate rotation to verify the setup.

aws secretsmanager rotate-secret \
  --secret-id prod/mysql/credentials \
  --rotation-lambda-arn arn:aws:lambda:::function: \
  --rotation-rules AutomaticallyAfterDays=30 \
  --rotate-immediately

Secrets Manager will invoke the Lambda, which will:

1. Retrieve the current secret version.
2. Generate a new strong password.
3. Call rds:ModifyDBInstance to update the password for the specified username.
4. Store the new username/password pair as a new secret version with the staging label AWSPENDING.
5. If the database connection test succeeds, move AWSPENDING to AWSCURRENT and shift the previous current version to AWSPREVIOUS.

4. Verify the rotation succeeded

Run the following checks after the Lambda finishes (you can monitor progress in CloudWatch Logs for the function).

1. Check secret version stages:

aws secretsmanager describe-secret \
  --secret-id prod/mysql/credentials \
  --query 'VersionIdsToStages'

You should see one version with ["AWSCURRENT"] (the newest) and another with ["AWSPREVIOUS"] (the old password).

1. Inspect Lambda logs:

aws logs filter-log-events \
  --log-group-name /aws/lambda/ \
  --filter-pattern "SUCCESS" \
  --limit 5

Look for a log line similar to Rotation succeeded for secret prod/mysql/credentials.

1. Test database connectivity with the new credentials:

# Retrieve the current secret value
SECRET=$(aws secretsmanager get-secret-value \
  --secret-id prod/mysql/credentials \
  --query SecretString --output text)

USERNAME=$(echo $SECRET | jq -r .username)
PASSWORD=$(echo $SECRET | jq -r .password)
HOST=  # e.g., mydb.abcdefg.us-east-1.rds.amazonaws.com
DBNAME=

mysql -h $HOST -u $USERNAME -p$PASSWORD $DBNAME -e "SELECT VERSION();"

If the command returns a version string, the rotation propagated correctly.

Expected checks

• describe-secret shows AWSCURRENT pointing to the newest version.
• CloudWatch Logs for the rotation function contain a SUCCESS message and no stack traces.
• A manual MySQL login using the retrieved credentials succeeds.
• Secrets Manager metric RotateSecretSuccess (available in CloudWatch) increments.

Recovery options (rollback)

If the rotation fails, Secrets Manager leaves the AWSCURRENT version unchanged and retains the failed attempt as AWSPENDING (or may not create a new version at all). You can manually revert to the known‑good version:

1. Identify the version ID of the AWSPREVIOUS stage (from describe-secret output).
2. Promote it to current:

aws secretsmanager update-secret-version-stage \
  --secret-id prod/mysql/credentials \
  --version-stage AWSCURRENT \
  --move-to-version-id  \
  --remove-from-version-id 

After this step, applications retrieving the secret will receive the old credentials. Investigate the Lambda failure (check logs for ERROR or timeout), fix the underlying issue (e.g., VPC routing, IAM permissions, KMS key access), then re‑initiate rotation.

Limitations and practical verification

• The rotation Lambda must run in a VPC with a route to the RDS instance; otherwise the function will timeout and the rotation will fail.
• Minimum rotation interval is 1 day; you cannot set a shorter schedule.
• Only secrets that store a single username/password pair (or the JSON structure expected by the MySQL template) can be rotated with the provided template. Custom schemas require your own Lambda code.
• If the secret uses a customer‑managed KMS key, the Lambda role needs both kms:Decrypt (to read) and kms:GenerateDataKey (to write). Missing either permission causes the rotation to abort before touching the database.
• Practical verification: after each scheduled rotation, run the describe-secret command and check that the AWSCURRENT version timestamp has advanced. Combine this with a CloudWatch alarm on the RotateSecretSuccess metric to get notified of failures.

---

By following the steps above you achieve fully automated, auditable credential rotation for your RDS/MySQL databases, reduce the risk of static passwords, and have a clear path to rollback if something goes wrong.