Diagnosing intermittent Lambda-to-RDS connection timeouts in a VPC

The recognizable condition

Your Lambda function talks to an RDS or Aurora database in the same VPC, and most of the time it works. Under load, though, logs start showing sporadic connection timed out or could not connect to server errors. A bastion host in the same VPC connects fine every time. The database itself reports healthy CPU and memory. This pattern — intermittent, load-correlated, infrastructure that looks fine — almost always points to networking or connection exhaustion, not the database engine.

The useful takeaway: don't start by tuning the database. Start by deciding which of four fault domains you're in, because each has a different fix and a different diagnostic signal.

Symptom-to-cause map

Ordered checks

Work through these in order; each one narrows the fault domain cheaply.

1. Security groups, both directions. The Lambda function's security group needs egress to the database port (5432 for PostgreSQL, 3306 for MySQL), and the RDS security group needs ingress from the Lambda group. Reference the security group IDs against each other rather than CIDR blocks — it's both safer and self-documenting. Run aws ec2 describe-security-groups --group-ids sg-xxx in the CLI (read-only, no special risk) and confirm the rules match.
2. NACLs and route tables. Security groups are stateful; NACLs are stateless. If a NACL restricts traffic, you must allow the ephemeral port range (1024–65535) on the return path, not just the database port inbound. Fixing only one side of a NACL is the classic cause of "fixed but still broken" reports.
3. Connection math. Get the instance's limit with SHOW max_connections; (run from a bastion or CloudShell with database access). Then compare it against peak Lambda concurrency: reserved concurrency on the function, or account-level concurrency if unreserved. Each concurrent execution that opens its own connection consumes one slot. If peak concurrency exceeds max_connections, you've found your cause.
4. CloudWatch correlation. Overlay DatabaseConnections (RDS) with Lambda's ConcurrentExecutions over the incident window. If the error rate climbs exactly when connections approach the limit, that's confirmation. Metric names and dimensions vary by engine, so verify against current RDS documentation.
5. Minimal reproduction. Deploy a throwaway Lambda in the same subnets and security groups that does nothing but open a TCP socket to the RDS endpoint and port, then close. If this fails, the problem is pure networking. If it succeeds under load while your real function fails, the problem is connection management in your code.

VPC Flow Logs: the decisive tool

When checks 1–4 are ambiguous, enable VPC Flow Logs on the subnets involved (this is a configuration change, but it only adds logging — it doesn't alter traffic). Then query in CloudWatch Logs Insights:

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action
| filter dstPort = 5432
| sort @timestamp desc
| limit 50

Run this in the CloudWatch console against the flow log group. REJECT entries tell you traffic is being denied by a security group or NACL at that network interface. ACCEPT on both ends with continued failures means the problem is above the network layer — escalate rather than keep guessing.

Fixes mapped to findings

• Connection exhaustion: Put RDS Proxy between Lambda and the database. It pools and multiplexes connections so thousands of concurrent executions share a bounded number of database connections. Also create the database client outside the handler function so warm executions reuse it. Note that RDS Proxy adds cost and a small latency overhead, and it will not fix a security group misconfiguration — it only addresses connection churn.
• Security group issues: Replace broad 0.0.0.0/0 rules with group-to-group references. This is a state change; apply it during a low-traffic window and verify connectivity immediately after.
• NACL issues: Add explicit allow rules for ephemeral ports 1024–65535 in both directions on the relevant subnets.
• DNS issues: Enable enableDnsSupport and enableDnsHostnames on the VPC.
• Stale connections: Enable TCP keepalive or validate connections before use (most pooling libraries have a validate or testOnBorrow option), and set idle timeouts on the client shorter than the server's.

Verifying the fix

Replay the conditions that caused the failure: run a load test at the concurrency level that previously triggered timeouts, and watch Lambda errors and DatabaseConnections in CloudWatch. Error rates should return to baseline and connections should plateau below the limit. A fix that hasn't been tested at the failure-inducing load is a hypothesis, not a fix.

When to escalate to AWS Support

Open a support case when: flow logs show ACCEPT on both ends but connections still fail; you suspect hypervisor-level ENI issues on the Lambda side; or RDS metrics look healthy while connections refuse to establish. Attach flow log excerpts, Lambda request IDs with timestamps, and the minimal-reproduction results — this lets support skip the first round of questions. Note that Lambda's VPC networking changed significantly in 2019 (ENIs are now shared and cold-start impact is much lower), so disregard older guidance that attributes these symptoms to per-function ENI creation delays.