Nova API ↔ Placement Service: Does Shared Database Connection Pool Cause Latency Spikes Under Concurrent Requests?

Connection Pool Contention and Latency

Yes, a shared or undersized database connection pool can cause latency spikes under concurrent requests. When the combined demand for connections from the Nova API and the Placement service exceeds the max_pool_size, threads enter a wait state. This results in increased response times as requests block while waiting for a connection to be returned to the pool.

Likely Explanation of the Bottleneck

In OpenStack releases Rocky and later, the oslo.db connection pool manages how services interact with the database. If Nova API and Placement are configured with a pool size that does not account for their combined worker count, the following occurs:

• Resource Starvation: High concurrency in the Nova API can exhaust the pool, leaving the Placement service unable to acquire a connection for critical resource provider lookups.
• Wait-Time Spikes: Latency increases not because of slow query execution, but because of the time spent in the "connection acquisition" phase.
• Cascading Delays: Since Nova API often depends on Placement for scheduling and allocation, a bottleneck in the Placement service's ability to reach the DB creates a synchronous delay that bubbles up to the end-user API response.

Recommended Configuration Steps

To eliminate these spikes, the max_pool_size must be scaled to accommodate the total concurrency of both services. Assuming a standard deployment:

1. Calculate Total Workers: Sum the total number of API workers across all Nova API instances and Placement service instances.
2. Set Pool Size: Update nova.conf and placement.conf. A safe starting point is setting max_pool_size equal to or slightly greater than the total worker count to ensure no thread is blocked during peak concurrency.
3. Verify DB Limits: Ensure the database server's max_connections setting is higher than the aggregate max_pool_size of all distributed service instances to avoid shifting the bottleneck to the database engine.

Impact of Worker Architecture

The relationship between pool size and latency changes based on the server model:

• Eventlet-based: These use green threads; while they handle more concurrent connections, they can still saturate the pool if the underlying DB driver does not support non-blocking I/O efficiently.
• Multi-process WSGI: This creates a 1:1 relationship between a process and its potential connection usage. This typically makes latency spikes more predictable and directly tied to the number of processes.

Verification and Diagnostics

To confirm the pool is the culprit, monitor the database process list during a spike to see if connections are idling or if the application is failing to establish new ones. You can use a scoped command to check current DB connections:

# Example for MySQL/MariaDB to check active connections
mysql -u root -p -e "SHOW PROCESSLIST;"

Missing Diagnostic Detail: Are you using a database proxy (like MaxScale or PgBouncer)? If a proxy is present, the max_pool_size in the config files may be superseded by the proxy's own pooling logic, changing the recommendation for worker scaling.