Placement-backed scheduling in Nova: attributing latency after the Stein and Train transitions

Observables that split scheduler‑side from Placement‑side latency

In a fully upgraded Nova (Stein or later) the two components expose distinct timing metrics that can be compared directly:

• Nova‑scheduler metrics – scheduler.filtering and scheduler.weighing (RPC‑level timings). These represent the time spent evaluating filters and weighers on the scheduler side.
• Placement API metrics – placement.api.request_time and placement.db.query_time (Prometheus labels). These capture the round‑trip to Placement and the database query cost.

Collect a sample set of scheduling requests (e.g., via openstack server create or a workload generator) and record the two metric series. If the scheduler’s filtering/weighing times dominate, the bottleneck lies in Nova’s logic or its local state. If the Placement metrics are higher, the delay originates in the inventory service or its database. When both sides show similar latencies, the issue is likely shared or cascading.

Reconciling Allocation Ratios across nova.conf and Placement

After Train, the authoritative allocation ratio for a resource class is the one defined in Placement’s inventory. Nova falls back to the nova.conf value only when Placement is not queried for that class. The scheduler applies the stricter (i.e., higher reservation factor) of the two values. To resolve conflicts:

1. Run openstack placement inventory show <resource_class> to read the current ratio.
2. Compare it with [resource_class_allocation_ratio] in nova.conf.
3. If they differ, update the inventory to match the desired ratio (or adjust nova.conf if you intend to keep the legacy value).
4. Restart the scheduler to force a fresh inventory cache.

In practice, the scheduler will refuse a host if the inventory’s ratio permits fewer units than the request requires, even if nova.conf would allow more. Thus, the inventory is the source of truth for capacity calculations.

Distinguishing Capacity Shortfall from Placement Misconfiguration

When a scheduling request fails with No valid host found, the following diagnostics help identify the root cause:

• Placement candidate count – If the Placement API returns zero candidates, the inventory is truly exhausted.
• Scheduler filter failures – A high number of filtering failures in the scheduler logs, coupled with a non‑zero candidate count, indicates that filters (e.g., FilterScheduler constraints, custom filters) are rejecting hosts that Placement considered viable.
• Candidate count vs. filter failures – A low candidate count (<10) with many filter failures suggests mis‑configured placement constraints (e.g., over‑constrained availability zones). A high candidate count but many failures points to filter logic or weighers mis‑aligned with the inventory.
• Resource class inventory – Verify that the host’s inventory for the requested resource class is present and that the allocation ratio is not inadvertently set to a very low value.

Combining these observations with the metric comparison above provides a clear picture: if Placement reports capacity but the scheduler rejects hosts, the problem is with scheduler filters or a mis‑aligned allocation ratio. If Placement reports no candidates, the issue is a genuine capacity shortfall.

Quick Checklist for Operators

• Ensure nova-scheduler and placement are on compatible versions (Stein ≥19.0.0, Train ≥20.0.0).
• Enable Prometheus metrics for both services and collect a baseline.
• Verify that the allocation ratio in Placement matches the intended value; update if necessary.
• When encountering No valid host found, first check the Placement candidate count, then review scheduler filter logs.
• Consider raising the allocation ratio temporarily on a test host to see if the scheduler’s behavior changes; a change confirms the ratio is the governing factor.

These steps give operators a reproducible path to isolate latency, reconcile configuration drift, and differentiate real capacity limits from misconfiguration.