Fine‑Tuning OpenStack Nova Scheduling: Custom Filters and Weights Explained

Problem: Why the default Nova scheduler sometimes feels blind

When a tenant launches an instance, Nova’s scheduler decides which compute node will host it. In a 1,000‑node cloud, the default filter set can be too coarse: a single mis‑configured host, a recent upgrade, or a custom affinity rule can cause a launch to fail or to land on a sub‑optimal node. Operators need a way to influence that decision without rewriting the scheduler core.

Thesis: The filter‑and‑weight chain gives you fine‑grained, declarative control

Nova’s scheduling logic is split into two layers:

• Filters reject hosts that don’t meet a criterion (e.g., not enough RAM, not on the right availability zone).
• Weighers rank the remaining hosts so the scheduler picks the best one.

By adding or tweaking these plugins you can express domain‑specific policies (affinity, anti‑affinity, cost‑based placement) while keeping the core scheduler untouched.

Understanding the Default Filter Chain

The list of active filters is stored in the scheduler_default_filters configuration option in /etc/nova/nova.conf. The default value (Nova 24.x) is:

filter_scheduler_default_filters = ComputeFilter,AvailabilityZoneFilter,ImagePropertiesFilter,RamFilter,DiskFilter,GenericFilter,RequiredByFlavorFilter,GenericFilter,GenericFilter

When Nova starts, it imports each class by its fully‑qualified Python path. A typo in the path will abort the scheduler with a traceback, so always double‑check the import string.

Listing the Filters at Runtime

Run this command on the controller node:

openstack-config --get compute scheduler default_filters

It will print the current list. You can also query the scheduler API:

curl -s -H "X-Auth-Token: $TOKEN" http://$CONTROLLER:8774/v2.1/filters

Both approaches reveal which filters are active and in what order.

Weight Plugins: Turning “good enough” into “best fit”

After filtering, each host receives a numeric score from the weighers. The default weighers are:

• RamWeigher – prefers hosts with more free RAM.
• DiskWeigher – prefers hosts with more free disk space.
• CpuWeigher – prefers hosts with more free vCPUs.

Weights are applied in the order they appear in scheduler_default_weighers. The final host is the one with the highest summed score. You can override the default by editing the same configuration option:

filter_scheduler_default_weighers = RamWeigher,DiskWeigher

Removing CpuWeigher can reduce scheduling latency in CPU‑heavy workloads.

Adding a Custom Filter: An Affinity Example

Suppose you want to launch instances only on compute nodes that already host a certain tenant’s VM. Create a Python module my_affinity.py in /opt/nova/plugins:

from nova.scheduler.filters import BaseFilter

class AffinityFilter(BaseFilter):
    """Reject hosts that do not already run a VM from the same tenant."""

    def host_passes(self, host_state, spec_obj):
        # spec_obj contains the request data, including the tenant_id
        tenant_id = spec_obj.tenant_id
        # Count instances of this tenant on the host
        tenant_instances = sum(
            1 for inst in host_state.instances
            if inst.tenant_id == tenant_id
        )
        # If no instances exist, reject the host
        return tenant_instances > 0

Register the filter by adding its import path to scheduler_default_filters:

filter_scheduler_default_filters = ComputeFilter,AvailabilityZoneFilter,AffinityFilter

After editing nova.conf, restart the scheduler service:

systemctl restart openstack-nova-scheduler

Verify the new filter is active:

openstack-config --get compute scheduler default_filters

Now launch a test instance with a flavor that triggers the filter (e.g., a small RAM flavor). Check the scheduler logs at /var/log/nova/scheduler.log for lines like:

Filtering hosts by AffinityFilter: 3 hosts rejected

These entries confirm the filter ran and which hosts were discarded.

Trade‑offs and Limitations

• Performance impact: Filters that iterate over all instances on a host (like the affinity example) can add latency, especially in clouds with thousands of VMs. Profile with nova-scheduler --debug if you notice a 1–2 s delay per launch.
• Maintenance overhead: Custom code must be kept in sync with Nova upgrades. Use the plugins directory and a separate Git repo to isolate changes.
• Failure mode: If a filter returns False for all hosts, the launch fails with NoValidHost. Always test with a small subset of hosts before rolling out.
• Logging verbosity: The scheduler logs can become noisy. Adjust debug=true only during troubleshooting.

Actionable Closing: A Checklist for Scheduler Tuning

1. Document the current filter and weigher list: openstack-config --get compute scheduler default_filters and --get compute scheduler default_weighers.
2. Identify the policy you need (affinity, anti‑affinity, cost‑based). Write a lightweight filter if needed.
3. Add the filter path to scheduler_default_filters and restart the scheduler.
4. Run a test instance and inspect /var/log/nova/scheduler.log for filter execution and host scores.
5. Measure scheduling latency before and after the change.
6. Iterate: tweak weight multipliers or filter logic until the desired host selection pattern emerges.
7. Version‑control your nova.conf changes and custom plugin modules.
8. In production, monitor the scheduler’s health with openstack status and log rotation policies.

By treating the filter and weigher chain as a declarative policy engine, you can keep Nova’s core stable while tailoring placement to your operational needs.