Control Instance Placement with Nova Server Groups

Why Instance Placement Matters

In a multi‑tenant OpenStack cloud, the location of a virtual machine can affect licensing compliance, high‑availability (HA) guarantees, and performance isolation. For example, a database cluster should stay on the same physical server to reduce network latency, while a HA pair should never share a single host so that a host failure does not bring both instances down. Nova’s server groups give operators a declarative way to enforce these constraints without hard‑coding placement logic into each project.

What Are Server Groups?

A server group is a logical container that groups multiple instances and associates a policy with them. The four built‑in policies are:

• affinity – all members must be placed on the same compute host.
• anti‑affinity – members must be on distinct hosts.
• soft‑affinity – the scheduler prefers the same host but can violate the rule if resources are tight.
• soft‑anti‑affinity – the scheduler prefers different hosts but may place them together under pressure.

These policies are enforced by the FilterScheduler through two filters: ServerGroupAffinityFilter and ServerGroupAntiAffinityFilter. Soft policies add a weigher (ServerGroupSoftAffinityWeigher) that biases placement but does not mandate it.

Creating an Anti‑Affinity Group for an HA Pair

Below is a step‑by‑step example that shows how to create an anti‑affinity group and boot two instances that belong to it. The example assumes you have the OpenStack client installed and are logged in as a user with admin role.

# 1. Create the anti‑affinity group
openstack server group create \
  --policy anti-affinity \
  my-ha-group
# Expected output: a UUID, e.g. 123e4567-e89b-12d3-a456-426614174000

# 2. Note the group UUID for use as a hint
GROUP_ID=123e4567-e89b-12d3-a456-426614174000

# 3. Boot the first instance into the group
openstack server create \
  --image ubuntu-22.04 \
  --flavor m1.small \
  --hint group=$GROUP_ID \
  ha-instance-1

# 4. Boot the second instance into the same group
openstack server create \
  --image ubuntu-22.04 \
  --flavor m1.small \
  --hint group=$GROUP_ID \
  ha-instance-2

After the instances reach the ACTIVE state, verify that they are on different hosts:

# Show host assignment for each instance
openstack server show ha-instance-1 -c OS-EXT-SRV-ATTR:host -f value
openstack server show ha-instance-2 -c OS-EXT-SRV-ATTR:host -f value
# If the two commands return different host names, anti‑affinity succeeded.

For deeper diagnostics, inspect the scheduler log. A line like Filter ServerGroupAntiAffinityFilter returned 0 hosts indicates that no host satisfied the anti‑affinity constraint for a particular placement request.

Soft Policies for Cost‑Effective Packing

Soft affinity/anti‑affinity is useful when you want the scheduler to prefer a placement but can tolerate violations. For example, you might want to pack batch jobs onto a few hosts to free up capacity for critical workloads. The weight that determines how strongly the scheduler favors a soft policy is controlled by two global configuration options in nova.conf:

• soft_affinity_weight_multiplier – default 1.0
• soft_anti_affinity_weight_multiplier – default 1.0

Adjusting these values changes the scheduler’s preference across the entire cloud, so use them sparingly and monitor placement outcomes.

Operational Caveats and Trade‑offs

Group Size and Scheduler Latency

Each placement request must evaluate the group’s members against every candidate host. A group with hundreds of instances can noticeably increase scheduler latency and, if the compute pool is fragmented, may cause placement failures. Keep anti‑affinity groups small—ideally one per HA pair—and avoid adding unrelated instances to a single group.

Anti‑Affinity Across Failure Domains

The anti‑affinity policy only guarantees separation across compute hosts. If you need to enforce separation across racks or availability zones, pair server groups with host aggregates and the AggregateInstanceExtraSpecsFilter. For example, label aggregates with rack=1 and rack=2 and add the aggregate_instance_extra_specs:rack=1 filter to the scheduler.

Immutability of Membership

Once an instance is booted, its group membership cannot be changed. Moving an instance to a different group requires deleting and recreating the instance or performing a rebuild, which incurs downtime. Plan group membership carefully and document which instances belong to which groups.

Persistence Across Cell Migrations

Nova server groups live in the nova_api database. If you archive or migrate the database, or if you move cells, group information can be lost. Back up nova_api regularly and, if you plan a cell migration, recreate the groups in the new cell.

Best Practices for Production Environments

1. Automate with Heat – Use Heat templates to create server groups and launch instances with the correct --hint value. This eliminates manual drift and ensures consistent placement across projects.
2. Monitor Group Size – Track the num_instances per host and the size of each anti‑affinity group. Use openstack server group list --long to audit membership.
3. Tune Soft Weights Cautiously – If you find the scheduler is frequently violating soft policies, consider raising the weight multiplier, but remember it applies globally.
4. Validate Placement – After booting an instance, run openstack server show and check the host, then cross‑reference with the placement service via nova-manage placement heal_allocations to keep the allocation database in sync.
5. Backup nova_api before major changes – A simple pg_dump of the nova database protects group definitions during upgrades or migrations.

Conclusion

Nova server groups are a mature, declarative mechanism to enforce placement policies that are essential for HA, licensing, and performance isolation. By combining affinity/anti‑affinity policies with careful group sizing, host aggregates for failure‑domain separation, and automated provisioning via Heat, operators can reliably control where instances land while keeping the scheduler responsive. Remember that soft policies provide guidance, not guarantees, and that group membership is immutable once an instance is created. With these guidelines, you can make informed decisions about where to place your workloads and avoid costly placement surprises.