When to enable pytest-xdist and how to choose a distribution mode

The decision

Parallel test execution with pytest-xdist is useful when a suite is slow and tests are isolated. The useful takeaway is: enable xdist only if tests are deterministic, have no shared mutable state, and your CI can afford extra CPU and memory. Otherwise keep serial execution.

Constraints that block parallelism

Shared resources break xdist. Tests that write to the same file, use a single database schema, or depend on test ordering will fail intermittently under concurrency.

• Shared state: module-level globals, singletons, or fixtures with session scope that mutate.
• Resource limits: each worker is a separate Python process. Memory and CPU usage scale roughly with -n.
• Ordering assumptions: tests that rely on previous tests having run first are not safe.

Fixtures should be function scoped and clean up after themselves. Use temporary directories per test via tmp_path rather than a fixed path.

Supported options

pytest-xdist distributes tests across workers. The distribution mode controls granularity.

Mode	Command	Granularity	When it helps
Serial	pytest	one process, all tests	Default. Safe for shared state, easier debugging.
loadfile	pytest -n auto --dist=loadfile	whole files to workers	Good when files are independent and roughly equal size.
loadscope	pytest -n auto --dist=loadscope	class or module kept together	Preserves module-level fixtures and setup. Common default for xdist.
loadscope with loads	pytest -n auto --dist=loads	individual test items	Fine-grained balancing for heterogeneous test durations.

-n auto sets workers to number of CPUs. Use an explicit number in CI to respect quotas.

Trade-offs

Speed vs stability

Parallelism reduces wall-clock time for CPU-bound tests. It also exposes hidden coupling. Flaky failures often appear only under concurrency.

Resource usage

Each worker loads the test suite and application code. Memory usage multiplies. CI minutes may increase if you raise parallelism without a limit.

Debugging

Failures can be harder to reproduce because order changes. Re-run a failing test serially with -n 1 -k test_name to isolate.

Concrete implementation

Start conservative with loadscope and function-scoped fixtures.

[pytest]
addopts = -n auto --dist=loadscope --maxfail=1
asyncio_mode = auto

Place this in pytest.ini or pyproject.toml under [tool.pytest.ini_options]. Run from the repository root with a user that can execute pytest. No elevated permissions are required.

Example fixture pattern for isolation:

import pytest

@pytest.fixture
def temp_db(tmp_path):
    db_path = tmp_path / "test.db"
    # create fresh DB per test
    yield str(db_path)
    # cleanup handled by tmp_path teardown

Use @pytest.mark.parametrize for data-driven tests. xdist will distribute parametrized items according to the chosen distribution mode.

Mark tests that cannot run in parallel:

@pytest.mark.xdist_group(name="db")
def test_requires_shared_db():
    pass

Group marks keep related tests on the same worker.

Validation without claiming test results

Check isolation before committing to parallelism.

1. Run a serial baseline: pytest --durations=10 and note total duration.
2. Run with xdist: pytest -n auto --dist=loadscope --durations=10. Compare total duration. A reduction indicates effective parallelism.
3. Inspect for warnings about resource use or ordering. Absence of repeated failures across runs suggests isolation.
4. Create a canary test that writes to a shared path. If each test receives a unique temporary directory via tmp_path, the test should not interfere. Failure to isolate indicates shared state.

Limitations: xdist does not parallelize within a single test function. Tests that spawn subprocesses may oversubscribe CPUs. Network or database contention can still cause flakiness even with isolated fixtures.

Rollback is simple: remove -n and --dist from addopts or run pytest -n 0 to force serial execution.