PyPIUtilsupdated 06 Aug 2026

pytest-xdist

Item: pytest-xdist
Rating: 4.5
Author: Keyoor

pytest-xdist is the pytest plugin that runs your test suite across several processes so a suite that takes twelve minutes takes three. You add -n auto and it spawns one worker process per physical CPU core. Each worker is a full mini pytest runner: it does its own collection, reports the collected test ids back to a controller process, and then waits to be told which test index to run next. The controller does no collecting of its own and instead hands out work and merges the results back into a single report. Communication happens over execnet, the same library that lets xdist send tests to a subprocess running a different Python or to another machine entirely. That architecture is where both the speed and every one of the plugin's limitations come from.

pytest-xdist · v3.8.0 · 41.3M/wk · ★ 1,892 · pushed 03 Aug 2026 · pytest-dev/pytest-xdist · docs · reviewed by Keyoor

Verdict

The default way to cut pytest wall-clock time, and on an isolated CPU-bound suite -n auto is close to free money. The cost is real and lands later: no -s, no --pdb, session fixtures running once per worker, and any shared-state assumption in your suite turning into an intermittent CI failure.

API stability5/5The -n and --dist flags, the xdist_group mark, and the worker_id and testrun_uid fixtures have worked the same way for years, and 3.x releases have been additive. The one behaviour change worth knowing is 3.6, where workers began always running tests on the main thread, which fixed async frameworks and quietly changed timing for a few suites.

Docs4/5The Read the Docs site is short but unusually candid: there is a dedicated known-limitations page stating plainly that -s and --pdb do not work and that collection order must match, plus a how-it-works page explaining the controller and worker split. What it lacks is a practical guide to making a shared-database suite parallel-safe, which is the actual problem most readers arrive with.

Maintenance4/5Maintained under the pytest-dev organisation with the repo pushed 3 August 2026, but 3.8.0 dates from June 2025, so more than a year has passed without a release. The 280 open issues (316 counting PRs) on a repo this size reflect a plugin that works and is slowly triaged rather than one under active development.

Ecosystem5/5About 41M weekly downloads and effectively part of the standard pytest toolkit; pytest-cov, pytest-django, and most large plugins document how they behave under xdist, and the pytest_xdist_make_scheduler hook lets other plugins supply their own distribution logic.

Use it if

Your suite is CPU-bound and slow enough that wall-clock time is a real cost, and you have cores sitting idle while pytest runs one test at a time
Your tests are already isolated from each other: no shared temp paths, no ordering assumptions, no one global database that two tests would both truncate
You want CI runtime down without splitting the suite across more runners and paying for them
You have expensive module-level or class-level fixtures and can keep them in one process using --dist loadscope or --dist loadfile
Some tests genuinely must share a resource and you can tag them with @pytest.mark.xdist_group so they land on the same worker under --dist loadgroup

Skip it if

You need to debug: --pdb is disabled under distribution, and -s / --capture=no does nothing because execnet cannot forward worker stdout and stderr to your terminal, so every print you added is invisible
Your tests share mutable state: one database, one Redis, one fixture file, one output directory. Parallelising them turns deterministic failures into flakes that only reproduce on CI, and fixing that is a bigger job than the speedup is worth
Your suite is I/O-bound on an external service: eight workers hitting the same staging API do not finish faster, they get rate limited, and the async event loop cases are better served by making the tests concurrent inside one process
It is a small suite: worker startup plus a full collection in every worker is fixed overhead, so a two-second suite reliably gets slower with -n auto
Your parametrize values come from a set or any unordered iterable: workers must collect the same tests in the same order, and a differing collection aborts the whole run with an error that does not point at the offending parametrize
You want per-worker coverage or a specific test order: coverage needs pytest-cov's combine step, and ordering plugins that assume a single sequential run do not survive being sharded

Setup reality

pip install pytest-xdist pulls execnet 2.1 or newer and needs pytest 7.0 or newer on Python 3.9 or newer, then -n auto works immediately with no configuration. The work is not in installing it, it is in surviving it. Every worker imports your conftest.py and runs session-scoped fixtures independently, so a fixture that creates a database, binds a port, or writes to a fixed path now runs N times at once and collides; the standard fixes are the worker_id fixture to give each worker its own resource and the testrun_uid fixture to namespace the whole run. -n auto counts physical cores, while -n logical counts logical ones and needs Python 3.13 or the psutil extra to do it. Two smaller traps: workers with -p no:randomly or any collection that varies between processes will abort with a collection mismatch, and CI containers frequently report the host's core count rather than the cgroup limit, so -n auto oversubscribes a two-CPU runner into sixteen workers and everything gets slower. Pin the number explicitly in CI, or set PYTEST_XDIST_AUTO_NUM_WORKERS.

Patterns

Spread tests across coresrun-in-parallel

pytest -n auto        # one worker per physical core
pytest -n logical     # per logical core (needs Python 3.13 or psutil)
pytest -n 4           # exactly four
pytest -n 0           # plugin installed but disabled
pytest -n auto --maxprocesses 8

In CI, -n auto often reads the host's core count rather than the container's CPU limit, so a 2-CPU runner spawns sixteen workers that all fight for the same core. Pin the number, or set PYTEST_XDIST_AUTO_NUM_WORKERS in the job environment. -n 0 is the quick way to prove a failure is caused by parallelism.

Pick how tests are grouped onto workerschoose-distribution-mode

pytest -n 4 --dist load        # default: any test to any free worker
pytest -n 4 --dist loadfile    # all tests in a file stay together
pytest -n 4 --dist loadscope   # grouped by module, or by class for methods
pytest -n 4 --dist loadgroup   # grouped by the xdist_group mark
pytest -n 4 --dist worksteal   # even split, idle workers steal work
pytest -n 4 --dist each        # run the whole suite on every worker

load ignores your fixtures entirely, so an expensive module-scoped fixture can be built by all four workers. loadfile and loadscope trade some balance for fixture reuse. worksteal handles suites with wildly uneven test durations better than load while keeping similar fixture reuse. each is for running the same suite against several interpreters, not for speed.

Keep related tests in the same processpin-tests-to-one-worker

import pytest

@pytest.mark.xdist_group(name="payments-db")
def test_creates_invoice():
    ...

@pytest.mark.xdist_group(name="payments-db")
def test_voids_invoice():
    ...

# pytest -n 4 --dist loadgroup

The mark only has an effect under --dist loadgroup; everywhere else it is silently ignored. Since 3.7 multiple xdist_group marks on one test (from parametrize or a fixture) are merged into a combined group name, and the mark order does not matter because names are sorted lexically first.

Give each worker its own database or portisolate-per-worker-resources

import pytest

@pytest.fixture(scope="session")
def database_url(worker_id):
    if worker_id == "master":      # running with -n0
        return "postgres:///test"
    return f"postgres:///test_{worker_id}"   # gw0, gw1, gw2 ...

@pytest.fixture(scope="session")
def http_port(worker_id):
    offset = 0 if worker_id == "master" else int(worker_id.removeprefix("gw"))
    return 8000 + offset

This is the change that makes a suite parallel-safe, and it is on you, not the plugin. worker_id is "master" when xdist is off, which is the branch people forget and then cannot run -n0 locally. The same values are also on PYTEST_XDIST_WORKER and PYTEST_XDIST_WORKER_COUNT for code that cannot take a fixture.

Create one shared resource per test runnamespace-a-whole-run

import pytest

@pytest.fixture(scope="session", autouse=True)
def schema(testrun_uid, worker_id):
    name = f"app_{testrun_uid}"
    if worker_id in ("master", "gw0"):
        create_schema(name)
    wait_for_schema(name)
    yield name

testrun_uid is identical across every worker in one run and different between runs, so it is what you use to name something shared rather than something per-worker. Creating it from a single worker still needs a lock or a wait, because there is no ordering guarantee between workers; the docs use a POSIX semaphore for exactly this.

Branch on whether xdist is activedetect-xdist-at-runtime

import xdist

def pytest_configure(config):
    if xdist.is_xdist_worker(config):
        return          # controller-only setup, skip in workers
    start_shared_docker_stack()

@pytest.fixture
def slow_thing(request):
    if xdist.is_xdist_controller(request.config):
        pytest.skip("controller does not run tests")

conftest.py is imported by the controller and by every worker, so anything with a side effect at import or in pytest_configure happens N+1 times. is_xdist_worker and is_xdist_controller are the guards for that; get_xdist_worker_id gives the raw gw0-style id.

Deal with a segfaulting testhandle-worker-crashes

pytest -n 4                        # crashed worker is restarted automatically
pytest -n 4 --max-worker-restart 2 # give up after two restarts
pytest -n 4 --max-worker-restart 0 # never restart, fail the run

# find the culprit once you know the file
pytest -n 0 tests/test_native_bindings.py -x

A worker that dies (segfault in a C extension, an os._exit, an OOM kill) is restarted and its test is reported as failed, which means a genuinely crashing test can quietly restart workers all run long and just look slow. Setting --max-worker-restart 0 turns that into a hard failure you will actually notice.

Make every worker collect the same testsfix-collection-mismatch

# breaks: set iteration order can differ between processes
@pytest.mark.parametrize("param", {"a", "b"})
def test_thing(param): ...

# fine: deterministic order
@pytest.mark.parametrize("param", ["a", "b"])
def test_thing(param): ...

@pytest.mark.parametrize("param", sorted(load_names()))
def test_other(param): ...

Workers collect independently and the controller aborts if the lists differ in content or order. Sets, dict iteration over data loaded at import, parametrize values built from a glob, and anything keyed on a random seed all produce this. The error names the mismatch but not the parametrize that caused it, so grep for set literals and unsorted globs first.

Collect coverage across workerscoverage-with-xdist

pytest -n auto --cov=myapp --cov-report=term-missing

# .coveragerc
[run]
parallel = true
concurrency = multiprocessing
sigterm = true

pytest-cov handles the per-worker data files and the combine step, but only if coverage is set up for parallel mode; otherwise workers overwrite each other's .coverage file and the report is a fraction of reality. Check the total against a -n0 run once before you trust it in a CI gate.

Get output back when a parallel test failsdebug-under-xdist

# none of these do what you want with -n > 0:
pytest -n 4 -s          # capture=no is not forwarded from workers
pytest -n 4 --pdb       # disabled while distributing

# reproduce serially instead:
pytest -n 0 -s --pdb tests/test_thing.py::test_case

# or use logging, which is captured and reported normally:
pytest -n 4 --log-cli-level=DEBUG

execnet cannot forward worker stdout and stderr, which is why print debugging silently stops working the moment you add -n. Failure tracebacks and captured logs do come back through the report channel, so switching from print to logging is the practical workaround when a failure only reproduces in parallel.

Smooth out startup and chunkingtune-scheduling

pytest -n 16 --ramp=10s          # stagger the first test of each worker
pytest -n 8 --dist load --maxschedchunk 1   # hand out one test at a time
pytest -n 8 --dist loadscope --no-loadscope-reorder

--ramp stops sixteen workers from hammering a database or a container registry in the same second at startup. --maxschedchunk 1 helps when a few tests are far slower than the rest and the default chunking leaves one worker with all of them. --no-loadscope-reorder, added in 3.8, keeps file order under loadscope when relative ordering matters more than balance.

Send tests to a different Python or another machinerun-on-other-interpreters

pytest -d --tx popen//python=python3.13
pytest -d --tx 3*popen//python=python3.12
pytest -d --tx ssh=user@host//python=python3.12 --rsyncdir myapp --rsyncdir tests

This is the execnet layer showing through, and it is the original reason the plugin exists. The ssh form rsyncs the directories you name to the remote box, so the remote environment has to already have your dependencies installed; in practice most teams reach for containers in CI instead and only use --tx for cross-interpreter checks.

Alternatives

Package	Registry	Pick it when
pytest-split	PyPI	You want to shard across several CI machines by recorded test duration rather than across cores on one machine
pytest-testmon	PyPI	The goal is a faster feedback loop locally, and running only the tests affected by your change beats running all of them in parallel
pytest-forked	PyPI	You only want each test in its own forked process for crash isolation, with no distribution or scheduling involved