hypothesis
Hypothesis is property-based testing for Python. Instead of writing a test with three example inputs you picked by hand, you describe the shape of all valid inputs with a strategy (st.integers(), st.text(), st.lists(st.floats())) and assert something that should hold for every one of them. Hypothesis then generates around a hundred inputs per run, biased toward the values that break code: zero, empty strings, NaN, surrogate pairs, huge lists. When it finds a failure it does the part that makes the whole approach worth it: it shrinks the input to the smallest example that still fails, so instead of a 400-element list you get [0, 0]. It saves that failing example in a local database and replays it first on every later run, which turns a random discovery into a permanent regression test.
The best testing tool most Python teams are not using, and the shrinking is what makes it practical rather than academic. Reach for it on pure functions with clear invariants, keep it away from tests that talk to a database, and put it in CI with its own settings profile.
Use it if
- You have code with a round trip: encode and decode, serialize and parse, compress and decompress, or a to_dict and from_dict pair, where the property writes itself as decode(encode(x)) == x
- You are rewriting or optimizing a function and can keep the old one around, since asserting new(x) == old(x) over generated inputs catches differences no hand-written case would
- You are handling messy input domains where edge cases are the bugs: unicode normalization, float rounding, timezone arithmetic, date parsing, or anything that accepts user-supplied strings
- You have a stateful component such as a cache, a queue or a connection pool, where hypothesis.stateful can generate sequences of operations and check invariants after each step
- You cannot state a property: if the only thing you can assert is that a specific input produces a specific output, an ordinary parametrized pytest case is clearer and 100 times faster to run
- The code under test touches a database, network or filesystem: every test body runs around 100 times, the default 200ms per-example deadline will fire, and function-scoped pytest fixtures are not recreated between examples, so state leaks from one generated case into the next
- Your team treats a red build as a broken build: a Hypothesis test that passed for six months can fail today because it finally generated the right input, which is correct behavior but needs agreement before it lands in CI
- You need reproducible CI failures without extra work: the example database lives in a local .hypothesis directory that CI throws away, so reproducing a CI failure means copying the @reproduce_failure blob or wiring up a shared database
- You mainly want realistic-looking fixture data such as names, addresses and emails: Faker or factory_boy are built for that, while Hypothesis deliberately generates hostile data instead
Setup reality
pip install hypothesis needs Python 3.10 or newer and pulls only sortedcontainers, and the pytest plugin registers itself through an entry point so there is nothing to add to your config. The first thing that surprises people is the .hypothesis directory it creates in your working directory to store failing examples; add it to .gitignore, and remember that it is per-machine, so a colleague or a CI runner does not inherit your regression cases. The second is the deadline: any single example slower than 200ms raises DeadlineExceeded, which usually fires on the first call because of one-time import or connection cost rather than a real performance problem, and the fix is settings(deadline=None) or a warmup. The third is fixtures: pytest function-scoped fixtures run once for the whole test, not once per generated example, and Hypothesis emits the function_scoped_fixture health check to tell you so; suppressing that check without understanding it is how a database test starts passing for the wrong reason. Extras are installed as hypothesis[cli] for the ghostwriter command, plus numpy, pandas, django, redis, lark, dateutil, pytz and crosshair for the corresponding integrations. In CI, register a profile in conftest.py with more examples and no deadline, and use --hypothesis-show-statistics when a test is mysteriously slow or discarding most of its inputs.
Patterns
Write your first property testgiven-basics
from hypothesis import given, strategies as st
@given(st.lists(st.integers()))
def test_sort_is_idempotent(xs):
once = sorted(xs)
assert sorted(once) == once
@given(st.text(), st.text())
def test_concat_length(a, b):
assert len(a + b) == len(a) + len(b)Each test runs about 100 generated inputs by default. The decorated function is still a normal function, so pytest and unittest collect it as usual, and you can call it with no arguments to run the whole generation loop yourself.
Build a strategy for your own typecompose-strategies
from dataclasses import dataclass
from hypothesis import given, strategies as st
@dataclass
class Order:
id: int
items: list[str]
total_cents: int
# simple case: builds() fills the constructor
orders = st.builds(
Order,
id=st.integers(min_value=1),
items=st.lists(st.text(min_size=1), min_size=1),
total_cents=st.integers(min_value=0, max_value=10**6),
)
# when fields depend on each other, use @composite
@st.composite
def consistent_orders(draw):
items = draw(st.lists(st.text(min_size=1), min_size=1, max_size=5))
prices = draw(st.lists(st.integers(0, 5000), min_size=len(items), max_size=len(items)))
return Order(id=draw(st.integers(min_value=1)), items=items, total_cents=sum(prices))
@given(consistent_orders())
def test_total_matches_items(order):
assert order.total_cents >= 0Inside @composite the first parameter is the draw function and callers never pass it. Keep draws unconditional where you can: branching heavily on drawn values makes shrinking slower and the reported example less minimal.
Generate values from type annotationsinfer-from-types
from hypothesis import given, strategies as st
@given(st.from_type(Order)) # reads the dataclass annotations
def test_roundtrip(order):
assert Order(**order.__dict__) == order
# teach it about a type it cannot infer
class UserId(str):
pass
st.register_type_strategy(
UserId,
st.uuids().map(lambda u: UserId(str(u))),
)from_type resolves dataclasses, NamedTuples, TypedDicts and most typing constructs, and falls back to builds() on the constructor signature. register_type_strategy is global and process-wide, so put it in conftest.py rather than inside a test module that may not be imported.
Discard inputs you do not want to testfilter-and-assume
from hypothesis import assume, given, strategies as st
# preferred: constrain the strategy itself
@given(st.integers(min_value=1))
def test_reciprocal(n):
assert 0 < 1 / n <= 1
# next best: filter on the strategy
@given(st.integers().filter(lambda n: n % 2 == 0))
def test_even(n):
assert n % 2 == 0
# last resort: assume() inside the body
@given(st.integers(), st.integers())
def test_distinct(a, b):
assume(a != b)
assert a - b != 0Every discarded example is wasted work, and if too many get thrown away the filter_too_much health check fails the test. Narrowing the strategy with min_value or a map is always cheaper than filtering, and filtering is cheaper than assume().
Tune example count and deadlines per environmentsettings-and-profiles
# conftest.py
import os
from hypothesis import HealthCheck, Phase, settings
settings.register_profile("dev", max_examples=25)
settings.register_profile(
"ci",
max_examples=1000,
deadline=None,
derandomize=True,
suppress_health_check=[HealthCheck.too_slow],
)
settings.load_profile("ci" if os.getenv("CI") else "dev")
# per test
@settings(max_examples=500, phases=[Phase.generate, Phase.shrink])
@given(st.text())
def test_parser(s):
...The default deadline is 200 milliseconds per example, which the first example often blows through on import or connection setup rather than on real slowness. derandomize=True makes CI runs deterministic by deriving the seed from the test rather than the clock, at the cost of no longer finding new inputs over time.
Pin regression cases alongside generated onesexplicit-examples
from hypothesis import example, given, strategies as st
@given(st.text())
@example("")
@example("\u0000")
@example("e\u0301") # combining accent, the bug from ticket 812
def test_slugify(s):
assert " " not in slugify(s)Explicit examples run first, before any generated input, so a known regression fails fast. They are also the honest way to keep a bug Hypothesis once found, since the .hypothesis database is local and will not survive a fresh checkout.
Reproduce a failure that only happened in CIreproduce-a-failure
from hypothesis import reproduce_failure, seed, given, strategies as st
# paste the blob Hypothesis printed in the CI log
@reproduce_failure("6.165.2", b"AXicY2BkAAAAEwAD")
@given(st.integers())
def test_thing(n):
...
# or fix the seed for a whole test
@seed(1234)
@given(st.integers())
def test_thing_seeded(n):
...The blob is version-specific and the decorator errors if the installed Hypothesis differs, which is deliberate: an old blob does not describe the same choices in a newer generator. Both decorators are debugging aids and should be removed before merging, since they pin the test to a single input.
Handle the function-scoped fixture health checkpytest-fixtures
import pytest
from hypothesis import HealthCheck, given, settings, strategies as st
@pytest.fixture
def db():
conn = connect()
yield conn
conn.rollback()
# this fixture runs ONCE for the whole test, not once per example
@settings(suppress_health_check=[HealthCheck.function_scoped_fixture], deadline=None)
@given(st.text())
def test_insert(db, s):
db.execute("INSERT INTO t (v) VALUES (?)", (s,))
db.rollback() # reset state yourself, per exampleSuppressing the health check does not change the behavior it warns about: the fixture body still runs once, so anything stateful leaks between generated examples and a test can pass only because of an earlier example. Reset inside the test body, or keep database work in ordinary example-based tests.
Generate sequences of operations against a state machinestateful-testing
from hypothesis import strategies as st
from hypothesis.stateful import Bundle, RuleBasedStateMachine, invariant, rule
class CacheMachine(RuleBasedStateMachine):
keys = Bundle("keys")
def __init__(self):
super().__init__()
self.cache = LRUCache(maxsize=3)
self.model = {}
@rule(target=keys, k=st.text(min_size=1), v=st.integers())
def put(self, k, v):
self.cache.put(k, v)
self.model[k] = v
return k
@rule(k=keys)
def get(self, k):
if k in self.cache:
assert self.cache.get(k) == self.model[k]
@invariant()
def never_over_capacity(self):
assert len(self.cache) <= 3
TestCache = CacheMachine.TestCaseAssign the generated TestCase to a module-level name or pytest will not collect it. Bundles feed values produced by one rule into another, and invariants are checked after every step; stateful_step_count (default 50) caps how long each generated sequence gets.
Draw more values in the middle of a testinteractive-draws
from hypothesis import given, note, strategies as st
@given(st.data())
def test_insert_then_query(data):
table = data.draw(st.lists(st.integers(), min_size=1), label="rows")
index = data.draw(st.integers(0, len(table) - 1), label="index")
note(f"looking up position {index} in {table}")
assert lookup(table, index) == table[index]st.data() is the escape hatch for when a later value depends on an earlier one and @composite does not fit. Pass label= so the failure output tells you which draw produced which value, and use note() to attach extra context that is printed only on failure.
Generate numpy arrays and dataframesnumpy-arrays
import numpy as np
from hypothesis import given
from hypothesis.extra.numpy import array_shapes, arrays, floating_dtypes
@given(arrays(
dtype=np.float64,
shape=array_shapes(min_dims=1, max_dims=2, max_side=8),
))
def test_normalize_keeps_shape(a):
assert normalize(a).shape == a.shapeThe default element strategy for a float dtype includes NaN and both infinities, which is exactly the point and also why a naive test fails immediately; pass elements=st.floats(allow_nan=False, allow_infinity=False) when the function genuinely does not accept them. hypothesis.extra.pandas offers the same for series, columns and dataframes.
Have Hypothesis draft the test for youghostwriter
pip install 'hypothesis[cli]'
hypothesis write gzip.compress
hypothesis write --roundtrip json.dumps json.loads
hypothesis write --equivalent mymodule.old_parse mymodule.new_parse
hypothesis write mypackage.utils > tests/test_utils_generated.pyThe ghostwriter reads type hints and signatures and prints a test module to stdout, which you then edit; it guesses the property, so treat the output as a first draft rather than a finished test. The roundtrip and equivalent modes are the two that most often produce something worth keeping.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| faker | PyPI | You want plausible fake data for fixtures and demos rather than adversarial inputs designed to break your parser. |
| schemathesis | PyPI | The thing you want to fuzz is an HTTP API with an OpenAPI or GraphQL schema; it is built on Hypothesis and derives the strategies for you. |
| crosshair-tool | PyPI | You want symbolic execution that reasons about all paths instead of sampling inputs; it also plugs in as a Hypothesis backend. |
| pytest | PyPI | Your cases are genuinely example-based; parametrize is simpler, faster and easier for a reviewer to read. |