great-expectations
A data quality framework built around Expectations, which are declarative assertions about a dataset such as this column has no nulls or this value stays between two bounds. You register a Data Source, define a Batch Definition that says which slice of data to look at, collect Expectations into a Suite, wrap that in a Validation Definition, and run one or more of those through a Checkpoint that can trigger Actions on the result. It also renders Data Docs, static HTML reports of what was checked and what failed. It is a testing framework for data rather than a schema validator, and the ceremony exists so the same suite can run against pandas, Spark and a dozen SQL backends.
The most complete data quality framework in Python, and worth its weight when checks must be shared, reported on and run across several backends. For a single pandas pipeline the six-object setup and the dependency tree cost more than they return, and pandera is the better fit.
Use it if
- You need data quality checks that produce a durable, shareable artefact rather than an assertion that only lives in a log line, which is what Data Docs and Checkpoint results give you
- The same checks have to run against more than one backend, since a suite written once works across pandas, Spark and SQLAlchemy-supported warehouses
- You want a large library of ready-made assertions instead of writing them: dozens of ExpectColumn checks cover nulls, sets, ranges, regexes, quantiles, distributions and multi-column relationships
- You are wiring checks into an orchestrator, where a Checkpoint with Actions is a natural unit for Airflow, Dagster or Prefect to call
- You want a light dependency. The base install pulls altair, scipy, marshmallow, mistune, pydantic, ruamel.yaml, jinja2, tqdm, tzlocal, numpy and pandas before you have added a single backend extra, which is a lot to accept for checking that a column is not null
- You are following older material. Version 1.0 replaced the 0.x API wholesale, so anything mentioning great_expectations init, DataContext v2 or v3 checkpoint YAML, or expectation suite JSON edited by hand, will not run. The volume of stale tutorials, blog posts and Stack Overflow answers is genuinely a cost
- You only need to validate a DataFrame's schema and dtypes in code. pandera does that in a few lines with type-checked models and no Data Context on disk
- You need a stable minor version to pin against. There have been 364 releases, with 1.18.x, 1.19.x and 1.20.0 all landing between June and August 2026, so tracking the latest means regular upgrade work
- Governance matters to your dependency choices: the GitHub repository now resolves to fivetran/great_expectations, so the project sits inside a commercial vendor that also sells GX Cloud, and a cloud extra ships in the package. The core stays Apache-2.0, but the roadmap is not community-owned
- Your Python is outside 3.10 through 3.13. Python 3.14 works only behind the GX_PYTHON_EXPERIMENTAL environment variable at install time
Setup reality
pip install great_expectations is one line and a large tree, and you almost always need an extra as well (postgresql, snowflake, bigquery, spark, athena and about a dozen others), each of which pins its own SQLAlchemy range; the redshift, teradata and clickhouse extras still pin SQLAlchemy below 2.0, which will fight anything else in the environment that has moved on. The first architectural decision is the Data Context: gx.get_context() gives you an ephemeral in-memory context that vanishes when the process exits, gx.get_context(mode='file') writes a gx/ directory you commit, and there is a cloud mode too. Getting this wrong is why suites disappear between runs. After that the object chain is fixed and each link must exist before the next: Data Source, then Data Asset, then Batch Definition, then Suite, then Validation Definition, then Checkpoint. For in-memory pandas the dataframe is not attached until run time, so you pass batch_parameters={'dataframe': df} to the run call rather than to the asset. Names are identifiers: adding a suite with a name that already exists raises rather than replacing, so idempotent setup scripts need add_or_update or a try around it. Validation against a SQL backend issues real queries, and an expectation over a large table without a partitioner will scan it. Data Docs are static HTML written to disk and are not built unless an UpdateDataDocsAction is in the checkpoint. Expect the first working pipeline to take longer than the documentation suggests, mostly spent discovering which of the six objects you skipped.
Patterns
Choose a Data Contextcreate-context
import great_expectations as gx
# in-memory, discarded when the process ends
context = gx.get_context()
# persisted to a gx/ directory you can commit
context = gx.get_context(mode='file')The default is ephemeral. Suites and checkpoints created against it are gone on the next run, which is the most common reason a setup script appears to do nothing.
Check a DataFrame with one expectationvalidate-dataframe-quickly
import great_expectations as gx
import great_expectations.expectations as gxe
import pandas as pd
df = pd.read_csv('orders.csv')
context = gx.get_context()
batch = (
context.data_sources.add_pandas('local')
.add_dataframe_asset('orders')
.add_batch_definition_whole_dataframe('all')
.get_batch(batch_parameters={'dataframe': df})
)
result = batch.validate(gxe.ExpectColumnValuesToNotBeNull(column='order_id'))
print(result.success)This is the shortest honest path from a DataFrame to a result. The dataframe is passed at get_batch time, not when the asset is created.
Group expectations into a suitebuild-expectation-suite
suite = context.suites.add(gx.ExpectationSuite(name='orders_daily'))
suite.add_expectation(gxe.ExpectColumnValuesToNotBeNull(column='order_id'))
suite.add_expectation(gxe.ExpectColumnValuesToBeUnique(column='order_id'))
suite.add_expectation(
gxe.ExpectColumnValuesToBeBetween(column='amount', min_value=0, max_value=100000)
)
suite.add_expectation(
gxe.ExpectColumnValuesToBeInSet(column='status', value_set=['new', 'paid', 'refunded'])
)context.suites.add raises if the name already exists rather than replacing it. Setup scripts that run twice need add_or_update or a delete first.
Wire a Validation Definition into a Checkpointrun-checkpoint
batch_definition = (
context.data_sources.add_pandas('local')
.add_dataframe_asset('orders')
.add_batch_definition_whole_dataframe('all')
)
validation = context.validation_definitions.add(
gx.ValidationDefinition(data=batch_definition, suite=suite, name='orders_daily_check')
)
checkpoint = context.checkpoints.add(
gx.Checkpoint(name='orders_daily_cp', validation_definitions=[validation])
)
result = checkpoint.run(batch_parameters={'dataframe': df})
print(result.success)The chain is Data Source, Asset, Batch Definition, Suite, Validation Definition, Checkpoint, and every link must be created before the next. Skipping one is the usual source of a confusing error.
Validate a warehouse tableconnect-sql-database
data_source = context.data_sources.add_postgres(
name='warehouse',
connection_string='postgresql+psycopg2://user:pass@host:5432/analytics',
)
asset = data_source.add_table_asset(name='orders', table_name='public.orders')
batch_definition = asset.add_batch_definition_whole_table('full_table')Needs the matching extra, here great_expectations[postgresql]. Each expectation runs real SQL, so a whole-table definition on a large table means a full scan per check.
Validate only one day of a tablepartition-by-date
daily = asset.add_batch_definition_daily(
name='by_order_date', column='order_date'
)
batch = daily.get_batch(
batch_parameters={'year': 2026, 'month': 8, 'day': 7}
)Partitioned definitions are how you keep validation cheap on a large table. Without batch_parameters you get the most recent partition, not the whole table.
Validate the result of a queryquery-based-asset
asset = data_source.add_query_asset(
name='recent_paid_orders',
query="SELECT * FROM public.orders WHERE status = 'paid' AND order_date >= CURRENT_DATE - 7",
)
batch_definition = asset.add_batch_definition_whole_table('last_7_days')Query assets let you check a joined or filtered view without creating a database view. The query is re-run for every expectation in the suite.
Get the rows that failedread-failed-rows
from great_expectations import ResultFormat
result = batch.validate(
gxe.ExpectColumnValuesToBeInSet(column='status', value_set=['new', 'paid']),
result_format=ResultFormat.COMPLETE,
)
print(result.result['partial_unexpected_list'])The default result format summarises. COMPLETE returns every unexpected value, which is what you want when debugging and not what you want in a log line on a large table.
Write the HTML reportgenerate-data-docs
from great_expectations.checkpoint import UpdateDataDocsAction
checkpoint = context.checkpoints.add(
gx.Checkpoint(
name='orders_daily_cp',
validation_definitions=[validation],
actions=[UpdateDataDocsAction(name='update_docs')],
)
)
checkpoint.run(batch_parameters={'dataframe': df})
context.open_data_docs()Data Docs are not built unless this action is on the checkpoint. They are static HTML on disk, so serving them to a team means copying the directory somewhere.
Reuse a suite with runtime thresholdsparameterise-expectations
suite.add_expectation(
gxe.ExpectColumnValuesToBeBetween(
column='amount', min_value=0, max_value={'$PARAMETER': 'max_amount'}
)
)
result = batch.validate(suite, expectation_parameters={'max_amount': 250000})The $PARAMETER syntax keeps one suite usable across environments. Forgetting to supply a parameter at run time fails the validation rather than skipping the check.
Stop an orchestrated job when checks failfail-pipeline-on-error
result = checkpoint.run(batch_parameters={'dataframe': df})
if not result.success:
failures = [
r.expectation_config.type
for run in result.run_results.values()
for r in run.results
if not r.success
]
raise ValueError(f'data quality failed: {failures}')checkpoint.run returns a result object; it does not raise. Nothing stops your pipeline unless you check result.success yourself.
Keep suites and checkpoints across runspersist-configuration
context = gx.get_context(mode='file', project_root_dir='.')
# later, in a different process
context = gx.get_context(mode='file', project_root_dir='.')
checkpoint = context.checkpoints.get('orders_daily_cp')
checkpoint.run(batch_parameters={'dataframe': df})File mode writes a gx/ directory holding the definitions. Commit it, but keep connection strings in environment variables rather than in the config files it generates.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| pandera | PyPI | You want DataFrame schema and value checks declared in Python next to the code, with type checking and no persistent context |
| soda-core | PyPI | You prefer checks written in a small YAML language run by a CLI, so analysts can edit them without touching Python |
| dbt-core | PyPI | Your data already lives in a warehouse and tests belong next to the models that build it, in SQL |