deltalake
deltalake is the Python binding for delta-rs, a native Rust implementation of the Delta Lake transaction-log protocol. It reads and writes ACID tables over Parquet on local disks, S3-compatible stores, Azure, GCS, HDFS, and other backends without starting Spark or a JVM. The Python API exposes Arrow, pandas, DataFusion SQL, time travel, change data feed, merge, update, delete, schema evolution, compaction, Z-ordering, vacuum, constraints, and table metadata. It is a storage-format engine, not a hosted catalog or distributed compute service.
The strongest way to use Delta Lake from Python without Spark, with a surprisingly complete operations surface. It is still a native storage engine that makes you own protocol compatibility, object-store credentials, file maintenance, and single-process resource limits.
Use it if
- A Python or Rust service must read and write Delta Lake tables without carrying Spark and a JVM
- Your pipeline works naturally with Arrow, pandas, Polars, DuckDB, Dask, or DataFusion and needs transactional object-store data
- You need Delta operations such as merge, change data feed, time travel, schema enforcement, compaction, or vacuum in a single-node process
- An embedded native engine is a better operational fit than submitting jobs to a separate Spark cluster
- You only need local analytics over immutable files: plain Parquet with PyArrow or DuckDB avoids a transaction log, checkpoints, vacuum policy, protocol versions, and object-store commit concerns
- Your tables depend on every Databricks or Spark Delta feature: the project's feature table marks Identity Columns unsupported, and the Python source declares a finite set of reader and writer features rather than universal protocol support
- You need distributed execution across a large cluster: delta-rs embeds DataFusion and Arrow in one process, while delta-spark supplies Spark's scheduler and executor model
- Your deployment cannot accept native wheels around 45 to 53 MB or is on an unlisted platform: PyPI 1.6.2 publishes compiled wheels for selected CPython 3.10+ macOS, Linux, musl, and Windows targets, with no source distribution
- Your workload produces constant tiny appends but has no maintenance budget: the docs prescribe compaction for small files and vacuum for obsolete files, and both operations consume compute and require retention decisions
Setup reality
The core package requires Python 3.10 or newer and installs a compiled Rust extension plus arro3-core. Current PyPI artifacts are platform wheels roughly 45 to 53 MB; there is no source distribution, so unsupported architectures need a Rust build from the repository rather than a normal pip fallback. PyArrow and pandas are optional extras. Install deltalake[pyarrow,pandas] when using to_pyarrow_table, datasets, or to_pandas; current metadata requires PyArrow 21 or newer for that extra. The first write can accept Arrow-compatible data, but schema, nullability, timestamp precision, partition columns, and append compatibility still need deliberate tests. Remote tables add object-store configuration. S3, Azure, GCS, HDFS, MinIO, R2, OneLake, and LakeFS are supported, yet credentials and endpoint flags vary by backend. The docs explicitly say delta-rs does not read local .aws/config or .aws/creds files; use environment credentials, metadata, profiles or web identity, or storage_options, and never log the latter. Object-store permissions must cover both data files and _delta_log commits. Delta protocol compatibility is per feature, not just a file extension, so inspect dt.protocol() before adopting tables written by newer engines. A DeltaTable represents one loaded snapshot. Call update_incremental or reopen it to see later commits from another writer. Reading into pandas materializes data; use a PyArrow dataset, batches, DataFusion QueryBuilder, columns, filters, and partition pruning for larger tables. Appends and row updates create more Parquet files while overwrites logically remove old ones. Run optimize.compact or z_order to address small files. vacuum defaults to a dry run and the normal retention window is one week; setting dry_run=False physically removes files and can make time travel impossible. Concurrent writers, retention, checkpoints, catalogs, credentials, storage costs, and observability remain your operations problem.
Patterns
Create a partitioned table from Arrow datacreate-delta-table
import pyarrow as pa
from deltalake import write_deltalake
data = pa.table({
'event_date': ['2026-08-08', '2026-08-08'],
'user_id': [1, 2],
'amount': [12.5, 9.0],
})
write_deltalake(
's3://analytics/events',
data,
partition_by=['event_date'],
storage_options=storage_options,
)The default mode is error when the table already exists. Choose partitions with care; high-cardinality partition columns create many small directories and files.
Append rows and add compatible columnsappend-with-schema-merge
write_deltalake(
table_uri,
new_batch,
mode='append',
schema_mode='merge',
storage_options=storage_options,
)Schema merge is explicit. Existing types, nullability, constraints, and table protocol still govern whether the append can commit.
Replace only rows matching a predicateoverwrite-matching-partition
write_deltalake(
table_uri,
replacement,
mode='overwrite',
predicate="event_date = '2026-08-08'",
storage_options=storage_options,
)Predicate overwrite is safer than a full overwrite only when every input row satisfies the predicate. Test failure behavior before using it in a partition replacement job.
Read selected rows without materializing the tableread-filtered-columns
import pyarrow.dataset as ds
from deltalake import DeltaTable
dt = DeltaTable(table_uri, storage_options=storage_options)
dataset = dt.to_pyarrow_dataset()
for batch in dataset.to_batches(
columns=['user_id', 'amount'],
filter=(ds.field('event_date') == '2026-08-08') & (ds.field('amount') > 10),
batch_size=50_000,
):
process(batch)Install deltalake[pyarrow]. Dataset filters use partitions and file statistics for pruning, then push remaining filters into the scan.
Run SQL through the embedded query enginequery-with-datafusion
from deltalake import DeltaTable, QueryBuilder
dt = DeltaTable(table_uri, storage_options=storage_options)
reader = (
QueryBuilder()
.register('events', dt)
.execute('SELECT user_id, sum(amount) total FROM events GROUP BY user_id')
)
result = reader.read_all()QueryBuilder uses embedded DataFusion and returns an Arrow RecordBatchReader. It is single-process analytics, not a distributed SQL service.
Load a historical table snapshottime-travel-table
from datetime import datetime, timezone
from deltalake import DeltaTable
dt = DeltaTable(table_uri, storage_options=storage_options)
dt.load_as_version(12)
old_schema = dt.schema()
dt.load_as_version(datetime(2026, 8, 1, tzinfo=timezone.utc))Historical snapshots fail after required data or log files have been vacuumed. A DeltaTable stays on the loaded snapshot until explicitly updated.
Inspect recent Delta commitsinspect-commit-history
dt = DeltaTable(table_uri, storage_options=storage_options)
for commit in dt.history(limit=10):
print(commit['version'], commit.get('operation'), commit.get('timestamp'))History is returned newest first. Commit metadata is user and engine supplied, so tolerate absent optional fields.
Upsert an Arrow table with MERGEmerge-upsert-rows
metrics = (
dt.merge(
source=updates,
predicate='target.id = source.id',
source_alias='source',
target_alias='target',
)
.when_matched_update_all()
.when_not_matched_insert_all()
.execute()
)
print(metrics)The source and target field names must align for the all-column helpers. MERGE commits in one transaction but may rewrite affected files.
Apply SQL predicates to updates and deletesupdate-and-delete-rows
update_metrics = dt.update(
predicate="status = 'pending' AND created_at < '2026-01-01'",
new_values={'status': 'expired'},
)
delete_metrics = dt.delete(
predicate="status = 'expired' AND created_at < '2025-01-01'"
)Omitting the delete predicate deletes every row. Both operations can rewrite entire files that contain matching records, not only isolated row bytes.
Compact files and Z-order common filterscompact-small-files
compact_metrics = dt.optimize.compact(
partition_filters=[('event_date', '=', '2026-08-08')]
)
zorder_metrics = dt.optimize.z_order(
['user_id'],
partition_filters=[('event_date', '=', '2026-08-08')],
)Z-order also compacts and rewrites file order. More concurrency increases memory use, and concurrent non-append operations can make optimization fail.
Preview and execute vacuumvacuum-obsolete-files
candidates = dt.vacuum(retention_hours=168)
review(candidates)
if approved(candidates):
deleted = dt.vacuum(retention_hours=168, dry_run=False)Vacuum is a dry run by default. Physical deletion can break time travel and readers still using older snapshots; do not disable retention enforcement casually.
Read change data between versionsstream-change-data-feed
dt = DeltaTable(table_uri, storage_options=storage_options)
changes = dt.load_cdf(
starting_version=20,
ending_version=25,
columns=['id', 'status', '_change_type', '_commit_version'],
predicate="status <> 'unchanged'",
)
for batch in changes:
process_changes(batch)The table must have delta.enableChangeDataFeed=true before the relevant commits. CDF cannot reconstruct changes from before enablement or after required history is removed.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| delta-spark | PyPI | A Spark cluster and full JVM Delta ecosystem are already the execution and compatibility standard |
| pyiceberg | PyPI | The organization standardizes on Apache Iceberg catalogs and wants a Python-native client for that table format |
| duckdb | PyPI | Local or embedded analytics over Parquet matters more than transactional lakehouse writes |
| pyarrow | PyPI | You need columnar memory and Parquet I/O without a Delta transaction layer |