deltalake review
deltalake 1.6.3 is the Python binding for delta-rs, a Rust implementation of the Delta Lake transaction protocol. It opens and writes Parquet-backed Delta tables without launching Spark or a JVM. The Python surface covers Arrow and pandas reads, append and overwrite writes, merge, delete, update, time travel, change data feed, compaction, Z-ordering, vacuum, constraints, and object-store access. Release 1.6.3 adds lazy snapshot replay, V2 checkpoint reading and writing, change-feed support for deletion vectors, and custom Unity Catalog tokens. It also fixes deletes on string columns and optimize failures on Spark-written tables.
deltalake 1.6.3 installed in 0.9 seconds and imported in 0.20 seconds in our sandbox, but its 4 installed packages occupied 140 MB because the package carries a native storage engine. It is a good fit for Delta Lake work without Spark when one process, protocol feature checks, object-store credentials, and file maintenance are acceptable.
We installed it
| Install | ✓ · 0.9s | 4 packages on disk · 140 MB · 1 deprecation warning |
| Import | ✓ | import deltalake in 0.20s · compiled extensions · py.typed · requires Python >=3.10 |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does deltalake install cleanly?
Yes. In a fresh container with an empty cache, pip install deltalake finished in 0.9s, leaving 4 packages and 140 MB on disk. pip-audit reported no known vulnerabilities. The install printed 1 deprecation warning.
What does deltalake need to run?
Python >=3.10, and a platform wheel with compiled extensions. In our run import deltalake succeeded in 0.20s, and the package ships py.typed for type checkers.
deltalake or delta-spark: which should you use?
delta-spark: Use it when Spark's distributed scheduler and JVM Delta implementation are already operational requirements. deltalake 1.6.3 installed in 0.9 seconds and imported in 0.20 seconds in our sandbox, but its 4 installed packages occupied 140 MB because the package carries a native storage engine.
When should you not use deltalake?
Immutable local files are enough. PyArrow or DuckDB can query Parquet without a Delta log, checkpoints, protocol versions, or vacuum policy.
Use it if
- A Python service needs transactional Delta Lake reads and writes without a Spark cluster.
- Your data already moves through Arrow, pandas, Polars, DuckDB, Dask, or DataFusion.
- The job needs merge, time travel, change data feed, compaction, or vacuum from one process.
- An embedded native engine fits deployment better than submitting work to a JVM service.
- Immutable local files are enough. PyArrow or DuckDB can query Parquet without a Delta log, checkpoints, protocol versions, or vacuum policy.
- Your tables use Delta features absent from delta-rs. The project's feature table is the authority; format compatibility does not mean every Databricks or Spark feature is implemented.
- The workload needs distributed execution across a cluster. delta-rs runs Arrow and DataFusion work inside one process, while delta-spark uses Spark executors.
- Your target cannot use the published compiled extension. Version 1.6.3 ships native code, so platform and architecture support must be checked before deployment.
- Writers create many tiny files and nobody can own compaction or retention. Append, optimize, and vacuum choices directly affect read cost and historical snapshots.
Setup reality
We installed deltalake 1.6.3 in 0.9 seconds in a clean Python 3.12 container. Four packages occupied 140 MB, and the installer printed one deprecation warning. pip-audit found 0 known vulnerabilities. The distribution has 4 direct dependencies, includes a compiled extension and py.typed, and requires Python 3.10 or later. import deltalake succeeded in 0.20 seconds.
PyArrow and pandas are optional extras, despite being central to many examples. Install deltalake[pyarrow,pandas] when those conversion methods are part of the job; the current PyArrow extra starts at version 21. Cloud tables need backend credentials plus access to both Parquet objects and _delta_log. Supply secrets through environment, workload identity, or storage_options and keep storage_options out of logs.
A DeltaTable holds one snapshot. It will not notice another writer's commit until update_incremental runs or the table is reopened. pandas conversion materializes rows in memory; use Arrow datasets or batches with selected columns and filters when 140 MB of installed code is the smaller part of the workload. Version 1.6.3 introduces lazy snapshot materialization, but callers still need to bound scan output.
Concurrent commits can conflict when merge, delete, update, or optimize rewrites overlapping files. Repeated appends create small files, so schedule optimize.compact or Z-order only after measuring the read pattern. Vacuum is a preview by default; dry_run=False physically removes old files and can break time travel or readers pinned to earlier versions. Release 1.6.3 also changes CDF behavior to respect in-commit timestamps and adds deletion-vector handling, which deserves an integration test before upgrading a CDC consumer.
Patterns
Write a partitioned Delta table create-table
import pyarrow as pa
from deltalake import write_deltalake
batch = pa.table({
'event_date': ['2026-08-25', '2026-08-25'],
'user_id': [1, 2],
'amount': [12.5, 9.0],
})
write_deltalake(table_uri, batch, partition_by=['event_date'])The default mode rejects an existing table. Partitioning by a high-cardinality field can turn each write into many small files.
Append with explicit schema evolution append-schema
write_deltalake(
table_uri,
new_batch,
mode='append',
schema_mode='merge',
storage_options=storage_options,
)schema_mode='merge' can add compatible fields; it does not bypass existing types, nullability, constraints, or protocol features.
Replace one slice of a table overwrite-predicate
write_deltalake(
table_uri,
replacement,
mode='overwrite',
predicate="event_date = '2026-08-25'",
storage_options=storage_options,
)Every input row must satisfy the predicate. Test that failure path before using this for production partition replacement.
Read filtered Arrow batches scan-arrow-batches
import pyarrow.dataset as ds
from deltalake import DeltaTable
dt = DeltaTable(table_uri, storage_options=storage_options)
for batch in dt.to_pyarrow_dataset().to_batches(
columns=['user_id', 'amount'],
filter=ds.field('event_date') == '2026-08-25',
batch_size=50_000,
):
process(batch)This path needs the pyarrow extra. Columns, partition filters, and file statistics reduce materialization compared with to_pandas().
Pick up another writer's commit refresh-snapshot
dt = DeltaTable(table_uri, storage_options=storage_options)
current = dt.version()
dt.update_incremental()
if dt.version() != current:
reload_derived_state(dt)A DeltaTable remains on its loaded snapshot until it is updated or reopened; later commits do not appear automatically.
Load a table version time-travel
from deltalake import DeltaTable
dt = DeltaTable(table_uri, version=42, storage_options=storage_options)
print(dt.version(), dt.schema())Version 42 can be opened only while its required log and data files remain. Vacuum may remove the files behind an older snapshot.
Read recent commits inspect-history
dt = DeltaTable(table_uri, storage_options=storage_options)
for commit in dt.history(limit=10):
print(commit['version'], commit.get('operation'), commit.get('timestamp'))History is newest first. Optional metadata comes from writers and may be absent, so use get() for those fields.
Upsert an Arrow source merge-upsert
metrics = (
dt.merge(
source=updates,
predicate='target.id = source.id',
source_alias='source',
target_alias='target',
)
.when_matched_update_all()
.when_not_matched_insert_all()
.execute()
)The all-column helpers expect source and target names to line up. Merge can rewrite every touched Parquet file.
Delete rows with a SQL predicate delete-rows
metrics = dt.delete(
predicate="status = 'expired' AND created_at < '2025-01-01'"
)Leaving predicate unset deletes all rows. Version 1.6.3 fixes delete handling for string columns, so test that case when upgrading.
Compact one partition compact-files
metrics = dt.optimize.compact(
partition_filters=[('event_date', '=', '2026-08-25')]
)Compaction rewrites files and can conflict with overlapping write operations. Run it after measuring file counts and query cost.
Reorder files for a common filter z-order-files
metrics = dt.optimize.z_order(
['user_id'],
partition_filters=[('event_date', '=', '2026-08-25')],
)Z-order includes file rewriting and compaction. More worker concurrency also means more process memory.
Review then remove obsolete files vacuum-files
candidates = dt.vacuum(retention_hours=168)
review(candidates)
if approved(candidates):
deleted = dt.vacuum(retention_hours=168, dry_run=False)The first call is a dry run. Physical deletion can break time travel and any reader still holding an earlier table snapshot.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| delta-spark | PyPI | Use it when Spark's distributed scheduler and JVM Delta implementation are already operational requirements. |
| pyiceberg | PyPI | Use it when the table standard is Apache Iceberg and catalog integration matters more than Delta compatibility. |
| duckdb | PyPI | Use it for embedded analytics over Parquet when transactional lake writes are unnecessary. |
| pyarrow | PyPI | Use it for Arrow memory and Parquet I/O without Delta commits, checkpoints, or maintenance operations. |
More data guides
numpy · fsspec · pandas · sqlalchemy · pyarrow · lxml · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.