PyPIDataupdated 05 Aug 2026

pyarrow

Item: pyarrow
Rating: 4.5
Author: Keyoor

PyArrow is the Python binding for Apache Arrow, the columnar in-memory format that most of the data ecosystem now speaks. In practice it is three things: the fastest mainstream Parquet and CSV reader/writer for Python, a Table structure that moves between pandas, Polars, DuckDB, and Spark with zero or minimal copying, and a dataset API that scans folders of Parquet files with filter and column pushdown so you never load what you do not need. If you touch Parquet from Python, you are almost certainly running this code, directly or under pandas.

pyarrow · v25.0.0 · 101.0M/wk · ★ 16,987 · pushed 05 Aug 2026 · apache/arrow · docs · reviewed by Keyoor

Verdict

The plumbing of the modern Python data stack: if Parquet or cross-engine interchange is involved, install it and use it directly for real control over scans and memory. If you only ever want dataframe ergonomics, let Polars or DuckDB carry their own Arrow and skip the direct dependency.

API stability4/5Core APIs (Table, parquet, dataset) have been stable for years, but the project ships several major versions annually and deprecations do land; ecosystem pin conflicts are the real-world symptom.

Docs4/5arrow.apache.org/docs/python has a solid user guide and complete API reference, but it documents a huge surface and the cookbook-style answers for common tasks take some hunting.

Maintenance5/5Apache Software Foundation project with corporate-backed full-time contributors, pushed August 2026, and a predictable release cadence; about 2.5k open issues and PRs reflect scale, not neglect.

Ecosystem5/5Roughly 101M weekly downloads and it is the interchange layer pandas, DuckDB, Polars, Spark, and most warehouses agree on; Arrow support is now a checkbox feature across the data world.

Use it if

You read or write Parquet files: pq.read_table and pq.write_table are the reference implementation and what pandas calls under the hood
You pass data between engines (pandas to DuckDB to Polars to Spark) and want the interchange to be a pointer handoff instead of a serialize/deserialize round trip
You scan folders of Parquet larger than RAM: pyarrow.dataset pushes filters and column selection into the file scan so only matching row groups are read
You want pandas 2.x string and nullable dtypes backed by Arrow memory (dtype_backend="pyarrow") for lower memory use on text-heavy frames

Skip it if

You just want dataframes: Polars and DuckDB ship their own Arrow engine inside a single install, and you may never need to touch pyarrow's API directly
Wheel size matters: the pyarrow wheel is 36 to 50 MB depending on platform because it bundles the Arrow C++ libraries, which is painful in Lambda layers and slim containers
You need a version you can pin and forget: Arrow ships several major versions per year (25.0.0 landed July 2026), and libraries pinning different majors regularly produce resolver conflicts
Your Parquet needs are small and write-only: fastparquet is a much lighter dependency if you only read and write modest files from pandas
The API is deliberately low-level: no joins-and-groupby dataframe ergonomics; compute is function-style (pc.filter, pc.sum) and mid-level devs often reach for it expecting pandas and bounce off

Setup reality

pip install pyarrow pulls a large prebuilt wheel (36 to 50 MB) and works fine on manylinux, macOS, and Windows. The pain starts off the paved road: Alpine/musl or unusual architectures mean building from source, which requires the Arrow C++ libraries, CMake, and real patience. The other recurring annoyance is version pinning: because majors ship frequently, one dependency pinning pyarrow<20 while another wants >=24 is a resolver fight you will eventually have. Table.to_pandas() copies by default, so peak memory can double on conversion unless you use types_mapper or self_destruct.

Patterns

Read a Parquet file into a Tableread-parquet

import pyarrow.parquet as pq

table = pq.read_table("data.parquet", columns=["user_id", "amount"])
df = table.to_pandas()

Pass columns= to avoid reading the whole file; to_pandas() copies memory by default, so peak RAM is roughly table plus dataframe.

Write a Table to Parquet with compressionwrite-parquet

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({"id": [1, 2, 3], "name": ["a", "b", "c"]})
pq.write_table(table, "out.parquet", compression="zstd")

Default compression is snappy; zstd usually gives noticeably smaller files for a small CPU cost and is supported by every modern reader.

Build a Table from Python databuild-table

import pyarrow as pa

table = pa.table({
    "ts": pa.array([1690000000, 1690000060], type=pa.timestamp("s")),
    "value": [1.5, 2.5],
})
print(table.schema)

Plain lists get types inferred; pass pa.array(..., type=...) when you need a specific type instead of casting afterwards.

Convert between pandas and Arrowpandas-round-trip

import pandas as pd
import pyarrow as pa

df = pd.DataFrame({"x": [1, 2], "s": ["a", "b"]})
table = pa.Table.from_pandas(df, preserve_index=False)

df2 = table.to_pandas(self_destruct=True, split_blocks=True)

preserve_index=False stops pandas indexes leaking into your Parquet schema; self_destruct frees Arrow memory during conversion to cut peak usage.

Scan a Parquet folder with filter pushdowndataset-filter-pushdown

import pyarrow.dataset as ds
import pyarrow.compute as pc

dataset = ds.dataset("s3://bucket/events/", format="parquet")
table = dataset.to_table(
    columns=["user_id", "amount"],
    filter=(pc.field("country") == "DE") & (pc.field("amount") > 100),
)

Filters are evaluated against Parquet row group statistics, so non-matching chunks are never read; this is the main reason to use the dataset API over read_table.

Write a hive-partitioned datasetpartitioned-dataset

import pyarrow as pa
import pyarrow.dataset as ds

ds.write_dataset(
    table,
    "warehouse/events",
    format="parquet",
    partitioning=ds.partitioning(
        pa.schema([("year", pa.int16()), ("country", pa.string())]),
        flavor="hive",
    ),
    existing_data_behavior="overwrite_or_ignore",
)

Partition columns are moved into directory names (year=2026/country=DE) and dropped from the files; readers reconstruct them automatically. The hive flavor needs a schema; field_names only works with the default directory flavor.

Read CSV with the multithreaded Arrow readerread-csv-fast

import pyarrow.csv as pv

table = pv.read_csv(
    "big.csv",
    convert_options=pv.ConvertOptions(strings_can_be_null=True),
)

Usually several times faster than pandas.read_csv because it parses in parallel; column types are inferred per block, so pass explicit column_types for messy files.

Filter and aggregate without pandascompute-kernels

import pyarrow.compute as pc

mask = pc.greater(table["amount"], 100)
big = table.filter(mask)
total = pc.sum(big["amount"]).as_py()

Compute kernels run vectorized C++ over Arrow memory; .as_py() converts a scalar result back to a Python value.

Stream row batches into one Parquet filechunked-parquet-writer

import pyarrow.parquet as pq

with pq.ParquetWriter("out.parquet", schema) as writer:
    for batch_table in produce_tables():
        writer.write_table(batch_table)

Every batch must match the schema passed to ParquetWriter exactly, including nullability; cast first or the write raises.

Zero-copy read of an Arrow IPC filememory-map-ipc

import pyarrow as pa

with pa.memory_map("data.arrow", "r") as source:
    table = pa.ipc.open_file(source).read_all()

Memory-mapped IPC (Feather V2) files load without copying; startup is near-instant even for multi-GB files because pages fault in on access.

Define a schema and cast a table to itschema-cast

import pyarrow as pa

schema = pa.schema([
    ("id", pa.int64()),
    ("price", pa.decimal128(10, 2)),
    ("created", pa.timestamp("us", tz="UTC")),
])
fixed = table.cast(schema)

cast raises on lossy conversions by default; this is the standard way to stop schema drift across daily Parquet writes.

Load Parquet into pandas with Arrow-backed dtypespandas-arrow-dtypes

import pandas as pd

df = pd.read_parquet("data.parquet", dtype_backend="pyarrow")
print(df.dtypes)

Arrow-backed strings use far less memory than object dtype, but some older pandas code paths still silently convert back to NumPy; measure before assuming.

Alternatives

Package	Registry	Pick it when
polars	PyPI	You want a full dataframe API (joins, group_by, lazy queries) on Arrow memory in one install, without pandas.
duckdb	PyPI	You want to run SQL directly over Parquet files or Arrow tables; the wheel is smaller and the query engine is included.
fastparquet	PyPI	You only need pandas-to-Parquet round trips and want a far lighter dependency than the Arrow C++ stack.