pyarrow review
PyArrow is the Python access layer for Apache Arrow's typed columnar buffers and the C++ readers, writers, kernels, and filesystem code built around them. It supplies arrays, record batches, tables, Parquet and CSV I/O, Arrow IPC, partitioned dataset scans, compute functions, and cloud filesystem clients. Version 25.0.1 repairs silent wrong double values while reading Parquet on Arm SVE, fixes a crash when the first library load happens on a non-main thread that exits, makes `to_pylist()` bypass per-element Scalar objects, and limits the Feather deprecation to V1. Our Python 3.12 import completed in 0.18 seconds. This is a compiled data engine rather than a pandas-style analysis interface.
Our PyArrow 25.0.1 install finished in 1.3 seconds with zero audit findings, but its single package occupied 152 MB, so it earns a place only where Parquet, Arrow IPC, or dataset scans pay for that footprint. Choose pandas or DuckDB when ordinary table work or SQL is the actual job.
We installed it
| Install | ✓ · 1.3s | 1 package on disk · 152 MB |
| Import | ✓ | import pyarrow in 0.18s · compiled extensions · requires Python >=3.10 |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does pyarrow install cleanly?
Yes. In a fresh container with an empty cache, pip install pyarrow finished in 1 seconds, leaving 1 package and 152 MB on disk. pip-audit reported no known vulnerabilities.
What does pyarrow need to run?
Python >=3.10, and a platform wheel with compiled extensions. In our run import pyarrow succeeded in 0.18s.
pyarrow or polars: which should you use?
polars: Choose it for a DataFrame-first API with lazy queries while retaining Arrow and Parquet interchange. Our PyArrow 25.0.1 install finished in 1.3 seconds with zero audit findings, but its single package occupied 152 MB, so it earns a place only where Parquet, Arrow IPC, or dataset scans pay for that footprint.
When should you not use pyarrow?
Your service only moves small dictionaries or JSON and never reads Parquet or Arrow IPC; the one installed package occupied 152 MB in our sandbox
Discussed on
Use it if
- You exchange tables with DuckDB, Polars, pandas, Spark, or another Arrow-aware system and need to retain Arrow schemas and buffers at the boundary
- Parquet reads need column selection, row filters, partition pruning, encryption, or explicit types that a one-line DataFrame loader hides
- A directory of partitioned files must be scanned in record batches without first turning every value into a Python object
- Python needs Arrow IPC files or streams for cross-process or cross-language transfer under one fixed schema
- Your service only moves small dictionaries or JSON and never reads Parquet or Arrow IPC; the one installed package occupied 152 MB in our sandbox
- A static-type gate requires complete inline annotations; our wheel had no `py.typed` marker, and Arrow 25.0 withheld annotations because coverage is incomplete
- Your target has no published wheel and you will not own a C++ and CMake build; the official source path builds Arrow C++ before it builds PyArrow
- You need DataFrame labels, group-by workflows, indexing, and plotting as the main interface; pandas or Polars fits that work directly
- You need parallel incremental CSV ingestion; `csv.open_csv()` is single-threaded and freezes inferred types after its first input block
Setup reality
In our fresh Python 3.12 Bookworm sandbox, pyarrow==25.0.1 installed successfully in 1.3 seconds. The unprivileged container had 3 CPUs, 8 GB RAM, and no cache. Installation left one package using 152 MB, and the distribution declared zero direct dependencies. pip-audit found 0 known vulnerabilities. import pyarrow finished in 0.18 seconds. The wheel contains compiled .so files, so the low package count hides a large native payload.
Python 3.10 is the floor. PyPI publishes 25.0.1 wheels for supported CPython releases on Windows, macOS, manylinux, and musllinux, and the install page recommends a 64-bit system. If no wheel matches, the source route asks you to configure and build Arrow C++ with CMake before building PyArrow; optional components alter that build. A Windows wheel that will not import may also need Microsoft's current Visual C++ Redistributable.
Local Parquet and IPC work needs no credentials or config file. S3 is different: S3FileSystem consults AWS environment variables, config files, and instance metadata, while the bucket region must be correct. Set ARROW_S3_LOG_LEVEL before the first S3 call if you need SDK logs. Our installed-package probe found no license value, although current PyPI metadata and the repository identify Apache-2.0. The 25.0.1 wheel also lacked a py.typed marker.
Dataset operations draw on native CPU and I/O thread pools, so cap them in shared workers. write_dataset() may change row order when threading is on unless preserve_order=True, which can slow the write. Existing destination data raises an error by default; other modes may keep unrelated files or delete matching partitions. Parquet pre-buffering adds background readahead, so disable duplicate readahead in another filesystem layer. csv.open_csv() remains single-threaded and fixes inferred types after its first block. Track Arrow allocations with pa.total_allocated_bytes() alongside process RSS.
Patterns
Build a table under an explicit schema define-typed-table
import pyarrow as pa
schema = pa.schema([
pa.field("order_id", pa.int64(), nullable=False),
pa.field("amount", pa.decimal128(12, 2)),
pa.field("created_at", pa.timestamp("us", tz="UTC")),
])
table = pa.Table.from_pylist(records, schema=schema)`Table.from_pylist()` applies the supplied field types and nullability instead of inferring a schema from whichever records arrive first.
Read selected Parquet data read-filtered-parquet
import pyarrow.parquet as pq
table = pq.read_table(
"warehouse/orders/",
columns=["order_id", "amount"],
filters=[("year", "=", 2026), ("amount", ">", 100)],
)The column list limits materialized fields, while filters can skip partitions and Parquet row groups when their metadata proves there is no match.
Write Parquet with deliberate encoding write-parquet
import pyarrow.parquet as pq
pq.write_table(
table,
"orders.parquet",
compression="zstd",
use_dictionary=["country"],
write_statistics=True,
row_group_size=128_000,
)Compression, dictionary columns, statistics, and row-group size become file-format choices that every downstream reader inherits.
Scan a partitioned dataset in batches scan-dataset-batches
import pyarrow.dataset as ds
dataset = ds.dataset(
"warehouse/events/",
format="parquet",
partitioning="hive",
)
scanner = dataset.scanner(
columns=["user_id", "amount"],
filter=(ds.field("country") == "DE") & (ds.field("amount") > 100),
batch_size=64_000,
)
for batch in scanner.to_batches():
consume(batch)`Scanner.to_batches()` avoids first materializing one full Table; partition keys and Parquet statistics determine how much source data can be skipped.
Append files with unique names append-partitioned-dataset
from uuid import uuid4
import pyarrow.dataset as ds
run_id = uuid4().hex
ds.write_dataset(
table,
"warehouse/orders/",
format="parquet",
partitioning=["year", "month"],
partitioning_flavor="hive",
basename_template=f"{run_id}-{{i}}.parquet",
existing_data_behavior="overwrite_or_ignore",
)`overwrite_or_ignore` leaves unrelated files in place and replaces name collisions. A unique basename prevents a retry or later append from reusing `part-0.parquet`.
Control a pandas round trip convert-pandas
import pandas as pd
import pyarrow as pa
table = pa.Table.from_pandas(frame, preserve_index=False)
round_trip = table.to_pandas(types_mapper=pd.ArrowDtype)`preserve_index=False` drops pandas index storage, and `pd.ArrowDtype` keeps nullable Arrow-backed dtypes where pandas supports them.
Stream CSV under fixed column types stream-csv-batches
import pyarrow as pa
from pyarrow import csv
convert = csv.ConvertOptions(column_types={
"order_id": pa.int64(),
"amount": pa.decimal128(12, 2),
})
reader = csv.open_csv("orders.csv", convert_options=convert)
for batch in reader:
consume(batch)`open_csv()` is single-threaded and freezes inferred types after the first block. Explicit `column_types` stops later rows from changing the schema.
Filter with Arrow compute kernels run-compute-kernels
import pyarrow as pa
import pyarrow.compute as pc
active = pc.equal(table["status"], "active")
minimum = pa.scalar(100, type=table["amount"].type)
expensive = pc.greater(table["amount"], minimum)
selected = table.filter(pc.and_(active, expensive))The comparisons return Arrow boolean data, and `Table.filter()` keeps rows where the combined mask is true without a Python row loop.
Send record batches as an IPC stream write-ipc-stream
import pyarrow as pa
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, batch.schema) as writer:
writer.write_batch(batch)
payload = sink.getvalue()
with pa.ipc.open_stream(payload) as reader:
received = reader.read_all()Every batch in one IPC stream must use the schema passed to `new_stream()`. Stream readers only require sequential reads.
Open an IPC file through a memory map memory-map-ipc-file
import pyarrow as pa
with pa.OSFile("events.arrow", "wb") as sink:
with pa.ipc.new_file(sink, table.schema) as writer:
writer.write_table(table)
with pa.memory_map("events.arrow", "rb") as source:
reader = pa.ipc.open_file(source)
first_batch = reader.get_batch(0)The IPC file format supports random batch access, and a memory-mapped source lets returned batches reference mapped data without a fresh read allocation.
Read Parquet through the native S3 client read-s3-parquet
import pyarrow.parquet as pq
from pyarrow import fs
bucket = "analytics-prod"
region = fs.resolve_s3_region(bucket)
s3 = fs.S3FileSystem(region=region)
table = pq.read_table(
f"{bucket}/events/date=2026-09-02/",
filesystem=s3,
columns=["user_id", "event"],
)`S3FileSystem` uses the AWS credential chain when keys are omitted. Pass a path without `s3://` when a filesystem object is supplied separately.
Process one Parquet batch at a time iterate-parquet-batches
import pyarrow.parquet as pq
parquet = pq.ParquetFile("orders.parquet")
for batch in parquet.iter_batches(
batch_size=64_000,
columns=["order_id", "amount"],
use_threads=True,
):
consume(batch)`iter_batches()` limits each yielded RecordBatch and can read columns in parallel. Retaining old batches still grows the caller's memory use.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| polars | PyPI | Choose it for a DataFrame-first API with lazy queries while retaining Arrow and Parquet interchange. |
| pandas | PyPI | Choose it when labeled indexes, grouping, joins, plotting hooks, and notebook analysis are the main job. |
| duckdb | PyPI | Choose it to query Parquet and other files with SQL without making Arrow objects your application interface. |
| fastparquet | PyPI | Choose it as a narrower Parquet engine when Arrow IPC, compute kernels, and cloud filesystems are outside the requirement. |
More data guides
numpy · fsspec · pandas · sqlalchemy · lxml · s3fs · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.