polars
Polars is a DataFrame library and query engine written in Rust with Python bindings. Instead of pandas-style step-by-step mutation, you compose expressions (pl.col, filter, group_by, agg) that a query optimizer rewrites and executes multi-threaded across all cores. It has both eager and lazy modes, a streaming engine for datasets larger than RAM, optional NVIDIA GPU execution, and stores data in the Apache Arrow columnar format for zero-copy interop. In practice it is the main serious alternative to pandas for tabular work in Python, usually chosen for speed.
The best reason to leave pandas: real speed, a coherent API, and larger-than-RAM execution from one pip install. Adopt it for new pipelines you control end to end; think twice mid-project if everything downstream expects pandas objects.
Use it if
- Your pandas workloads are slow or memory-bound: multi-threaded execution and query optimization routinely turn minutes into seconds
- You process files bigger than RAM: scan_parquet or scan_csv plus collect(engine='streaming') handles larger-than-memory queries
- You are starting a new data pipeline and can adopt the expression API from day one, where its consistency is an asset not a migration
- You want the same engine from multiple languages: Python, Rust, Node.js, R, and SQL bindings all exist
- Your stack assumes pandas objects everywhere: plenty of libraries in the scientific Python world accept only pandas DataFrames, so you end up calling to_pandas() at every boundary and paying conversions
- Your team knows pandas and your data is small: the expression API is a genuinely different mental model, and on a 10k-row CSV the speedup is unobservable while the retraining cost is real
- You lean on pandas-specific machinery: there is no index, no inplace mutation idiom, and row-wise .apply habits translate badly
- You depend on the sheer volume of pandas answers: two decades of Stack Overflow and tutorials cover pandas; Polars docs are good but the searchable long tail is thinner
- You need very old platforms: current wheels need Python 3.10+, and pre-2011 CPUs without AVX2 need the separate legacy build
Setup reality
pip install polars gets you a prebuilt wheel with the Rust engine inside, no compiler needed, and import polars just works. The friction is elsewhere: I/O and interop extras are optional dependencies you discover one error at a time (pyarrow for to_pandas, connectorx or adbc for databases, fsspec for cloud storage), old CPUs without AVX2 need the polars-lts-cpu build, and expect more than 4.2 billion rows to require the polars-u64-idx variant. Releases are frequent, so pin versions; deprecation warnings between minors are common even post-1.0.
Patterns
Read a CSV eagerlyread-csv
import polars as pl
df = pl.read_csv("orders.csv")
print(df.head())read_csv infers dtypes from a sample; pass schema_overrides for columns it guesses wrong, like zip codes becoming integers.
Lazy query with filter, group_by, and agglazy-query
import polars as pl
df = (
pl.scan_parquet("orders.parquet")
.filter(pl.col("status") == "shipped")
.group_by("customer_id")
.agg(
pl.col("amount").sum().alias("total"),
pl.len().alias("n_orders"),
)
.sort("total", descending=True)
.collect()
)scan_* returns a LazyFrame and nothing reads from disk until collect(); the optimizer pushes the filter down so only matching row groups load.
Derive new columns with expressionsadd-columns
df = df.with_columns(
(pl.col("amount") * 1.2).alias("amount_with_tax"),
pl.col("name").str.to_uppercase().alias("name_upper"),
)Expressions in one with_columns call run in parallel and cannot see each other; chain a second with_columns to reference a column you just made.
Conditional values with when/then/otherwiseconditional-column
df = df.with_columns(
pl.when(pl.col("amount") > 100)
.then(pl.lit("big"))
.otherwise(pl.lit("small"))
.alias("size")
)Wrap literal strings in pl.lit(); a bare string inside then() is read as a column name and gives confusing results.
Join two DataFramesjoin-frames
orders = pl.read_parquet("orders.parquet")
customers = pl.read_parquet("customers.parquet")
full = orders.join(customers, on="customer_id", how="left")how accepts inner, left, right, full, semi, anti, and cross; unlike pandas there is no index, joins are always on named columns.
Group-wise values without collapsing rowswindow-functions
df = df.with_columns(
pl.col("amount").sum().over("customer_id").alias("customer_total"),
pl.col("amount").rank(descending=True).over("customer_id").alias("rank_in_customer"),
)over() is the window equivalent of a group_by-then-join back; it keeps the original row count.
Process larger-than-RAM data with the streaming enginestreaming-large-files
result = (
pl.scan_csv("huge_250gb.csv")
.filter(pl.col("country") == "US")
.group_by("state")
.agg(pl.col("revenue").sum())
.collect(engine="streaming")
)Streaming only helps if you start from scan_* sources; read_csv already loaded everything into memory before you could stream.
Convert to and from pandaspandas-interop
import polars as pl
import pandas as pd
pdf = pd.DataFrame({"a": [1, 2, 3]})
df = pl.from_pandas(pdf)
back = df.to_pandas() # requires pyarrowto_pandas() needs pyarrow installed and copies data out of Arrow; do it once at the boundary, not inside loops.
Write results to Parquetwrite-parquet
df.write_parquet("out.parquet", compression="zstd")
# lazy version, streams to disk without materializing:
lf.sink_parquet("out.parquet")sink_parquet on a LazyFrame writes without collecting the whole result into memory, which is the point of a streaming pipeline.
Query DataFrames with SQLrun-sql
df = pl.read_parquet("orders.parquet")
result = pl.sql(
"SELECT customer_id, SUM(amount) AS total FROM df GROUP BY customer_id"
).collect()pl.sql sees DataFrames and LazyFrames from the surrounding scope by variable name and returns a LazyFrame, so remember collect().
String cleanup with the str namespacestring-operations
df = df.with_columns(
pl.col("email").str.to_lowercase().str.strip_chars().alias("email_clean"),
pl.col("phone").str.replace_all(r"[^0-9]", "").alias("phone_digits"),
)str.replace uses regex by default and replaces the first match only; use str.replace_all for every occurrence.
Fill and filter nullshandle-nulls
df = df.with_columns(
pl.col("amount").fill_null(0),
pl.col("category").fill_null(pl.lit("unknown")),
)
non_null = df.filter(pl.col("customer_id").is_not_null())Polars separates null (missing) from NaN (float not-a-number); fill_null does not touch NaN, that is fill_nan.