pandas
pandas is the standard Python library for working with tabular data. Its DataFrame and Series structures give you labeled rows and columns plus readers and writers for CSV, Excel, SQL, Parquet, JSON and HDF5. You use it to load, clean, reshape, join, group and aggregate data that fits in memory, and it sits underneath most of the Python data stack, from scikit-learn workflows to one-off analysis notebooks. Development started at the hedge fund AQR in 2008 and it is now a NumFOCUS-sponsored community project.
Still the default for in-memory data work in Python, with unmatched documentation and ecosystem gravity, and the 3.0 defaults fix its oldest footguns. But for large data or performance-sensitive pipelines, start with polars or duckdb instead of trying to make pandas fast.
Use it if
- You do exploratory analysis or data cleaning on datasets that fit comfortably in RAM
- You need glue between formats and systems: CSV or Excel in, Parquet or SQL out, DataFrames into scikit-learn or plotting libraries
- You work with time series and want resampling, rolling windows and date arithmetic that already handle the edge cases
- You want the API with the largest body of examples, courses and answers in data tooling
- Your data is bigger than memory or your groupbys and joins are slow: pandas is mostly single-threaded, and polars or duckdb run the same workloads several times faster on large tables
- You are upgrading a legacy codebase casually: pandas 3.0 (January 2026) turned copy-on-write on permanently, made the dedicated string dtype the default and removed long-deprecated APIs, so old 1.x/2.x code and a decade of Stack Overflow answers no longer match current behavior
- The job is simple record shaping in a service: the csv module and plain dicts are lighter and easier to reason about than the index/axis model
- You want strict schemas and lazy, optimizable query plans; pandas executes eagerly and validates nothing unless you bolt on extra tools
Setup reality
pip install pandas just works; prebuilt wheels exist for every mainstream platform so there is no compiler step. The friction is elsewhere: pandas 3.0 requires Python 3.11+, and the optional engines are separate installs you discover via ImportError at runtime (pyarrow for Parquet, openpyxl for Excel, SQLAlchemy for SQL, matplotlib for .plot). Migrating to 3.0 needs a real deprecation pass: chained assignment silently stops writing under copy-on-write, and code that relied on object-dtype strings can behave differently under the new string dtype.
Patterns
Load a CSV with explicit typesread-csv
import pandas as pd
df = pd.read_csv(
'sales.csv',
dtype={'store_id': 'int32'},
parse_dates=['sold_at'],
)In pandas 3 text columns load as the dedicated str dtype, not object; code that checked dtype == object for strings needs updating.
Filter rows with boolean masksfilter-rows
recent = df[df['sold_at'] >= '2026-01-01']
big = df[(df['amount'] > 100) & (df['region'] == 'EU')]Combine conditions with & and | plus parentheses; Python's and/or raise ValueError on Series.
Write to a subset without the chained-assignment trapassign-values
df.loc[df['amount'] < 0, 'amount'] = 0
# WRONG under copy-on-write (writes to a discarded copy):
# df[df['amount'] < 0]['amount'] = 0Copy-on-write is always on in pandas 3, so chained assignment never updates the original; use one .loc call for the whole operation.
Group and aggregate with named columnsgroupby-agg
summary = (
df.groupby('region', as_index=False)
.agg(total=('amount', 'sum'), orders=('order_id', 'nunique'))
)Named aggregation keeps flat column names; without it multi-stat aggs produce a MultiIndex that surprises everyone downstream.
Join two DataFrames safelymerge-join
orders = pd.merge(
orders, customers,
on='customer_id', how='left',
validate='many_to_one',
)validate='many_to_one' raises immediately if the right side has duplicate keys, which otherwise silently multiplies your rows.
Handle missing datamissing-values
df['qty'] = df['qty'].fillna(0)
df = df.dropna(subset=['customer_id'])pandas 3 no longer silently downcasts dtypes after fillna, and inplace-style idioms from old tutorials are gone; assign the result back.
Parse timestamps and bucket by perioddatetime-handling
df['ts'] = pd.to_datetime(df['ts'], format='%Y-%m-%d %H:%M:%S', utc=True)
monthly = df.set_index('ts').resample('MS')['amount'].sum()Pass format when you know it; inference on messy columns is slow and can misread day/month order.
Pivot long data into a wide tablepivot-reshape
wide = df.pivot_table(
index='region', columns='month',
values='amount', aggfunc='sum', fill_value=0,
)pivot_table aggregates duplicates; plain .pivot raises on duplicate index/column pairs, which is usually what tells you the data is not what you thought.
Store results as Parquetwrite-parquet
df.to_parquet('out/sales.parquet', index=False)
back = pd.read_parquet('out/sales.parquet')Requires pyarrow installed separately; prefer Parquet over CSV for anything you plan to re-read, it keeps dtypes and is far smaller.
Prefer vectorized ops over applyvectorize-not-apply
import numpy as np
# slow: row-wise Python loop
# df['margin'] = df.apply(lambda r: r['rev'] - r['cost'], axis=1)
# fast: whole-column arithmetic
df['margin'] = df['rev'] - df['cost']
df['tier'] = np.where(df['margin'] > 100, 'high', 'low')apply(axis=1) runs a Python function per row and is often 100x slower than column arithmetic; reach for it last.
Cut memory with categoricalscategory-dtype
df['region'] = df['region'].astype('category')
df.info(memory_usage='deep')Big win only when the column has few unique values; comparisons and merges between category and str columns need matching categories.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| polars | PyPI | You want multicore speed and lazy execution on large tables with a stricter, more consistent API |
| duckdb | PyPI | You would rather write SQL over Parquet/CSV files and only surface results as DataFrames |
| dask | PyPI | You need the pandas API but the data is bigger than one machine's memory |