awswrangler
awswrangler, published by AWS as the AWS SDK for pandas, is a library of functions that move pandas DataFrames in and out of AWS data services without you writing the boto3 plumbing. It is organized by service: wr.s3, wr.athena, wr.catalog, wr.redshift, wr.dynamodb, wr.timestream, wr.opensearch, wr.neptune and about a dozen more. One call to wr.s3.to_parquet writes a partitioned Parquet dataset, creates the Glue Data Catalog table, and registers the partitions. One call to wr.athena.read_sql_query submits the query, polls until it finishes, finds the result files on S3, and gives you a DataFrame with real dtypes. Underneath it is boto3 for the API calls and PyArrow for the file formats, so nothing magic is happening; the value is that hundreds of lines of polling loops, pagination, type coercion and temp-file cleanup have already been written and tested against the actual services.
If your analytics stack is S3, Glue and Athena, awswrangler removes a genuinely tedious layer of boto3 code and the Glue Catalog integration on writes is worth the install on its own. Just budget for the extras system, the pyarrow deployment weight, and the fact that everything here still runs in one process's memory.
Use it if
- Your data lake is S3 plus Glue Catalog plus Athena and you keep rewriting the same start_query_execution, poll, then read the result CSV off S3 dance
- You write partitioned Parquet datasets and want the Glue table created, partitions registered, and old partitions replaced by the same call that writes the files (mode="overwrite_partitions")
- You bulk load Redshift and want the S3 staging, the COPY statement, the DISTKEY and SORTKEY DDL, the upsert by primary key, and the temp file cleanup handled in one function
- You run in AWS Lambda, Glue Python Shell, or SageMaker notebooks, where AWS publishes prebuilt layers and wheels so you skip the pyarrow packaging problem
- You only touch one AWS service: installing awswrangler drags in boto3, botocore, pandas, numpy and pyarrow for what may be a single Athena query, and pyathena is a much smaller DB-API driver that does the same job
- Cold start matters in your Lambda: importing pandas and pyarrow costs real startup time on every cold invocation, and a boto3-only handler that writes JSON to S3 starts far faster
- Your results do not fit in memory: this is pandas, so a large Athena result set will exhaust the process unless you pass chunksize and iterate, or switch to the Ray and Modin backend, which is a separate install and only covers a subset of the APIs
- You need vendor support commitments: this is an AWS Professional Services open source initiative under Apache 2.0, not a supported AWS SDK, so an edge case in the Athena type mapping goes to GitHub issues rather than your AWS support plan
- Your data is not on AWS: every function signature takes a boto3 session and encodes AWS service semantics, so there is no portability story here and nothing transfers to GCS or Azure
Setup reality
pip install awswrangler gets you S3, Athena, Glue, DynamoDB, Timestream and CloudWatch. Since 3.0, everything else is an extra you must ask for by name: pip install 'awswrangler[redshift,postgres,opensearch]'. Forget the extra and you get an ImportError at call time, not install time, which is a fun thing to discover in a Glue job at 2am. The base install pulls pyarrow, which is around 40 MB of wheel, so a zipped Lambda deployment blows past the 50 MB direct upload limit fast; use the managed layer AWS publishes per Python version and region instead of packaging it yourself, and pin the layer version because it also pins the pandas and pyarrow inside it. Python 3.10 through 3.14. Credentials come from the normal boto3 chain, but every function also accepts boto3_session, and in multi-account setups you will want that explicitly rather than relying on the default. The Athena default path (ctas_approach=True) writes a temporary CTAS table, so the execution role needs Glue CreateTable and DeleteTable plus write access to the query output bucket, not just Athena StartQueryExecution.
Patterns
Write a partitioned Parquet dataset and register it in Gluewrite-parquet-dataset
import awswrangler as wr
wr.s3.to_parquet(
df=df,
path="s3://my-lake/events/",
dataset=True,
partition_cols=["dt"],
mode="overwrite_partitions",
database="analytics",
table="events",
compression="snappy",
)dataset=True is what turns on partitioning, Glue registration and the mode argument; without it path is treated as a single file and partition_cols is rejected. mode="overwrite" deletes the whole prefix, not just the partitions in your frame, so use overwrite_partitions for incremental loads. The write may mutate your DataFrame in place, so pass df.copy() if you still need the original.
Read only the partitions you needread-parquet-with-partition-filter
df = wr.s3.read_parquet(
path="s3://my-lake/events/",
dataset=True,
columns=["user_id", "amount"],
partition_filter=lambda p: p["dt"] >= "2026-08-01",
)Partition values arrive as strings no matter what the column type is in Glue, so comparing p["year"] to the integer 2026 silently matches nothing. The filter only runs when dataset=True; on a plain path it is ignored. Passing columns matters more than it looks: Parquet is columnar, so it cuts bytes off the wire, not just off the frame.
Run an Athena query with bound parametersquery-athena
df = wr.athena.read_sql_query(
"SELECT user_id, sum(amount) AS total FROM events WHERE dt = :dt GROUP BY user_id",
database="analytics",
params={"dt": "2026-08-01"},
paramstyle="named",
workgroup="primary",
)paramstyle="named" does client-side substitution with quoting, which is still far better than f-strings. The default ctas_approach=True wraps your SQL in a CREATE TABLE AS to get typed Parquet output, which means the role needs Glue table create and delete rights, and it will not work for DDL, INSERT, or queries against views with certain functions. Set ctas_approach=False for those and accept the CSV-typed results.
Pull a large Athena result without the CTAS tableunload-large-results
df = wr.athena.read_sql_query(
"SELECT * FROM big_table WHERE dt = '2026-08-01'",
database="analytics",
ctas_approach=False,
unload_approach=True,
s3_output="s3://my-lake/athena-unload/",
)unload_approach uses Athena UNLOAD to write Parquet directly and skips creating a temporary Glue table, so it is the right choice when the execution role cannot create tables. It cannot be combined with ctas_approach, and UNLOAD refuses to write into a non-empty prefix, so give it a path you control and expect it to fail on reruns if you leave files behind.
Iterate a result set that does not fit in memorystream-results-in-chunks
for chunk in wr.athena.read_sql_query(
"SELECT * FROM events",
database="analytics",
chunksize=100_000,
):
process(chunk)
for chunk in wr.s3.read_parquet("s3://my-lake/events/", dataset=True, chunked=100_000):
process(chunk)Passing chunksize changes the return type from DataFrame to Iterator[DataFrame], which will break any caller doing df.shape on the result. chunked=True (rather than an integer) yields one frame per Parquet file, so chunk sizes are whatever your writer produced; pass an integer if you need predictable memory.
Bulk load Redshift through S3 COPY with an upsertload-redshift
con = wr.redshift.connect("my-glue-connection")
try:
wr.redshift.copy(
df=df,
path="s3://my-lake/stage/redshift/",
con=con,
schema="public",
table="users",
iam_role="arn:aws:iam::111122223333:role/RedshiftCopyRole",
mode="upsert",
primary_keys=["user_id"],
)
finally:
con.close()Needs pip install 'awswrangler[redshift]'; the ImportError only appears when you call it. wr.redshift.connect reads a Glue Catalog connection so credentials stay out of your code. copy stages Parquet at path then issues COPY, and keep_files defaults to False so the staging files are deleted afterwards. mode="overwrite" drops and recreates the table by default, which loses grants and views; pass overwrite_method="truncate" or "delete" to keep the table object.
Append to or merge into an Athena Iceberg tablewrite-iceberg-table
wr.athena.to_iceberg(
df=df,
database="analytics",
table="users_iceberg",
temp_path="s3://my-lake/tmp/iceberg/",
merge_cols=["user_id"],
merge_condition="update",
schema_evolution=True,
)This works by writing the frame to temp_path and running INSERT INTO or MERGE INTO through Athena, so you pay for an Athena query per write and it is wrong for row-at-a-time workloads. Passing merge_cols turns it into a MERGE; without them you get plain appends and duplicate rows. schema_evolution defaults to False, so a new column in your frame is an error until you turn it on.
Inspect and clean up Glue Catalog objectscatalog-housekeeping
wr.catalog.create_database(name="analytics", exist_ok=True)
if wr.catalog.does_table_exist(database="analytics", table="events"):
print(wr.catalog.get_table_types(database="analytics", table="events"))
wr.catalog.delete_table_if_exists(database="analytics", table="scratch")
wr.catalog.delete_all_partitions(database="analytics", table="events")delete_table_if_exists removes the catalog entry only; the Parquet files stay on S3 and the next crawler run or CTAS will happily find them again. Use wr.s3.delete_objects for the data. Column names get sanitized to lowercase with underscores when a write registers a table, so a frame column named "User ID" becomes user_id in Athena and your later select by original name fails.
Target a specific account, region or endpointscope-boto3-session
import boto3
import awswrangler as wr
session = boto3.Session(profile_name="data-prod", region_name="eu-west-1")
df = wr.s3.read_parquet("s3://prod-lake/events/", dataset=True, boto3_session=session)
# or set defaults once for the whole process
wr.config.database = "analytics"
wr.config.s3_endpoint_url = "http://localhost:9000" # MinIO or LocalStackPassing boto3_session per call is the only safe pattern in code that crosses accounts; the global wr.config is process-wide state and will surprise anything running in threads. s3_endpoint_url is what makes LocalStack and MinIO work in tests, but Athena and Glue need their own endpoint overrides, so a full offline test of a lake pipeline is still not straightforward.
Pull DynamoDB items into a DataFrameread-dynamodb
df = wr.dynamodb.read_items(
table_name="orders",
partition_values=["customer#42"],
columns=["order_id", "total", "created_at"],
consistent=True,
)
wr.dynamodb.put_df(df=df, table_name="orders")read_items refuses to scan unless you pass allow_full_scan=True, which is a deliberate guardrail against accidentally reading a billion-item table. Numbers come back as Decimal because that is what DynamoDB stores, so cast the columns before doing arithmetic. put_df batches writes but does not retry your capacity errors forever, so throttling on a provisioned table will surface as an exception.
Scale the same calls across a Ray clusterrun-on-ray
# pip install 'awswrangler[modin,ray]'
import modin.pandas as pd
import awswrangler as wr
print(wr.engine.get()) # ray, if ray is importable at import time
df = wr.s3.read_parquet("s3://my-lake/events/", dataset=True) # returns a modin frameThe engine is chosen at import time by whether ray is installed, so simply having ray in the environment changes the return type of every read from a pandas DataFrame to a modin one. Only a subset of the API is distributed; the rest silently falls back to single-node. Verify with wr.engine.get() rather than assuming, and pin the ray version because the supported range is narrow.
List, size and read raw CSV objectsread-csv-and-list-objects
keys = wr.s3.list_objects("s3://my-lake/raw/", suffix=".csv")
sizes = wr.s3.size_objects(keys)
df = wr.s3.read_csv(
path=keys,
dtype={"user_id": "string"},
parse_dates=["created_at"],
use_threads=True,
)read_csv accepts a prefix or an explicit list of keys and concatenates them, so one bad file in a prefix poisons the whole read. Extra keyword arguments pass straight through to pandas.read_csv, which means dtype and parse_dates behave exactly as you expect. use_threads=True downloads in parallel and will happily saturate a small Lambda's network and memory budget; pass an integer to cap it.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| pyathena | PyPI | Athena is the only service you touch and you want a plain DB-API 2.0 or SQLAlchemy driver instead of a data lake toolkit |
| boto3 | PyPI | You are writing a small Lambda that moves objects around and cannot afford the pandas plus pyarrow import cost |
| duckdb | PyPI | You want to query Parquet on S3 directly with SQL and skip Athena, Glue Catalog, and the per-query billing entirely |
| polars | PyPI | The bottleneck is the pandas dataframe itself and you want lazy execution and lower memory on the same S3 Parquet files |