PyPIDataupdated 05 Aug 2026

s3fs

Item: s3fs
Rating: 4
Author: Keyoor

s3fs wraps Amazon S3 in a filesystem interface: ls, glob, open, put, get, and Python file objects over buckets and keys. It is the S3 backend of the fsspec ecosystem, which is why pandas, dask, xarray, and pyarrow can all read s3:// URLs directly; when you pass one of those, s3fs is what actually runs. Under the hood it is built on aiobotocore, so every call is async botocore with a sync facade on top. Its enormous download count comes from being a transitive dependency of half the data stack.

s3fs · v2026.7.0 · 164.4M/wk · ★ 1,044 · pushed 28 Jul 2026 · fsspec/s3fs · docs · reviewed by Keyoor

Verdict

If your data tools speak fsspec, s3fs is not really a choice, it is the plumbing, and it does that job well. Choose it deliberately when filesystem semantics fit your problem; choose boto3 when you need the actual AWS API, and keep them in separate environments if you can.

API stability4/5The fsspec filesystem API has been stable for years and calendar releases rarely break user code; the churn lives in the aiobotocore/botocore pins underneath, not the interface.

Docs3/5The readthedocs site covers the API and common credential setups, but async usage, caching behavior, and the dependency-pinning story are thin and mostly learned from GitHub issues.

Maintenance4/5Actively maintained inside the fsspec org with monthly-ish calendar releases (2026.7.0 current, pushed July 2026) and Anaconda-sponsored upkeep, though the issue tracker moves slowly.

Ecosystem5/5It is the default S3 layer for pandas, dask, xarray, pyarrow, and anything else fsspec-aware, which is why it sees over 160M downloads a week.

Use it if

You want pandas, dask, polars, xarray, or pyarrow to read and write s3:// paths directly, with credentials and listing handled for you
You think in files and directories: fs.glob('bucket/logs/2026-*/*.json') and fs.open(...) read more naturally than boto3's paginated client calls
You need the same code to run against real S3 and an S3-compatible store like MinIO or Cloudflare R2 by switching endpoint_url
You are already inside fsspec land (dask or intake) where s3fs is the assumed S3 driver anyway

Skip it if

You need the full breadth of AWS S3 API features (bucket policies, lifecycle rules, multipart tuning, presigned POST): that is boto3's job, s3fs only covers filesystem-shaped operations
Your environment already depends on a specific boto3 version: s3fs hard-pins aiobotocore, which pins botocore, and the pip resolver fights you over it regularly
You are chasing maximum throughput on large scans: the filesystem abstraction adds listing and caching overhead, and Rust-based obstore benchmarks faster for bulk object access
You only make a handful of get/put calls: a 10-line boto3 client wrapper avoids the whole aiobotocore dependency chain

Setup reality

pip install s3fs is easy until it meets your other pins: it requires aiobotocore within a hard range and fsspec at an exact calendar minor (2026.7.x needs fsspec 2026.7.0), and because aiobotocore pins botocore, installing a newer boto3 alongside is the classic conflict that has its own FAQ entry. Conda solves this more gracefully than pip. Credentials come from the normal AWS chain (env vars, ~/.aws, instance roles) which mostly just works, but debugging permission errors through the filesystem abstraction is worse than through raw boto3, and the directory-listing cache will happily show you stale results after another process writes.

Patterns

List and read objects like filesbasic-filesystem

import s3fs

fs = s3fs.S3FileSystem()  # credentials from env / ~/.aws / IAM role
print(fs.ls("my-bucket/raw"))

with fs.open("my-bucket/raw/data.csv", "rb") as f:
    head = f.read(1024)

Paths are 'bucket/key' with no s3:// prefix required (both work). ls results are cached; see the invalidate pattern below before you trust a listing twice.

Write an object with open()write-file

with fs.open("my-bucket/out/report.txt", "w") as f:
    f.write("hello from s3fs\n")

Writes buffer locally and upload on close as (possibly multipart) PUTs; nothing hits S3 until the block exits, so a crash mid-block leaves no partial object.

Read a public bucket without credentialsanonymous-public-bucket

fs = s3fs.S3FileSystem(anon=True)
files = fs.ls("noaa-ghcn-pds/csv")

anon=True skips the credential chain entirely. Without it, s3fs on a machine with no credentials fails with NoCredentialsError even for public buckets.

Point at MinIO, R2, or another S3 clonecustom-endpoint-minio

fs = s3fs.S3FileSystem(
    key="minioadmin",
    secret="minioadmin",
    endpoint_url="http://localhost:9000",
)
fs.mkdir("test-bucket")
fs.pipe("test-bucket/hello.txt", b"hi")

endpoint_url is the whole trick for S3-compatible stores. Some clones need client_kwargs={'region_name': 'us-east-1'} too, or botocore complains.

Read S3 directly from pandaspandas-storage-options

import pandas as pd

df = pd.read_parquet(
    "s3://my-bucket/data/part-0.parquet",
    storage_options={"profile": "analytics"},
)

pandas hands the URL to fsspec, which imports s3fs; storage_options is forwarded to the S3FileSystem constructor. If s3fs is not installed you get ImportError from deep inside pandas.

Glob keys by patternglob-find

paths = fs.glob("my-bucket/logs/2026-08-*/**/*.json")
all_keys = fs.find("my-bucket/logs")  # recursive, files only

Globbing is client-side: s3fs lists the prefix and filters, so a wide pattern over millions of keys is slow and costs LIST requests. Narrow the fixed prefix as much as possible.

Upload and download whole directoriesupload-download-trees

fs.put("./local_dir/", "my-bucket/backup/", recursive=True)
fs.get("my-bucket/backup/", "./restore/", recursive=True)

Transfers run concurrently through the async core, which is decently fast; but there is no sync/skip-existing logic, everything is re-copied every time.

Deal with the stale listing cachelisting-cache-invalidate

fs.ls("my-bucket/incoming")      # cached
# another process writes a new object...
fs.invalidate_cache("my-bucket/incoming")
fresh = fs.ls("my-bucket/incoming", refresh=True)

Directory listings are cached per instance; long-lived processes watching a prefix must invalidate_cache() or pass refresh=True, or new files never appear. Top source of confused bug reports.

Use s3fs from async codeasync-usage

import asyncio, s3fs

async def main():
    fs = s3fs.S3FileSystem(asynchronous=True)
    session = await fs.set_session()
    listing = await fs._ls("my-bucket")
    print(listing)
    await session.close()

asyncio.run(main())

In async mode the coroutine methods are the underscore-prefixed ones (_ls, _cat, _put); the plain names stay sync-only. Calling sync methods inside a running event loop deadlocks.

Read a specific object versionversioned-objects

fs = s3fs.S3FileSystem(version_aware=True)
versions = fs.object_version_info("my-bucket/config.json")
with fs.open("my-bucket/config.json", version_id=versions[-1]["VersionId"]) as f:
    old = f.read()

Only works on buckets with versioning enabled and version_aware=True set at construction; without the flag, version_id is ignored.

Open by URL through fsspecfsspec-generic-url

import fsspec

with fsspec.open("s3://my-bucket/raw/data.csv", "rt", anon=True) as f:
    for line in f:
        process(line)

fsspec.open dispatches to s3fs from the URL scheme; extra kwargs go to S3FileSystem. Handy for code that must also accept local paths or gcs:// URLs unchanged.

Alternatives

Package	Registry	Pick it when
boto3	PyPI	You need real AWS API coverage or your ops code already speaks boto3; it is the canonical SDK and never fights your pins the way aiobotocore does.
obstore	PyPI	You want the fastest object-store access from Python (Rust object_store bindings) and can trade the fsspec ecosystem for speed.
smart-open	PyPI	You just want open() that streams from S3, GCS, or HTTP with minimal ceremony and no filesystem semantics.
cloudpathlib	PyPI	You want pathlib-style Path objects for cloud storage with local caching, closer to stdlib ergonomics than a filesystem object.