PyPIDataupdated 05 Aug 2026

fsspec

Item: fsspec
Rating: 4
Author: Keyoor

fsspec defines one Python filesystem interface (open, ls, glob, get, put, and friends) and lets any storage backend implement it. Write code against fsspec and the same lines read local files, S3, GCS, Azure, HTTP, SFTP, zip archives, or memory, chosen by URL prefix like s3:// or gs://. The core package ships local, memory, HTTP, and archive backends; cloud backends live in sister packages such as s3fs and gcsfs. It is the storage layer that pandas, dask, pyarrow, zarr, and xarray use when you hand them a remote URL, which is why its download count dwarfs its star count.

fsspec · v2026.7.0 · 219.3M/wk · ★ 1,337 · pushed 30 Jul 2026 · fsspec/filesystem_spec · docs · reviewed by Keyoor

Verdict

The de facto standard storage abstraction in the Python data world, and if you use dask, pandas, or zarr with remote data you are already running it, so learning it is rarely wasted. Adopt it deliberately when you need multi-backend support; skip it for single-cloud apps where the provider SDK is more honest about what the store can do.

API stability4/5The AbstractFileSystem interface has been steady for years and CalVer releases are mostly additive; friction comes from backend version pinning and occasional behavior changes in caching and async details rather than API breaks.

Docs3/5ReadTheDocs covers the API and features like URL chaining and caching, but discovery is hard: real-world patterns live in the docs of consumers (dask, pandas) and in GitHub issues, and backend-specific behavior is documented unevenly.

Maintenance4/5Pushed July 2026 with monthly CalVer releases and Anaconda-sponsored maintainers, but 309 open issues (347 with PRs) against a small core team means many edge-case reports sit for a long time.

Ecosystem5/5Around 219 million weekly downloads and a backend ecosystem (s3fs, gcsfs, adlfs, and an implementations registry) plus first-class integration in pandas, dask, pyarrow, zarr, and xarray.

Use it if

Your code must read and write both local paths and object storage without two code paths; fsspec.open('s3://bucket/key') and open('file.txt') converge on one API
You use pandas, dask, polars, pyarrow, or zarr with remote data: storage_options you pass there are fsspec arguments, so learning it explains those errors
You need the extras around remote files: transparent compression by extension, local caching of remote reads (simplecache::), or reading inside zip/tar archives via chained URLs
You are writing a library and want users to bring their own storage; subclassing AbstractFileSystem gets you every backend feature for free

Skip it if

You only ever touch local files: pathlib and open() are simpler, faster to reason about, and do not add a dependency with 300+ open issues
You only use one cloud and want its full feature set: boto3 (or the provider SDK) exposes multipart tuning, ACLs, and presigned URLs that the generic interface abstracts away or hides
You need strict version discipline: the core is CalVer and backends like s3fs pin fsspec to matching releases, so mixed pins across your dependency tree regularly produce resolver conflicts
Filesystem semantics matter to you precisely: backends differ on atomicity, listing consistency, and what glob or mtime mean on object stores, and code assuming POSIX behavior breaks quietly

Setup reality

pip install fsspec gets you local, memory, and archive backends only; every interesting protocol is an extra install (s3fs for s3://, gcsfs for gs://, adlfs for abfs://, aiohttp for http://), and the error when one is missing names the package but still interrupts your day. Versions are CalVer and backends release in lockstep, so upgrading s3fs usually drags fsspec along. Directory listings are cached per filesystem instance, which surprises people polling for new files until they call invalidate_cache(). Async use has its own rules: you must ask for an async instance and not mix it with the sync API casually.

Patterns

Open a file by URL, any backendopen-remote-file

import fsspec

with fsspec.open("s3://mybucket/data.csv", "r") as f:
    header = f.readline()

with fsspec.open("data/local.csv", "r") as f:
    print(f.readline())

fsspec.open returns an OpenFile; the real file object only exists inside the with block. s3:// requires pip install s3fs, gs:// needs gcsfs, http(s):// needs aiohttp.

Get a filesystem object and browse itfilesystem-instance

import fsspec

fs = fsspec.filesystem("s3", anon=False)
print(fs.ls("mybucket/raw"))
print(fs.glob("mybucket/raw/*.parquet"))
print(fs.info("mybucket/raw/part-0.parquet"))

Instances are cached: calling filesystem() twice with the same arguments returns the same object, including its directory-listing cache.

Copy files between remote and localdownload-upload

import fsspec

fs = fsspec.filesystem("s3")
fs.get("mybucket/model.bin", "/tmp/model.bin")          # download
fs.put("/tmp/results.json", "mybucket/out/results.json")  # upload
fs.get("mybucket/data/", "/tmp/data/", recursive=True)

get and put take (remote, local) and (local, remote) respectively; recursive=True copies trees. Trailing slashes matter for whether the source dir itself is included.

Cache remote files locally with a chained URLcache-remote-reads

import fsspec

with fsspec.open(
    "simplecache::s3://mybucket/big.parquet",
    s3={"anon": True},
    simplecache={"cache_storage": "/tmp/fsspec-cache"},
) as f:
    data = f.read()

The :: chain composes filesystems; simplecache downloads whole files once, filecache adds expiry, blockcache caches ranges. Per-layer options are passed as keyword dicts named after the protocol.

Read a file inside a zip on remote storageread-inside-archive

import fsspec

with fsspec.open("zip://inner/data.csv::s3://mybucket/archive.zip") as f:
    print(f.read(100))

fs = fsspec.filesystem("zip", fo="archive.zip")
print(fs.ls("/"))

URL chaining reads right to left: fetch archive.zip from S3, then resolve inner/data.csv inside it. Zip requires random access, so the outer backend must support seeks.

Read and write compressed files transparentlycompression-on-open

import fsspec

with fsspec.open("data.csv.gz", "rt", compression="infer") as f:
    print(f.readline())

with fsspec.open("out.txt.zst", "wt", compression="zstd") as f:
    f.write("hello")

compression='infer' picks the codec from the file extension. Non-gzip codecs (zstd, lz4, snappy) need their Python packages installed separately.

Use the memory filesystem in testsmemory-filesystem-tests

import fsspec

fs = fsspec.filesystem("memory")
with fs.open("/bucket/sample.txt", "wb") as f:
    f.write(b"test data")

assert fs.cat("/bucket/sample.txt") == b"test data"

memory:// is process-global and persists between filesystem() calls in one process, so clean up between tests (fs.rm or fs.store.clear()) to avoid cross-test leakage.

Pass fsspec options through pandaspandas-storage-options

import pandas as pd

df = pd.read_parquet(
    "s3://mybucket/data.parquet",
    storage_options={"anon": True},
)
df.to_csv("s3://mybucket/out.csv", storage_options={"profile": "prod"})

storage_options in pandas, dask, and polars is forwarded to the fsspec backend constructor; the valid keys are whatever s3fs/gcsfs accept, not pandas options.

List files with sizes and typeslist-with-details

import fsspec

fs = fsspec.filesystem("s3")
for entry in fs.ls("mybucket/logs", detail=True):
    print(entry["name"], entry["size"], entry["type"])

files = fs.find("mybucket/logs")  # recursive file list

detail=True returns dicts instead of path strings. find() walks recursively and returns files only; use fs.du() for total sizes.

See new files after a listing was cachedinvalidate-listing-cache

import fsspec

fs = fsspec.filesystem("s3")
fs.ls("mybucket/incoming")
# another process uploads a file...
fs.invalidate_cache("mybucket/incoming")
print(fs.ls("mybucket/incoming"))  # now fresh

Directory listings are cached on the instance, and instances themselves are cached, so a polling loop that never invalidates can miss new files forever. This is the top fsspec bug report.

Open a set of files matching a patternopen-many-files

import fsspec

files = fsspec.open_files("s3://mybucket/logs/2026-08-*.jsonl", "rt")
for of in files:
    with of as f:
        process(f)

open_files expands the glob once and returns lazy OpenFile objects, which is how dask builds partitioned reads; each must still be used as a context manager.

Register a custom filesystem backendcustom-filesystem

from fsspec.spec import AbstractFileSystem
from fsspec.registry import register_implementation

class MyFS(AbstractFileSystem):
    protocol = "myfs"

    def ls(self, path, detail=False, **kwargs):
        ...

register_implementation("myfs", MyFS)
# now fsspec.open("myfs://...") works everywhere

Implement _open, ls, and info and most derived operations (glob, find, walk, cat) come from the base class. Registration makes the protocol usable from pandas and dask too.

Alternatives

Package	Registry	Pick it when
smart-open	PyPI	You mainly want streaming open() for large files on S3/GCS/Azure/HTTP with minimal API surface, not a full filesystem.
cloudpathlib	PyPI	You want pathlib-style Path objects (CloudPath) for cloud storage with local caching, and prefer that idiom to filesystem objects.
obstore	PyPI	You want a fast Rust-backed object store client with a smaller, stricter API and are willing to leave the fsspec ecosystem hooks behind.
boto3	PyPI	You are S3-only and need the full AWS surface (presigned URLs, ACLs, multipart control) rather than a generic interface.