PyPIDataupdated 06 Aug 2026

gcsfs

Item: gcsfs
Rating: 4.3
Author: Keyoor

gcsfs makes Google Cloud Storage look like a filesystem. It implements the fsspec interface, so once you build a GCSFileSystem you get ls, glob, walk, open, cat, cp, mv, rm, du, and the rest of the vocabulary you already know, backed by GCS HTTP calls instead of a disk. The bigger payoff is indirect: because pandas, dask, pyarrow, zarr, xarray, and polars all resolve paths through fsspec, installing gcsfs is what makes pd.read_parquet('gs://bucket/data.parquet') work at all. Files opened for reading behave like Python file objects with block caching, so gzip and csv readers can stream from them, and writes buffer in memory and flush in chunks. Underneath it is async, built on aiohttp, and it exposes both a synchronous API and an async one where every method is the same name with a leading underscore.

gcsfs · v2026.7.0 · 34.9M/wk · ★ 394 · pushed 05 Aug 2026 · fsspec/gcsfs · docs · reviewed by Keyoor

Verdict

The standard way to make GCS look like a filesystem to the Python data stack, and effectively mandatory if anything in your stack reads gs:// URLs. If nothing does, skip it and use google-cloud-storage, because the abstraction costs you eight transitive dependencies and a caching model you have to remember.

API stability4/5The fsspec interface it implements barely moves, and GCSFileSystem's constructor arguments have been stable for years. The wobble is underneath: the default implementation recently became ExtendedGcsFileSystem for all bucket types, documented with a warning and an environment variable to revert, and the calendar versioning means a floor bump on fsspec every month.

Docs4/5gcsfs.readthedocs.io covers every credential mode, async usage, proxies, endpoint overrides, retry behaviour with the exact backoff formula, and has dedicated pages for HNS buckets, rapid storage, and the prefetcher. The README is thinner and reads like marketing, and most method-level detail lives in the inherited fsspec docs rather than here.

Maintenance4/5Pushed the same week as this writing, with monthly releases in step with fsspec and visible investment from Google and Anaconda in benchmarks, zonal bucket support, and zero-copy write paths. Against that, 86 open issues (97 counting PRs), and a changelog where a large share of entries are benchmark infrastructure rather than user-facing fixes.

Ecosystem5/5Around 34.9M weekly downloads and the only real answer for gs:// support in pandas, dask, pyarrow, zarr, xarray, and polars, which all discover it through the fsspec entry point. It sits in the fsspec organisation alongside s3fs and adlfs, so the interface is shared across clouds.

Use it if

You are reading or writing GCS from the Python data stack and want gs:// URLs to just resolve in pandas, dask, pyarrow, polars, or xarray without writing any GCS code
You need one code path across local disk, S3, and GCS: fsspec.open('gs://...') and fsspec.open('s3://...') behave the same, which is how you make a pipeline portable between clouds
You want filesystem verbs GCS does not natively provide: recursive glob, walk, du, and bulk cp, mv, and rm that fan out concurrently instead of looping one object at a time
You are storing zarr arrays or a partitioned parquet dataset on GCS and need a mutable key-value mapping via fs.get_mapper()
You are on a Hierarchical Namespace bucket and want atomic directory renames, which recent versions route through the native GCS Folders API instead of copy-then-delete

Skip it if

You are doing plain object get and put inside one application. google-cloud-storage is Google's own client, has fewer moving parts, and gcsfs now depends on it anyway, so the filesystem abstraction is pure overhead if nothing else consumes it
You need a lean dependency tree. Installing gcsfs pulls aiohttp, requests, decorator, fsspec, google-auth, google-auth-oauthlib, google-cloud-storage 3.11 or newer, and google-cloud-storage-control. That is a lot of transitive surface for reading a file
You cannot live with calendar versioning in lockstep with fsspec. gcsfs 2026.7.0 requires fsspec 2026.6.0 or newer, releases follow fsspec monthly, and mixing pinned versions of gcsfs, s3fs, and fsspec in one environment is a recurring resolver fight
You expect POSIX semantics. On a standard flat bucket there are no real directories, mkdir creates a bucket rather than a folder, rename is a copy followed by a delete that is neither atomic nor cheap, and there is no append and no locking
You will be surprised by the listing cache. Directory listings never expire by default, so an object another process wrote after your first ls stays invisible until you call fs.invalidate_cache() or construct the filesystem with cache_timeout
Your workload is thousands of small objects on a hot path. Every operation is an HTTP round trip, and while the project has added prefetching and zero-copy paths, a Python client over REST is still a different performance class from local disk or a mounted filesystem
You want a stable default implementation. Recent versions switched the default entry point to ExtendedGcsFileSystem for all bucket types to support HNS and zonal buckets, with the docs offering GCSFS_EXPERIMENTAL_ZB_HNS_SUPPORT=false as the escape hatch if that change breaks you

Setup reality

pip install gcsfs or conda install -c conda-forge gcsfs, Python 3.10 or newer, and then most of the work is credentials. With token=None gcsfs tries your gcloud application default credentials, then the metadata service, then falls back to anonymous, which is convenient right up until it silently lands on anonymous and you get a confusing 403 instead of an auth error. Supplying the project explicitly is recommended even though it is only needed for bucket-level calls, because omitting it raises a warning. Other token values are 'cloud' for the metadata service, 'anon' for public data, 'browser' for an OAuth flow cached in ~/.gcs_tokens, 'cache' to reuse that, or a dict, a raw token string, a path to a JSON key, or a google.auth credentials object. Note that ~ is not expanded for you, so pass os.path.expanduser() results. Two more environment surprises: aiohttp ignores HTTP_PROXY and HTTPS_PROXY unless you pass session_kwargs={'trust_env': True}, and fsspec caches filesystem instances by constructor arguments, so calling GCSFileSystem() again with different credentials can hand you back the old instance. If you use the async API you must pass asynchronous=True and await the client creation before the first call, and open() has no async form at all. Retries default to 6 with exponential backoff and jitter and no total deadline, so a persistently failing call takes a while to give up.

Patterns

Connect and list a bucketcreate-filesystem

import gcsfs

fs = gcsfs.GCSFileSystem(project='my-google-project')

fs.ls('my-bucket')                 # ['my-bucket/data.csv', ...]
fs.ls('my-bucket/logs', detail=True)  # dicts with size, updated, etag
fs.exists('my-bucket/data.csv')
fs.info('my-bucket/data.csv')

Paths have no gs:// prefix once you hold a filesystem object; the scheme is only for URLs handed to fsspec or pandas. Omitting project raises a warning and disables bucket listing and creation, but object operations still work.

Open objects like filesread-and-write-files

with fs.open('my-bucket/data.txt', 'rb') as f:
    content = f.read()

with fs.open('my-bucket/out.txt', 'wb') as f:
    f.write(b'hello')

# text mode and compression are handled by fsspec
with fs.open('my-bucket/rows.csv', 'r', encoding='utf-8') as f:
    header = f.readline()

Writes buffer in memory and only become visible in GCS when the file is closed or a block boundary flushes, so an exception mid-write can leave nothing behind. Always use the context manager rather than a bare open.

Read and write dataframes straight from gs:// URLsuse-with-pandas

import pandas as pd

df = pd.read_parquet('gs://my-bucket/events/date=2026-08-01/')

df.to_csv(
    'gs://my-bucket/exports/summary.csv',
    storage_options={'token': 'anon'},
)

You never import gcsfs here; pandas asks fsspec for a gs handler and fsspec finds gcsfs by entry point. storage_options is passed straight to the GCSFileSystem constructor, which is where credentials, project, and requester_pays go.

Pick an authentication mode explicitlychoose-credentials

import os
import gcsfs

fs = gcsfs.GCSFileSystem(token=None)        # ADC, then metadata, then ANONYMOUS
fs = gcsfs.GCSFileSystem(token='cloud')     # metadata service only
fs = gcsfs.GCSFileSystem(token='anon')      # public buckets
fs = gcsfs.GCSFileSystem(
    token=os.path.expanduser('~/keys/sa.json'),
    project='my-google-project',
)

The default chain silently falls through to anonymous access, which turns a missing-credentials bug into a 403 on your own bucket. In production, name the mode. Also note that ~ is not expanded for you, hence the expanduser call.

See objects another process just wroteinvalidate-the-cache

fs.ls('my-bucket/incoming')     # cached from here on
# ... another job uploads a file ...
fs.ls('my-bucket/incoming')     # still the old listing

fs.invalidate_cache('my-bucket/incoming')
fs.ls('my-bucket/incoming')     # fresh

# or set an expiry up front
fs = gcsfs.GCSFileSystem(cache_timeout=60)  # seconds; <= 0 disables caching

Directory listings are cached with no expiry by default. This is the single most common gcsfs surprise: a polling loop that never sees new files. cache_timeout=0 turns caching off entirely at the cost of an extra request per listing.

Upload and download whole treesbulk-transfer

fs.put('local_dir/', 'my-bucket/backup/', recursive=True)
fs.get('my-bucket/backup/', 'restored/', recursive=True)

fs.cp('my-bucket/a/', 'my-bucket/b/', recursive=True)
fs.rm('my-bucket/tmp/', recursive=True)

These fan out concurrently under the async layer, which is the main speed win over looping with the plain client. On a standard flat bucket, cp and mv are copy-then-delete per object and are neither atomic nor free; on an HNS bucket a directory rename becomes a single metadata call.

Read or write many small objects in one gobulk-bytes

blobs = fs.cat(['my-bucket/a.json', 'my-bucket/b.json'])
# {'my-bucket/a.json': b'...', 'my-bucket/b.json': b'...'}

fs.pipe({'my-bucket/x.txt': b'one', 'my-bucket/y.txt': b'two'})

head = fs.cat_file('my-bucket/big.csv', start=0, end=4096)

cat and pipe issue their requests concurrently, so they are dramatically faster than a loop of open calls for hundreds of small objects. cat_file with start and end sends a ranged request and avoids downloading the whole object.

Find objects by patternglob-and-walk

fs.glob('my-bucket/logs/2026-*/*.json')
fs.find('my-bucket/logs')            # every object below the prefix

for root, dirs, files in fs.walk('my-bucket/logs'):
    print(root, len(files))

All three enumerate objects by prefix, so cost scales with the number of keys under the path, not with how many match your pattern. A glob at the root of a bucket with millions of objects will take a long time and then cache the whole listing in memory.

Expose a prefix as a key-value storezarr-and-mappers

store = fs.get_mapper('my-bucket/arrays/experiment.zarr')

import zarr
z = zarr.open(store, mode='w', shape=(1000, 1000), chunks=(100, 100))
z[:] = 42

get_mapper returns a MutableMapping, which is what zarr and similar chunked formats expect. The older gcsfs.mapping.GCSMap function still exists but only forwards to get_mapper for backward compatibility.

Hand out a temporary download linksigned-urls

url = fs.sign('my-bucket/reports/q3.pdf', expiration=3600)

This delegates to google-cloud-storage and needs credentials that can sign, meaning a service account key or the IAM signBlob permission. Default application credentials from a user account will fail here even though every other call works.

Read a specific generation of an objectobject-versions

fs = gcsfs.GCSFileSystem(project='my-google-project', version_aware=True)

fs.ls('my-bucket/data.csv', versions=True)

with fs.open('my-bucket/data.csv#1712345678901234', 'rb') as f:
    old = f.read()

version_aware must be set on the filesystem, and it changes how paths are split, so the trailing #generation syntax only works when it is on. The bucket also has to have object versioning enabled or there is nothing to read.

Call gcsfs from async codeasync-usage

import asyncio, gcsfs

async def main():
    fs = gcsfs.GCSFileSystem(asynchronous=True)
    await fs._set_session()         # create the aiohttp client first
    paths = await fs._ls('my-bucket')
    data = await fs._cat_file('my-bucket/data.json')
    return paths, data

asyncio.run(main())

Every synchronous method has an underscore-prefixed coroutine twin, except open, which has no async form; download to a temporary file instead. Touching fs.session before the client exists raises 'Please await _connect* before anything else', which is the error that tells you asynchronous=True was set but the session was never created.

Alternatives

Package	Registry	Pick it when
google-cloud-storage	PyPI	You are writing GCS-specific code and want Google's supported client with resumable uploads and full bucket administration.
smart-open	PyPI	You only need to stream single files by URL across GCS, S3, HTTP, and local paths with minimal dependencies.
universal-pathlib	PyPI	You want pathlib.Path semantics over gs:// URLs; it sits on fsspec and still uses gcsfs underneath.
fsspec	PyPI	You want the generic filesystem API and caching layers, and will add the backend for whichever cloud you land on.