google-cloud-storage
The official Google-maintained Python client for Google Cloud Storage. You get a Client, then Bucket and Blob objects that wrap the JSON API: upload and download files or streams, list objects by prefix, read and write object metadata, manage ACLs and IAM, and mint signed URLs so a browser can upload or download directly without going through your server. It handles credential discovery, request signing, resumable uploads, chunked downloads, and crc32c checksum verification for you. If your Python code touches a GCS bucket, this is the library, and buckets are also reachable through gcsfs or smart_open, which are wrappers over the same API rather than replacements.
The right default for GCS from Python, kept current by Google with releases most months. Plan for the synchronous API, the size of the dependency chain, and the fact that signed URLs on keyless hosts need an IAM permission you have to request deliberately.
Use it if
- You read or write objects in Google Cloud Storage from Python and want the supported client rather than raw REST calls you sign yourself
- You run on Cloud Run, GKE, Cloud Functions, or Compute Engine, where the metadata server supplies credentials and this library picks them up with no configuration
- You need signed URLs so browsers can upload or download directly against the bucket without proxying bytes through your app
- You want correctness details handled for you: resumable uploads, retries with backoff, generation preconditions for safe overwrites, and crc32c verification on both sides of a transfer
- You need async. The Client, Bucket, and Blob API is synchronous and blocks the event loop. There is an asyncio module now, but it covers specific paths such as appendable writers and multi-range downloads rather than being a drop-in async client, so asyncio services usually reach for gcloud-aio-storage or a thread pool
- You mostly want a filesystem: gcsfs plugs into pandas, dask, and anything fsspec-aware, so pd.read_parquet('gs://bucket/file.parquet') just works instead of a download-to-temp-file dance
- Container size or cold start time is tight. The dependency chain pulls google-auth, google-api-core, google-cloud-core, requests, and google-crc32c, and the grpc extra adds grpcio and protobuf on top, which is tens of megabytes in a function image
- You need signed URLs on a host without a private key file. Compute or workload identity credentials cannot sign locally, so you must enable the IAM Credentials API and grant the service account the token creator role on itself, which is exactly the permission most security reviews push back on
- You need one storage API across clouds: this is GCS only, while smart_open and fsspec give you the same call shape for S3, Azure Blob, and local paths
- You are pinned to Python 3.9 or older, or to code written against version 2: 3.x requires Python 3.10 or newer, changed the default checksum strategy for uploads and downloads to auto, removed the num_retries and text_mode arguments, renamed from_string to from_uri, and turned retries on by default for uploads, blob deletes, and metadata updates
Setup reality
pip install google-cloud-storage needs Python 3.10 or newer and drags in google-auth, google-api-core, google-cloud-core, requests, and google-crc32c; the grpc extra adds grpcio and protobuf if you want it. The install is the easy part. Credentials come from Application Default Credentials, which means a GOOGLE_APPLICATION_CREDENTIALS path to a key file, a gcloud auth application-default login on your laptop, or the metadata server in Google-hosted environments, and the failure mode when none of them resolve is a DefaultCredentialsError at client construction rather than at the call site. The project is inferred from those credentials, so a key file from the wrong project fails later with a confusing 403 instead of an obvious error. Local testing against fake-gcs-server needs the STORAGE_EMULATOR_HOST environment variable, which is not in the README. One more thing worth knowing: this package used to live in googleapis/python-storage and was folded into the googleapis/google-cloud-python monorepo, so the issue tracker you land on now carries hundreds of open issues spanning every Google Cloud Python library, not just storage.
Patterns
Create a client and reference a bucketclient-setup
from google.cloud import storage
client = storage.Client(project="my-project")
bucket = client.bucket("my-bucket") # no API call
blob = bucket.blob("reports/june.csv") # no API callclient.bucket() only builds a local reference. Use client.get_bucket() when you actually want to confirm the bucket exists and fetch its metadata, since it costs a request.
Upload a file and a stringupload-file
blob = bucket.blob("reports/june.csv")
blob.upload_from_filename("june.csv")
bucket.blob("config.json").upload_from_string(
'{"mode": "live"}', content_type="application/json"
)Since 3.0 the default checksum strategy is auto, so uploads are verified without you asking. Content type is guessed from the filename on upload_from_filename but defaults to text/plain on upload_from_string.
Download to a file or into memorydownload-file
blob = bucket.blob("reports/june.csv")
blob.download_to_filename("/tmp/june.csv")
data = bucket.blob("config.json").download_as_bytes()
text = bucket.blob("config.json").download_as_text()download_as_bytes pulls the whole object into memory, which is how workers get killed on large files. Since 3.0, download_to_filename removes the empty destination file when the object turns out to be missing.
Stream a large object line by linestream-large-object
with bucket.blob("logs/2026-08-05.jsonl").open("r") as f:
for line in f:
handle(line)
with bucket.blob("out/report.csv").open("w") as f:
writer = csv.writer(f)
writer.writerows(rows)blob.open() gives you a file object backed by chunked reads and resumable writes, so memory stays flat. A write is only committed when the context manager exits cleanly; an exception cancels the upload.
List objects under a prefixlist-blobs
for blob in client.list_blobs("my-bucket", prefix="logs/2026/"):
print(blob.name, blob.size, blob.updated)
# folder-style listing
iterator = client.list_blobs("my-bucket", prefix="logs/", delimiter="/")
names = [b.name for b in iterator]
folders = list(iterator.prefixes)The iterator pages transparently, so listing a huge prefix keeps making requests as you loop. iterator.prefixes is only populated after you have consumed the iterator.
Generate a V4 signed URLsigned-url
from datetime import timedelta
url = bucket.blob("reports/june.csv").generate_signed_url(
version="v4",
expiration=timedelta(minutes=15),
method="GET",
)Signing needs a private key, so this works when the client was built from a service account JSON file. V4 URLs cap out at seven days of validity.
Sign a URL from Cloud Run or GKE with no key filesigned-url-without-key
import google.auth
from google.auth.transport import requests as ga_requests
from datetime import timedelta
credentials, _ = google.auth.default()
credentials.refresh(ga_requests.Request())
url = bucket.blob("reports/june.csv").generate_signed_url(
version="v4",
expiration=timedelta(minutes=15),
service_account_email=credentials.service_account_email,
access_token=credentials.token,
)This routes signing through the IAM Credentials API, so that API must be enabled and the service account needs roles/iam.serviceAccountTokenCreator on itself. Without the role you get a 403 that names signBlob, not signed URLs.
Avoid clobbering with generation preconditionsoverwrite-preconditions
# create only if the object does not exist yet
blob.upload_from_filename("june.csv", if_generation_match=0)
# update only if nobody changed it since we read it
blob.reload()
blob.upload_from_string(new_body, if_generation_match=blob.generation)Without a precondition, two concurrent writers silently produce a last-write-wins result. A mismatch raises google.api_core.exceptions.PreconditionFailed, which you can catch and retry around.
Tune retries and timeouts per callretry-and-timeout
from google.cloud.storage.retry import DEFAULT_RETRY
retry = DEFAULT_RETRY.with_deadline(30.0)
blob.upload_from_filename("june.csv", retry=retry, timeout=60)Since 3.0 uploads, blob deletes, and metadata updates retry by default, where version 2 left them unretried. Pass retry=None on an operation you would rather fail fast than repeat.
Read and update object metadataobject-metadata
blob = bucket.get_blob("reports/june.csv") # fetches metadata, None if missing
print(blob.content_type, blob.size, blob.crc32c, blob.generation)
blob.content_type = "text/csv"
blob.cache_control = "public, max-age=3600"
blob.metadata = {"source": "nightly-job"}
blob.patch()bucket.blob() returns an unpopulated reference whose attributes are all None until reload() or get_blob(). Use patch() rather than update() so you do not wipe fields you never read.
Upload or download many files in parallelparallel-transfer
from google.cloud.storage import transfer_manager
results = transfer_manager.upload_many_from_filenames(
bucket,
["a.csv", "b.csv", "c.csv"],
source_directory="./out",
max_workers=8,
)
for name, result in zip(["a.csv", "b.csv", "c.csv"], results):
if isinstance(result, Exception):
print("failed", name, result)Failures come back inside the results list instead of raising, so a loop that ignores the return value will report success on a partial upload. The default worker type is processes, which pickles the client and does not suit every runtime.
Turn a gs:// URI into a blobblob-from-uri
from google.cloud import storage
blob = storage.Blob.from_uri("gs://my-bucket/reports/june.csv", client=client)
if blob.exists():
blob.delete()This was Blob.from_string() before 3.0 and the old name is gone, which is why copied snippets from older tutorials raise AttributeError.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| gcsfs | PyPI | You want gs:// paths to behave like files for pandas, dask, pyarrow, or anything else built on fsspec |
| gcloud-aio-storage | PyPI | You need genuinely async GCS calls inside an asyncio service and can accept a community-maintained client |
| smart_open | PyPI | You stream large objects across GCS, S3, and local paths and want one open() call shape for all of them |