cloudpathlib
cloudpathlib gives Amazon S3, Google Cloud Storage, Azure Blob Storage, and HTTP URLs objects that feel much like Python's pathlib.Path. You can join a cloud URI with `/`, call read_text, open, glob, copy, upload_from, or download_to, and let the library dispatch to the right provider SDK. Remote files are staged through a configurable local cache, while AnyPath can accept either an ordinary filesystem path or a supported cloud URI and return the matching path type.
A good adapter when pathlib-shaped application code matters more than exposing every cloud-storage control. Install only the provider extras you need, configure the cache deliberately, and use native SDKs for async workloads or advanced transfer semantics.
Use it if
- Your code already accepts pathlib-style objects and you want local, S3, GCS, and Azure paths behind a similar interface
- You need ordinary file operations such as open, glob, read_text, upload_from, and download_to more than provider-specific storage features
- A local on-disk cache helps repeated reads or lets libraries that require filesystem paths consume cloud objects
- You want test doubles that replace cloud clients with the package's LocalS3Client, LocalGSClient, or LocalAzureBlobClient
- You need a complete pathlib replacement: the README's compatibility table marks chmod, symlink_to, readlink, owner, as_posix, cwd, home, and several other filesystem operations unsupported
- Your service is async-first: the public file operations and provider SDK integrations are synchronous, so large transfers and listings block unless you move them off the event loop
- You need one abstraction across many storage systems: the README lists S3, GCS, Azure, and read-oriented HTTPS support, while FTP is still described as on the way
- You need provider-native controls on every request: wrapping boto3, google-cloud-storage, or azure-storage can hide pagination, retries, conditional requests, metadata, request costs, and service-specific error details
- You cannot tolerate local staging: open and several pathlib-like operations download objects into a cache, and stat can download an entire file when the provider-specific implementation lacks a metadata shortcut
Setup reality
The base `pip install cloudpathlib` is intentionally not enough for S3, GCS, or Azure access. Install the matching extra, such as `cloudpathlib[s3]`, `cloudpathlib[gs]`, `cloudpathlib[azure]`, or `cloudpathlib[all]`; otherwise the README says you can only build against the base classes. Those extras bring boto3, google-cloud-storage, or both Azure blob packages, so environment size follows the provider SDK. Authentication is delegated to those SDKs: AWS profiles and environment variables, Google Application Default Credentials or a service-account file, and Azure account URLs, credentials, or `AZURE_STORAGE_CONNECTION_STRING`. Azure raises when credentials are absent, while S3 and GCS can fall back to anonymous access for public data. Reads and writes use a local cache. The default is a temporary directory, a supplied cache directory implies persistent mode, and `close_file` cleanup only works when callers use `CloudPath.open`. Conflicting local and remote modifications raise instead of silently choosing a winner unless you explicitly force an overwrite. Directory operations map onto object-key prefixes, not real directories, and a seemingly cheap pathlib call may list keys, make metadata requests, or transfer a whole object. Set timeouts and retry behavior on the provider client where supported, watch cache disk usage, and never assume local filesystem atomicity or rename behavior applies to object storage.
Patterns
Create the provider path from a URIdispatch-cloud-uri
from cloudpathlib import CloudPath
path = CloudPath('s3://reports-prod/2026/summary.json')
print(type(path).__name__) # S3Path
print(path.bucket, path.key)CloudPath dispatches from the URI prefix. The matching provider extra must be installed before the concrete client can be created.
Accept a local path or cloud URIaccept-local-or-cloud
from cloudpathlib import AnyPath
def load_text(location: str) -> str:
path = AnyPath(location)
return path.read_text(encoding='utf-8')AnyPath returns pathlib.Path for local inputs and CloudPath for recognized cloud prefixes; the same call may therefore perform network I/O.
Read and write a text objectread-write-text
from cloudpathlib import S3Path
source = S3Path('s3://my-bucket/input/message.txt')
text = source.read_text(encoding='utf-8')
(source.parent / 'output.txt').write_text(text.upper(), encoding='utf-8')Reads are cached locally and writes upload on close. This is synchronous I/O and should not run directly on an async event loop.
Use an open-style context managerstream-with-open
from cloudpathlib import CloudPath
path = CloudPath('gs://analytics/events.ndjson')
with path.open('rt', encoding='utf-8') as stream:
for line in stream:
process(line)The object is downloaded to the local cache before normal file reading. It is not a provider-native streaming body.
Find objects recursivelyglob-cloud-objects
from cloudpathlib import CloudPath
root = CloudPath('azure://exports/2026/')
for item in root.glob('**/*.parquet'):
print(item, item.stat().st_size)Globbing maps to remote listings and can be expensive on a broad prefix. The generic stat implementation may download a file, as the source warns.
Download to a local destinationdownload-file-or-tree
from pathlib import Path
+from cloudpathlib import CloudPath
remote = CloudPath('s3://my-bucket/releases/v4/')
local = remote.download_to(Path('artifacts/v4'))
print(local)A directory download walks children recursively. Budget for listings, transfer time, and existing local files before using this on a large prefix.
Upload a file or directoryupload-file-or-tree
from cloudpathlib import CloudPath
destination = CloudPath('gs://my-bucket/site/')
uploaded = destination.upload_from('dist')
print(uploaded)Directories are traversed recursively. If local cache and cloud copies have diverged, the library can require force_overwrite_to_cloud=True instead of guessing.
Bind an S3 path to an explicit profileuse-aws-profile
from cloudpathlib import S3Client, S3Path
client = S3Client(profile_name='production')
path = S3Path('s3://private-reports/latest.csv', client=client)
print(path.read_text())Passing a client makes credential and cache ownership explicit. Avoid embedding access keys in source; boto3's normal profile and role chain still applies.
Use unsigned requests for public S3 dataread-public-s3
from cloudpathlib import S3Client, S3Path
client = S3Client(no_sign_request=True)
public = S3Path('s3://drivendata-public-assets/', client=client)
print([p.name for p in public.iterdir()])Unsigned mode deliberately skips credentials and can access only public resources. It should not be used as a fallback for a broken private-bucket credential setup.
Keep a controlled persistent cacheconfigure-persistent-cache
from cloudpathlib import FileCacheMode, S3Client, S3Path
client = S3Client(
local_cache_dir='/var/tmp/report-cache',
file_cache_mode=FileCacheMode.persistent,
)
report = S3Path('s3://reports/monthly.pdf', client=client)Persistent mode is never removed by cloudpathlib. Monitor disk use and arrange cleanup; a persistent mode without local_cache_dir is rejected.
Clear cached copies after a batchclear-local-cache
from cloudpathlib import CloudPath
path = CloudPath('s3://reports/big.csv')
try:
consume(path.read_bytes())
finally:
path.clear_cache()Clearing removes local cached data, not the cloud object. Other path objects sharing a client may depend on the same cache directory.
Use the local S3 test implementationmock-s3-in-tests
from pathlib import Path
+from cloudpathlib.local import LocalS3Client
+
client = LocalS3Client(local_storage_dir=Path('tests/fixtures/storage'))
path = client.CloudPath('s3://fixtures/input.txt')
assert path.read_text() == 'hello'The local clients mimic cloudpathlib behavior, not every S3 permission, consistency, metadata, or provider error. Keep a smaller set of tests against the real service.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| fsspec | PyPI | You need a wider protocol ecosystem, dataframe integrations, async-capable backends, or file-like URLs more than pathlib fidelity |
| s3fs | PyPI | Your scope is S3 and you want fsspec integration for pandas, Dask, and related data tools |
| smart-open | PyPI | You mostly stream large objects through open-style readers and writers without a persistent local cache |
| universal-pathlib | PyPI | You want pathlib-like objects backed by the broader fsspec protocol catalog |