PyPIDataupdated 06 Aug 2026

smart-open

Item: smart-open
Rating: 3.8
Author: Keyoor

smart_open gives you one open() function that works whether the file lives on local disk, in S3, in Google Cloud Storage, in Azure Blob Storage, on HDFS or WebHDFS, behind an http(s) URL, or on an SFTP host. You pass a URI instead of a path, get back a normal file object, and iterate it line by line or read it in chunks without pulling the whole thing into memory. It also handles compression on the way past: a name ending in .gz, .bz2, .xz, .zst, or .lz4 is decompressed on read and compressed on write with no extra code. The point is that your data-loading code stops branching on where the bytes are. Swap the URI, keep the loop. It is a drop-in stand-in for the builtin open(), and falls back to the builtin for local paths.

smart-open · v8.0.1 · 17.0M/wk · ★ 3,455 · pushed 04 Aug 2026 · piskvorky/smart_open · docs · reviewed by Keyoor

Verdict

The cleanest answer when you want the file-object model and nothing else: one open(), any URI, transparent compression, minimal install. Reach for fsspec or s3fs instead the moment you need to list, glob, or delete, or when the rest of your stack already speaks fsspec.

API stability3/5open() itself has looked the same for years, but 8.0.0 in June 2026 removed a batch of long-deprecated entry points and changed s3.iter_bucket to take a session_kwargs dict; the project keeps a MIGRATING file documenting every major back to v2, which tells you how often majors break call sites.

Docs3/5The README is unusually good, with runnable doctests for compression, S3, GCS, and Azure, and there are separate HOWTO and EXTENDING files; the actual API reference is a generated help.txt with no hosted site, so per-backend transport_params are hard to find.

Maintenance5/5Pushed August 2026 with only 2 open issues (3 counting PRs) against 3,455 stars, releases roughly monthly through 2026, and a changelog that links every change to a PR; nearly all recent work comes from one contributor, so the bus factor is the risk, not the backlog.

Ecosystem4/5About 17 million weekly downloads, much of it transitive (gensim, from the same author, pins smart_open>=1.8.1), and documented extras for eight URI schemes; it stays a single-function library, so extension means register_compressor or writing a transport per EXTENDING.md rather than installing plugins.

Use it if

The same loading code has to run against a local file in development and an s3:// or gcs:// object in production, and you do not want an if-statement per backend
You process objects too large to hold in memory and want line iteration, chunked read(), and seek() without downloading the whole file first
You write results straight to object storage and would rather not assemble boto3 multipart-upload boilerplate around a BytesIO wrapper
Your files are compressed with mixed extensions and you want decompression picked from the filename in both directions, or forced with compression='.gz' when the extension lies

Skip it if

You need a filesystem, not a file: there is no ls, glob, rm, exists, or rename here, only open() plus the S3-specific iter_bucket helper, so fsspec or s3fs is the better fit for anything catalog-shaped
You already pass paths into pandas, dask, or pyarrow: those go through fsspec, so adding smart_open means two storage abstractions in one project with two sets of credential quirks
You are S3-only and already carry boto3 for presigned URLs, tagging, or lifecycle work: one extra dependency to save a few lines around upload_fileobj is a thin trade
You maintain older code: 8.0.0 (June 2026) deleted the top-level smart_open.smart_open() wrapper, smart_open.s3_iter_bucket, concurrency.create_pool, and the s3://key:secret@host:port@bucket URI form, so most tutorials written before mid-2026 no longer run
You want a documentation site: the API reference is a generated help.txt plus help('smart_open') in the REPL, so learning the transport_params for your backend means reading a text dump or the source

Setup reality

pip install smart-open installs essentially nothing (only wrapt) because every backend is an optional extra, which is the right default and also the first thing that bites you. Reading an s3:// URI without pip install 'smart_open[s3]' fails at call time, not at import, so a missing extra shows up in production rather than in CI. Pick your extras explicitly, as in pip install 'smart_open[s3,gcs,zst]', or take 'smart_open[all]' and accept boto3, google-cloud-storage, azure-storage-blob, paramiko, and requests all arriving together. A few transports need packages that are not extras at all: HTTP Kerberos auth wants requests-kerberos, SSH GSSAPI wants paramiko[gssapi], and hdfs:// shells out to the Hadoop hdfs command-line client, which has to be installed and on $PATH. Credentials are delegated to the underlying SDK, so S3 follows the usual boto3 chain, GCS follows google-cloud-storage, and Azure infers nothing at all, meaning you must build a BlobServiceClient and pass it in transport_params. Python 3.10 or newer since 7.5.1.

Patterns

Iterate an S3 object line by linestream-s3-lines

# pip install 'smart_open[s3]'
from smart_open import open

for line in open('s3://commoncrawl/robots.txt'):
    print(repr(line))
    break

Shadowing the builtin open is the intended usage and keeps local paths working. Nothing is downloaded up front: the reader pulls ranges as you consume, so breaking early costs you one request, not the whole object.

Stream data into S3 with your own boto3 clientwrite-to-s3

import boto3, botocore
from smart_open import open

config = botocore.client.Config(
    max_pool_connections=64,
    retries={'max_attempts': 6, 'mode': 'adaptive'},
)
client = boto3.Session().client('s3', config=config)

with open('s3://bucket/out.jsonl', 'wb',
          transport_params={'client': client}) as fout:
    fout.write(b'{"a": 1}\n')

Writes go out as a multipart upload, so the object does not appear until close() succeeds; always use a with block. Pass your own client when you need endpoint, retry, or pool settings, since the default client is built with boto3 defaults.

Read and write compressed files by extensiontransparent-compression

from smart_open import open

with open('data.txt.gz', encoding='utf-8') as fin:
    with open('s3://bucket/data.txt.bz2', 'w') as fout:
        for line in fin:
            fout.write(line)

Detection is purely by suffix (.gz, .bz2, .xz, .zst, .lz4) in both directions. .zst needs the zst extra on Python below 3.14, and .lz4 needs the lz4 extra; without them the suffix is simply not recognized.

Force or disable compression when the extension liescontrol-compression

from smart_open import open

# a .gz file named .gzip
with open('archive.gzip', compression='.gz') as fin:
    head = fin.read(32)

# read the raw compressed bytes
with open('archive.gz', 'rb', compression='disable') as fin:
    magic = fin.read(2)  # b'\x1f\x8b'

The default is compression='infer_from_extension'. compression='disable' replaced the old ignore_extension=True argument, which went away with the top-level smart_open() wrapper in 8.0.0.

Point at MinIO, Ceph, or DigitalOcean Spacescustom-s3-endpoint

import boto3
from smart_open import open

client = boto3.client(
    's3',
    endpoint_url='https://ams3.digitaloceanspaces.com',
    aws_access_key_id='user',
    aws_secret_access_key='secret',
)
for line in open('s3://bucket/key.txt',
                 transport_params={'client': client}):
    print(line)

8.0.0 removed the s3://key:secret@host:port@bucket URI form and the s3u:// scheme, so a custom endpoint now has to come from a client you build. The simpler s3://key:secret@bucket/key credential form still parses.

Read a specific S3 object versions3-object-version

from smart_open import open

with open('s3://bucket/key.txt?versionId=abc123') as fin:
    print(fin.read())

# equivalent explicit form
with open('s3://bucket/key.txt', 'rb',
          transport_params={'version_id': 'abc123'}) as fin:
    fin.read()

Parsing versionId out of the query string was added in 8.0.0 to match the AWS CLI and s3fs. On older releases the query string is treated as part of the key and you get a 404.

Skip multipart upload for small writessingle-part-upload

import tempfile
from smart_open import open

with tempfile.TemporaryFile() as tmp, open(
    's3://bucket/small.txt', 'wb',
    transport_params={'multipart_upload': False, 'writebuffer': tmp},
) as fout:
    fout.write(b'hello world!')

Single-part saves billable API calls and allows seek() before the upload commits, but the whole payload is buffered first. Point writebuffer at a temporary file instead of the default in-memory BytesIO when the object is large.

Read many keys from a bucket in paralleliter-s3-bucket

from smart_open import s3

for key, content in s3.iter_bucket(
    'silo-open-data',
    prefix='Official/annual/monthly_rain/',
    accept_key=lambda k: '/201' in k,
    workers=8,
    key_limit=3,
):
    print(key, len(content))

This hands you whole object bodies in memory, not streams, so cap it with key_limit or accept_key on large buckets. In 8.0.0 credentials moved into a single session_kwargs dict, and the old top-level smart_open.s3_iter_bucket shim is gone.

Read from GCS and Azure Blob Storagegcs-and-azure

import os
from azure.storage.blob import BlobServiceClient
from smart_open import open

for line in open('gcs://my_bucket/my_file.txt'):
    print(line)

client = BlobServiceClient.from_connection_string(
    os.environ['AZURE_STORAGE_CONNECTION_STRING'])
for line in open('azure://mycontainer/myfile.txt',
                 transport_params={'client': client}):
    print(line)

gcs:// is the canonical scheme since 8.0.0, with gs:// kept as an alias. Azure cannot infer credentials at all, so passing a BlobServiceClient is mandatory; Azure is also the only backend supporting append mode via 'ab'.

Read over HTTP or SFTP with the same callhttp-and-sftp

from smart_open import open

for line in open('http://example.com'):
    print(line[:15])
    break

with open('sftp://user:pass@host/path/file.csv') as fin:
    header = fin.readline()

HTTP and HTTPS are read-only. SFTP needs the ssh extra (paramiko); credentials in the URI end up in tracebacks and shell history, so prefer an ssh config entry or key agent.

Tune the compression levelcompression-kwargs

from smart_open import open

with open('out.txt.gz', 'wb',
          compression_kwargs={'compresslevel': 6}) as fout:
    fout.write(b'hello world')

The dict is forwarded as-is to the underlying library, so spell the option its way: compresslevel for gzip and bz2, preset for xz, level for zstd, compression_level for lz4. Added in 8.0.0.

Add support for another compression formatregister-compressor

import lzma
from smart_open import open, register_compressor

def _handle_xz(file_obj, mode, **kwargs):
    return lzma.open(filename=file_obj, mode=mode, **kwargs)

register_compressor('.xz', _handle_xz)

The handler receives the already-opened byte stream, so it works for remote URIs too. .xz ships registered by default; this is the pattern for in-house formats. Registration is process-global, so do it once at import time.

Alternatives

Package	Registry	Pick it when
fsspec	PyPI	You want one abstraction over many backends plus a real filesystem API (ls, glob, rm) and local caching, and can take on a much larger surface area.
s3fs	PyPI	You are S3-only but need listing, globbing, and deletion, and want paths that pandas, dask, and pyarrow already accept out of the box.
cloudpathlib	PyPI	You prefer pathlib-style CloudPath objects with local file caching over a bare open() replacement, and your code already thinks in Path terms.
boto3	PyPI	S3 is your only backend and you already need presigned URLs, object tagging, or lifecycle calls, so a streaming wrapper adds a dependency without removing boto3.