PyPIUtilsupdated 06 Aug 2026

h5py

Item: h5py
Rating: 4.5
Author: Keyoor

h5py is a thin Python wrapper over the HDF5 C library. It maps HDF5's two core concepts onto types you already know: groups behave like dictionaries, so f['experiments/run_42'] navigates a tree inside a single file, and datasets behave like NumPy arrays that live on disk, so ds[100:200, :] reads exactly that slab and nothing else. That last part is the whole point. A 400 GB dataset opens instantly and costs you only the bytes you slice. On top of that you get attributes for metadata on any group or dataset, per-dataset chunking and gzip or lzf compression, resizable datasets you can append to, virtual datasets that stitch many files into one logical array, single-writer multiple-reader mode for live monitoring, and an MPI build for parallel HPC writes. HDF5 files are a portable format read by C, Fortran, MATLAB, Julia, R, and Java, which is often the real reason a project picks it.

h5py · v3.16.0 · 10.7M/wk · ★ 2,242 · pushed 06 Aug 2026 · h5py/h5py · docs · reviewed by Keyoor

Verdict

The default and correct way to touch HDF5 from Python, stable for over a decade and maintained by people who know the C library well. Pick it when the format is a requirement or when you need out-of-core slicing on a local disk, and pick Zarr instead the moment the data moves to object storage.

API stability5/5The File, Group, Dataset trio has been unchanged in shape since 2.x; the one disruptive change in living memory was 3.0 returning bytes for strings instead of str, and that shipped in 2020 with the asstr() escape hatch alongside it

Docs4/5docs.h5py.org has a genuine quick start, separate reference pages for datasets, filters, virtual datasets, SWMR, and MPI, plus per-release what's-new notes; the weak spot is performance, where chunk shape and cache sizing get described but not really taught, so people learn it from HDF Group articles

Maintenance4/5Pushed the same week, 3.16.0 released March 2026, and a steady two-to-three releases a year since 2021 with wheels for every new Python within weeks; the backlog is large at 295 open issues (345 counting PRs) and the team is a handful of volunteers, so niche bugs sit

Ecosystem5/5About 10.7M weekly downloads and a dependency of pandas HDFStore, keras, scanpy, deeplabcut, and much of the scientific Python stack; HDF5 itself is readable from essentially every language used in scientific computing

Use it if

Your arrays do not fit in RAM and you need to slice arbitrary regions off disk without loading the whole thing
You have to exchange numeric data with C, Fortran, MATLAB, Julia, or an instrument vendor's tooling, and HDF5 is the format everyone already reads
You want one self-describing file holding many arrays plus their metadata, instead of a directory of .npy files and a README nobody updates
You are reading files someone else produced: MATLAB v7.3 .mat files, neuroimaging or genomics archives, older Keras model weights, or telescope and detector output are all HDF5
You are on an HPC cluster with MPI and need many ranks writing into one file through parallel HDF5

Skip it if

Your data lives in object storage: HDF5 assumes a POSIX file it can seek around in, and reading one over S3 turns a slice into a long chain of small range requests; Zarr was designed for that access pattern instead
You need concurrent writers: HDF5 gives you one writer or many readers, never both without SWMR, and SWMR itself forbids creating or resizing anything except appending along one axis while readers are attached; two processes opening the same file for writing corrupts it
You want threads: the C library h5py links against is built non-threadsafe, so h5py serializes every call behind a global lock and extra threads buy you nothing on read-heavy work
You are storing tables rather than arrays: there is no query engine, no index, and no predicate pushdown, so a columnar format like Parquet through pyarrow is a better fit for anything you filter or group by
You mix scientific packages carelessly: h5py wheels bundle their own HDF5 build, and importing it alongside a netCDF4 or pytables wheel that bundled a different HDF5 version can produce a library version mismatch abort rather than a Python exception
You expect files to shrink: HDF5 does not reclaim space when you delete a dataset, so a file that gets rewritten repeatedly grows until you run h5repack on it
You need a specific HDF5 build (MPI, the ROS3 driver, a plugin filter): the convenient wheel does not have it, and you are compiling from source against a system HDF5 with a working compiler toolchain

Setup reality

pip install h5py normally just works, because the project ships wheels with HDF5 statically bundled for Linux, macOS, and Windows; the cp312 Linux wheel for 3.16.0 is about 5.2 MB and pulls in only NumPy 1.21.2 or newer. Python 3.10 is the floor. The trouble starts when you need something the wheel does not carry. MPI support means installing an HDF5 built with parallel enabled, then HDF5_MPI=ON CC=mpicc pip install --no-binary=h5py h5py, and any mismatch between your mpi4py, your MPI runtime, and that HDF5 shows up as a link error or a silent hang at file open. Mixing conda and pip in one environment is the other classic failure: conda's h5py links a shared libhdf5 that a pip-installed netCDF4 or tables can shadow, and you get an abort message about a mismatched HDF5 library version at import time. Day to day, the annoying part is that HDF5 needs files closed properly. Always use a with block, because a handle left open in a notebook keeps the file locked against other processes, and a crashed writer can leave a file no reader can open. Also budget an afternoon for learning chunk shapes, because the default layout is fine for whole-array reads and terrible for the axis you actually slice.

Patterns

Create a file and write a datasetcreate-and-write

import h5py
import numpy as np

with h5py.File("run.h5", "w") as f:
    f.create_dataset("temperature", data=np.random.rand(1000, 512))
    grp = f.create_group("metadata")
    grp.create_dataset("sensor_ids", data=np.arange(512))

Mode 'w' truncates an existing file without asking; use 'w-' (or the alias 'x') to fail instead when the file already exists. The with block matters more here than usual, because an unclosed handle leaves the file locked against other processes.

Pick the right open modeopen-modes

h5py.File("run.h5", "r")    # read only, fails if missing (the default)
h5py.File("run.h5", "r+")   # read/write, fails if missing
h5py.File("run.h5", "a")    # read/write, create if missing
h5py.File("run.h5", "w")    # create, truncate if exists
h5py.File("run.h5", "w-")   # create, fail if exists

Only one process may hold the file in a write mode at a time. A second writer does not queue, it raises a file-locking error, and forcing past it with HDF5_USE_FILE_LOCKING=FALSE is how files get corrupted.

Read only the slab you needlazy-slicing

with h5py.File("run.h5", "r") as f:
    ds = f["temperature"]          # no data read yet
    print(ds.shape, ds.dtype, ds.nbytes)

    window = ds[500:600, :]        # reads 100 rows off disk
    everything = ds[...]           # reads all of it into RAM
    one_channel = ds[:, 42]

ds is a lazy handle, not an array; numpy functions on it will trigger a full read. The handle dies with the file, so returning ds from a with block and slicing it later raises a closed-file error. Copy what you need before leaving the block.

Chunk and compress a dataset for how you will read itchunking-and-compression

with h5py.File("run.h5", "w") as f:
    ds = f.create_dataset(
        "frames",
        shape=(10_000, 512, 512),
        dtype="uint16",
        chunks=(1, 512, 512),      # one frame per chunk
        compression="gzip",
        compression_opts=4,
        shuffle=True,
    )
    ds[0] = frame

Chunk shape should match your read pattern: chunks=(1, 512, 512) makes reading one frame cheap and reading one pixel across time very expensive. Any read touching a chunk decompresses the whole chunk, so tiny chunks mean overhead and huge ones mean wasted I/O. compression='lzf' is far faster than gzip and compresses less.

Append rows to a growing datasetresizable-append

with h5py.File("log.h5", "a") as f:
    if "events" not in f:
        ds = f.create_dataset(
            "events", shape=(0, 4), maxshape=(None, 4),
            dtype="f4", chunks=(1024, 4),
        )
    ds = f["events"]
    n = ds.shape[0]
    ds.resize(n + len(batch), axis=0)
    ds[n:] = batch

maxshape=None on an axis means unlimited, and it must be set at creation; you cannot make a fixed dataset resizable afterwards. Resizable datasets are always chunked, so pick a chunk row count that matches your batch size rather than letting h5py guess.

Attach metadata to groups and datasetsattributes

with h5py.File("run.h5", "a") as f:
    ds = f["temperature"]
    ds.attrs["units"] = "celsius"
    ds.attrs["sample_rate_hz"] = 250.0
    ds.attrs["calibration"] = np.array([1.0, 0.02])
    f.attrs["created"] = "2026-08-06T09:00:00Z"

    print(dict(ds.attrs))

Attributes are meant for small values; anything over about 64 KB spills into a separate header block and slows every open of that object. String attributes written from Python 3 read back as str, unlike string datasets.

Store and read text without getting bytes backstrings

with h5py.File("run.h5", "a") as f:
    dt = h5py.string_dtype(encoding="utf-8")        # variable length
    f.create_dataset("labels", data=["alpha", "beta"], dtype=dt)

    raw = f["labels"][0]              # b'alpha'
    text = f["labels"].asstr()[0]     # 'alpha'
    all_text = f["labels"].asstr()[:]

This is the single most common h5py surprise: since 3.0 string datasets read back as bytes, and asstr() is the wrapper that decodes them. Use h5py.string_dtype(length=16) for fixed-width strings, which store more compactly but truncate silently.

Walk an unfamiliar fileexplore-file

def describe(name, obj):
    if isinstance(obj, h5py.Dataset):
        print(f"{name:<45} {str(obj.shape):<18} {obj.dtype} {obj.compression}")
    else:
        print(f"{name}/")

with h5py.File("mystery.h5", "r") as f:
    print(list(f.keys()))
    f.visititems(describe)

visititems skips soft and external links to avoid cycles; use visititems_links if you need to see them. For a quick look from the shell, the HDF5 tools h5ls -r and h5dump -H are faster than writing this.

Reference data in another place or another filelinks-and-external-files

with h5py.File("index.h5", "a") as f:
    f["latest"] = h5py.SoftLink("/runs/2026-08-06")
    f["calibration"] = h5py.ExternalLink("calib_2026.h5", "/coefficients")

    # hard link: two names, one dataset
    f["alias"] = f["runs/2026-08-06/temperature"]

An ExternalLink resolves relative to the linking file's directory, so moving one file and not the other gives a KeyError that looks like missing data. A hard link means deleting one name does not free the space until every name is gone.

Present many files as one arrayvirtual-dataset

layout = h5py.VirtualLayout(shape=(len(files), 512, 512), dtype="uint16")

for i, path in enumerate(files):
    layout[i] = h5py.VirtualSource(path, "frames", shape=(512, 512))

with h5py.File("combined.h5", "w", libver="latest") as f:
    f.create_virtual_dataset("all_frames", layout, fillvalue=0)

No data is copied; the virtual file is tiny and reads pass through to the sources, so the source paths must stay valid and reachable. Gaps read back as fillvalue rather than raising, which makes a missing source file easy to miss.

Read a file while a writer is still appendingswmr-live-read

# writer
with h5py.File("live.h5", "w", libver="latest") as f:
    ds = f.create_dataset("stream", (0,), maxshape=(None,), dtype="f8")
    f.swmr_mode = True                 # no new objects after this point
    for chunk in source:
        n = ds.shape[0]
        ds.resize(n + len(chunk), axis=0)
        ds[n:] = chunk
        ds.flush()

# reader, in another process
with h5py.File("live.h5", "r", libver="latest", swmr=True) as f:
    ds = f["stream"]
    ds.refresh()
    print(ds.shape)

Create every dataset before setting swmr_mode; after it, adding groups, datasets, or attributes is forbidden and only appends along an existing unlimited axis are allowed. Readers see nothing new until they call refresh(), and the writer must flush().

Process a dataset larger than memoryiterate-chunks

with h5py.File("huge.h5", "r", rdcc_nbytes=256 * 1024**2) as f:
    ds = f["frames"]

    total = 0.0
    for sel in ds.iter_chunks():
        total += ds[sel].sum()

    # or fixed-size windows when the dataset is contiguous
    for start in range(0, ds.shape[0], 512):
        block = ds[start:start + 512]

iter_chunks yields slice tuples aligned to the stored chunks, so each read decompresses each chunk exactly once. rdcc_nbytes raises the per-dataset chunk cache from its 1 MB default, which matters a lot when your access pattern revisits chunks.

Alternatives

Package	Registry	Pick it when
zarr	PyPI	Your arrays live in S3, GCS, or another object store, or you need concurrent writers to different chunks
pyarrow	PyPI	Your data is tabular and you want Parquet with column pruning, predicate pushdown, and dataframe interop
tables	PyPI	You want HDF5 with a query language, indexes, and a pandas HDFStore-compatible layer on top
netCDF4	PyPI	You are in climate, ocean, or atmospheric science where NetCDF4 conventions and dimension coordinates are the standard