PyPIUtilsupdated 05 Aug 2026

joblib

Item: joblib
Rating: 4.5
Author: Keyoor

joblib does three unglamorous things that show up in almost every Python data script. Parallel and delayed turn a for loop into a parallel loop across processes or threads with one line. Memory caches a function's return value on disk keyed by its arguments, so an expensive step runs once and is read back on the next run. dump and load write and read Python objects, especially large numpy arrays, faster and smaller than plain pickle, with optional compression and memory-mapped reads. It is the parallelism and caching layer inside scikit-learn, which is why it appears in so many dependency trees.

joblib · v1.5.3 · 49.2M/wk · ★ 4,381 · pushed 05 Aug 2026 · joblib/joblib · docs · reviewed by Keyoor

Verdict

The default answer for parallelizing a loop and caching an expensive function in a single-machine Python script, and it earns its place through scikit-learn. Treat it as a loop accelerator, not a job system: the moment you need retries, scheduling, or more than one machine, you have outgrown it.

API stability5/5Parallel, delayed, Memory, dump, and load have been call-compatible for years; 1.5 removed one deprecated Memory argument and dropped Python 3.8 and PyPy, and parallel_backend still works alongside the newer parallel_config.

Docs4/5joblib.readthedocs.io covers each feature with runnable examples and a gallery, plus a specific page on shared memory and oversubscription; the README is only build instructions, and the finer points of backend selection are still learned mostly by experiment.

Maintenance4/5Pushed August 2026 with 1.5.3 in December 2025 and regular vendored loky and cloudpickle bumps; the tracker carries 371 open issues and 66 open PRs, so bug reports can sit for a long time.

Ecosystem5/5Roughly 49M downloads a week, pulled in by scikit-learn and much of the scientific Python stack, with pluggable backends that dask and Ray both implement.

Use it if

You have an embarrassingly parallel loop over independent items and want processes without writing multiprocessing.Pool boilerplate or worrying about pickling closures
You are iterating on a notebook or script and want expensive steps (downloads, feature extraction, model fits) cached to disk between runs, keyed automatically by arguments
You save and load numpy-heavy objects such as fitted scikit-learn models and want memory-mapped loads so several processes share one copy of a large array
You want one backend switch (loky processes, threads, or dask) that your library users can override from the outside with parallel_config, instead of hard-coding a pool

Skip it if

You need work spread across machines, retries, scheduling, or a durable queue: joblib is a single-machine loop helper, not a task system; use dask.distributed, Ray, or Celery
Your tasks are tiny. Every loky task pays pickling plus interprocess transfer, so parallelizing a microsecond function makes it slower; vectorize with numpy or move the loop into numba or Cython first
joblib.load runs pickle, so loading a .joblib file from an untrusted source executes arbitrary code; there is no safe-mode loader, unlike safetensors for model weights
Memory keys the cache on the function's source code plus a hash of its arguments. Change a global, a closed-over variable, or a helper the function calls and the cache does not notice, so you get stale results with no warning
Your code is async: there is no asyncio backend, and running Parallel inside an event loop blocks it
You are on Python 3.9 or older, or on PyPy, both of which the current line no longer supports

Setup reality

pip install joblib pulls nothing else in: 1.5.3 has no required dependencies because loky and cloudpickle are vendored inside the wheel. The friction is at runtime. The default loky backend spawns fresh interpreters, so your module is imported again in every worker and any top-level side effect runs again; scripts need the if __name__ == '__main__' guard or they fork bomb themselves. Arrays bigger than max_nbytes (default 1M) are written to a memory-mapped temp file so workers can share them, which joblib puts in /dev/shm only when more than 2GB is free there, otherwise silently in /tmp; containers with a small /dev/shm therefore get disk-backed instead of RAM-backed sharing. And if your inner function already calls into a threaded BLAS, running it under n_jobs=-1 oversubscribes the CPU and can be slower than serial until you cap threads with inner_max_num_threads.

Patterns

Run a loop across processesparallel-loop

from math import sqrt
from joblib import Parallel, delayed

results = Parallel(n_jobs=-1)(
    delayed(sqrt)(i ** 2) for i in range(10)
)
print(results)  # [0.0, 1.0, 2.0, ...]

delayed(f)(x) only records the call; Parallel runs it. n_jobs=-1 means all cores, -2 means all but one. In a script, wrap this in if __name__ == '__main__' or the spawned workers re-import and re-run your module.

Use threads when the work releases the GILthreads-instead-of-processes

from joblib import Parallel, delayed

# I/O bound or numpy-heavy: threads avoid pickling cost
results = Parallel(n_jobs=8, prefer='threads')(
    delayed(fetch_url)(u) for u in urls
)

prefer='threads' is a hint a caller can override; backend='threading' is a hard choice. Threads only help if the function releases the GIL (I/O, numpy, compiled extensions), otherwise you get serial execution with extra overhead.

Set the backend for a whole block of codeconfigure-backend-globally

from joblib import parallel_config, Parallel, delayed

with parallel_config(backend='threading', n_jobs=4):
    # any Parallel() inside here, including inside scikit-learn,
    # picks up these settings
    model.fit(X, y)
    Parallel()(delayed(score)(m) for m in models)

parallel_config replaced parallel_backend in 1.3 and is how you tune a library that calls Parallel internally without patching it. parallel_backend still works but is the older, narrower API.

Consume results as they finish instead of waitingstream-results-as-generator

from joblib import Parallel, delayed

out = Parallel(n_jobs=4, return_as='generator_unordered')(
    delayed(process)(chunk) for chunk in chunks
)
for result in out:
    write_to_db(result)

return_as='generator' keeps input order, 'generator_unordered' yields whichever finishes first. Both keep peak memory down because results are not accumulated in a list, but you must consume the generator or the workers stay blocked.

Cache an expensive function on disk between runscache-function-to-disk

from joblib import Memory

memory = Memory('./.joblib_cache', verbose=0)

@memory.cache
def build_features(dataset_path, window):
    return expensive_pipeline(dataset_path, window)

features = build_features('data/raw.parquet', window=30)

The key is a hash of the arguments plus the function's own source. Editing the body invalidates the entry, but changing a global or a helper it calls does not, which is the classic way to get a stale answer.

Exclude arguments from the cache keyignore-cache-arguments

from joblib import Memory

memory = Memory('./.joblib_cache')

@memory.cache(ignore=['conn', 'verbose'])
def load_rows(conn, table, since, verbose=False):
    return conn.read(table, since)

Without ignore, joblib tries to hash the database connection, which either fails or produces a new key on every run so nothing is ever a hit. ignore takes parameter names, not values.

Make cached results expire after a whileexpire-cached-results

import datetime
from joblib import Memory, expires_after

memory = Memory('./.joblib_cache')

@memory.cache(cache_validation_callback=expires_after(days=7))
def fetch_exchange_rates(day):
    return http_get_rates(day)

# housekeeping: keep the cache directory under 500MB
memory.reduce_size(bytes_limit='500M')
memory.clear(warn=False)  # nuke everything

Nothing evicts on its own: the cache directory grows until you call reduce_size or clear. bytes_limit belongs on reduce_size, not on the Memory constructor, where it was removed in 1.5.

Persist a model or array to disksave-and-load-objects

import joblib

joblib.dump(model, 'model.joblib')
model = joblib.load('model.joblib')

# compressed: smaller file, slower write
joblib.dump(model, 'model.joblib.lz4', compress=('lz4', 3))

This is pickle underneath with a faster path for numpy arrays, so loading a file you did not create runs arbitrary code. The extension picks the codec: .gz, .bz2, .xz, .lzma, and .lz4 with the optional lz4 package installed.

Share one big array across processes on loadmemory-map-large-arrays

import joblib

joblib.dump(big_array, 'big.joblib')

# each worker maps the same file instead of copying it into RAM
arr = joblib.load('big.joblib', mmap_mode='r')

mmap_mode only works on uncompressed dumps of plain numpy arrays. The array is read-only under 'r'; use 'r+' or 'c' (copy-on-write) if you need to write, and expect page faults instead of an upfront load cost.

Fingerprint arbitrary Python objects, numpy includedhash-any-object

import numpy as np
from joblib import hash as jhash

a = np.zeros(1000)
b = np.zeros(1000)
print(jhash(a) == jhash(b))  # True, content-based

This is the hashing Memory uses for cache keys, exposed directly. Unlike Python's hash it handles unhashable values and looks at numpy array contents rather than identity, which makes it useful for change detection in pipelines.

Stop nested BLAS threads from fighting your workerscontrol-oversubscription

from joblib import parallel_config, Parallel, delayed

with parallel_config(n_jobs=8, inner_max_num_threads=1):
    Parallel()(delayed(fit_one)(p) for p in grid)

numpy, scipy, and scikit-learn already thread internally. Eight workers each spawning eight BLAS threads on an eight-core box thrashes; inner_max_num_threads caps the nested pools for the duration of the block.

See progress and cap memory used by queued tasksshow-progress-and-tune-dispatch

from joblib import Parallel, delayed

Parallel(n_jobs=4, verbose=10, batch_size=1, pre_dispatch='2*n_jobs')(
    delayed(process_image)(p) for p in paths
)

verbose prints completion counts to stderr, higher is chattier. pre_dispatch limits how many tasks are materialized ahead of the workers, which matters when each task argument is large; batch_size='auto' is right unless tasks are very uneven.

Alternatives

Package	Registry	Pick it when
dask	PyPI	The data or the task graph outgrows one machine, or you want dependencies between tasks rather than a flat loop.
ray	PyPI	You want a cluster-scale actor and task runtime with fault tolerance rather than a parallel for loop.
loky	PyPI	You only want the reusable process pool executor without the caching and serialization parts of joblib.
diskcache	PyPI	You want on-disk caching with expiry and cross-process safety, and none of the parallelism.