PyPIUtilsupdated 07 Aug 2026

diskcache

Item: diskcache
Rating: 4
Author: Keyoor

diskcache is a cache that lives on your local disk instead of in memory or in a separate server process. It is pure Python with no dependencies: keys and metadata go into a SQLite database in a directory you choose, and values larger than 32 KB are written as individual files alongside it. On top of that store it gives you the things a cache needs (expiry, tags, size limits, several eviction policies, hit and miss statistics) plus a memoize decorator, a Django cache backend, persistent Deque and Index types that behave like collections.deque and dict across processes, and cross-process locks, semaphores and throttles. Nothing to install beyond pip, nothing to run, and the cache survives process restarts.

diskcache · v5.6.3 · 9.4M/wk · ★ 2,902 · pushed 10 Aug 2024 · grantjenks/python-diskcache · docs · reviewed by Keyoor

Verdict

For a single machine that needs a cache bigger than memory, this is hard to beat: no server, no dependencies, and a sensible API with a Django backend included. Check eviction_policy, size_limit and how you handle Timeout before shipping it, and choose Redis instead the moment a second server enters the picture.

API stability5/5The 5.x surface has not changed since 2020 and the last release was August 2023, so upgrades cost nothing. Some of that stability is simply a project that stopped moving.

Docs5/5A full tutorial, complete API reference, published cache and Django benchmarks, two case studies and thorough docstrings, which is more than most libraries this size ship. The docs are frozen at the 5.6 era, so nothing addresses recent Python versions.

Maintenance2/5Last release August 2023, last commit August 2024, 51 open issues out of 74 open issues and PRs, and a single author. Being pure Python with no dependencies means it keeps running on new interpreters, but do not expect fixes or merged pull requests.

Ecosystem4/5Around 9.38M weekly downloads, a Django cache backend in the box, and it is the default local cache inside a number of other tools. Extensibility stops at subclassing Disk; there is no plugin system or third-party backend ecosystem.

Use it if

You want a cache larger than RAM on a single machine and do not want to operate Redis or Memcached for it: no server, no C compiler, no configuration file
You are on Django and the built-in file-based cache backend is too slow: diskcache.DjangoCache is a drop-in BACKEND with sharding and real eviction instead of random culling
You need the cache to survive restarts and be visible to sibling processes: multiple processes can open the same directory, and Lock, RLock, BoundedSemaphore, throttle and barrier are built on the same store
You want persistent memoization with invalidation: cache.memoize gives you a decorator whose entries outlive the process, can be tagged for bulk eviction, and can compute the exact cache key so you can delete one result
Your values are large blobs such as rendered pages, model outputs or downloaded files: anything above disk_min_file_size becomes its own file, so big values do not bloat the SQLite rows

Skip it if

More than one machine needs the same cache: this is a local directory, so a three-node deployment means three independent caches with three different views and three separate warmups. That is what Redis is for
You expected LRU: the default eviction_policy is least-recently-stored, so under pressure the oldest written entries go first regardless of how hot they are, and size_limit defaults to 1 GiB. Both are easy to change and both surprise people who never looked
You need failures to be loud under contention: Cache methods default to retry=False and raise diskcache.Timeout when SQLite is busy, while FanoutCache catches that same timeout and returns the default value, so a saturated cache is indistinguishable from a cold one in your metrics
You are writing async code: every call blocks. Inside asyncio you have to push each operation into a thread executor, which erases much of the speed advantage over a network cache
You want an actively developed dependency: the last release 5.6.3 is from August 2023, the last commit is August 2024, the trove classifiers stop at Python 3.9, and there are 51 open issues out of 74 open issues and PRs. It is stable and dependency-free so it keeps working, but nobody is merging your patch
Untrusted input can reach the cache directory or the values: the default Disk pickles values, so anything that can write into the directory can execute code in your process on the next read

Setup reality

pip install diskcache is pure Python with zero runtime dependencies and no build step, which is the entire pitch. The cost is in the defaults you silently accept: size_limit is 1 GiB, eviction_policy is least-recently-stored rather than LRU, cull_limit is 10 so only ten expired or evicted rows are cleaned up per write, and disk_min_file_size is 32 KB, above which each value becomes a separate file in the cache directory. The directory has to be on a real local filesystem, because SQLite in WAL mode over NFS or some container overlay mounts will lock up or corrupt. A Cache keeps a thread-local SQLite connection and detects a PID change so it reconnects after a fork, but you should still close the cache (or use it as a context manager) before spawning workers. Expired entries are not swept on a timer; they are removed lazily during writes, so a read-heavy cache holds dead rows until you call expire() yourself. Django users set BACKEND to diskcache.DjangoCache with a LOCATION directory that is both writable and persistent, which on many container images is neither the image layer nor /tmp.

Patterns

Open a cache and read and write keysbasic-get-and-set

from diskcache import Cache

with Cache('/var/tmp/myapp-cache') as cache:
    cache['user:1'] = {'name': 'Ada'}
    cache.set('token', 'abc', expire=300)

    user = cache.get('user:1', default=None)
    if 'token' in cache:
        print(cache['token'])

    cache.delete('token')
    print(len(cache), cache.volume())

The context manager closes the SQLite connection; a long-lived process can keep one Cache open for its lifetime instead. get() and set() default to retry=False, so under write contention they raise diskcache.Timeout rather than blocking, and you have to decide whether that is a miss or an error.

Expire entries and evict a whole group at onceexpiry-and-tags

from diskcache import Cache

cache = Cache('/var/tmp/myapp-cache', tag_index=True)

cache.set('page:/home', html, expire=60, tag='pages')
cache.set('page:/about', html2, expire=60, tag='pages')

# publish new content: drop every page in one call
cache.evict('pages')

# reclaim rows for entries that already expired
cache.expire()

evict() is only efficient when the cache was created with tag_index=True; without it the call scans. Expiry is lazy, so entries past their expire time still take disk space until a write triggers culling or you call expire() from a periodic job.

Cache function results across restartsmemoize-function-results

from diskcache import Cache

cache = Cache('/var/tmp/myapp-cache')

@cache.memoize(expire=3600, tag='reports', typed=True)
def build_report(customer_id, month):
    return expensive_query(customer_id, month)

build_report(42, '2026-07')

# invalidate exactly one result
cache.delete(build_report.__cache_key__(42, '2026-07'))

# or bypass the cache entirely
fresh = build_report.__wrapped__(42, '2026-07')

__cache_key__ and __wrapped__ are the two attributes that make this usable in production: one lets you invalidate a single call, the other lets you skip the cache. typed=True keeps f(3) and f(3.0) apart. Arguments must be picklable, so passing a database session or an open file fails at set time.

Use FanoutCache when many processes writeshard-for-write-concurrency

from diskcache import FanoutCache

cache = FanoutCache(
    '/var/tmp/myapp-cache',
    shards=8,       # eight SQLite databases
    timeout=1.0,    # per-operation SQLite timeout, default 0.010
    size_limit=4 * 2**30,
)

cache.set('k', 'v', expire=60)

Sharding cuts write-lock contention because each key lands in one of N databases. The trap is error handling: FanoutCache catches the SQLite timeout that Cache would raise and quietly returns the default (or False from set), so overload shows up as a falling hit rate rather than an exception. Raise the timeout above the 10 ms default before you trust it.

Share a dict or a deque between processespersistent-dict-and-deque

from diskcache import Deque, Index

index = Index('/var/tmp/myapp-index')
index['last_run'] = '2026-08-07'
index.setdefault('runs', 0)
print(dict(index))

queue = Deque(directory='/var/tmp/myapp-queue')
queue.append('job-1')
queue.appendleft('job-0')
job = queue.popleft()

Index and Deque are MutableMapping and Sequence backed by a Cache with no eviction, so they persist rather than expire. Read-modify-write on a value is not atomic: index['runs'] += 1 from two processes can lose an update, so use cache.incr or index.transact() for counters.

Increment safely and batch writesatomic-counters-and-transactions

from diskcache import Cache

cache = Cache('/var/tmp/myapp-cache')

cache.incr('hits', delta=1, default=0)
cache.decr('quota:42', delta=1, default=100)

with cache.transact():
    cache.set('a', 1)
    cache.set('b', 2)
    cache.incr('writes', 2)

incr and decr are single SQLite statements, so they are safe across processes; a get-then-set pair is not. transact() wraps several operations in one SQLite transaction, which is both atomic and much faster than the same writes done individually.

Coordinate processes with a lock or throttlecross-process-lock

from diskcache import Cache, Lock, throttle

cache = Cache('/var/tmp/myapp-cache')

with Lock(cache, 'nightly-job', expire=600):
    run_nightly_job()

@throttle(cache, count=10, seconds=1)
def call_partner_api(payload):
    return requests.post(URL, json=payload)

Always pass expire on a Lock, otherwise a process that dies while holding it leaves the lock held forever with no way to notice. These primitives poll the SQLite store rather than using OS locks, so they work across processes on one machine and not across machines.

Store and read large values as filesstream-large-values

from diskcache import Cache

cache = Cache('/var/tmp/myapp-cache', disk_min_file_size=2**16)

with open('report.pdf', 'rb') as f:
    cache.set('report:42', f, read=True, expire=86400)

handle = cache.get('report:42', read=True)
if handle is not None:
    with handle:
        for chunk in iter(lambda: handle.read(65536), b''):
            send(chunk)

read=True on set stores the file's bytes without loading them all into memory, and read=True on get returns a file handle instead of bytes. Close that handle, since it is a real open file. Values above disk_min_file_size always become separate files, so the cache directory holds many small files and needs a filesystem with inodes to spare.

Store JSON instead of picklescustom-serialization

from diskcache import Cache, JSONDisk

cache = Cache('/var/tmp/myapp-cache', disk=JSONDisk, disk_compress_level=1)
cache['config'] = {'retries': 3, 'region': 'ap-south-1'}

The default Disk pickles values, which means a cache directory writable by anything untrusted is a code execution path, and cached objects break when you rename or move their class. JSONDisk (zlib-compressed JSON) avoids both at the cost of only supporting JSON types; for anything else, subclass Disk and override put, get, store and fetch.

Use the cache as a persistent work queuequeue-with-push-and-pull

from diskcache import Cache

cache = Cache('/var/tmp/myapp-queue')

key = cache.push({'job': 'resize', 'id': 7}, prefix='jobs')

key, value = cache.pull(prefix='jobs', default=(None, None))
if key is not None:
    handle(value)

push and pull give a FIFO queue whose keys are monotonically increasing integers (or prefix-integer strings), and pull is atomic so several workers can drain the same prefix. There is no acknowledgement step: once pulled, an item is gone, so a worker that crashes mid-job loses it.

Wire it up as a Django cachedjango-cache-backend

# settings.py
CACHES = {
    'default': {
        'BACKEND': 'diskcache.DjangoCache',
        'LOCATION': '/var/lib/myapp/cache',
        'TIMEOUT': 300,
        'SHARDS': 8,
        'DATABASE_TIMEOUT': 0.5,
        'OPTIONS': {'size_limit': 2**32},
    }
}

DjangoCache is a FanoutCache underneath, so the same silent-timeout behavior applies and DATABASE_TIMEOUT is worth raising from the 10 ms default. LOCATION must be a writable path that survives deploys; pointing it at the image layer means the cache resets on every release, and pointing it at /tmp means the OS may clear it under you.

Track hit rate and keep the directory healthymeasure-and-maintain

from diskcache import Cache

cache = Cache('/var/tmp/myapp-cache', statistics=True)

hits, misses = cache.stats(enable=True, reset=True)
print(hits, misses, cache.volume(), len(cache))

# periodic maintenance job
cache.expire()          # drop entries past their expire time
cache.cull()            # evict down to size_limit
warnings = cache.check(fix=True)   # verify and repair the directory

Statistics are off by default and cost a write per lookup when enabled, so turn them on deliberately. volume() estimates total bytes including the value files, and it can drift from reality after a crash until check(fix=True) reconciles the directory with the database.

Alternatives

Package	Registry	Pick it when
cachetools	PyPI	The working set fits in RAM, you do not need persistence, and you want plain in-process LRU, LFU or TTL dictionaries.
redis	PyPI	More than one process on more than one machine has to see the same cache, or you need pub/sub and atomic server-side operations.
joblib	PyPI	You are caching expensive numeric or scientific function results on disk and want NumPy-aware storage and hashing.
sqlitedict	PyPI	You want a persistent dict backed by SQLite rather than a cache, with no eviction, expiry or size limit involved.