PyPIUtilsupdated 06 Aug 2026

ijson

Item: ijson
Rating: 3.8
Author: Keyoor

ijson parses JSON incrementally instead of loading the whole document into memory. You hand it a file-like object opened in binary mode and it reads in 64 KB chunks, emitting parser events as it goes, so a 40 GB API dump costs you roughly the size of one record rather than the size of the file. The usual entry point is ijson.items(f, 'earth.europe.item'), which builds real Python objects only for the parts of the tree matching a dotted prefix and throws the rest away. Below that sit ijson.kvitems for key-value pairs inside one huge object, ijson.parse for prefixed events, and ijson.basic_parse for raw tokens. Under the hood it picks the fastest available backend, normally a C extension around the YAJL library, and falls back to a pure Python parser when nothing else is installed.

ijson · v3.5.1 · 12.4M/wk · ★ 1,091 · pushed 01 Aug 2026 · ICRAR/ijson · docs · reviewed by Keyoor

Verdict

The right tool when the JSON is too big to load and you know which part you want, and the C backend keeps it fast enough to be practical on real dumps. If the file fits in RAM, or the data is really newline-delimited records, you are paying incremental-parsing overhead for nothing.

API stability5/5The items, kvitems, parse and basic_parse quartet has been the public surface since 3.0 in 2019, and 3.x additions such as from_iter and transparent async support were layered on without breaking the older explicit *_async functions

Docs3/5The README is genuinely thorough on prefixes, events, options, backends, capabilities and performance tips, and includes a real FAQ, but it is the whole documentation: no API reference site, no docstring-generated pages, and the coroutine push interface gets two examples and no further explanation

Maintenance4/5Pushed 2026-08-01 with 3.5.1 released July 2026, wheels built for Python 3.14, and 10 open issues (11 including PRs); it is effectively a one-maintainer project inherited from the original author in 2019, with a self-maintained YAJL fork carrying fixes for known CVEs

Ecosystem3/5Around 12.4M downloads a week and the default answer for streaming JSON in Python, but there are no plugins or integrations to speak of, and neighbours like json-stream, jsonlines and msgspec each cover part of the same ground

Use it if

The JSON file is bigger than the RAM you are willing to spend on it, for example a multi-gigabyte export where json.load would need several times the file size in heap
You only want a slice of a large document: ijson.items with a prefix like 'results.item.id' skips building objects for everything else in the tree
You are consuming a streaming HTTP response and want to start processing before the download finishes, which ijson.from_iter plus requests iter_content or httpx aiter_bytes gives you directly
You need to react to structure rather than values, such as counting occurrences of a key or converting JSON to XML on the fly, which ijson.parse and ijson.basic_parse do without allocating objects at all

Skip it if

The file comfortably fits in memory. json.loads and especially orjson beat ijson by a wide margin on whole-document parsing, because incremental parsing costs per-event Python overhead that a bulk parser never pays
Your data is JSON Lines or newline-delimited records. ijson fails on the second top-level value with a trailing garbage error until you pass multiple_values=True, and even then splitting on newlines and calling orjson.loads per line is simpler and faster
You expect floats. Non-integer numbers come back as decimal.Decimal by default, which silently breaks numpy conversion, JSON round-tripping and arithmetic against floats. use_float=True fixes it but then integers above 2^64 raise an overflow error instead
You want a query language. The prefix is a dotted string with no validation, so one typo yields zero items with no exception and no warning, and there is no wildcard, filter or slice syntax the way jq or JSONPath has
You need the parse to be portable across backends without checking. The pure Python fallback is a great deal slower and does not support C-style comments, and you land on it silently if no wheel matches your platform
You need a deep bench behind the project. It is around 1.1k stars, was handed over by the original author in 2019, and is now effectively maintained by one person, with 10 open issues (11 including PRs)

Setup reality

pip install ijson is usually instant because 3.5.1 ships 90 binary wheels covering CPython 3.9 through 3.14 on manylinux, musllinux, macOS universal2 and Windows. When no wheel matches your platform, pip builds the yajl2_c extension, which wants a C compiler and the YAJL development headers, and if that fails you end up on a much slower backend without being told. Print ijson.backend after import to see which one you actually got, and set the IJSON_BACKEND environment variable to pin it in production. The other trap is input type: pass a file opened with 'rb'. Handing ijson a text-mode file works, but it re-encodes every chunk back to UTF-8 internally and only emits a warning that is invisible by default. Python 3.9 is the floor.

Patterns

Iterate objects inside a huge arraystream-array-items

import ijson

with open('dump.json', 'rb') as f:
    for record in ijson.items(f, 'results.item'):
        handle(record)

Open in binary mode. The literal 'item' segment is how you say array element, so results.item means every element of the results array. A prefix that matches nothing yields an empty iterator with no error, which is the number one reason people think ijson is broken.

Find the right prefix by dumping eventsdiscover-prefixes

import ijson

with open('dump.json', 'rb') as f:
    for prefix, event, value in ijson.parse(f):
        print(prefix, event, value)
        # stop early once you have seen enough structure

# or from the shell:
# head -c 4000 dump.json | python -m ijson.dump -m parse

Do this before writing the items() call. The ijson.dump module reads stdin and prints the prefix, event and value for each token, which is far faster than guessing at nesting levels.

Walk key-value pairs of an object too big to builditerate-object-members

import ijson

with open('index.json', 'rb') as f:
    for key, value in ijson.kvitems(f, 'documents'):
        print(key, value['title'])

kvitems yields the members of the object at the prefix one at a time, so a top-level map with a million keys never becomes a single Python dict. Each individual value is still fully materialized, so this only helps when the container is the big thing.

Count occurrences without allocating objectscount-without-building

import ijson

with open('dump.json', 'rb') as f:
    events = ijson.basic_parse(f)
    total = sum(1 for event, value in events
                if event == 'map_key' and value == 'error')

basic_parse skips prefix bookkeeping and object construction entirely, so it is the fastest path in the library. Event names are start_map, end_map, start_array, end_array, map_key, null, boolean, integer, double, number and string.

Parse a streaming HTTP body with requestsstream-http-response

import ijson
import requests

with requests.get(url, stream=True) as resp:
    resp.raise_for_status()
    f = ijson.from_iter(resp.iter_content(chunk_size=64 * 1024))
    for city in ijson.items(f, 'earth.europe.item'):
        handle(city)

from_iter adapts any byte iterator into the file-like object ijson wants. Prefer iter_content over resp.raw because requests then handles gzip and chunked transfer encoding for you; resp.raw hands you the undecoded stream.

Consume an async response with httpxasync-streaming

import httpx
import ijson

async def load(url):
    async with httpx.AsyncClient() as client:
        async with client.stream('GET', url) as resp:
            resp.raise_for_status()
            f = ijson.from_iter(resp.aiter_bytes())
            async for obj in ijson.items(f, 'earth.europe.item'):
                await handle(obj)

Since 3.1 the normal functions detect an async source and return an async iterator, so the explicit items_async variant is optional. Event interception by chaining one function's output into another is not supported on the async path.

Handle concatenated or newline-separated JSON valuesmultiple-top-level-values

import ijson

with open('stream.json', 'rb') as f:
    for obj in ijson.items(f, '', multiple_values=True):
        handle(obj)

Without multiple_values=True this raises a trailing garbage error on the second value. An empty prefix selects each top-level value. If your file is genuinely one JSON object per line, jsonlines or a plain loop with orjson.loads is less machinery.

Get float instead of Decimal for numbersfloats-not-decimals

import ijson

with open('metrics.json', 'rb') as f:
    for row in ijson.items(f, 'series.item', use_float=True):
        assert isinstance(row['value'], float)

The default is decimal.Decimal, which is precise but slow and does not compare or serialize the way callers expect. use_float=True is faster, but a value like 1e400 raises an overflow error, and integers at or above 2^64 do the same. Check your data range first.

See and pin the parsing backendselect-backend

import ijson

print(ijson.backend)        # e.g. 'yajl2_c'
print(ijson.ALL_BACKENDS)   # ('yajl2_c', 'yajl2_cffi', 'yajl2', 'yajl', 'python')

# force one explicitly
backend = ijson.get_backend('yajl2_cffi')
for obj in backend.items(f, 'item'):
    handle(obj)

# or from the environment, before import:
# IJSON_BACKEND=yajl2_c python app.py

Backends are tried in descending speed order and the first importable one wins, so a missing C extension quietly downgrades you to the pure Python parser. Assert on ijson.backend at startup in production if throughput matters.

Push chunks in instead of letting ijson readpush-data-in

import ijson

@ijson.coroutine
def print_cities():
    while True:
        obj = (yield)
        if obj['type'] == 'city':
            print(obj['name'])

coro = ijson.items_coro(print_cities(), 'earth.europe.item')
for chunk in socket_chunks():
    coro.send(chunk)
coro.close()

Use this when something else owns the read loop, such as a websocket handler or a message consumer. Every iterator has a *_coro twin that takes a target instead of a file. ijson.sendable_list() is a simpler target when you just want to accumulate results, as long as you clear it between sends.

Catch truncated and malformed inputhandle-parse-errors

import ijson
from ijson import IncompleteJSONError, JSONError

try:
    with open('maybe-truncated.json', 'rb') as f:
        for obj in ijson.items(f, 'item'):
            handle(obj)
except IncompleteJSONError:
    print('stream ended mid-document')
except JSONError as e:
    print(f'invalid JSON: {e}')

IncompleteJSONError subclasses JSONError, so order the handlers accordingly. An IncompleteJSONError with an empty message usually means invalid UTF-8 rather than truncation; pipe the input through iconv -f utf8 -t utf8 -c or a lenient incremental decoder first.

Measure before tuning buffer sizetune-throughput

# compare methods and backends against your own file
# python -m ijson.benchmark my/file.json -m items -p results.item

import ijson

with open('big.json', 'rb') as f:
    for obj in ijson.items(f, 'results.item', buf_size=1 << 20, use_float=True):
        handle(obj)

buf_size defaults to 65536. Raising it helps on fast local disks and hurts on high-latency sources where you want to start work sooner. The bundled benchmark module accepts your own file, so guess less.

Alternatives

Package	Registry	Pick it when
orjson	PyPI	The document fits in memory and you want raw speed, or you are parsing newline-delimited records one line at a time
json-stream	PyPI	You want a lazy dict-and-list interface you can index and iterate like normal Python instead of learning prefixes and events
jsonlines	PyPI	Your file is one JSON value per line and you want a reader that handles the framing rather than a streaming parser
msgspec	PyPI	You want fast parsing plus schema validation into typed structs, and can afford to hold each message in memory