fastavro review
fastavro reads and writes Apache Avro data from Python dictionaries. It covers object container files, single records without a container header, Avro JSON, schema resolution, logical types, compression codecs, canonical forms, and schema fingerprints. Our Python 3.12 test imported version 1.12.2 in 0.12 seconds and found compiled extensions, so deployments depend on a suitable wheel or a working native build path. The current release adds Windows ARM64 wheels. It does not implement Avro RPC.
fastavro 1.12.2 installed in 0.4 seconds, used 13 MB, and imported in 0.12 seconds in our sandbox with 0 audit findings. Pick it for fast Avro files and schemaless records, but walk away if you need Avro RPC or generated record classes.
We installed it
| Install | ✓ · 0.4s | 1 package on disk · 13 MB |
| Import | ✓ | import fastavro in 0.12s · compiled extensions · py.typed · requires Python >=3.9 |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does fastavro install cleanly?
Yes. In a fresh container with an empty cache, pip install fastavro finished in 0.4s, leaving 1 package and 13 MB on disk. pip-audit reported no known vulnerabilities.
What does fastavro need to run?
Python >=3.9, and a platform wheel with compiled extensions. In our run import fastavro succeeded in 0.12s, and the package ships py.typed for type checkers.
fastavro or avro: which should you use?
avro: Use Apache's Python implementation when project ownership and RPC support matter more than fastavro's compiled read and write path. fastavro 1.12.2 installed in 0.4 seconds, used 13 MB, and imported in 0.12 seconds in our sandbox with 0 audit findings.
When should you not use fastavro?
You need Avro RPC or protocol support, because the README lists every RPC feature as missing
What does fastavro block_reader’s offset attribute mean?
fastavro’s block_reader() returns Block objects whose offset attribute is the block’s byte offset from the beginning of the Avro file. Read it alongside size, the block size in bytes, and num_records, the number of records in that block.
from fastavro import block_reader
with open("large.avro", "rb") as fo:
for block in block_reader(fo):
print(block.offset, block.size, block.num_records)
for record in block:
process(record)
The decisive caveat is granularity: offset describes an Avro container block, not an individual record. block_reader() is therefore useful for building a block index or assigning independently decodable blocks to workers, but it does not provide a per-record offset. Iterating the Block yields its records.
fastavro supports file reading by blocks, but its repository is in maintenance mode, with even simple fixes or features potentially delayed.
Use it if
- Your Python service must exchange Avro container files with Spark, Hadoop, or JVM applications that already use Avro schemas
- You need schemaless record encoding for Kafka values and already have a separate way to distribute the writer schema
- You consume records written under older schemas and need Avro defaults, aliases, and reader-schema resolution
- You need canonical schema text or fingerprints so equivalent schemas receive the same identifier
- You need Avro RPC or protocol support, because the README lists every RPC feature as missing
- You want active feature development; the maintainer labels fastavro as maintenance mode and warns that Python updates, security work, and small fixes can be delayed
- Your workload scans a few columns from large analytical datasets; Parquet through pyarrow fits column selection better than a row-oriented Avro file
- You expect schema-generated Python models, because fastavro works with mappings and its type marker does not turn Avro records into domain classes
- Your production image cannot use compiled Python extensions or build them when no wheel matches the platform; the package we installed included .so files
Setup reality
In a fresh Python 3.12 container, fastavro 1.12.2 installed successfully in 0.4 seconds. One package occupied 13 MB afterward. The distribution declares 6 direct dependencies, requires Python 3.9 or newer, includes compiled .so files, and ships py.typed. import fastavro completed in 0.12 seconds. pip-audit reported 0 known vulnerabilities.
The basic file API needs no credentials or config file. Open Avro containers with rb or wb, and parse a schema once if it will be reused. parse_schema() adds internal data to its returned mapping, so retain an untouched schema when another tool expects the original JSON. Optional Snappy, Zstandard, and LZ4 paths need their matching codec packages; null, deflate, bzip2, and xz are listed by the project without the same external-package warning.
A schemaless payload carries neither a container header nor its writer schema. Put a schema ID beside the bytes or use a registry-aware layer, otherwise a consumer cannot reliably decode the record. Reader-schema evolution also follows Avro rules: a new field needs a default when older records lack it. Writes accept mapping-shaped records, and strict=True is worth enabling when extra or missing fields should stop the write instead of slipping into a data pipeline.
Patterns
Write a container file write-container-file
from fastavro import parse_schema, writer
schema = {
"type": "record",
"name": "Reading",
"fields": [
{"name": "sensor_id", "type": "string"},
{"name": "value", "type": "double"},
],
}
records = [{"sensor_id": "north-1", "value": 18.4}]
with open("readings.avro", "wb") as output:
writer(output, parse_schema(schema), records, strict=True)Use a binary handle. With strict=True, missing and unexpected record fields raise an error before bad data is accepted.
Iterate records from a container read-container-file
from fastavro import reader
with open("readings.avro", "rb") as source:
avro_reader = reader(source)
print(avro_reader.writer_schema)
print(avro_reader.codec)
for record in avro_reader:
consume(record)The reader consumes from the file's current position. Seek back and construct another reader when a second pass is required.
Encode one headerless record encode-one-record
from io import BytesIO
from fastavro import parse_schema, schemaless_writer
parsed = parse_schema(schema)
buffer = BytesIO()
schemaless_writer(buffer, parsed, {"sensor_id": "north-1", "value": 18.4})
payload = buffer.getvalue()These bytes do not contain the schema. Store or transmit a schema identifier alongside the payload.
Decode one headerless record decode-one-record
from io import BytesIO
from fastavro import schemaless_reader
record = schemaless_reader(BytesIO(payload), parsed_schema)Decoding must use the writer schema that produced the bytes. Passing an unrelated current schema can misread data or fail.
Supply a reader schema for old data evolve-reader-schema
reader_schema = {
"type": "record",
"name": "Reading",
"fields": [
{"name": "sensor_id", "type": "string"},
{"name": "value", "type": "double"},
{"name": "unit", "type": "string", "default": "C"},
],
}
with open("old.avro", "rb") as source:
for record in reader(source, reader_schema=reader_schema):
print(record["unit"])The default supplies unit for records written before that field existed. Without it, schema resolution cannot produce the new field.
Append records to an existing file append-container-file
from fastavro import writer
more = [{"sensor_id": "north-2", "value": 19.1}]
with open("readings.avro", "a+b") as output:
writer(output, None, more)The read-capable append mode lets fastavro inspect the header. A None schema reuses the schema already stored in that file.
Write with Zstandard compression compress-container-file
with open("readings.avro", "wb") as output:
writer(output, parsed_schema, records, codec="zstandard", codec_compression_level=6)Install the Zstandard codec dependency first. A plain fastavro install does not guarantee that this optional codec can load.
Emit Avro JSON write-avro-json
from fastavro import json_writer, parse_schema
with open("readings.json", "w", encoding="utf-8") as output:
json_writer(output, parse_schema(schema), records)Avro JSON applies Avro's rules for unions and logical types. It is different from serializing the original dictionaries with json.dump.
Name the selected union branch disambiguate-union
record = {
"payload": ("Open", {"path": "/tmp/report"})
}Tuple notation removes ambiguity when two named record branches accept the same mapping shape. It can be disabled with disable_tuple_notation.
Validate candidate records validate-before-write
from fastavro import parse_schema
from fastavro.validation import validate, validate_many
parsed = parse_schema(schema)
valid = validate(candidate, parsed, raise_errors=False)
validate_many(batch, parsed, strict=True)Validation is a separate walk over the record. Keep it at input boundaries or in tests instead of repeating it on trusted hot-path data.
Fingerprint canonical schema text fingerprint-schema
from fastavro.schema import fingerprint, to_parsing_canonical_form
canonical = to_parsing_canonical_form(schema)
schema_id = fingerprint(canonical, "CRC-64-AVRO")Hash the parsing canonical form. Hashing the original JSON makes formatting and object-key order part of the identifier.
Read one Avro block at a time iterate-blocks
from fastavro import block_reader
with open("readings.avro", "rb") as source:
for block in block_reader(source):
print(block.offset, block.size, block.num_records)
for record in block:
consume(record)Block offsets and sizes are useful for partitioning a large container after its header has been read.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| avro | PyPI | Use Apache's Python implementation when project ownership and RPC support matter more than fastavro's compiled read and write path. |
| pyarrow | PyPI | Use it for Parquet datasets and analytical jobs that read selected columns instead of full Avro records. |
| confluent-kafka | PyPI | Use it when Kafka transport and Confluent Schema Registry serializers should come from the same client package. |
More data guides
numpy · fsspec · pandas · sqlalchemy · pyarrow · lxml · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.