protobuf review
protobuf 7.36.0 is the Python runtime that generated Protocol Buffer classes use to encode, decode, inspect, and convert typed messages. You define field numbers and types in a `.proto` file, run the separate `protoc` compiler, then import the resulting `_pb2.py` module. The bytes can be read by protobuf runtimes in other languages, provided every side follows the same schema evolution rules. Release 7.36.0 adds Python buffer assignment for repeated scalar fields, corrects out-of-range `pop()` errors, protects descriptor-pool caches with a mutex, and fixes upb memory bugs.
protobuf 7.36.0 installed in 0.2 seconds and used 2 MB in our sandbox, with 0 direct dependencies and 0 audit findings, making the runtime cheap once a schema toolchain already exists. Install it for gRPC or a cross-language wire contract; use JSON or msgpack when compiler and compatibility work would outweigh that benefit.
We installed it
| Install | ✓ · 0.2s | 1 package on disk · 2 MB |
| Import | ✓ | import google in 0.02s · compiled extensions · requires Python >=3.10 |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does protobuf install cleanly?
Yes. In a fresh container with an empty cache, pip install protobuf finished in 0.2s, leaving 1 package and 2 MB on disk. pip-audit reported no known vulnerabilities.
What does protobuf need to run?
Python >=3.10, and a platform wheel with compiled extensions. In our run import google succeeded in 0.02s.
protobuf or flatbuffers: which should you use?
flatbuffers: Use it when reading serialized data without a separate decode step is worth a more specialized generated API. protobuf 7.36.0 installed in 0.2 seconds and used 2 MB in our sandbox, with 0 direct dependencies and 0 audit findings, making the runtime cheap once a schema toolchain already exists.
When should you not use protobuf?
Choose JSON with a validator for a small internal API when readable logs and no code-generation step matter more than a binary contract.
Discussed on
- hnParsing Protobuf at 2+GB/S: How I Learned to Love Tail Calls in C586 points
- hnParsing protobuf at 2+GB/s: how I learned to love tail calls in C (2021)366 points
- hnFinding a $5,000 Google Maps XSS by fiddling with Protobuf304 points
- hnA new ProtoBuf generator for Go296 points
- hnGo Protobuf: The New Opaque API287 points
Use it if
- A Python service shares a versioned message contract with clients or services written in other languages.
- Your application uses gRPC, whose request and response objects are generated from protobuf schemas.
- Old readers must accept newer messages and preserve fields they do not recognize while forwarding them.
- Your team can pin the compiler, regenerate code in CI, and review field-number changes as part of API work.
- Choose JSON with a validator for a small internal API when readable logs and no code-generation step matter more than a binary contract.
- Choose msgpack for schema-free dictionaries and lists; protobuf requires owned `.proto` files and generated classes.
- Do not install this package expecting a compiler. The README directs non-C++ users to separate prebuilt `protoc` binaries.
- Avoid protobuf when generated files cannot be rebuilt during upgrades. The runtime checks generated-code compatibility and may reject stale `_pb2.py` modules at import time.
- Skip it for loosely governed documents. Deleted field numbers must stay reserved, and changing a field's number changes its wire identity.
Setup reality
Our Python 3.12 sandbox installed protobuf 7.36.0 in 0.2 seconds. The result was 1 package occupying 2 MB, with 0 direct dependencies and 0 known vulnerabilities from pip-audit. It requires Python 3.10 or newer and includes compiled .so extensions. The package has no py.typed marker. import google completed successfully in 0.02 seconds.
The install supplies the Python runtime only. Download a released protoc binary, or use a separate compiler package, and run it with stable include and output paths. A source such as billing/invoice.proto produces an _pb2.py module whose imports reflect the schema tree. Generate .pyi files with --pyi_out when your compiler supports that output, since this wheel itself does not advertise typing through py.typed.
Compiler output and runtime versions are coupled by a documented compatibility window. Upgrade them together and regenerate checked-in modules before deploying. A stale generated module can fail during import, long before a request reaches the parser. Schema changes carry their own permanent bookkeeping: never recycle a removed field number, reserve deleted names and numbers, and use optional when code must distinguish an absent scalar from its default value.
Generated messages have container rules that surprise Python users. Mutate nested messages or call CopyFrom() instead of assigning a new object. Use add() for repeated messages. Parsing into an existing instance merges fields, so clear it first or allocate a new message. Protobuf JSON also has its own names, integer encodings, and presence behavior; test JSON gateways separately from the binary path.
Patterns
Generate Python modules and stubs compile-schema
protoc -I schemas \
--python_out=src \
--pyi_out=src \
schemas/acme/person.protoThe protobuf wheel does not contain `protoc`. Pin a released compiler in CI and keep its generated output inside the runtime compatibility window.
Create a generated message construct-message
from acme import person_pb2
person = person_pb2.Person(
id=42,
name="Ada",
emails=["ada@example.com"],
)Unset scalar fields read as schema defaults. Use `optional` in the schema when zero or an empty string must differ from absence.
Serialize and parse binary data serialize-message
payload = person.SerializeToString()
copy = person_pb2.Person()
copy.ParseFromString(payload)`ParseFromString()` merges into its receiver. Allocate a new message or call `Clear()` before reusing an instance.
Set a nested message populate-nested-message
profile = person_pb2.Profile()
profile.address.city = "Delhi"
address = person_pb2.Address(city="Mumbai")
profile.address.CopyFrom(address)Direct assignment to a message field raises an error. Mutate its fields or copy another message of the same generated type.
Add repeated message values append-repeated-message
member = team.members.add()
member.id = 7
member.name = "Lin"
team.labels.extend(["backend", "on-call"])Repeated message containers use `add()`. Repeated scalar containers accept familiar operations such as `append()` and `extend()`.
Write and inspect a map use-map-field
job.tags["region"] = "ap-south-1"
for key, value in job.tags.items():
print(key, value)Reading a missing scalar map key can insert its default value. Test membership before lookup when mutation would be a bug.
Convert a message to protobuf JSON convert-json
from google.protobuf import json_format
text = json_format.MessageToJson(
person,
preserving_proto_field_name=True,
)
restored = json_format.Parse(text, person_pb2.Person())The JSON mapping is defined by protobuf and differs from a plain object dump. Unknown JSON fields raise unless parsing enables `ignore_unknown_fields`.
Distinguish zero from absent check-field-presence
# schema: optional int32 retries = 1;
request = jobs_pb2.Request()
assert not request.HasField("retries")
request.retries = 0
assert request.HasField("retries")`HasField()` works for message fields and presence-enabled scalars. A plain proto3 scalar has no explicit presence.
Find the active oneof member inspect-oneof
payment.card.token = "tok_123"
assert payment.WhichOneof("method") == "card"
payment.bank_account = "DE89..."
assert payment.WhichOneof("method") == "bank_account"Setting one member clears the previous member in that `oneof`. `WhichOneof()` returns `None` when none is present.
Convert an aware datetime handle-timestamp
from datetime import datetime, timezone
from google.protobuf.timestamp_pb2 import Timestamp
ts = Timestamp()
ts.FromDatetime(datetime.now(timezone.utc))
value = ts.ToDatetime(tzinfo=timezone.utc)Passing `tzinfo` to `ToDatetime()` returns an aware datetime. Without it, the result represents UTC in a naive object.
Print a message for debugging format-text
from google.protobuf import text_format
print(text_format.MessageToString(person))
parsed = person_pb2.Person()
text_format.Parse('id: 42 name: "Ada"', parsed)Text format needs the schema and is intended for protobuf-aware tools. Do not treat it as a stable replacement for the binary encoding.
Apply a partial update merge-messages
current = person_pb2.Person(id=42, name="Old")
patch = person_pb2.Person(name="New")
current.MergeFrom(patch)`MergeFrom()` combines fields according to protobuf merge rules. Presence controls whether a scalar in the patch can intentionally overwrite a value with its default.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| flatbuffers | PyPI | Use it when reading serialized data without a separate decode step is worth a more specialized generated API. |
| msgpack | PyPI | Use it for compact binary dictionaries and lists when you do not want schemas or code generation. |
| marshmallow | PyPI | Use it for Python object validation and serialization when other languages do not share the contract. |
More utils guides
lru-cache · type-fest · ajv · p-limit · find-up · js-yaml · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.