msgpack review
msgpack 1.2.1 encodes Python dictionaries, sequences, strings, bytes, numbers, booleans, and None into MessagePack binary values. packb and unpackb handle a single object, while Packer and Unpacker support repeated values and incremental streams. Extension type codes let applications represent values such as decimals without wrapping them in string-keyed maps. The 1.2.1 release fixes a segfault reached by calling Unpacker.unpack() or skip() after a decode failure, but the changelog still says that failed Unpacker instances must be discarded. It now requires Python 3.10 or later.
msgpack 1.2.1 installed as one 2 MB package in 0.2 seconds and imported in 0.13 seconds with zero audit findings in our sandbox. Choose it for controlled cross-language binary messages; choose JSON or a schema-aware codec when inspection or field validation matters more.
We installed it
| Install | ✓ · 0.2s | 1 package on disk · 2 MB |
| Import | ✓ | import msgpack in 0.13s · compiled extensions · requires Python >=3.10 |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does msgpack install cleanly?
Yes. In a fresh container with an empty cache, pip install msgpack finished in 0.2s, leaving 1 package and 2 MB on disk. pip-audit reported no known vulnerabilities.
What does msgpack need to run?
Python >=3.10, and a platform wheel with compiled extensions. In our run import msgpack succeeded in 0.13s.
msgpack or msgspec: which should you use?
msgspec: Use it when MessagePack bytes must decode directly into validated Struct types. msgpack 1.2.1 installed as one 2 MB package in 0.2 seconds and imported in 0.13 seconds with zero audit findings in our sandbox.
When should you not use msgpack?
Operators need to inspect payloads with logs, curl, and browser tools; JSON remains readable without a decoder
Discussed on
- hnMessagePack: like JSON, but fast and small326 points
- hnIt's like JSON. but fast and small.188 points
- hnMessagePack: It's like JSON, but fast and small.102 points
- hnMsgPack vs. JSON: Cut your client-server exchange traffic by 50%78 points
- hnMessagePack: Fast and Compact Serialization50 points
Use it if
- Profiles show serialization cost on frequent messages and every participating language has a compatible MessagePack implementation
- Payloads contain byte strings that would otherwise need base64 when sent as JSON
- A schema-free binary value format is acceptable and Python pickle is unsuitable for cross-language exchange
- A socket or file contains consecutive values that should be decoded incrementally through Unpacker
- Operators need to inspect payloads with logs, curl, and browser tools; JSON remains readable without a decoder
- Decoded messages must become validated typed structures; msgspec can apply a declared Struct during MessagePack decoding
- The runtime is PyPy, where the README says msgpack._cmsgpack is unsupported and msgpack.fallback is used
- Schema evolution must reject missing or renamed fields automatically; MessagePack maps provide no schema registry or validation layer
- A Windows deployment lacks a compatible wheel and cannot install Visual Studio or the Windows SDK for the extension build
Setup reality
Our install of msgpack 1.2.1 finished in 0.2 seconds on Python 3.12. It left one package using 2 MB, declared zero direct dependencies, and imported in 0.13 seconds. pip-audit found zero known vulnerabilities. The wheel contains compiled .so files, requires Python >=3.10, and does not ship py.typed. The measured package metadata did not provide a usable license value.
There are no credentials or config files. CPython normally loads msgpack._cmsgpack from a wheel. PyPy uses msgpack.fallback, and a Windows source build needs Visual Studio or the Windows SDK. If speed motivated adoption, verify msgpack.Packer.module in the deployed image. Type checking requires third-party stubs or a local protocol because the installed distribution has no py.typed marker.
Specify use_bin_type and raw at integration boundaries. Modern defaults distinguish text from bytes; pre-1.0 peers may treat both as raw. Datetime packing accepts aware datetimes and recovers the instant in UTC, not the original timezone object. Every implementation must share extension-code assignments and matching default or ext_hook behavior. A default callback must raise TypeError for values it cannot encode.
For untrusted bytes, cap max_buffer_size and keep strict_map_key unless non-string keys are part of a controlled contract. unpackb accepts one object and raises ExtraData when more bytes follow; Unpacker is the right tool for consecutive frames. OutOfData means feed another chunk. After any other unpacking exception, create a new Unpacker as the 1.2.1 changelog instructs, even though that release fixed the earlier segfault on accidental reuse.
Patterns
Pack one value and read it back pack-unpack-bytes
import msgpack
data = msgpack.packb({"id": 1, "tags": ["a", "b"]})
print(type(data), len(data)) # <class 'bytes'> 18
back = msgpack.unpackb(data)
print(back) # {'id': 1, 'tags': ['a', 'b']}dumps and loads are aliases. unpackb requires exactly one encoded value and raises ExtraData when trailing messages remain.
Feed socket chunks to Unpacker stream-unpack
import msgpack
unpacker = msgpack.Unpacker()
while chunk := sock.recv(65536):
unpacker.feed(chunk)
for message in unpacker:
handle(message)Iteration pauses on an incomplete value and continues after feed() receives more bytes. Replace the Unpacker after any exception other than OutOfData.
Write consecutive values to a file pack-to-file
import msgpack
with open("events.msgpack", "wb") as f:
for event in events:
msgpack.pack(event, f)
with open("events.msgpack", "rb") as f:
for event in msgpack.Unpacker(f):
handle(event)pack writes one value to a file-like object. Unpacker(file) is the documented reader for files containing multiple consecutive values.
Encode an aware datetime datetime-support
import datetime
import msgpack
now = datetime.datetime.now(datetime.timezone.utc)
packed = msgpack.packb({"created": now}, datetime=True)
back = msgpack.unpackb(packed, timestamp=3)
print(back["created"]) # datetime in UTCdatetime=True rejects naive values. timestamp=3 returns a UTC datetime for the same instant and does not restore the original timezone object.
Convert Decimal through hooks custom-types-hooks
import decimal
import msgpack
def default(obj):
if isinstance(obj, decimal.Decimal):
return {"__dec__": str(obj)}
raise TypeError(f"Unknown type: {obj!r}")
def object_hook(obj):
if "__dec__" in obj:
return decimal.Decimal(obj["__dec__"])
return obj
raw = msgpack.packb({"price": decimal.Decimal("1.10")}, default=default)
print(msgpack.unpackb(raw, object_hook=object_hook))default must raise TypeError for unsupported values. object_hook runs for every decoded map, so its marker check should be narrow.
Assign an extension code to array data ext-type
import array
import msgpack
def default(obj):
if isinstance(obj, array.array) and obj.typecode == "d":
return msgpack.ExtType(42, obj.tobytes())
raise TypeError(f"Unknown type: {obj!r}")
def ext_hook(code, data):
if code == 42:
a = array.array("d")
a.frombytes(data)
return a
return msgpack.ExtType(code, data)
values = array.array("d", [1.2, 3.4])
assert msgpack.unpackb(msgpack.packb(values, default=default), ext_hook=ext_hook) == valuesAll producers and consumers must reserve code 42 for the same payload. Negative extension codes are reserved by the MessagePack specification.
Bound allocation while decoding harden-untrusted-input
import msgpack
try:
payload = msgpack.unpackb(
body,
max_buffer_size=1 * 1024 * 1024,
strict_map_key=True,
)
except (msgpack.exceptions.UnpackException, ValueError) as exc:
reject(exc)max_buffer_size also limits container preallocation. strict_map_key accepts only str and bytes keys by default to reduce hash-denial-of-service exposure.
Return tuples instead of lists tuples-not-lists
import msgpack
packed = msgpack.packb([[1, 2], [3, 4]])
print(msgpack.unpackb(packed, use_list=False))
# ((1, 2), (3, 4))use_list=False changes every decoded array to a tuple. Consumers that append or replace elements will need adjustment.
Keep one Packer across records reuse-packer
import msgpack
packer = msgpack.Packer(buf_size=1024 * 1024)
with open("out.msgpack", "wb") as f:
for record in records:
f.write(packer.pack(record))buf_size controls the initial internal allocation. autoreset=False changes pack() behavior by accumulating output until the caller reads and resets it.
Read a container one member at a time read-container-headers
import msgpack
unpacker = msgpack.Unpacker(open("big.msgpack", "rb"))
count = unpacker.read_array_header()
for _ in range(count):
row = unpacker.unpack()
if row["kind"] != "click":
continue
handle(row)read_array_header returns the member count. skip() advances past an encoded value without constructing its Python representation.
Set flags for an old raw-format peer legacy-raw-format
import msgpack
# writing for an old peer that has no bin type
old = msgpack.packb([b"spam", "eggs"], use_bin_type=False)
# reading from one: everything string-ish arrives as bytes
print(msgpack.unpackb(old, raw=True)) # [b'spam', b'eggs']Version 1.0 changed the defaults to use_bin_type=True and raw=False. Pin both flags when an older peer treats text and bytes as raw values.
Check which Packer implementation loaded verify-c-extension
import msgpack
module = msgpack.Packer.__module__
if module != 'msgpack._cmsgpack':
raise RuntimeError(f"compiled msgpack extension missing: {module}")The README says PyPy uses msgpack.fallback. Check Packer.__module__ in environments where the compiled extension is a deployment requirement.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| msgspec | PyPI | Use it when MessagePack bytes must decode directly into validated Struct types. |
| orjson | PyPI | Use it when readable JSON and existing HTTP tooling matter more than a binary value format. |
| cbor2 | PyPI | Use it when an IETF-standardized CBOR format is required by the surrounding protocol. |
More utils guides
lru-cache · type-fest · ajv · p-limit · find-up · js-yaml · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.