pyzmq
pyzmq is the Python binding for ZeroMQ, a C messaging library that gives you sockets with built-in patterns instead of raw byte streams. You create a Context, ask it for a socket of a given type, and bind or connect it over TCP, IPC, or in-process transports. The socket type decides the semantics: REQ and REP enforce a strict request then reply alternation, PUB and SUB fan out with topic filtering, PUSH and PULL round-robin work across workers, and DEALER and ROUTER give you fully asynchronous routing with explicit peer identities. Messages are whole frames, not streams, so you never write length-prefix framing yourself, and multipart messages let you build envelopes. There is no broker process and no server to run: ZeroMQ is a library that turns your own processes into the topology. pyzmq also ships an asyncio-aware layer, a Poller for multiplexing, CURVE encryption bindings, and a green variant for gevent.
The right tool when you want socket-level messaging patterns between processes you own and no broker to operate. The cost is that reliability, retries, and back pressure become your protocol design work, so anything needing durable delivery belongs on a broker instead.
Use it if
- You need fast in-process, inter-process, or cross-host messaging between components you control, and running RabbitMQ or Kafka for that would be more infrastructure than the problem deserves
- The messaging shapes you want are already socket types: fan-out with PUB and SUB, work distribution with PUSH and PULL, or asynchronous request routing with DEALER and ROUTER
- You want message framing, automatic reconnection, and queueing handled for you rather than writing a length-prefix protocol over a raw TCP socket
- You are building an asyncio service and want the same socket types inside it, which zmq.asyncio provides with awaitable send and recv
- You need to talk to something already speaking ZeroMQ, which includes Jupyter kernels, many trading and robotics stacks, and a lot of scientific tooling
- You need durability: ZeroMQ keeps messages in memory only, so a crashed or restarted process loses whatever was queued, and there are no acknowledgements, no dead-letter handling, and no replay
- You expect a PUB socket to be reliable: subscribers that connect after a publisher starts miss everything sent before the handshake completes (the slow joiner problem), and once the send high water mark is hit a PUB socket silently drops messages rather than blocking
- You want managed infrastructure or an operations story: there is no broker to monitor, which also means no queue depth dashboards, no management UI, and no server-side routing rules
- Your team is not ready to design a protocol: sockets are primitives, so heartbeating, retries, request timeouts, and worker liveness are all yours to build, and the ZeroMQ Guide is a book-length read for a reason
- You share sockets across threads: a Context is thread-safe but a Socket is not, and the usual crash people hit is two threads calling recv on one socket
- You need the C library to be moving: the latest libzmq release, v4.3.5, dates from October 2023, so pyzmq updates mostly bring build, platform, and typing improvements rather than new protocol capability
- You are relying on IPC transport on Windows: the official Windows wheels ship with IPC disabled because of unavoidable crashes, so ipc:// endpoints only work if you build from source
Setup reality
pip install pyzmq installs a wheel on macOS, Windows, Linux, and Android for CPython and PyPy, including a CPython 3.12 stable-ABI wheel since 27.0 that covers newer interpreters, plus free-threaded builds. That means no compiler for most people, and the wheel bundles libzmq so there is no system package to install first. Compiling is the awkward path: pip install --no-binary=pyzmq pyzmq needs a C compiler, cmake, and either a system libzmq or a bundled build that also fetches libsodium, and building from a git checkout additionally needs a recent Cython. What actually bites in production is not installation. Sockets are not thread-safe while the Context is, so each thread gets its own socket. Sockets default to a linger of -1, meaning context.term() blocks forever if anything is unsent, so set linger or use context.destroy(linger=0) at shutdown. Every socket has a send and receive high water mark that defaults to 1000 messages, and hitting it either blocks or, on PUB and SUB, silently drops. And recv_pyobj runs pickle on whatever arrives, which the pyzmq docs explicitly discourage without message authentication.
Patterns
REQ and REP for simple synchronous callsrequest-reply
# server.py
import zmq
with zmq.Context() as ctx, ctx.socket(zmq.REP) as sock:
sock.bind('tcp://127.0.0.1:5555')
while True:
request = sock.recv_string()
sock.send_string(f'echo: {request}')
# client.py
import zmq
with zmq.Context() as ctx, ctx.socket(zmq.REQ) as sock:
sock.connect('tcp://127.0.0.1:5555')
sock.send_string('hello')
print(sock.recv_string())REQ and REP enforce strict alternation: two sends in a row on a REQ socket raise ZMQError. If the peer dies mid-exchange the socket is stuck in the wrong state forever, which is why production code uses DEALER and ROUTER instead.
PUB and SUB with topic filteringpublish-subscribe
# publisher
import time, zmq
ctx = zmq.Context()
pub = ctx.socket(zmq.PUB)
pub.bind('tcp://*:5556')
time.sleep(0.2) # let subscribers finish connecting
while True:
pub.send_multipart([b'temps.kitchen', b'21.5'])
time.sleep(1)
# subscriber
sub = ctx.socket(zmq.SUB)
sub.connect('tcp://127.0.0.1:5556')
sub.subscribe(b'temps.') # prefix match; b'' means everything
while True:
topic, payload = sub.recv_multipart()A SUB socket with no subscribe call receives nothing at all. Filtering is a byte prefix, not a pattern. The sleep is the standard workaround for the slow joiner problem: messages published before the subscriber's handshake completes are simply gone.
Fan work out to a pool of workerspush-pull-pipeline
# ventilator
ctx = zmq.Context()
push = ctx.socket(zmq.PUSH)
push.bind('tcp://*:5557')
for i in range(1000):
push.send_json({'task': i})
# worker (run several)
pull = ctx.socket(zmq.PULL)
pull.connect('tcp://127.0.0.1:5557')
while True:
task = pull.recv_json()
handle(task)PUSH round-robins across connected PULL peers, so a slow worker still gets its equal share and holds up its own queue. Nothing is acknowledged: a worker that dies mid-task takes that task with it.
Asynchronous routing with explicit peer identitiesdealer-router
# broker side
router = ctx.socket(zmq.ROUTER)
router.bind('tcp://*:5558')
identity, empty, payload = router.recv_multipart()
router.send_multipart([identity, b'', b'ack:' + payload])
# client side
dealer = ctx.socket(zmq.DEALER)
dealer.setsockopt(zmq.IDENTITY, b'worker-7')
dealer.connect('tcp://127.0.0.1:5558')
dealer.send_multipart([b'', b'job-42'])
empty, reply = dealer.recv_multipart()ROUTER prepends the sender's identity frame on receive and strips it on send, which is how replies find their way back. The empty delimiter frame is a convention inherited from REQ and REP; drop it and a REQ peer on the other end will not parse your reply.
Multiplex several sockets with a timeoutpoll-multiple-sockets
import zmq
poller = zmq.Poller()
poller.register(sub, zmq.POLLIN)
poller.register(pull, zmq.POLLIN)
while True:
events = dict(poller.poll(timeout=1000)) # milliseconds
if sub in events:
handle_event(sub.recv_multipart())
if pull in events:
handle_task(pull.recv_json())
if not events:
heartbeat()poll() takes milliseconds and returns an empty list on timeout, which is where periodic work like heartbeating belongs. A Poller can also register plain file descriptors, so you can mix ZeroMQ sockets with regular sockets in one loop.
Use ZeroMQ inside an asyncio serviceasyncio-sockets
import asyncio
import zmq
import zmq.asyncio
async def main():
ctx = zmq.asyncio.Context()
sock = ctx.socket(zmq.PULL)
sock.bind('tcp://127.0.0.1:5559')
try:
while True:
msg = await sock.recv_multipart()
asyncio.create_task(handle(msg))
finally:
sock.close()
ctx.term()
asyncio.run(main())zmq.asyncio.Context is a different class from zmq.Context, and sockets made by one do not work with the other. The socket is still not safe to await from two tasks at once, so keep one reader per socket and hand work off with create_task.
Stop recv from blocking foreverreceive-timeout
import zmq
sock.setsockopt(zmq.RCVTIMEO, 2000) # ms
sock.setsockopt(zmq.SNDTIMEO, 2000)
try:
reply = sock.recv_string()
except zmq.Again:
print('no reply within 2s')
# one-off non-blocking read
try:
reply = sock.recv(zmq.NOBLOCK)
except zmq.Again:
reply = NoneEvery blocking call defaults to waiting forever. A timeout surfaces as zmq.Again, which subclasses ZMQError with errno EAGAIN, so catch Again specifically rather than swallowing all ZMQError.
Shut down without hangingcontext-shutdown
import zmq
ctx = zmq.Context()
sock = ctx.socket(zmq.PUSH)
sock.setsockopt(zmq.LINGER, 1000) # wait at most 1s for unsent messages
sock.connect('tcp://127.0.0.1:5557')
...
sock.close()
ctx.term()
# blunt version for shutdown paths: close every socket and drop pending sends
ctx.destroy(linger=0)LINGER defaults to -1, meaning infinite, so ctx.term() with an unsent message and no reachable peer hangs your process at exit with no error. Setting LINGER on every socket or calling destroy(linger=0) is the fix.
Bound the queue instead of eating memoryhigh-water-mark
sock.setsockopt(zmq.SNDHWM, 10_000)
sock.setsockopt(zmq.RCVHWM, 10_000)
# detect back pressure instead of blocking
try:
sock.send_json(payload, flags=zmq.NOBLOCK)
except zmq.Again:
metrics.increment('zmq.backpressure')The default is 1000 messages in each direction. When the send HWM is reached, PUSH and DEALER block while PUB silently discards, so a publisher outrunning its subscribers loses data with no exception anywhere. HWM must be set before bind or connect.
Send structured data safelyserialize-payloads
sock.send_json({'id': 7, 'ok': True})
msg = sock.recv_json()
# or bring your own codec
import msgpack
sock.send(msgpack.packb(obj, use_bin_type=True))
obj = msgpack.unpackb(sock.recv(), raw=False)
# avoid unless every peer is authenticated:
# sock.send_pyobj(obj); obj = sock.recv_pyobj()recv_pyobj unpickles whatever bytes arrive, which is remote code execution if anything untrusted can reach the socket. The pyzmq docs discourage it without message authentication; send_json or msgpack costs nothing extra and does not have that property.
Encrypt and authenticate with CURVEcurve-encryption
import zmq
server_public, server_secret = zmq.curve_keypair()
client_public, client_secret = zmq.curve_keypair()
# server
rep = ctx.socket(zmq.REP)
rep.curve_secretkey = server_secret
rep.curve_publickey = server_public
rep.curve_server = True
rep.bind('tcp://*:5560')
# client
req = ctx.socket(zmq.REQ)
req.curve_secretkey = client_secret
req.curve_publickey = client_public
req.curve_serverkey = server_public
req.connect('tcp://127.0.0.1:5560')Set every CURVE option before bind or connect. This encrypts the link and proves the server's identity, but it does not restrict which clients may connect: for that you run a zmq.auth authenticator with an allow list of client public keys.
Forward between two socketsproxy-device
import zmq
ctx = zmq.Context()
frontend = ctx.socket(zmq.XSUB)
frontend.bind('tcp://*:5561') # publishers connect here
backend = ctx.socket(zmq.XPUB)
backend.bind('tcp://*:5562') # subscribers connect here
try:
zmq.proxy(frontend, backend) # blocks forever
except KeyboardInterrupt:
pass
finally:
frontend.close()
backend.close()
ctx.term()An XSUB and XPUB proxy gives publishers and subscribers one stable endpoint each so neither side needs to know the other's address. zmq.proxy never returns, so run it in its own process or thread, or use zmq.devices.ProcessDevice to get that for free.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| pika | PyPI | You want a real broker with durable queues, acknowledgements, and routing rules, and can run RabbitMQ |
| nats-py | PyPI | You want brokered pub/sub and request/reply with a lightweight server and optional persistence through JetStream |
| pynng | PyPI | You want the same scalability-protocol ideas from nanomsg-next-generation, with a cleaner story around thread safety and context objects |