One API spans a 46.5 MB distributed server
Mesh LLM's 46.5 MB checkout implements three execution choices behind an OpenAI-compatible API. A model can stay on the receiving machine, a peer can serve it elsewhere, or the Skippy runtime can divide model layers among several nodes. Clients keep using the same /v1 routes, so applications need not know which computer has the model or whether several GPUs are involved.
Our scan counted 3,469 files and about 867,362 lines of source across the Rust host, native runtime, web console, documentation, and support code. Commands cover local-only serving, private invites, public discovery, model publishing, agent launchers, plugins, diagnostics, and management. This is an inference platform even when its first chat request resembles ordinary local model software.
Seventy-two reviewed model families can use staged serving
The README lists 72 model families in its reviewed support set and 89 certified rows in the larger parity inventory. When one node cannot hold a model, package-backed splits assign contiguous layer ranges to suitable peers. The coordinator starts downstream stages first and only publishes the initial route after every required stage reports ready.
Placement does not make networking free. The split guide prefers directly reachable, low-latency UDP paths and treats relay-only peers as unsuitable for stage work. That condition was outside our 348-test sandbox run. Open issue 1300 gives a concrete failure: a cloud provider remapped the container's UDP port, the invite advertised the wrong endpoint, and the otherwise nearby peer fell back to a relay before split admission excluded it.
What happened when we ran it
Our run installed 805 packages in 37 seconds, then completed the build in 723 seconds. That measurement setup used commit f219b03 in an unprivileged container with 3 CPUs and 12 GB of RAM, without secrets. Both stages succeeded, but more than 12 minutes for the build makes source changes expensive compared with installing a published executable.
cargo test finished in 130 seconds with 348 passed and 0 failed out of 348. The measured checkout also contained 65 CI workflow files, which is unusually extensive visible automation. There was no tests directory, so the passing cases live elsewhere in the Rust workspace rather than under that conventional path. No Dockerfile was present either.
The 3,469-file scan counted about 867,362 lines of source, much more than a small local runner. Our sandbox measured install, build, and tests only. We did not download a model, start a mesh, split layers, join the public network, or measure generation speed. The successful 348-test result therefore says the checked-out workspace compiled and passed its automated cases. It does not prove that a particular GPU mix, model package, or network path will serve correctly.
Release binaries avoid the 723-second source build
The 723-second build is avoidable for ordinary users. The project publishes executable installers, Homebrew packages, Ubuntu and Arch packages, OCI images, checksums, and SBOMs through its release process and packaging repository. Source builders need just, CMake, Rust, Node.js 24, and npm. Native CUDA, ROCm, and Vulkan runtimes bring their own compiler or development-file requirements, while Metal is limited to macOS.
After installation, mesh-llm setup chooses a backend, and serve --auto can obtain a suitable model, join a public mesh, and expose inference on port 9337 plus the console on port 3131. A private mesh instead prints an invite token. Local-only mode skips discovery, QUIC, peer maintenance, split planning, plugins, and the console, but it requires absolute, non-symlinked model paths and refuses to distribute a model that does not fit locally.
Version 0.76.0 encrypts transport without proving the peer
Version 0.76.0 documents QUIC encryption for prompts, responses, and split activations between nodes; an iroh relay forwards ciphertext. That protects traffic in transit. The mesh guide separately says signed release attestation proves that a trusted signer published a binary, not that the remote host is running unmodified code. Trust flags, owner identities, private invites, and model choice still decide which peers should handle sensitive prompts.
Public discovery uses Nostr by default, while mDNS can keep discovery and startup on the LAN. Blackboard posts on a public mesh are visible to every peer, and the guide explicitly warns against sending secrets or customer data there. Our 12 GB sandbox did not test either network mode. Operators should start private, inspect direct paths on every node, and expose the HTTP listener beyond loopback only when another machine genuinely needs it.
Version 0.76.0 is active and still has hardware gaps
Mesh LLM released v0.76.0 on September 10, 2026, and GitHub recorded another push on September 11. The repository had 3,376 stars, 152 open issues and pull requests, and 101 open issues when fetched. That is active development rather than a dormant tag. The release added major serving and cache work while fixing many split, runtime, packaging, and security defects, a pace that calls for pinned versions and staged upgrades.
Documentation is broad, though one section has already aged. The current README still describes three Windows problems as blockers under a v0.76.0 release candidate, while issues 1510, 1511, and 1512 are now closed and stable v0.76.0 is published. A newer open report, issue 1811, says Windows reports 4 GB for a 16 GB AMD card and may reject models based on that value. Windows operators should check the live issue state and verify detected capacity on their own hardware.
Several machines must justify an 805-package codebase
Mesh LLM makes sense when an 805-package install and distributed networking solve a constraint you truly have: no single node can hold the model, or several model hosts need one API and one control surface. Exo is the closer comparison for Apple-heavy clusters, while Ollama removes most of this machinery for one machine. vLLM fits a more conventional serving fleet. The passed 348-test suite earns Mesh LLM a serious trial, but the trial should reproduce your exact model, GPU mix, and network path before adoption.