SMG needs inference workers before its routing can help
Shepherd Model Gateway is a candidate for teams evaluating cache-aware routing across an existing inference fleet. Its gateway quick start launches SMG with worker URLs, then sends a chat request through the gateway. The workers serve the model. A reader seeking a standalone model runner should account for that separate serving requirement.
The README describes cache-aware routing that tracks each worker's KV-cache state in radix trees. It also documents HTTP and gRPC transport and separation of prompt processing from token generation. These are capabilities to evaluate against an actual deployment. Our sandbox did not measure cache reuse, request latency or GPU utilization.
What happened when we ran it
Our supplied run checked out commit bc9ea78 on October 19, 2026. Installation completed in 28 seconds and recorded 1,012 dependencies. The build and test steps each timed out at 900 seconds. The test-log tail still showed compilation, including pyo3-build-config, smg-radix-tree, pyo3, smg-client and openapi-gen. It contains no completed suite summary, so it establishes neither a passing-test count nor a failing-test count.
The sandbox configuration specified 3 CPUs, 12 GB RAM and image lab-rust:1. That image label does not establish the actual Rust or operating-system version. The checkout contained 2,293 files, about 927,503 source lines and 44.8 MB. The scan found 42 CI workflow files, no Dockerfile and no tests directory. Those scan results do not establish test coverage or invalidate the README's published-container installation path.
The excerpt identifies no compiler error or reason for the timeouts. Source-build completion and correctness remain unverified, and no dependency-audit result was supplied. This run limits what we can conclude about the product; it does not demonstrate a serving failure.
Decide whether the routing policy justifies a gateway
The README lists cache_aware, least_load, round_robin and passthrough among its routing policies. For a fleet handling repeated prompt prefixes, the documented cache-aware policy gives operators a specific capability to evaluate. For a single engine, first identify which routing requirement would justify operating another service.
The documented backends include SGLang, vLLM, TensorRT-LLM, TokenSpeed and MLX, alongside cloud providers and compatible endpoints. A backend listing does not verify a particular engine build or deployment configuration. Acceptance checks should cover worker registration, streaming, cancellation and retry ownership. These are proposed checks, not results from our run.
SMG also documents MCP tool discovery and execution with approval policies and audit logging. Chat history can use PostgreSQL, Oracle, Redis or memory. Decide whether these responsibilities belong in your gateway before enabling them.
Check worker registration and gateway readiness separately
Issue 3496 reports a dynamic HTTP worker-registration failure against v1.12.0. According to the reporter, registration probes /health before configurable health settings take effect. A backend exposing /healthz while returning 404 at /health remained pending. This is an attributed user report, not a failure reproduced in our sandbox. It does not establish whether commit bc9ea78 or v1.13.0 has the same behavior.
The v1.13.0 release notes, dated October 15, 2026, describe a separate gateway-readiness concern. Helm's readiness probe moved to /readiness. For a router whose workers register only through the admin API, the notes instruct operators to set router.readinessProbe.path to /health. That setting controls gateway readiness; it is not evidence that the worker-registration report has been fixed.
The same release notes describe startup failures for unreachable PostgreSQL or Redis stores, a mesh that cannot start and malformed settings. Read those requirements against your deployment before upgrading. The measured checkout is identified by commit bc9ea78; this review does not equate it with the v1.13.0 release artifact. The project's release benchmarks are not measurements from our sandbox and do not establish performance for your fleet.
Choose by the operating problem
LiteLLM documents virtual keys and spend tracking. Consider it when provider access administration and spending controls drive the gateway decision. Envoy describes itself as an edge, middle and service proxy. Consider it for general service traffic when model-worker policy can remain elsewhere. These are evaluation suggestions based on the projects' descriptions. Neither alternative was tested or benchmarked in this run.
For SMG, begin with one existing worker and the registration path your deployment will use. Complete source-build and correctness validation, or validate the distributed artifact you intend to deploy. Before replacing a routing layer, check streamed requests, cancellation and worker failure against that artifact. A successful route through your intended worker configuration is the next useful result.