Short answer: compare complete requests before adopting cache affinity
llm-d Router is worth evaluating for operators placing requests across a pool of inference replicas; teams serving one instance should use their serving engine alone. Repeated context gives cache-aware routing something to exploit, but the adoption decision should turn on client latency, aggregate throughput and running requests per replica. Compare the proposed policy with the existing load-balancing policy using the same requests and serving capacity.
Historical reports show why that comparison matters. In issue #355, approximate prefix routing concentrated a single-prefix workload onto two of eight replicas and lost throughput. In issue #1289, precise multimodal routing obtained the expected cache hits while overall latency and throughput worsened. These reports identify two useful acceptance checks: whether cache affinity leaves capacity idle, and whether request preprocessing consumes the time saved inside the model server.
Endpoint picking adds cache state to replica selection
llm-d Router comprises a proxy and an Endpoint Picker that selects model-server pods using metrics, cache affinity and configured policies. The llm-d architecture reference describes that division. The proxy accepts requests and consults the picker through ext-proc. Operators with an existing gateway can use this distinction to identify which component supplies the placement decision and which component carries the traffic.
A repeated prompt illustrates the scheduling choice: one replica may hold useful cached context while another has more available capacity. Cache and load scoring influence which endpoint wins. The llm-d Router README also documents request priority and model-name rewriting. Selecting a serving replica and choosing a different language model according to a question's complexity are different decisions, so an evaluation should define which behavior it needs.
Standalone Mode uses a self-managed Envoy proxy, either in the EPP Pod or as a separate service. Gateway Mode connects the EPP through an InferencePool referenced by an HTTPRoute. The README recommends Gateway Mode for production environments and describes the standalone sidecar topology as suitable for basic testing and local evaluation. For manually configured Envoy, it specifies FULL_DUPLEX_STREAMED for both request and response body modes. Establish that request path before tuning scheduling weights.
What people use llm-d-router for
Snap, ZenteiQ and Google's demonstration team describe llm-d uses involving service-mesh integration, synthetic-data generation and shared agent traffic. Their accounts appear in the Google Cloud llm-d 1.0 announcement. They give operators concrete workloads to compare with their own: integration into an existing proxy environment, sustained generation where aggregate token capacity matters, and agent traffic spanning different inference engines and accelerator types.
- Snap reports integrating llm-d into production AI infrastructure and its Envoy-based service mesh, with prefix-cache hit rates reaching 75-80%. Snap's statement in Google's announcement
- ZenteiQ reports using GKE Inference Gateway powered by llm-d for synthetic scientific training data, with over 4× higher aggregate inference throughput and daily capacity increasing from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. ZenteiQ's statement
- Google reports demonstrating 1,000 sandboxed agents reaching one llm-d endpoint across vLLM, SGLang, TPUs and GPUs. Google's agent demonstration
What happened when we ran it
Our sandbox built llm-d-router in 239 seconds, then its test command failed with exit 1. The separate November 13, 2026 CLI recipe recorded checkout commit 3d0b21d51d29 in an unprivileged golang:1.24-bookworm container with 2 CPUs, 4 GB RAM, no published ports and no secrets. Its installation command used @latest. The CLI run record identifies the environment and recipe as quickstart-cli.
| CLI step | Recorded outcome |
|---|---|
| Clone | The quickstart-cli recipe cloned the repository in 2 seconds. |
| Install | go install github.com/llm-d/llm-d-router/...@latest recorded 293.0 MB installed in 252.5 seconds; the package path came from the manifest. |
| Version | coordinator --version printed usage and exited 2 in 0.1 seconds. |
| Help | coordinator --help exited 0 in 0.1 seconds. |
| Usage | The step failed because the recipe found no runnable coordinator command in the README's usage/examples sections. |
| Teardown | The recorded teardown completed successfully. |
Sources: clone record, installation record, version-step failure, version-failed finding, help output, usage failure and teardown record.
The CLI recipe reached command-line inspection: coordinator --help succeeded, while coordinator --version printed usage and returned exit 2. The subsequent usage step stopped because the recipe found no runnable coordinator example in the README's usage/examples sections. The installation path came from the manifest. The README instead documents deployment through Envoy and serving infrastructure, so command installation and router deployment are separate steps.
Our earlier source sandbox used commit 2539e00 on October 6, 2026, with 3 CPUs, 8 GB RAM and image label golang:1.24-bookworm. We measured successful installation and build command outcomes, followed by a failed test command. Installation took 47 seconds with 383 dependencies recorded; building took 239 seconds; testing took 263 seconds. The harness recorded exit codes 0, 0 and 1 respectively, with no step marked timed out.
| Source step | Outcome | Duration |
|---|---|---|
| Install | Succeeded; 383 dependencies recorded | 47 seconds |
| Build | Succeeded | 239 seconds |
| Tests | Failed, exit 1; recorded package-summary counts: 210 passed, 11 failed, 221 total | 263 seconds |
The supplied failing-log tail names TestStandaloneChart, github.com/llm-d/llm-d-router/test/integration/epp and the sidecar TestE2E. TestStandaloneChart shows 0.00 seconds, and TestE2E shows 20.00 seconds. The tail also records passing framework, framework/net, integration and tokenizerbench packages. The previously published account additionally names TestCoordinatorE2E and TestEndToEnd. Those two names come from that account; the supplied raw measurement tail shows the other named failures directly.
Where people get stuck: cache affinity can leave replicas idle
Issue #355 reports approximate prefix routing using only two of eight replicas for a single-prefix workload. In the historical scheduler discussion, mnmehta describes Qwen/Qwen3-0.6B on H200 replicas using the llm-d 0.2 path. The approximate configuration lost throughput with one or two shared prefixes, then improved throughput from four through 512 prefixes in the reported table. Prefix diversity therefore belongs in the workload definition.
| Unique prefixes | Default EPP, requests/s | Approximate prefix EPP, requests/s | Reported change |
|---|---|---|---|
| 1 | 351.9 | 127.11 | -63.88% |
| 2 | 278.35 | 220.94 | -20.63% |
| 4 | 265.09 | 287.49 | +8.45% |
The maintainer explains that low cache utilization and waiting queues left the algorithm favoring reuse while requests concentrated on fewer replicas. The suggested adjustment was active-request scoring. Compare running requests per replica alongside waiting queues and cache utilization: in the reported workload, the waiting queue stayed low even while the two selected replicas carried more running queries than replicas in the more evenly distributed configuration.
In multimodal evaluation #1289, capri-xiyue reports expected cache hits alongside worse overall latency and throughput. At 35 requests/s with four 720p images per request, the record says the vLLM-backed token producer could introduce more than 30 seconds of latency. The reported errors include DataProducer execution timed out: context deadline exceeded and PrefixCacheMatchInfo not found for endpoint, assigning score 0. Include preprocessing in client-observed latency.
| Symptom | What the report says | Practical response |
|---|---|---|
| Shared prefixes concentrate traffic | #355 reports two replicas used out of eight and recommends active-request scoring | Compare running requests per replica with the load-based baseline |
| Image routing gets cache hits but slows requests | #1289 reports render/tokenization overhead consuming the benefit | Include preprocessing in client-observed latency |
| A second model breaks gateway traffic | #158 records a historical multi-route problem and an Istio workaround | Compare route matching and pool attachment |
In multi-model issue #158, users report failures with two active HTTPRoutes and traffic returning after removing one route. jgchn reports responses for both models after switching to Istio on Kind. Preserve the working single-model route, add the second route deliberately, then inspect route matching and the gateway-to-pool attachment before changing scheduling policies elsewhere in the request path.
At volume, fairness and adapter placement need explicit policies
Enabling flowControl retains strict first-come, first-served defaults in the development flow-control guide. The flow-control documentation separates global fairness, arrival ordering and saturation detection. Its tenant-equity configuration changes the defaults to prioritize fairness between tenants. The guide also recommends a concurrency detector to avoid telemetry lag risks. Select those policies against the service behavior intended for shared capacity rather than assuming that enabling queuing chooses tenant isolation.
The standalone recipe requests 4 vCPU and 8 GiB each for EPP and Envoy, totaling 8 vCPU and 16 GiB per Pod. The guide connects a Pending Pod with Insufficient cpu to the node's allocatable capacity for the full request. When applying that recipe, count both containers in the scheduling requirement. The combined allocation exceeds the resources assigned to either of our command-level sandbox containers.
LoRA placement issue #709 asks for bounded adapter placement with expansion under load while respecting adapter capacity. It distinguishes preferring an already-loaded adapter from deciding where adapters should reside. Its reported problems include hot replicas, eviction churn and starvation when active adapters exceed slots. Include adapter residency, reloads and per-adapter waiting time alongside aggregate throughput when evaluating a workload with many adapters.
Security defaults depend on the component and release
The archived routing-sidecar documentation describes optional SSRF protection with a host-based allowlist, while the v1.0.0 router changelog lists target-port enforcement. The archived sidecar README says its allowlist tracks hosts or IPs rather than specific ports. The router release changelog includes fix(sidecar): enforce target ports in the SSRF allowlist. Identify the deployed component and release before applying either description to a configuration.
The CLI help says an empty certificate path generates a self-signed certificate suitable only for testing. That behavior appears in the recorded coordinator help. A deployment review should record the image, certificate configuration and target-selection controls together. The help output identifies a concrete configuration choice, and the v1.0.0 changelog identifies a documented security change to inspect when comparing an inherited deployment with its intended release.
Names and versions: the endpoint-picker image is one component
llm-d-router-endpoint-picker names the published EPP container package; llm-d Router names the proxy-plus-picker system. The EPP package record includes v1.0.0 images. The README terminology defines the complete router and its picker separately. Selecting that image is one deployment choice; the proxy, serving endpoints and gateway resources determine the surrounding request path.
The README says core Endpoint Picker code and request-management APIs moved from Gateway API Inference Extension into this repository. GAIE retains InferencePool and defines the Endpoint Picker Protocol. The archived routing-sidecar repository points to the historical inference-scheduler name. For a migration record, identify the router component, image tag, chart, API group, gateway and inference engine. That gives an operator a clearer configuration record than one project name covering the entire deployment.
llm-d-router vs vLLM, plain Services and the GAIE reference picker
vLLM is a model-serving engine, while llm-d Router supplies request placement across serving replicas. The architecture reference separates model servers from routing, and the older architecture description explicitly names vLLM and SGLang as inference engines. Choose the operating scope first: a single serving instance, a coordinated replica pool, or a reference implementation for protocol work. Then compare the routing policies relevant to that scope.
| Option | Why choose it | Decision boundary |
|---|---|---|
| Plain Kubernetes Service or round-robin balancing | Use a simple distribution baseline | Compare complete requests under the actual prefix distribution |
| vLLM alone | Serve the model on one instance | Add coordination when placement across replicas becomes useful |
| GAIE reference EPP | Exercise protocol conformance | Its documentation directs production users toward another implementation |
| llm-d Standalone Mode | Use a self-managed Envoy deployment | Configure the proxy topology and resource requests |
| llm-d Gateway Mode | Integrate with a separately managed gateway | Connect Gateway, HTTPRoute, InferencePool and EPP |
GAIE describes its reference Endpoint Picker as a lightweight implementation for conformance testing. Its project introduction directs production users toward their own implementation or an existing one such as llm-d-router. That makes the reference picker useful for protocol work. For operating a serving pool, compare the deployment and scheduling behavior of the implementation selected to sit behind those APIs.
What the other reviews get wrong
The supplied v1.0 architecture marks Workload Variant Autoscaler deprecated, and the v1.0.0 release notes list endpoint pinning and topology-aware routing. The older v0.7 architecture points to v0.9 as latest and presents complementary autoscaling approaches. Use the supplied v1.0 architecture's deprecation notice when interpreting that older recommendation, and use the named router release when discussing pinning or topology features.
The older Red Hat explanation describes a kgateway-based gateway. The current router README supports multiple proxy choices, including self-managed and cloud-managed options. Its standalone path uses self-managed Envoy, while Gateway Mode connects the picker through Gateway API resources. Treat the older explanation as an account of its stated architecture; choose a gateway from the integration documentation for the components in the intended deployment.
Solo.io's session-token example says subsequent prompts reach the same decode worker. Session-affinity issue #815 distinguishes soft scoring from hard affinity. The v1.0.0 release notes subsequently list an endpoint-pin-screener that pins a request to one endpoint, plus topology-aware P/D routing in the coordinator. Identify the plugin and release when discussing affinity rather than treating every scorer as a pinning mechanism.
Questions people ask
Check route attachment for multi-model failures, running-request distribution for concentrated prefixes, and preprocessing for slow image requests. Issue #158 supplies the route symptom, #355 supplies the replica-concentration example, and #1289 supplies the render/tokenization overhead report. Those checks locate the part of the request path to investigate before changing a scheduling weight. The FAQ below turns each report into a specific configuration or measurement decision.
What is llm-d-router, and how does it differ from llm-d-router-endpoint-picker?
llm-d Router is the proxy-plus-picker system; llm-d-router-endpoint-picker is its EPP container package. Select the picker image together with the surrounding proxy topology and serving endpoints. For Gateway Mode, record the Gateway, HTTPRoute and InferencePool attachment as well. The README's terminology distinguishes the routing system from the component that supplies scoring and placement decisions.
Why does my EPP work with one model but fail when I deploy a second model?
Issue #158 reports failures with two active HTTPRoutes and responses for both models after switching to Istio on Kind. Start with the working single-model path, add the second route, and compare route matching and pool attachment with that report. Removing one route restored traffic for a participant, making the second attachment a useful debugging boundary.
How can I tune prefix-cache-aware routing when one or two shared prefixes reduce throughput?
The maintainer in #355 recommends active-request scoring after requests concentrated on two of eight replicas. Inspect running requests per replica alongside cache utilization and waiting queues. Include one- and two-prefix workloads in the comparison: those were the cases where the reported approximate configuration lost throughput, while the table showed gains beginning at four unique prefixes.
Does precise prefix-cache routing improve multimodal performance after render and tokenization overhead?
The evaluator in #1289 reports worse overall latency and throughput despite expected cache hits. Include render/tokenization time in client-observed latency rather than choosing a policy from server TTFT alone. The reported case used 35 requests/s and four 720p images per request, with token production introducing more than 30 seconds of latency.
How much CPU and memory does the EPP need?
The development flow-control recipe requests 4 vCPU and 8 GiB for EPP, plus the same allocation for Envoy. Size the node against the combined 8 vCPU and 16 GiB Pod request when using that recipe. The guide ties Pending Pods with insufficient CPU to the full request fitting within node allocatable capacity.
How can I guarantee session affinity instead of merely increasing an endpoint's score?
Issue #815 identifies the scorer as soft affinity; the v1.0.0 release notes list an endpoint-pin-screener that pins a request to one endpoint. Select the mechanism by the required behavior and deployed release. Include endpoint loss and saturation in acceptance criteria before promising session continuity to callers, since those conditions affect the usefulness of a placement rule.
What we have not tested yet
The supplied evidence leaves the revision of the @latest coordinator executable unknown. Checkout commit 3d0b21d51d29 identifies the recorded checkout, while the installation command selected commands separately. Neither run exercised inference through a deployed router, and command installation does not provide a standalone inference deployment. The golang:1.24-bookworm image labels record sandbox configuration rather than an inventory of the actual runtime and operating-system versions used by every step.
The recorded 210 passed, 11 failed and 221 total are package-summary outcomes, not an inventory of 221 individual tests. The incomplete failing-log tail lacks assertion details and setup output needed to diagnose the failed packages. No new execution, reconciliation of unique tests or corrected-code execution accompanies this revision. The supplied evidence also contains no independent current-release latency benchmark, EPP memory profile, failover result or security audit. The flow-control recipe's resource requests are allocations rather than measured consumption.
The multimodal record supplies no measured text-only performance comparison or demonstrated recovery after increasing render CPU. It reports a multimodal fix in PR #1504 and a separate text-input error filed as #1540. The multi-model thread supplies a workaround rather than a demonstrated current kgateway fix. The active-request recommendation in #355 has no successful retest in the supplied extract, and the LoRA discussion describes proposed placement work rather than completed GPU results.
The v1.0.0 release documents pinning, topology-aware routing and target-port enforcement, but our runs supply no implementation or failure-behavior assessment of those changes. The archived sidecar's host-only controls therefore do not describe current images. Historical topology issue #545 describes independent prefill/decode selection. Conditional-prefill issue #382 ends by asking whether the corresponding vLLM support merged, leaving that dependency unresolved in the supplied evidence.
This pack contains no screenshot records, answer-quality question set or video transcript evidence. Snap and ZenteiQ's results are attributed company statements, and Google's agent example is a vendor demonstration. Before changing a routing policy, replay the same requests through the existing and proposed configurations, including low-diversity prefixes and actual image traffic. Record client latency, throughput and running requests per replica to judge complete-request outcomes.


