An AI policy layer for Envoy shops
Envoy AI Gateway is easiest to understand as an extension of an existing network stack. Envoy Proxy still moves requests, Envoy Gateway still turns Kubernetes Gateway API resources into proxy configuration, and this project adds the model-specific pieces. Its controller manages AI resources. An external processor reads and changes model requests and responses. Optional rate-limit services charge against token usage rather than treating every call as equal.
The main README describes two gateway tiers. A central tier handles authentication, broad routing, and global limits. A second tier sits at the entrance to a self-hosted inference cluster and can make finer routing decisions. Teams can use only the pieces they need, but the design clearly targets a platform shared by many applications, models, or tenants.
Provider coverage is wide. The documented list includes OpenAI, Azure OpenAI, Gemini, Vertex AI, Bedrock, Anthropic, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok, and Tetrate's service. Common client formats can stay stable while the gateway maps requests, credentials, model names, and responses to different backends. That is useful when an application should fail over without carrying provider code in every service.
What happened when we ran it
We cloned commit c217da8 into an unprivileged Debian container with 3 CPUs, 8 GB of RAM, Go 1.24, and no secrets. The checkout was 35.2 MB, containing 1,558 files and about 221,489 lines of source. It had seven CI workflow files, a Dockerfile, and a tests directory.
Installation succeeded in 96 seconds and installed 541 packages. The build then succeeded in 273 seconds. That gives the project credit for a working dependency and compile path in a clean container.
The test step told a different story. It ran for 668 seconds and exited with code 1. The Go test summary counted 12 packages: 6 passed and 6 failed. The supplied tail shows several internal helper packages with no test files, followed by successful results for internal end-to-end helpers and test servers, then a final FAIL. It does not show the failed package names or assertion messages. We cannot say whether the failures came from missing services, the sandbox, or code defects. The honest result is simply that the checked-out suite did not pass.
Policy that understands model traffic
A normal reverse proxy can route by host and path, but model traffic carries costs and failure modes inside the body and streaming response. Envoy AI Gateway can route by requested model, rewrite provider formats, attach upstream credentials, and apply fallback. Version 1.1.0 added a stream idle timeout that can move a request to another backend if no first token has arrived. Once a stream has begun, the documented behavior is a 504 rather than pretending another backend can continue it.
Token-aware controls are a larger reason to choose this project. The gateway records input, output, and total token metadata. Operators can connect that cost to rate-limit rules and tenant identity. OpenTelemetry metrics, traces, and access logs expose model names and token usage without asking every application team to implement the same instrumentation. The latest release also added token-counting routes across several provider shapes and optional GenAI tracing conventions.
Those controls need careful verification. Open issue 2551 reports that a distinct per-tenant quota rule can charge real token cost to a shared bucket while each tenant's own bucket only receives the request count. The report concerns v1.0.0 and remains open. Anyone using gateway counters for billing or hard tenant budgets should inspect the current behavior with separate test identities and Redis counters before relying on it.
MCP is more than a pass-through route
The MCP gateway can combine several remote MCP servers behind one streamable HTTP endpoint. Tool names receive backend prefixes, and the gateway merges tool lists and streaming notifications. Policies can filter tools by exact name or pattern, require OAuth scopes or JWT claims, evaluate CEL expressions, and inject or forward credentials to selected backends. Version 1.1.0 added hostname scoping and backend selection when a session initializes.
This gives a platform team one place to decide which agent can call which tool. It also creates protocol-sensitive failure cases. Open issue 2568 says a Java servlet backend that returns application/json;charset=UTF-8 can initialize successfully yet disappear from the merged tool list because one path compares the entire content-type string. The report says no client or external-processor error is logged. Test each SDK family directly and through aggregation.
Kubernetes is the price of admission
The documentation is candid about prerequisites. You need a Kubernetes cluster, kubectl, Helm, a compatible Envoy Gateway deployment, the AI Gateway CRDs, and the controller. The guide recommends a fresh Envoy Gateway install to avoid configuration conflicts. Its basic example uses a mock model, then real provider guides add secrets and backend resources. Extra traffic features bring extra Helm values and services.
This is reasonable for an established cluster team and excessive for one application calling two providers. LiteLLM or Portkey's gateway will usually get that smaller team to a common API faster. Envoy AI Gateway pays off when Gateway API resources, Envoy behavior, centralized identity, and cluster operations already belong to the platform.
Project activity supports continued evaluation. GitHub recorded a push on August 26, 2026, five days after the v1.1.0 release. The API's combined issues-and-pull-requests count was 266. New fixes and maintainer work were active on August 27. The project has reached a stable 1.x API, but its operational surface and our failed tests still call for a pinned, staged rollout rather than a fleet-wide switch.