PyPIAI / MLupdated 05 Aug 2026

litellm

Item: litellm
Rating: 4.3
Author: Keyoor

An open source AI gateway that gives you one OpenAI-format interface to 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and more). You use it two ways: as a Python SDK where litellm.completion() replaces per-provider SDKs, or as a self-hosted proxy server that your whole org points OpenAI clients at, adding virtual keys, spend tracking, load balancing, guardrails, and an admin dashboard. Provider errors are mapped to OpenAI-style exceptions so retry and fallback logic is written once.

litellm · v1.95.0 · 166.5M/wk · ★ 55,571 · pushed 05 Aug 2026 · BerriAI/litellm · docs · reviewed by Keyoor

Verdict

The pragmatic choice the moment you talk to more than one provider or need a gateway with keys, budgets, and fallbacks; adoption and maintenance pace back that up. For a single-provider app it is more machinery than the problem needs.

API stability3/5completion() has kept its OpenAI-shaped contract across years of 1.x releases, but the surface area is enormous and near-daily releases mean behavior in less-traveled providers and proxy features shifts more often than the core.

Docs4/5docs.litellm.ai is broad, with per-provider pages, a proxy end-to-end tutorial, and published benchmarks; depth is uneven across the 100+ providers and some pages lag the code given the release pace.

Maintenance5/5Pushed to within the hour of this review, 55k+ stars, backed by a YC W23 company with an enterprise tier funding full-time development.

Ecosystem5/5The README lists Netflix, Stripe, and OpenHands among OSS adopters, and it plugs into A2A agents, MCP tools, and the major agent SDKs; it is the default multi-provider layer in much of the Python LLM stack.

Use it if

You call two or more LLM providers and want one request/response format instead of juggling per-provider SDKs and auth patterns
You need fallbacks, retries, and load balancing across model deployments, which the Router handles for you
A platform team wants a central gateway with virtual keys, per-team budgets, and spend tracking rather than API keys scattered across services
You want to switch models via a config or model string without touching application code

Skip it if

You only call one provider; the official SDK is thinner, better typed, and gets new features (new endpoints, betas) before the translation layer does
You want a small dependency footprint; litellm is a large package that moves extremely fast, with releases landing near-daily, so unpinned installs drift and pinned ones go stale in weeks
You rely on provider-specific capabilities at the edges; the unified interface papers over real differences, and non-portable params either pass through untyped or get dropped
You need an agent framework; litellm is a call layer and gateway, not chains, memory, or tool orchestration

Setup reality

pip install litellm, export your provider API keys as env vars, and completion(model="provider/model") works in minutes. The proxy is where setup gets real: pip install 'litellm[proxy]' pulls a much bigger dependency tree, the quick start (litellm --model gpt-4o) is one command, but a production gateway means writing a config.yaml, standing up Postgres for virtual keys and spend tracking, and running it in Docker. Release velocity is the recurring annoyance: pin your version, read changelogs before bumping, and expect the occasional regression in less-used providers.

Patterns

Call any provider in OpenAI formatbasic-completion

import os
from litellm import completion

os.environ["ANTHROPIC_API_KEY"] = "sk-..."

response = completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

The model string is "provider/model"; auth comes from the provider's standard env var, so nothing else changes when you swap providers.

Stream tokens as they arrivestreaming

from litellm import completion

stream = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku"}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="")

delta.content is None on some chunks (role headers, finish chunks), so guard before concatenating.

Async calls with acompletionasync-completion

import asyncio
from litellm import acompletion

async def main():
    response = await acompletion(
        model="gemini/gemini-2.5-flash",
        messages=[{"role": "user", "content": "Hi"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())

acompletion mirrors completion exactly; combine with asyncio.gather for concurrent calls instead of threads.

Retries and timeouts on a callretries-timeout

from litellm import completion

response = completion(
    model="openai/gpt-4o",
    messages=msgs,
    timeout=30,
    num_retries=2,
)

Because provider errors are mapped to OpenAI-style exception types, retry behavior is consistent no matter which provider actually failed.

Fall back to another providerfallback-models

from litellm import completion

response = completion(
    model="openai/gpt-4o",
    messages=msgs,
    fallbacks=["anthropic/claude-sonnet-4-20250514", "gemini/gemini-2.5-flash"],
)

Fallbacks trigger on errors from the primary model; each fallback model needs its own provider key set in the environment.

Load balance across deploymentsload-balancing-router

from litellm import Router

router = Router(model_list=[
    {"model_name": "gpt-4o", "litellm_params": {"model": "azure/gpt-4o-eu", "api_key": key1, "api_base": base1}},
    {"model_name": "gpt-4o", "litellm_params": {"model": "openai/gpt-4o"}},
])

response = router.completion(model="gpt-4o", messages=msgs)

Deployments sharing a model_name form a pool; the Router spreads load and cools down deployments that error, which is how you survive per-key rate limits.

Run the proxy (AI gateway)proxy-quickstart

pip install 'litellm[proxy]'
litellm --model gpt-4o
# then point any OpenAI client at it:
# client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")

The one-liner is for trying it out; virtual keys, budgets, and spend tracking require a config.yaml plus a Postgres database.

Compute the cost of a responsecost-tracking

from litellm import completion, completion_cost

response = completion(model="openai/gpt-4o", messages=msgs)
usd = completion_cost(completion_response=response)
print(f"${usd:.6f}")

Pricing comes from a bundled model-price map; brand-new models may be missing until you upgrade litellm.

Embeddings through the same interfaceembeddings

from litellm import embedding

result = embedding(
    model="text-embedding-3-small",
    input=["good morning", "good night"],
)
vectors = [d["embedding"] for d in result.data]

Same provider-prefix convention as completion; response shape follows the OpenAI embeddings format regardless of provider.

Catch provider errors uniformlyerror-handling

import litellm
from litellm import completion

try:
    response = completion(model="anthropic/claude-sonnet-4-20250514", messages=msgs)
except litellm.RateLimitError:
    ...  # back off
except litellm.AuthenticationError:
    ...  # bad or missing key
except litellm.APIConnectionError:
    ...  # network / provider down

Every provider's errors are mapped onto OpenAI-style exception classes, so one except block covers all backends.

Drop params a provider does not supportdrop-unsupported-params

import litellm

litellm.drop_params = True

# now e.g. passing frequency_penalty to a provider
# that lacks it is dropped instead of raising

Convenient for portable code, but it fails silently by design; leave it off in tests so you notice which knobs a provider actually ignores.

Alternatives

Package	Registry	Pick it when
openai	PyPI	You only talk to OpenAI (or OpenAI-compatible endpoints) and want the first-party SDK
anthropic	PyPI	You only call Claude and want first-party types, streaming helpers, and day-one feature support
langchain	PyPI	You want a full framework (chains, agents, retrieval) rather than just a unified call layer