litellm review
LiteLLM translates a common OpenAI-shaped request into calls for many model providers. The Python SDK exposes completion, acompletion, embeddings, image and audio helpers, error mapping, costs, and a Router for retries, fallbacks, and deployment pools. The separately deployed proxy adds an OpenAI-compatible gateway, virtual keys, budgets, logs, guardrails, caching, and an admin interface. Version 1.97 adds Cursor model-name handling, endpoint and auto-router controls, more guardrail and spending features, and several provider adapters. It also signs release images, redacts credential headers in logging copies, corrects cost attribution, and fixes client cleanup and routing cases.
LiteLLM earns its weight when multi-provider routing or a shared gateway solves a concrete platform problem. For one provider, the 158 MB measured environment, slow import, translation edge cases, and upgrade pace are hard to justify over the official SDK.
We installed it
| Install | ✓ · 3.2s | 49 packages on disk · 158 MB |
| Import | ✓ | import litellm in 7.23s · compiled extensions · py.typed · requires Python >=3.10, <3.15 |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does litellm install cleanly?
Yes. In a fresh container with an empty cache, pip install litellm finished in 3 seconds, leaving 49 packages and 158 MB on disk. pip-audit reported no known vulnerabilities.
What does litellm need to run?
Python >=3.10, <3.15, and a platform wheel with compiled extensions. In our run import litellm succeeded in 7.23s, and the package ships py.typed for type checkers.
litellm or openai: which should you use?
openai: Choose it for OpenAI or an OpenAI-compatible endpoint when first-party types and features are enough. LiteLLM earns its weight when multi-provider routing or a shared gateway solves a concrete platform problem.
When should you not use litellm?
The application calls one provider and depends on its newest beta endpoints or exact response types; the first-party SDK will expose them sooner and with less translation
Discussed on
- hnTell HN: Litellm 1.82.7 and 1.82.8 on PyPI are compromised938 points
- hnMalicious litellm_init.pth in litellm 1.82.8 PyPI package – credential stealer739 points
- hnMy minute-by-minute response to the LiteLLM malware attack441 points
- hnMercor says it was hit by cyberattack tied to compromise LiteLLM151 points
- hnShow HN: liteLLM Proxy Server: 50+ LLM Models, Error Handling, Caching140 points
Use it if
- An application must switch or fall back between several LLM providers while keeping one message and error shape
- Multiple deployments of a model need load balancing, cooldowns, retry limits, and provider-aware routing
- A platform team needs a self-hosted gateway with virtual keys, team budgets, usage records, and policy hooks
- Existing OpenAI clients should point at a central service that selects credentials and upstream models
- The application calls one provider and depends on its newest beta endpoints or exact response types; the first-party SDK will expose them sooner and with less translation
- Install weight and startup matter; our base SDK install used 158 MB and importing litellm took 7.23 seconds before any model request
- Provider-specific parameters must fail loudly when unavailable; portability switches can drop unsupported inputs and hide a capability mismatch
- You expect an agent runtime, prompt graph, retrieval framework, or conversation memory; LiteLLM handles calls and gateway policy rather than application orchestration
- The team cannot track a fast release stream and a large issue queue; provider and proxy behavior changes frequently, so pinned upgrades need integration tests
Setup reality
Our fresh Python 3.12 install of LiteLLM 1.97.0 succeeded in 3.2 seconds. It installed 49 packages consuming 158 MB, while pip-audit found no known vulnerabilities. Metadata lists 87 direct dependency entries and supports Python 3.10 through below 3.15. The package includes compiled extensions and py.typed; its measured license field was unknown. Importing litellm worked but took 7.23 seconds in the clean container, which is significant for short jobs and cold-start functions.
SDK setup starts with provider credentials such as OPENAI_API_KEY, ANTHROPIC_API_KEY, or cloud-specific identity, then a provider-prefixed model name. Keep secrets in the environment or a secret manager and pass only provider options that you have tested. One normalized response does not make provider semantics identical: tool choice, JSON modes, caching, safety filters, token accounting, streaming events, and error details vary. Leave drop_params disabled in tests so an unsupported argument becomes visible.
The proxy is an application, not a thin import. A production gateway needs a reviewed config, its own master-key and virtual-key policy, persistent storage for budgets and spend, optional Redis for shared state, TLS, authentication, log redaction, database migrations, backups, health checks, and capacity planning. Version 1.97 release images are signed with cosign and the notes recommend verifying against the immutable commit-pinned public key. The release documentation also updates a Helm resource example to 4 GiB, which is a warning against tiny defaults.
Retries and fallbacks can duplicate billable or state-changing requests, so restrict automatic replay based on provider behavior and application idempotency. Cost calculation comes from a model price map that can lag or reload; use provider invoices for financial truth. Streaming failures may surface after partial output. Router and proxy caches hold network clients and connections, while 1.97 includes fixes around eviction and self-healing. Pin an exact version, canary it with real providers, and test auth, tools, streams, budgets, and failure routes before rollout.
Patterns
Send a provider-prefixed chat request call-chat-completion
from litellm import completion
response = completion(
model='anthropic/claude-sonnet-4-20250514',
messages=[{'role': 'user', 'content': 'Explain this traceback.'}],
timeout=30,
)
print(response.choices[0].message.content)Set the provider's credential outside code. Confirm the model identifier and supported parameters in the provider page.
Consume incremental content stream-chat-output
stream = completion(
model='openai/gpt-5.6-luna',
messages=messages,
stream=True,
)
for chunk in stream:
text = chunk.choices[0].delta.content
if text is not None:
print(text, end='', flush=True)Some chunks contain roles, tool data, or finish metadata instead of text. Handle a provider error after partial output.
Await a completion in async code call-async-model
from litellm import acompletion
response = await acompletion(
model='gemini/gemini-2.5-flash',
messages=messages,
timeout=20,
)
text = response.choices[0].message.contentUse acompletion inside an event loop. Bound application concurrency separately to avoid provider rate-limit spikes.
Classify provider failures handle-normalized-errors
import litellm
try:
response = litellm.completion(model=model, messages=messages)
except litellm.AuthenticationError:
raise RuntimeError('provider credential rejected')
except litellm.RateLimitError as error:
schedule_retry(error)
except litellm.APIConnectionError:
mark_provider_unavailable()Mapped classes simplify policy, but preserve original metadata in logs after removing credentials and prompt content.
Balance one logical model across deployments route-deployment-pool
from litellm import Router
router = Router(model_list=[
{
'model_name': 'support-model',
'litellm_params': {'model': 'openai/gpt-5.6-luna'},
},
{
'model_name': 'support-model',
'litellm_params': {'model': 'azure/support-gpt'},
},
])
response = router.completion(model='support-model', messages=messages)Each deployment needs valid credentials and compatible output behavior. A shared name does not prove feature parity.
Declare a bounded fallback path configure-model-fallback
router = Router(
model_list=model_list,
fallbacks=[{'primary': ['backup']}],
num_retries=1,
timeout=25,
)
response = router.completion(model='primary', messages=messages)Fallbacks can add latency and cost. Test tool calls, structured output, safety settings, and token limits on every target.
Create embeddings through one response shape request-embeddings
from litellm import embedding
response = embedding(
model='openai/text-embedding-3-small',
input=['first document', 'second document'],
)
vectors = [row['embedding'] for row in response.data]Embedding dimensions and normalization differ by model. Store the exact model and dimension beside a vector index.
Estimate one response cost calculate-estimated-cost
from litellm import completion_cost
estimated_usd = completion_cost(completion_response=response)
print(f'{estimated_usd:.6f}')The calculation uses LiteLLM pricing data. Reconcile provider billing for budgets or customer charges.
Run a local gateway trial start-local-proxy
# Install the documented proxy extra first.
# pip install 'litellm[proxy]'
litellm --model openai/gpt-5.6-luna --port 4000This is a local trial. Production virtual keys, spend records, shared routing, and the dashboard need persistent infrastructure and auth policy.
Point an OpenAI client at the proxy call-proxy-with-openai-client
from openai import OpenAI
client = OpenAI(
api_key='LITELLM_VIRTUAL_KEY',
base_url='https://gateway.example.com/v1',
)
response = client.chat.completions.create(
model='support-model',
messages=messages,
)Use a scoped virtual key rather than the proxy master key. TLS and server-side authorization remain deployment responsibilities.
Verify the signed proxy image verify-release-image
cosign verify \
--key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
ghcr.io/berriai/litellm:v1.97.0The 1.97 release notes recommend the commit-pinned key URL because the commit is immutable. Also pin the image digest in deployment.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| openai | PyPI | Choose it for OpenAI or an OpenAI-compatible endpoint when first-party types and features are enough |
| instructor | PyPI | Choose it when typed structured extraction is the main need and provider routing is secondary |
| portkey-ai | PyPI | Choose it when a managed AI gateway and its observability controls fit better than operating LiteLLM proxy |
More ai / ml guides
openai · mcp · huggingface-hub · @modelcontextprotocol/sdk · scikit-learn · tiktoken · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.