groq
groq is the official Python client for the Groq API, the hosted inference service that runs open-weight models (Llama, gpt-oss, Qwen, Whisper) on Groq's custom LPU hardware at unusually high tokens per second. The SDK is generated with Stainless, so it looks and behaves almost exactly like the OpenAI Python SDK: a Groq and an AsyncGroq client over httpx, typed request params via TypedDict, Pydantic response models, chat completions, streaming, tool use, and audio transcription. If you have used the openai package, you already know this one.
A clean, well-typed official client that stays out of your way; adopt it if Groq is your primary provider. If Groq is just one backend among several, the openai SDK with a custom base_url or litellm saves you the extra dependency.
Use it if
- You want very low latency inference on open-weight models without renting GPUs or running vLLM yourself
- You need fast, cheap Whisper transcription through client.audio.transcriptions
- You want a typed sync plus async client with retries, timeouts, and streaming handled for you
- Your app is latency sensitive (voice agents, autocomplete, real-time chat) where Groq's speed is the whole point
- You already use the openai SDK: Groq's API is OpenAI compatible, so pointing base_url at https://api.groq.com/openai/v1 covers most use cases without a second dependency
- You need frontier proprietary models (Claude, GPT-5, Gemini): Groq only serves its own catalog of open-weight models
- You call several providers: litellm or a gateway gives you one interface instead of one vendor SDK per provider
- You depend on a specific model long term: Groq rotates and deprecates catalog models on fairly short notice, so pinned model IDs need monitoring
Setup reality
pip install groq is light: httpx, pydantic, typing-extensions, Python 3.10+. You need a GROQ_API_KEY from console.groq.com. The real friction is rate limits: the free tier has tight per-model requests-per-minute and tokens-per-minute caps, so loops hit 429s quickly; the SDK retries twice by default with backoff, which can mask the problem until it doesn't. Because the code is Stainless-generated, minor releases sometimes shuffle type names, and the optional aiohttp backend is a separate extra (pip install groq[aiohttp]).
Patterns
Basic chat completionchat-completion
import os
from groq import Groq
client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
chat_completion = client.chat.completions.create(
messages=[{"role": "user", "content": "Explain the importance of low latency LLMs"}],
model="openai/gpt-oss-20b",
)
print(chat_completion.choices[0].message.content)api_key defaults to the GROQ_API_KEY env var, so the argument is optional. Model IDs come from the catalog at console.groq.com/docs/models and do change over time.
Async usage with AsyncGroqasync-client
import asyncio
from groq import AsyncGroq
async def main() -> None:
client = AsyncGroq()
chat_completion = await client.chat.completions.create(
messages=[{"role": "user", "content": "Say hello"}],
model="openai/gpt-oss-20b",
)
print(chat_completion.choices[0].message.content)
asyncio.run(main())The async client mirrors the sync API exactly. For high concurrency you can swap the HTTP backend to aiohttp via http_client=DefaultAioHttpClient().
Stream tokens as they arrivestreaming
from groq import Groq
client = Groq()
stream = client.chat.completions.create(
messages=[{"role": "user", "content": "Write a haiku about speed"}],
model="openai/gpt-oss-20b",
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")delta.content is None on some chunks (role headers, final chunk), hence the or-empty-string guard.
Function calling with toolstool-use
import json
from groq import Groq
client = Groq()
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
resp = client.chat.completions.create(
messages=[{"role": "user", "content": "Weather in Mumbai?"}],
model="llama-3.3-70b-versatile",
tools=tools,
tool_choice="auto",
)
call = resp.choices[0].message.tool_calls[0]
print(call.function.name, json.loads(call.function.arguments))tool_calls is None when the model answers directly, so check before indexing. Arguments arrive as a JSON string, not a dict.
Force JSON outputjson-mode
import json
from groq import Groq
client = Groq()
resp = client.chat.completions.create(
messages=[
{"role": "system", "content": "Reply in JSON with keys name and year."},
{"role": "user", "content": "When was Python released and by whom?"},
],
model="llama-3.3-70b-versatile",
response_format={"type": "json_object"},
)
print(json.loads(resp.choices[0].message.content))JSON mode requires the word JSON to appear in your messages and cannot be combined with streaming on all models. It guarantees syntax, not your schema.
Transcribe audio with Whisperaudio-transcription
from groq import Groq
client = Groq()
with open("meeting.mp3", "rb") as f:
transcription = client.audio.transcriptions.create(
file=("meeting.mp3", f.read()),
model="whisper-large-v3-turbo",
response_format="verbose_json",
)
print(transcription.text)File params accept bytes, a Path, or a (filename, contents, media type) tuple. verbose_json adds segment timestamps.
Handle API errors by typeerror-handling
import groq
from groq import Groq
client = Groq()
try:
client.chat.completions.create(
messages=[{"role": "user", "content": "hi"}],
model="openai/gpt-oss-20b",
)
except groq.RateLimitError:
print("429: back off before retrying")
except groq.APIStatusError as e:
print(e.status_code, e.response)
except groq.APIConnectionError as e:
print("network problem:", e.__cause__)All errors inherit from groq.APIError. RateLimitError is the one you will actually see on the free tier.
Configure retries and timeoutsretries-and-timeouts
import httpx
from groq import Groq
client = Groq(max_retries=5, timeout=20.0)
# or per request:
resp = client.with_options(max_retries=0, timeout=httpx.Timeout(60.0, connect=5.0)).chat.completions.create(
messages=[{"role": "user", "content": "hi"}],
model="openai/gpt-oss-20b",
)Default is 2 retries with exponential backoff on 408/409/429/5xx. Set max_retries=0 in tests so failures surface immediately.
Send an image to a multimodal modelvision-image-input
import base64
from groq import Groq
client = Groq()
with open("chart.png", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
resp = client.chat.completions.create(
model="meta-llama/llama-4-scout-17b-16e-instruct",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What does this chart show?"},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
],
}],
)
print(resp.choices[0].message.content)Only some catalog models accept images; check the model page. Content becomes a list of parts instead of a plain string.
Inspect raw HTTP response and headersraw-response-access
from groq import Groq
client = Groq()
response = client.chat.completions.with_raw_response.create(
messages=[{"role": "user", "content": "hi"}],
model="openai/gpt-oss-20b",
)
print(response.headers.get("x-request-id"))
completion = response.parse()
print(completion.choices[0].message.content)Useful for reading rate limit headers before you hit 429. parse() returns the normal typed object.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| openai | PyPI | You want one SDK for OpenAI and Groq both; set base_url to Groq's OpenAI-compatible endpoint. |
| litellm | PyPI | You call many providers and want unified routing, fallbacks, and cost tracking above them. |
| together | PyPI | You want another fast open-model host with a wider catalog, including image and embedding models. |