langsmith
langsmith is the Python client for LangSmith, LangChain's hosted platform for tracing, evaluating, and monitoring LLM applications. You decorate functions with @traceable or wrap your OpenAI or Anthropic client, and every call ships its inputs, outputs, latency, and token usage to a web UI where you can inspect nested traces. The same client manages datasets of test cases and runs evaluate() to score your app against them. It works with any LLM code, not just LangChain, but the data lands in LangSmith's cloud.
The lowest-friction observability if you are already on LangChain, and the trace UI is genuinely good for debugging agents. It is a client for a paid closed platform, so if self-hosting or data residency matters, start with langfuse or arize-phoenix instead of migrating later.
Use it if
- You already use LangChain or LangGraph: set two environment variables and every chain and agent step is traced with zero code changes
- You are debugging multi-step agents and need to see the exact prompt, tool calls, and intermediate outputs of a failed run, not just the final answer
- You want regression testing for prompts: datasets of examples plus evaluate() with scoring functions turn 'the prompt feels worse' into numbers
- Your team needs shared visibility: traces, feedback, and eval results live in one web UI instead of scattered log files
- You are not comfortable sending prompts and completions to a third-party SaaS: LangSmith is a commercial closed-source platform, and self-hosting is an enterprise-plan feature, unlike langfuse or arize-phoenix which you can run yourself for free
- You want OpenTelemetry-native observability that plugs into your existing Datadog or Grafana stack; LangSmith is its own silo with its own UI
- You only need basic request logging and cost tracking for a single provider; a thin logging wrapper or the provider dashboard covers that without a new vendor
- You dislike pre-1.0 dependencies in production: the SDK sits at 0.10.x, releases are frequent, and langchain-core pins it with a broad range that can drift under you
Setup reality
pip install langsmith is light (httpx, pydantic, orjson, zstandard). The friction is account and environment plumbing: you need an API key from smith.langchain.com, LANGSMITH_TRACING=true, LANGSMITH_API_KEY, and, for org-scoped keys, LANGSMITH_WORKSPACE_ID or every call 403s confusingly. Tracing is fire-and-forget in a background thread, so short-lived scripts and serverless functions can exit before traces flush; you end up calling client.flush() or wrapping with tracing_context. The free tier caps traces per month, and old LANGCHAIN_* env var names still float around tutorials next to the current LANGSMITH_* ones.
Patterns
Turn on tracing via environment variablesenable-tracing
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=ls_...
export LANGSMITH_WORKSPACE_ID=<workspace-id> # only for org-scoped keysWith these set, LangChain and LangGraph code is traced automatically with no code changes; non-LangChain code still needs @traceable or a wrapper.
Trace any function with @traceabletrace-function
from langsmith import traceable
@traceable
def retrieve(query: str) -> list[str]:
return search_index(query)
@traceable(run_type="chain", name="rag_pipeline")
def rag(question: str) -> str:
docs = retrieve(question)
return generate_answer(question, docs)Nested @traceable calls appear as child runs in one trace tree; run_type controls how the UI renders the step (chain, llm, tool, retriever).
Auto-trace an OpenAI clientwrap-openai
import openai
from langsmith.wrappers import wrap_openai
client = wrap_openai(openai.OpenAI())
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}],
)The wrapper logs prompts, completions, and token usage for every call; wrap_anthropic does the same for the Anthropic SDK.
Trace a block of code without a decoratortrace-context-manager
from langsmith import trace
with trace(name="summarize", run_type="chain", inputs={"doc_id": doc_id}) as run:
summary = do_summarize(doc)
run.end(outputs={"summary": summary})Useful when the unit of work is not a single function; anything traced inside the with block becomes a child run.
Create a dataset of test examplescreate-dataset
from langsmith import Client
client = Client()
ds = client.create_dataset("qa-regression")
client.create_examples(
dataset_id=ds.id,
examples=[
{"inputs": {"question": "What is LCEL?"}, "outputs": {"answer": "LangChain Expression Language"}},
{"inputs": {"question": "What is RAG?"}, "outputs": {"answer": "Retrieval augmented generation"}},
],
)create_dataset raises if the name already exists; use client.has_dataset / read_dataset to make setup scripts rerunnable.
Evaluate your app against a datasetrun-evaluation
from langsmith import Client
client = Client()
def correct(outputs: dict, reference_outputs: dict) -> bool:
return outputs["answer"].strip() == reference_outputs["answer"].strip()
results = client.evaluate(
lambda inputs: {"answer": my_app(inputs["question"])},
data="qa-regression",
evaluators=[correct],
experiment_prefix="prompt-v2",
)Each run becomes an experiment you can diff in the UI; evaluator functions receive inputs, outputs, and reference_outputs by parameter name.
Attach user feedback to a runlog-feedback
from langsmith import Client, get_current_run_tree, traceable
@traceable
def answer(q: str) -> str:
rt = get_current_run_tree()
app_state["last_run_id"] = rt.id
return llm_answer(q)
Client().create_feedback(run_id=app_state["last_run_id"], key="thumbs", score=1)Capture the run id at request time; searching for the run later by inputs is unreliable and slow.
Tag runs for filtering in the UIadd-metadata-tags
@traceable(tags=["prod"], metadata={"app_version": "1.4.2"})
def pipeline(q: str) -> str:
...
pipeline("hello", langsmith_extra={"metadata": {"user_id": "u_123"}})langsmith_extra sets per-call metadata at runtime without changing the function signature; decorator values apply to every call.
Make sure traces ship in short-lived processesflush-before-exit
from langsmith import Client
client = Client()
# ... traced work ...
client.flush()Traces upload from a background thread; Lambda functions and CLI scripts that exit fast silently drop traces unless you flush.
Evaluate an async targetasync-evaluate
from langsmith import aevaluate
async def target(inputs: dict) -> dict:
return {"answer": await my_async_app(inputs["question"])}
results = await aevaluate(
target,
data="qa-regression",
evaluators=[correct],
max_concurrency=8,
)aevaluate runs examples concurrently; keep max_concurrency below your provider rate limit or half the experiment errors out.
Scope tracing on or off around a blockdisable-tracing-locally
from langsmith import tracing_context
with tracing_context(enabled=False):
result = pipeline("not traced")
with tracing_context(enabled=True, project_name="experiments"):
result = pipeline("traced into a specific project")Handy in tests and data backfills where tracing every call would burn through the free-tier trace quota.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| langfuse | PyPI | You want an open source tracing and eval platform you can self-host with docker compose. |
| arize-phoenix | PyPI | You want OpenTelemetry-based tracing and evals that run locally in a notebook or your own infra. |
| opik | PyPI | You want Comet's open source LLM eval and tracing stack, self-hosted or hosted. |