langsmith review
We imported langsmith 0.11.1 in 0.14 seconds after a 43 MB installation. The Python SDK records nested runs for model calls, tools, retrieval, and ordinary functions, then sends them to the LangSmith service for inspection. Its Client also creates datasets, launches evaluations, stores human or automated feedback, and controls LangSmith sandboxes. LangChain is optional because @traceable and provider wrappers work in plain Python. Version 0.11.1 retries temporary sandbox WebSocket upgrade failures, creates download URLs for sandbox files, and fixes incomplete pytest-suite reporting. This package is useful for LLM development workflow, but it also creates a new hosted data path for prompts and outputs.
langsmith 0.11.1 installed in 0.7 seconds, occupied 43 MB, imported in 0.14 seconds, and had no pip-audit findings in our sandbox, making its real cost operational rather than installation friction. Adopt it when traces feed regular dataset evaluations; skip it when the same prompt data must remain inside an existing telemetry system.
We installed it
| Install | ✓ · 0.7s | 24 packages on disk · 43 MB |
| Import | ✓ | import langsmith in 0.14s · pure Python · py.typed · requires Python >=3.10 |
| Known vulns | 0 | (pip-audit) |
Answers from our run
Does langsmith install cleanly?
Yes. In a fresh container with an empty cache, pip install langsmith finished in 0.7s, leaving 24 packages and 43 MB on disk. pip-audit reported no known vulnerabilities.
What does langsmith need to run?
Python >=3.10, and nothing compiled: it is pure Python. In our run import langsmith succeeded in 0.14s, and the package ships py.typed for type checkers.
langsmith or langfuse: which should you use?
langfuse: Use it when an open-source tracing stack and a self-hosting path are deciding requirements. langsmith 0.11.1 installed in 0.7 seconds, occupied 43 MB, imported in 0.14 seconds, and had no pip-audit findings in our sandbox, making its real cost operational rather than installation friction.
When should you not use langsmith?
Prompts, retrieved text, tool arguments, and completions cannot be sent outside your current boundary; wrappers can record those fields unless you exclude or anonymize them
Use it if
- Developers need one trace tree that connects an agent decision to its model, retrieval, and tool runs
- Saved examples should become repeatable experiments with code evaluators or model judges
- A LangChain or LangGraph application should emit traces through supported environment settings
- User feedback must be attached to the exact run that produced an answer
- Prompts, retrieved text, tool arguments, and completions cannot be sent outside your current boundary; wrappers can record those fields unless you exclude or anonymize them
- Your existing OpenTelemetry collector already answers the tracing question and you have no need for LangSmith datasets, experiments, or annotation queues
- You only need provider latency and token totals; our clean installation added 24 packages and 43 MB for a much wider workflow
- A stable 1.x contract is mandatory; the current Python package is 0.11.1 and recent releases changed trace patch payloads, sandbox helpers, and integration behavior
- Every emitting environment cannot receive an API key and project routing; hosted tracing needs LANGSMITH_API_KEY, and organization-scoped keys also need a workspace ID
Setup reality
We installed langsmith 0.11.1 in 0.7 seconds in a clean Python 3.12 Bookworm container. It left 24 packages using 43 MB, and pip-audit found zero known vulnerabilities. The metadata contains 45 dependency declarations across the base package and optional extras. It is pure Python, requires Python 3.10 or later, includes py.typed, uses the MIT license, and imported in 0.14 seconds.
Hosted tracing needs LANGSMITH_TRACING=true plus LANGSMITH_API_KEY. Set LANGSMITH_PROJECT so production, staging, and local runs do not collect under the default project. An organization-scoped key also requires LANGSMITH_WORKSPACE_ID. Decorated functions carry context through normal calls, while queues and process boundaries need propagated trace headers or explicit parent information. Version 0.11.1 does not remove that routing work.
Provider wrappers can capture prompts, completions, retrieved passages, metadata, and tool arguments. Decide the exclusion or anonymization policy before sending the first production trace. Release 0.11.0 masked Anthropic MCP credentials and stopped storing mcp_servers metadata, evidence that wrapper versions affect what crosses the boundary. Avoid attaching whole request or user objects as metadata because retention and access rules then apply to those copies too.
The client batches writes away from the request path. A short CLI or serverless process may end before the upload completes, so call client.flush() or client.close() with an appropriate timeout. Evaluations can run examples and evaluators concurrently; set max_concurrency below provider quotas rather than matching all 3 sandbox CPUs. Sandbox work also needs network policy, timeouts, and artifact cleanup even though 0.11.1 retries transient WebSocket upgrades.
Patterns
Route traces to one project enable-hosted-tracing
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY='lsv2_...'
export LANGSMITH_PROJECT='checkout-agent'Organization-scoped keys also require LANGSMITH_WORKSPACE_ID; without LANGSMITH_PROJECT, runs go to the default project.
Build a nested run tree trace-nested-functions
from langsmith import traceable
@traceable(run_type='retriever')
def retrieve(query: str) -> list[str]:
return index.search(query)
@traceable(name='answer-order-question')
def answer(question: str) -> str:
return generate(question, retrieve(question))The inner decorated call becomes a child when Python execution context propagates normally; a queue boundary needs explicit trace context.
Record OpenAI calls wrap-openai-client
from openai import OpenAI
from langsmith.wrappers import wrap_openai
client = wrap_openai(OpenAI())
response = client.responses.create(
model='gpt-5-mini',
input='Summarize the failed checks',
)The wrapper can record request and response content. Apply input and output exclusions before processing secrets or personal data.
Trace work without decorating a function trace-code-block
from langsmith import trace
with trace('invoice-review', inputs={'invoice_id': invoice_id}) as run:
result = review_invoice(invoice_id)
run.end(outputs={'decision': result})A trace block is useful when the operation spans several calls; run.end() supplies the final output for that run.
Label one traced call attach-run-metadata
result = answer(
question,
langsmith_extra={
'tags': ['prod', 'billing'],
'metadata': {'release': release_id},
},
)Metadata is copied into the trace. Pass selected scalar fields instead of a complete request, session, or user object.
Store reference examples create-dataset
from langsmith import Client
client = Client()
dataset = client.create_dataset('refund-policy-v3')
client.create_examples(
dataset_id=dataset.id,
examples=[
{'inputs': {'question': 'Can I return ink?'}, 'outputs': {'allowed': True}},
{'inputs': {'question': 'Can I return opened software?'}, 'outputs': {'allowed': False}},
],
)create_dataset() creates another dataset on each run. Look up an existing versioned name when setup must be repeatable.
Evaluate a target against saved examples run-evaluation
from langsmith import Client
results = Client().evaluate(
lambda inputs: {'allowed': policy_agent(inputs['question'])},
data='refund-policy-v3',
evaluators=[is_correct],
experiment_prefix='prompt-2026-08-26',
max_concurrency=4,
)max_concurrency controls simultaneous example work. Count both target and evaluator model calls when setting it against provider limits.
Score a deterministic field write-code-evaluator
def is_correct(outputs: dict, reference_outputs: dict) -> dict:
passed = outputs['allowed'] is reference_outputs['allowed']
return {'key': 'policy_match', 'score': int(passed)}An exact code evaluator is reproducible and costs no model call; it fits structured fields better than a model judge.
Connect feedback to its run record-user-feedback
from langsmith import Client
Client().create_feedback(
run_id=run_id,
key='thumbs',
score=1,
comment='Answer matched the policy page',
)Persist the run_id with the response. Prompt text is not a safe lookup key when identical requests produce multiple runs.
Sample ordinary production traffic sample-traces
export LANGSMITH_TRACING_SAMPLING_RATE=0.10A 0.10 setting samples ordinary traces. Design a separate always-capture path for errors or user-reported runs that must reach review.
Turn tracing off inside a block suppress-one-workload
from langsmith import tracing_context
with tracing_context(enabled=False):
rebuild_search_index()The disabled context prevents a backfill or test section from filling a LangSmith project with low-value runs.
Wait for buffered trace writes flush-short-process
from langsmith import Client
client = Client()
run_batch()
client.flush(timeout=10)
client.close(timeout=10)The 10-second bounds stop a CLI from waiting forever; flush drains pending traces, and close also cleans up background workers.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| langfuse | PyPI | Use it when an open-source tracing stack and a self-hosting path are deciding requirements |
| arize-phoenix | PyPI | Use it for OpenTelemetry-based LLM traces and evaluations kept in your own environment |
| mlflow | PyPI | Use it when model tracking, registry work, and evaluation already live in MLflow |
| opik | PyPI | Use it when Comet's open-source LLM evaluation and tracing workflow fits the team |
More ai / ml guides
openai · mcp · huggingface-hub · scikit-learn · tiktoken · @modelcontextprotocol/sdk · the whole shelf →
How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.