mrkeyoor.com_
Thu 24 Sept 07:43 UTC
LLM Toolsevaluationupdated 26 Aug 2026

dspy review

DSPy is a Python framework for expressing language-model programs as typed modules and then optimizing their instructions or examples against a metric. It replaces hand-edited prompt strings with reusable signatures, predictors, retrieval steps, agents, and optimizers.

+115stars / 7d
Verdict

Our DSPy build finished in 10 seconds, yet 46 tests failed and 16 more hit collection or setup errors, so its ideas are easier to recommend than this exact checkout as a clean drop-in. Use DSPy when you can define a metric and want the framework to search for better instructions or demonstrations. Choose a simpler orchestration library when the prompt is small, the task has no reliable scorer, or repeated optimization calls are hard to budget.

We ran it

Lab card: what happened when we ran dspyScreenshot of dspy (dspy.ai)
Install✓ · 43s88 packages · 222 MB
Build✓ · 10s
Tests✗ · 569s1109 passed · 46 failed · 360 skipped · 16 errors of 1171 (pytest)
Known vulns1(pip-audit)
Repo567 files~73,695 lines of source · 23.6 MB · 5 CI workflows · tests dir

Answers from our run

Does dspy build from source?

Dependencies installed in 43 seconds (88 packages), and the build succeeded in 10 seconds. We cloned commit 4ed377e into a clean Debian container with 3 CPUs and no project-specific setup.

Do dspy's tests pass?

Not all of them: 1109 of 1171 passed and 46 failed when we ran the project's own test command (pytest), with 16 collection errors. Some failures need services or credentials a bare container does not have.

Does dspy have known vulnerabilities in its dependencies?

pip-audit flagged 1 known advisory in the dependency tree at the time of our run.

Who should not use dspy?

Teams without examples or a meaningful metric: DSPy's optimizers need a way to judge candidate programs.

What are the alternatives to dspy?

LangChain, Haystack, Promptflow. Our DSPy build finished in 10 seconds, yet 46 tests failed and 16 more hit collection or setup errors, so its ideas are easier to recommend than this exact checkout as a clean drop-in.

Setup3/5Simple pip entry, but optional extras and model setup appear quickly
Docs4/5Dedicated docs explain modules, optimizers, evaluation, and agents
Community5/537,596 stars, active maintainers, and detailed current issue work
Maturity4/5Large suite and 3.3.1 release, with some experimental paths

Discussed on

  1. hnDSPy: Framework for programming with foundation models141 points
  2. hnDspy3 points

Who it’s for

Python teams with a measurable task and a dataset they can use to evaluate outputs.
Researchers comparing prompt optimization, few-shot selection, and model behavior.
RAG or agent builders who want typed modules instead of long prompt templates.
Engineers prepared to treat evaluation code and metrics as part of the product.

Who it’s NOT for

Teams without examples or a meaningful metric: DSPy's optimizers need a way to judge candidate programs.
Developers wanting a low-concept wrapper around one model call: signatures, modules, adapters, and compilation add a new programming model.
Environments that install only the base extra but expect every feature: our suite reported that KNN few-shot requires NumPy through dspy[numpy] or a separate install.
Production users expecting every new agent path to be settled: the maintainers describe ReActV2 in 3.3.0 as experimental and are still collecting feedback before replacing ReAct.

Setup reality

Our sandbox install succeeded in 43 seconds, adding 88 packages and using 222 MB on disk. The package build completed in 10 seconds. Pytest ran for 569 seconds and exited 1: 1,109 tests passed, 46 failed, 360 skipped, and 16 collection or setup errors out of 1,171 tested cases.

The base command is pip install dspy, but real work needs model credentials or a local model endpoint. Optimization also needs examples, a scoring metric, and a budget for repeated model calls. Some features use optional extras.

The log tail shows several test servers exiting before port 48455 became ready and KNN few-shot raising an ImportError because NumPy was absent. It does not establish why the server process exited. pip-audit found 1 known vulnerability.

DSPy turns prompts into programs that can be optimized

DSPy asks developers to describe an input-output contract as a signature, compose predictors into Python modules, and evaluate the result with a metric. An optimizer can then search instructions, demonstrations, or other program choices. That is a different abstraction from putting a formatted string around each model call. It suits classifiers, retrieval pipelines, extraction, and agents when quality can be measured over a set of examples. The promise is less manual prompt editing, but the work moves into signatures, datasets, metrics, and evaluation.

A small application can start with dspy.Predict or ChainOfThought, then configure a language model once. Larger programs combine modules and retrieval, while optimizers compile the program against training examples. DSPy does not make an unclear task precise by itself. If the metric rewards the wrong thing or the examples miss production cases, optimization can improve the score you wrote while leaving the actual user problem untouched. The framework is most useful to teams already willing to maintain an evaluation set.

Version 3.3.1 is active, while ReActV2 remains experimental

The latest release is 3.3.1, published on 2026-08-21, and the last repository push was 2026-08-25. That combination indicates active maintenance. GitHub listed 637 open issues and PRs, a large queue that also reflects the project's usage and pace. Current work includes adapters, usage tracking, evaluation timeouts, GEPA optimization, and code-interpreter boundaries. This is a living research-backed framework rather than a frozen utility library.

Agent users should note that ReActV2 arrived as an experimental feature in 3.3.0. The maintainers are asking for feedback on prompt caching, native tool calling, and regressions before it replaces the earlier ReAct module. That wording matters: it is an experiment, not a completed migration. Teams shipping an agent should pin DSPy, choose the ReAct implementation deliberately, and cover tool-call behavior with their own evaluations instead of assuming the newest class is the stable default.

What happened when we ran it

Our sandbox cloned commit 4ed377e into an unprivileged Debian container with 3 CPUs and 8 GB of RAM. Installation took 43 seconds, pulled 88 Python packages, and occupied 222 MB on disk. The repository contained 567 files, about 73,695 lines of source, and used 23.6 MB when checked out. The package build then completed successfully in 10 seconds. pip-audit reported 1 known vulnerability.

Pytest ran for 569 seconds and exited 1. Of 1,171 tested cases, 1,109 passed, 46 failed, and 16 ended in collection or setup errors; another 360 were skipped. The log tail shows several client and streaming tests timing out because a server process exited before port 48455 became ready. It also shows the KNN few-shot test raising an ImportError because NumPy was missing and recommends dspy[numpy] or a direct NumPy install. The tail does not say why the server process exited.

Optional extras affect what the base install can verify

The public installation line is simply pip install dspy, and that is honest for core imports. Our 88-package base environment built correctly. The test result shows that the whole source tree expects more in at least one path: KNN few-shot refuses to run without NumPy. A developer who chooses features by reading examples can discover those optional requirements later, so install the matching extras and run the tests for the modules you use.

Model access is the larger configuration step. A real DSPy program needs credentials for a hosted model or a reachable local endpoint. Compiling a program may call the model many times across examples and candidates, which turns evaluation design into a cost and rate-limit decision. DSPy tracks usage, but an open pull request reports nested tracker totals and nonnumeric fields being combined incorrectly. If cost ceilings are contractual, verify totals outside the framework as well.

Current adapter issues can change typed output behavior

DSPy's adapters translate typed signatures into provider messages and parse responses back into Python values. That layer carries real correctness risk. Issue #10269 documented XML output containing ]]> failing to parse and empty optional containers returning None; a linked pull request supplies fixes and tests. Issue #10273 reports an lru_cache retaining up to 32 image objects, which can hold large base64 payloads in a long-running process.

These reports are specific enough to guide adoption. Teams using XMLAdapter should test code-like strings and empty containers. Image workloads should watch memory over repeated, distinct inputs. Neither concern disqualifies DSPy, and active fixes are a good sign, but typed signatures do not guarantee faithful round trips through every adapter. Pin the release, test the chosen provider adapter, and keep representative multimodal or structured examples in the same evaluation set used during optimization.

DSPy pays off only when the metric is worth trusting

DSPy's strongest idea is that prompts and demonstrations can be treated as parameters of a program. On a task with labeled examples and a defensible metric, that gives engineers a repeatable way to compare changes instead of polishing one prompt by feel. The 1,109 passing tests in our run and the published research show serious engineering behind the approach, even though the full suite was not green in our container.

For an internal extraction system, support classifier, or RAG pipeline with reviewable answers, the added concepts can pay for themselves. For a chatbot whose quality is subjective and changes by conversation, compilation may add expense without a trustworthy target. LangChain and Haystack are easier choices when the main job is wiring tools and data sources. Pick DSPy when evaluation already belongs in your development loop and you want the program, not a person, to search the prompt space.

Alternatives

ProjectWhat it isPick it when
LangChain gh↗A broad Python and JavaScript framework for model calls, tools, retrieval, and agents.pick this instead when integrations and explicit orchestration matter more than automatic prompt optimization.
Haystack gh↗A component pipeline framework centered on retrieval, generation, and production AI applications.pick this instead when you want inspectable data-flow pipelines and retrieval components without DSPy's compiler model.
PromptflowA toolkit for building, evaluating, tracing, and deploying LLM workflows.pick this instead when visualizable flows and operational evaluation are more important than learned prompt programs.

What people are saying

  1. [github-trending] stanfordnlp/dspy

Sources

  1. DSPy README
  2. DSPy 3.3.1 release
  3. DSPy documentation
  4. DSPy XMLAdapter issue #10269
  5. DSPy image cache issue #10273
  6. DSPy ReActV2 feedback issue #10203

More llm tools reviews

agent-beacon · MiMo-Code · pi-claude-bridge · treg · train-llm-from-scratch · mistral.rs · the whole board →