PyPIAI / MLupdated 18 Oct 2026

pydantic-evals review

Pydantic Evals defines evaluation cases, tasks and checks in Python. A case contains inputs and can include a reference output, metadata and its own evaluators. A dataset groups cases and shared checks; running a task against it produces a report. Evaluators return pass/fail assertions, numeric scores or categorical labels. This guide starts with a function that always returns 'Paris', so you can understand the evaluation machinery before connecting an AI application. We inspected the current official docs and PyPI metadata provided for this review but did not install or execute the package.

pydantic-evals · v2.57.0 · pydantic/pydantic-ai · docs · reviewed by Keyoor

Verdict

Pydantic Evals is a reasonable candidate when you want Python-defined evaluation cases and custom checks. Start with the fixed-output baseline, then replace its task with your function and add cases that expose meaningful failures. Use model judgments where exact rules are insufficient, and interpret their results separately. The documentation supports this starting workflow; actual release compatibility and operational fit still need validation in your environment.

Answers from our run

pydantic-evals or ragas: which should you use?

ragas: Consider ragas if you want to explore an experiment-centered evaluation workflow with custom metrics. Its official introduction describes experiments and metrics defined with decorators. This is a reason to inspect its workflow, not evidence that it evaluates more accurately than Pydantic Evals. No Ragas installation commands or provider setup are established here. Source: https://docs.ragas.io/en/stable/. Pydantic Evals is a reasonable candidate when you want Python-defined evaluation cases and custom checks.

When should you not use pydantic-evals?

You need to configure and run evaluations entirely through a web interface; the documented workflow uses Python.

API stability3/5Neutral editorial documentation-based assessment. The sources establish interfaces for cases, datasets, evaluators and reports, but do not establish compatibility across releases or the actual terms of a version policy. A public API and a stable classifier do not demonstrate backward compatibility, so this score carries no measured stability claim.

Docs4/5Editorial documentation-based assessment of the material available to a new reader. The overview supplies a complete baseline, while the evaluator guide explains return types, context, exceptions and report evaluators with code. These support a useful entry path. The dependency wording still needs interpretation beside release metadata, and runtime correctness was not measured.

Maintenance3/5Neutral editorial documentation-based assessment. PyPI records the 2.57.0 wheel upload on 16 October 2026, which establishes a release artifact at that time. It does not establish a sustained maintenance cadence, issue response, support quality or repository activity. Those unknowns prevent a stronger maintenance assessment.

Ecosystem3/5Neutral editorial documentation-based assessment. Documentation describes use with arbitrary task functions and optional observability, but integration possibilities do not establish adoption or successful interoperability. Stars and weekly downloads remain null, and no integration was executed. This score therefore conveys uncertainty rather than measured ecosystem strength.

Use it if

You want to keep evaluation cases and scoring logic in Python alongside the function you evaluate.
You have requirements that can be expressed as exact rules and want a deterministic baseline before adding subjective judgments.
You need shared dataset checks together with requirements attached to individual cases.
You want custom evaluators that return assertions, numeric scores or labels.

Skip it if

You need to configure and run evaluations entirely through a web interface; the documented workflow uses Python.
You require Python below 3.11; release 2.57.0 requires Python >=3.11.
You expect a type assertion or keyword match to establish factual accuracy, helpfulness or safety.
Your adoption decision requires demonstrated compatibility, performance or maintenance history; those remain unverified here.

Setup reality

Prerequisites and installation

Use Python 3.11 or newer in an isolated environment. The official overview documents pip install pydantic-evals. To select the release covered by the metadata here, use pip install pydantic-evals==2.57.0. Installation can download dependencies. The current documentation is separate evidence from the version-specific package metadata; the snippets below have documentation support, not demonstrated compatibility with that release.

Check the dependencies

The overview says pydantic-evals does not depend on the pydantic-ai distribution. However, release 2.57.0 metadata requires pydantic-ai-slim==2.57.0, along with anyio, logfire-api, pydantic, pyyaml and rich. The optional logfire extra adds logfire. Account for the slim dependency when reviewing your environment rather than interpreting the overview as an absence of AI-related dependencies.

Run the baseline

Save the first pattern as baseline.py and run python baseline.py in that environment. Its task returns a fixed string, and its evaluators use Python comparisons and a type check. The script contains no model call or provider credential setup and incurs no LLM inference charge through its own logic. It does not establish the network behavior of dependencies or an environment with telemetry already configured.

The documented example shows a score of 1.00 and a passing string-type assertion. Those results also follow from the code: the task returns exactly the reference answer. They are expected results, not measurements from this guide. The print call hides durations, which removes one changing display field.

Interpret failures

A False assertion means its rule did not pass. An evaluator exception is a different outcome: the evaluator overview says failures appear in report.cases[i].evaluator_failures, with the evaluator name, error message and stacktrace. Inspect these when a check produces no usable result. Keep input and output types consistent with your cases, and remember that substring scores only describe text matching.

Add a judge only when needed

LLMJudge is an optional check for criteria that require judgment. Before running it, configure the selected model and any external provider credentials, and review that provider's charges and data handling. The evaluator documentation lists monetary cost, non-determinism and possible bias among its disadvantages. The judge pattern below configures a rubric; it is not a complete provider setup.

How to read the ratings

Scores are editorial documentation-based assessments on a 1-5 scale, not measurements of product quality or prestige. A 1 means documented constraints substantially obstruct the stated workflow; 2 means material gaps limit it; 3 is neutral when evidence is incomplete, including unknown compatibility, maintenance or adoption; 4 means concrete documentation supports the assessed area with useful examples and limits; 5 requires unusually comprehensive direct evidence for that area. These scores are not an overall ranking. Documentation detail can support the documentation score, but integration possibilities alone cannot raise the ecosystem score above neutral.

Choose the relevant guide

Our Pydantic library guide covers parsing and validation with model types. Our Pydantic AI review covers building agents that call models and tools. This guide focuses on a different task: defining evaluation cases, running a fixed-output Python function and reading evaluator results before adding an optional model judge. If you need trace collection rather than a first evaluation dataset, see our Langfuse Python guide.

Patterns

Run the documented fixed-output evaluation run-deterministic-baseline

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance

case1 = Case(
    name='simple_case',
    inputs='What is the capital of France?',
    expected_output='Paris',
    metadata={'difficulty': 'easy'},
)

class MyEvaluator(Evaluator[str, str]):
    def evaluate(self, ctx: EvaluatorContext[str, str]) -> float:
        if ctx.output == ctx.expected_output:
            return 1.0
        elif (
            ctx.expected_output is not None
            and ctx.expected_output.lower() in ctx.output.lower()
        ):
            return 0.8
        else:
            return 0.0

dataset = Dataset(
    name='capital_quiz',
    cases=[case1],
    evaluators=[IsInstance(type_name='str'), MyEvaluator()],
)

async def guess_city(question: str) -> str:
    return 'Paris'

report = dataset.evaluate_sync(guess_city)
report.print(include_input=True, include_output=True, include_durations=False)

The complete simple_eval_complete.py example from the official overview, with annotation comments and displayed sample output omitted. The full passage establishes include_durations=False. The task ignores its input and always returns Paris, so a matching score demonstrates the scoring path, not a question-answering capability. Source: https://pydantic.dev/docs/ai/evals/evals/.

Name a case and attach reference data organize-case-metadata

from pydantic_evals import Case, Dataset

case1 = Case(
    name='simple_case',
    inputs='What is the capital of France?',
    expected_output='Paris',
    metadata={'difficulty': 'easy'},
)
dataset = Dataset(name='capital_quiz', cases=[case1])

The overview's dataset construction separates task inputs, the reference answer and descriptive metadata. This configures a dataset without running a task. The evaluator guide documents ctx.metadata for checks that need case metadata. Sources: https://pydantic.dev/docs/ai/evals/evals/ and https://pydantic.dev/docs/ai/evals/evaluators/overview/.

Add a shared string-type assertion add-type-assertion

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance

dataset = Dataset(
    name='capital_quiz',
    cases=[Case(inputs='What is the capital of France?', expected_output='Paris')],
)
dataset.add_evaluator(IsInstance(type_name='str'))

Adaptation of the overview's add_evaluator example, with dataset creation included and IsInstance imported as in its complete baseline. The assertion checks output type across cases. It does not check meaning. Run a task with evaluate_sync as in the first pattern. Source: https://pydantic.dev/docs/ai/evals/evals/.

Write a reusable keyword assertion require-output-keyword

from dataclasses import dataclass
from pydantic_evals.evaluators import Evaluator, EvaluatorContext

@dataclass
class HasKeyword(Evaluator):
    keyword: str

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return self.keyword in ctx.output

The evaluator guide's boolean-return example defines a pass/fail assertion. For string outputs, this is a case-sensitive substring check. Add HasKeyword(keyword='Paris') to a dataset's evaluators list to use it. A match says nothing about the surrounding statement's truth. Source: https://pydantic.dev/docs/ai/evals/evaluators/overview/.

Return a label based on output text categorize-output-text

from dataclasses import dataclass
from pydantic_evals.evaluators import Evaluator, EvaluatorContext

@dataclass
class SentimentClassifier(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> str:
        if 'error' in ctx.output.lower():
            return 'error'
        elif 'success' in ctx.output.lower():
            return 'success'
        return 'neutral'

The documented string-return example illustrates categorical labels. Despite its class name, it only searches for two words and is not a general sentiment classifier. If both words occur, the error branch wins. Source: https://pydantic.dev/docs/ai/evals/evaluators/overview/.

Distinguish exceptions from failed assertions inspect-evaluator-exceptions

from dataclasses import dataclass
from pydantic_evals.evaluators import Evaluator, EvaluatorContext

def risky_operation(output: str) -> bool:
    if 'error' in output:
        raise ValueError('Found error in output')
    return True

@dataclass
class RiskyEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        result = risky_operation(ctx.output)
        return result

The evaluator guide's error-handling example, with comments omitted. Attach this evaluator to a dataset and run a task to exercise it. The guide documents captured exceptions in report.cases[i].evaluator_failures; an exception differs from a returned False assertion. Source: https://pydantic.dev/docs/ai/evals/evaluators/overview/.

Configure a report-wide confusion matrix summarize-classification-results

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ConfusionMatrixEvaluator

dataset = Dataset(
    name='report_evaluator_example',
    cases=[
        Case(inputs='meow', expected_output='cat'),
        Case(inputs='woof', expected_output='dog'),
    ],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
        ),
    ],
)

The complete report-evaluator configuration appears in the current evaluator overview, including the import, constructor arguments and report_evaluators keyword. It compares outputs with reference labels after cases finish. This snippet only configures the dataset; provide a classification task to run it. Its documentation support does not establish compatibility history for release 2.57.0. Source: https://pydantic.dev/docs/ai/evals/evaluators/overview/.

Configure an optional subjective check configure-optional-model-judge

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import LLMJudge

dataset = Dataset(
    name='llm_judge_example',
    cases=[Case(inputs='What is 2+2?', expected_output='4')],
    evaluators=[
        LLMJudge(
            rubric='Response is factually accurate based on the input',
            include_input=True,
        )
    ],
)

The evaluator overview's LLMJudge configuration defines a rubric without running a task. It omits model and credential setup. Before execution, configure the selected model, supply any required external provider credentials and review charges. The documentation warns about monetary cost, non-determinism and bias. This is separate from the deterministic baseline. Source: https://pydantic.dev/docs/ai/evals/evaluators/overview/.

Alternatives

Package	Registry	Pick it when
ragas	PyPI	Consider ragas if you want to explore an experiment-centered evaluation workflow with custom metrics. Its official introduction describes experiments and metrics defined with decorators. This is a reason to inspect its workflow, not evidence that it evaluates more accurately than Pydantic Evals. No Ragas installation commands or provider setup are established here. Source: https://docs.ragas.io/en/stable/.
deepeval	PyPI	Consider deepeval if you want evaluation files collected through a Pytest-style workflow. Its official quickstart says deepeval test run collects and runs eval files the same way pytest would. The quickstart's optional GEval example requires OPENAI_API_KEY or another configured LLM; external provider credentials and charges need review before execution. No comparative accuracy or performance result is claimed. Source: https://deepeval.com/docs/getting-started.

More ai / ml guides

openai · mcp · huggingface-hub · scikit-learn · tiktoken · tokenizers · the whole shelf →

How this guide is made: grounded in the library's documentation, release notes, changelog, and issue history, on a fixed rubric — not a hands-on install of every release. The 50 most-downloaded entries are additionally install-verified in clean containers. Corrections: contact the desk.