# digline

> Regression testing for LLM applications, with the baseline in your repository, not on someone's server.

digline gives you an approved reference — the baseline — and on every change
tells you whether you are below it: which case, which check, by how much. The
baseline is a JSON file in your repository, so it goes through code review and
it rolls back with git. There is no hosted service that receives your payloads
and no usage data collection: the runs happen on your machines.

Every link below is the page's Markdown source. Drop the `.md` for the rendered
page — `/why.md` is `/why/`.

## Overview

- [Home](https://digline.dev/): What digline is in one screen: the sentence, what `digline compare` prints, and the two things that will never be on the roadmap
- [Start here](https://digline.dev/start.md): What a silent LLM regression is and why an ordinary test cannot see one, for a reader who has not met the problem yet — plus a short video of `digline compare` finding one
- [Why](https://digline.dev/why.md): Why an ordinary pass/fail test cannot see a quality regression — the same prompt scores 4 one morning and 3 the next, and the model moves under you — and what an approved baseline measures instead
- [digline for agents](https://digline.dev/agents.md): What a coding agent can reach on each surface — MCP, the operator loop, pytest, the Action — and why none of them lets it promote a baseline
- [About](https://digline.dev/about.md): Who builds digline, under which licence, and where the code is
- [Contact](https://digline.dev/contact.md): Where to write about digline: mail for questions, GitHub issues for bugs, the same address for security reports

## How digline compares

- [How digline compares](https://digline.dev/comparison/index.md): Which question each family of tools answers — snapshot testing, observability, exploration, benchmarking, and regression testing against an approved baseline — and why the last one is digline's: did it get worse than what I approved? Links to a page per tool underneath
- [promptfoo](https://digline.dev/comparison/promptfoo.md): What promptfoo does well — the prompt and provider matrix, its assertion library, `promptfoo redteam` — and where digline differs: a committed reference rather than a threshold, a noise floor measured per case, and a three-state verdict as the exit code
- [DeepEval](https://digline.dev/comparison/deepeval.md): What DeepEval does well — its metric library, `assert_test` inside pytest, `Synthesizer`, DeepTeam — and where digline differs: the approved run is a committed file rather than an official run on Confident AI
- [Braintrust](https://digline.dev/comparison/braintrust.md): What Braintrust does well — `Eval()`, experiments against a persistent baseline, Loop, hybrid self-hosting — and where digline differs: the same comparison, with the reference in git and no platform to run
- [LangSmith](https://digline.dev/comparison/langsmith.md): What LangSmith does well — tracing, datasets and experiments, `evaluate()`, annotation queues, online evaluation — and where digline differs: one pre-deploy question, a committed reference and an exit code
- [Langfuse](https://digline.dev/comparison/langfuse.md): What Langfuse does well — OpenTelemetry tracing, prompt management, batch evaluation over past traces, an MIT self-hosted core — and where digline differs: no platform at all, and the approval kept in git
- [Inspect AI](https://digline.dev/comparison/inspect-ai.md): What Inspect AI does well — `Task`, solvers and scorers, sandboxed execution, epochs and reducers — and where digline differs: an approved reference to regress against rather than a benchmark to score
- [Opik](https://digline.dev/comparison/opik.md): What Opik does well — Apache-2.0 tracing, experiments, guardrails, the Agent Optimizer, a self-hostable platform — and where digline differs: a file in your repository instead of a backend to deploy

## Handbook

- [Handbook](https://digline.dev/handbook/index.md): What the eight chapters cover and the order to read them in — about evaluating an LLM feature, not about the tool: nothing here needs digline installed
- [1. What you are actually shipping](https://digline.dev/handbook/01-what-you-are-shipping.md): Why a model call looks like a function and is not one, and what that costs you once it is in front of users
- [2. Cases](https://digline.dev/handbook/02-cases.md): What a case is, why almost every team has a prompt and no cases, and how to have twenty of them by this afternoon
- [3. Ground truth](https://digline.dev/handbook/03-ground-truth.md): Where the expected answer comes from when nobody gives you one, and why whether it can be collected is decided when you write the feature
- [4. Checks](https://digline.dev/handbook/04-checks.md): How to turn an output into a verdict, and the rule that saves the most time: use a model to judge only what nothing else can
- [5. The judge](https://digline.dev/handbook/05-the-judge.md): How to measure how often your judge disagrees with itself, and why that number has to exist before any score it produces can be read
- [6. The reference](https://digline.dev/handbook/06-the-reference.md): Why a threshold is not a reference, and how to choose the one number you agree to be measured against
- [7. Maintenance](https://digline.dev/handbook/07-maintenance.md): When to run the suite, what should make you look, and what to do the morning it turns red and you changed nothing
- [8. For teams building for others](https://digline.dev/handbook/08-for-teams-building-for-others.md): The two extra jobs a suite does when the LLM feature belongs to a client, and the one rule about their data that cannot be bent

## Essentials

- [Guide](https://digline.dev/product/guide.md): How to work with digline end to end, in the order the problems arrive: a first suite, a baseline, the judge's noise, sampling, thresholds, and the comparison in CI
- [Metrics](https://digline.dev/product/metrics.md): Which metric to reach for, per assertion and per aggregate: what each one takes, what it produces, and how it misleads you if you are not looking
- [Testing a selection](https://digline.dev/product/selection.md): How to test a system that selects items from its input: the offered set declared per case in a `suite.py`, and why an empty selection must never pass alone
- [Suites as data](https://digline.dev/product/declarative.md): How to write a suite as data in a `suite.toml` rather than a `suite.py`, and which suites should stay Python

## Compare

- [digline diff](https://digline.dev/product/diff.md): How to choose between two runs neither of which is a baseline — prompt A or prompt B, one model or another — with a report that names no reference and gates nothing
- [digline explain](https://digline.dev/product/explain.md): How to read one run back at length — what moved, against which measured interval, what differed underneath — from a command that states the facts and gives no advice

## History

- [digline log](https://digline.dev/product/log.md): How to tell whether the model behind your alias changed, read down the runs you already stored — what each side sent, what the provider said answered, and why most of that history is absence
- [digline register](https://digline.dev/product/register.md): How to write down what a person decided about a comparison — accepted, rejected or unsure — where the reason goes, and why no agent and no schedule may record one
- [digline view](https://digline.dev/product/view.md): How to read your runs, the baseline and a comparison in a local browser UI over `.digline/` — stdlib only, no JavaScript, no state of its own

## Maintenance

- [digline rejudge](https://digline.dev/product/rejudge.md): How to see what the same answers score under a changed judge, rubric or threshold without paying the target again, why that is a replay that declares itself rather than a cache, and why it cannot be promoted
- [digline migrate](https://digline.dev/product/migrate.md): How to bring stored runs and a baseline up to the schema version this release reads, and what a scan does with a document it cannot read

## Running it

- [The Docker image](https://digline.dev/product/docker.md): What the official `ghcr.io/digline/digline` image contains and what it deliberately does not, which tag to pin in CI, and how to derive it for a suite with dependencies of its own
- [digline over MCP](https://digline.dev/product/mcp.md): How a coding agent reads a digline result and measures a new one over MCP, and why the server has no tool that promotes a baseline
- [digline in pytest](https://digline.dev/product/pytest.md): How to gate a pytest run on a digline comparison — one row per check, a suspension as a skip, and no provider call unless you ask for one — and what pytest's single exit code cannot tell apart
- [The operator loop](https://digline.dev/product/operator.md): What an operator may decide alone — re-run, classify a draw from a drift, diff two candidates, all under a stopping rule declared before the first run — what it must escalate, why `promote` is absent from its surface rather than forbidden, and why the loop runs in your perimeter instead of as a service of ours

## Reference

- [digline API](https://digline.dev/product/api.md): What to import when writing a suite: what `digline.core` and `digline.run` export, the Judge protocols, and what an assertion returns
- [Security](https://digline.dev/product/security.md): digline's own threat model: the five surfaces an input can reach, why a suite file executing is a declared boundary rather than a gap, the six fixes of 0.7.1 and 0.7.2 and the four advisories, the delta-pass rule — and the attestations, pins and Scorecard run you can check

## Examples

- [Examples](https://digline.dev/product/examples/index.md): Every worked example by the question it answers, each a directory with its suite, its committed baseline and its CI job, and a whole product built on digline
- [A prompt, no application yet](https://digline.dev/product/examples/prompt-first.md): How to tell whether an edit to a prompt made the answers better or only different, when there is no application around it yet
- [A classifier](https://digline.dev/product/examples/classifier.md): How to keep an LLM classifier under control: several samples per case and a majority, precision over the whole set as the gate, thresholds measured rather than chosen
- [A RAG](https://digline.dev/product/examples/rag.md): How to check that a RAG does not make things up, with the retrieved passages frozen into each case so the generator is what is measured
- [An application digline cannot import](https://digline.dev/product/examples/external-app.md): How to test an application digline cannot import — Java, Go, a shell script behind a socket: a body it can post and a field it can read back
- [A LangChain pipeline](https://digline.dev/product/examples/langchain.md): How to tell what a LangChain upgrade changed, with digline importing the chain and calling it in process: no server, no HTTP, no port
- [A LangGraph agent](https://digline.dev/product/examples/langgraph.md): How to gate a LangGraph agent on what it did rather than what it said: which tools it called, in what order, and with which arguments, with the tools executing for real and the model scripted so CI needs no key
- [A LangChain4j service](https://digline.dev/product/examples/langchain4j.md): What to put in the repository of a LangChain4j service — Spring Boot or Quarkus — when the endpoint and not the framework is the contract
- [A LlamaIndex query engine](https://digline.dev/product/examples/llamaindex.md): How to tell whether a LlamaIndex RAG still answers from the right page, with retrieval running on every run instead of frozen into the cases
- [A team that does not write Python](https://digline.dev/product/examples/quickstart-toml.md): How a team that writes no Python gates a prompt in CI: a `suite.toml`, a `cases.json`, and no code at all
- [A suite watched on a schedule](https://digline.dev/product/examples/operator.md): The operator loop assembled: a suite re-run on a schedule within a declared stopping rule, a draw told from a drift, a three-layer alert

## Decisions

- [Decisions](https://digline.dev/product/adr/index.md): Every architecture decision behind digline, oldest first, each with its status and the date it was taken, and nothing deleted when a decision changes
- [ADR 0001 — Verdict, not score](https://digline.dev/product/adr/0001-verdict-not-score.md): Why an assertion produces a three-state verdict rather than a bare score, and why the comparison against the baseline lives in the core
- [ADR 0002 — Three worlds](https://digline.dev/product/adr/0002-three-worlds-and-where-the-data-lives.md): Where each kind of data lives across the developer's, the consultancy's and the customer's world, so that no customer's payload leaves its own
- [ADR 0003 — Artifacts](https://digline.dev/product/adr/0003-artifacts-travel-only-when-the-suite-says-so.md): Why a run records the verdicts and the commit but not the prompt that produced them, and which files a suite can ask to store
- [ADR 0004 — Plugins, target and judge](https://digline.dev/product/adr/0004-every-plugin-is-a-target-and-a-judge.md): Why every provider plugin ships both sides — the system under test and the judge that evaluates it — and the core implements neither
- [ADR 0005 — The configuration under test](https://digline.dev/product/adr/0005-the-configuration-of-the-system-under-test.md): Proposed, nothing implemented: how a run would record which model answered, at what temperature, under what token cap
- [ADR 0006 — Samples and the noise floor](https://digline.dev/product/adr/0006-repeated-samples-and-the-noise-floor.md): Why a case is run more than once, and why a drop is a regression only when it clears the noise floor those samples measure — the fix for a tool that cries wolf on its own measurement error
- [ADR 0007 — The declarative suite format](https://digline.dev/product/adr/0007-the-declarative-suite-format.md): What the TOML suite format can express, where it stops and Python takes over, and why both forms build the same objects and share one baseline
- [ADR 0008 — The two-run report](https://digline.dev/product/adr/0008-the-two-run-report.md): Why "should I switch?" is a different question from "did it get worse?", and why the report that answers it names neither run the reference and never gates a pipeline
- [ADR 0009 — Boundary semantics](https://digline.dev/product/adr/0009-boundary-semantics.md): One rule for every limit — compared at the precision the document stores, and inclusive — and why a threshold met exactly must pass while a delta must not be decided by a residue nobody can see
- [ADR 0010 — Per-group aggregates](https://digline.dev/product/adr/0010-per-group-aggregates.md): Why an aggregate over the whole run hides a class that is broken, and how a suite splits one by a variable its cases already carry without losing the noise floor
- [ADR 0011 — The MCP server](https://digline.dev/product/adr/0011-the-mcp-server.md): Why the MCP server has no tool that promotes a baseline — an absence, not a refusal — and what a run projected for an agent may and may not carry
- [ADR 0012 — The reading](https://digline.dev/product/adr/0012-the-reading.md): Why the command that reads a run back at length states no advice and quotes no judge, and why the facts it is built from are a type that cannot hold a payload rather than a filter that drops one
- [ADR 0013 — The pytest plugin](https://digline.dev/product/adr/0013-the-pytest-plugin.md): Why a pytest row is one check and not one case, why comparing is the default and running is a flag that refuses under --collect-only, and why there is no way to promote a baseline from a green test run
- [ADR 0014 — What may ride a schema bump](https://digline.dev/product/adr/0014-what-may-ride-a-schema-bump.md): Why a field may ride a schema bump only if it leaves `config_hash` alone, migrates without inventing a value and widens nothing that travels — and why a document now says which digline wrote it
- [ADR 0015 — The recorded output and the re-judge](https://digline.dev/product/adr/0015-the-recorded-output-and-the-declared-re-judge.md): Why the model's answers are recorded only on opt-in and released by no `Disclosure`, why a re-judge is a replay that declares itself rather than a cache, and why a replay can never become the baseline
- [ADR 0016 — The canary case](https://digline.dev/product/adr/0016-the-canary-case.md): How a suite notices that the model behind an alias changed when the provider says nothing: a case kept out of every metric that stops the pipeline when it moves beyond its noise, in either direction
- [ADR 0017 — The journal and the resumed run](https://digline.dev/product/adr/0017-the-journal-and-the-resumed-run.md): How a run killed mid-flight is finished rather than paid for again: a per-case journal that holds nothing the run file would not, a `--resume` refused unless every fact the run asserts is true of both halves, and why the resumed run carries no marker and no schema bump
- [ADR 0018 — The recorded trajectory](https://digline.dev/product/adr/0018-the-recorded-trajectory-and-the-agent-under-test.md): Why a suite that judged an agent's tool calls could never be re-judged, and what it cost to close that: the trajectory recorded beside the answer it belongs to, arguments that are payload and never travel, and a check for the right tool asked the wrong question
- [ADR 0019 — The reasoning operator](https://digline.dev/product/adr/0019-the-reasoning-operator.md): How the operator loop exercises a judgment somebody wrote down: a `[policy]` in `operator.toml` the agent may not edit, a decision that names the clause it applied and the policy's digest, clauses that may hold a cycle but never wake one, three floors none may lower, and an append-only journal where every hold is written
- [ADR 0020 — The reading across runs](https://digline.dev/product/adr/0020-the-reading-across-runs.md): How `digline log` reads the story of an alias down a suite's stored runs: a roll is only the same sent model answering as a different recorded one, seven absences each with its own sentence, a replay that asked the target nothing, a row with no score by type, and the MCP surface growing once to eight tools with `explain` beside it
- [ADR 0021 — The register](https://digline.dev/product/adr/0021-the-register.md): How a person's decision about a comparison is kept: `digline register` with a mandatory accepted, rejected or unsure, one committed line of counts and keys per disposition, the reason in the commit message, absent from the MCP, and one retention rule for both ledgers
- [ADR 0022 — The declared price](https://digline.dev/product/adr/0022-the-declared-price.md): How a suite prices an OpenAI-compatible aggregator, a gateway or a self-hosted model: `[target.pricing]` with four per-million rates, a declared price that wins over the plugin's list and enters `config_hash`, rates withheld at a named endpoint as a declared latch, and `promote --target` so a multi-target suite names what it signs
- [ADR 0024 — The judge as an instrument](https://digline.dev/product/adr/0024-the-judge-as-an-instrument.md): How the judge is measured rather than trusted: read the rendered judge prompt before re-running anything, a calibration case with a written answer whose score outside its declared band exits 2 with no baseline, `judge_samples` never reported without that result, and two readings printed without thresholds until data sizes them — the share of judged scores at exactly 0 or 1, and the run-to-run range
- [ADR 0025 — The tokens and the bill](https://digline.dev/product/adr/0025-the-tokens-and-the-bill.md): What a run cost, recorded instead of thrown away: token counts had never reached any document and the judge's spending had never reached one at all, so run-level totals now stand on every run — target and judge kept apart, because one is the thing measured and the other the instrument — with per-call counts only where the suite already records answers, a `counted` that admits when a total covers part of the run, and the totals crossing a boundary while the per-call counts do not
- [ADR 0026 — The thinking a model charged for](https://digline.dev/product/adr/0026-the-thinking-a-model-charged-for.md): The output tokens nobody reads, told apart from the ones they do: a model that reasons before it answers is billed for both, so a reply now records how many of its output tokens were thinking — a breakdown of a count already paid for, never added to it, which is the inverse of the cache-write case it will be mistaken for — with not-reported kept distinct from a measured zero, a reply claiming more thinking than output refused rather than clamped, and any unreported side making the total unreported
- [ADR 0027 — The run reconciles with what it was asked](https://digline.dev/product/adr/0027-the-run-reconciles.md): Whether a run recorded an answer to every question it asked: the driver reads its own dispatch back — a suspended case asked nothing, a calibration case its one check, every other case every assertion — and records each gap as an errored verdict naming the case and the check, so the run exits 2, cannot be promoted and says it is not a regression; with what no count can see, an exception the user's own code caught, stated rather than claimed

## Docs

- [Changelog](https://digline.dev/product/changelog.md): What changed for you in each release of digline
- [Roadmap](https://digline.dev/product/roadmap.md): What is being built next, as tracks and gates rather than dates, and the two things that will never be built

## Writing

- [Writing](https://digline.dev/blog/index.md): The posts, newest first, each with the runs behind it — written from the pipelines the author runs rather than about the library
- [The case that errors](https://digline.dev/blog/denominator-trap.md): Why a case that errors, is suspended or carries no label leaves a run-level aggregate's denominator and raises the score exactly when it was the failing case; three questions to ask of your own eval code, and what digline's compare and explain now say when the two sides counted different cases
- [Bad evals, my own](https://digline.dev/blog/bad-evals-my-own.md): Five exercises on two LLM judges the author runs, artifacts first and explanations after: a noise floor that itself moves across sixteen identical runs, prompt edits a small judge misreads, a precision measured on a censored ground truth, a suite whose class balance decides the answer, a one-case threshold that stops holding as the suite grows, and five gaps the exercise found in the tool
- [My LLM eval cried wolf](https://digline.dev/blog/my-llm-eval-cried-wolf.md): A worked example of a false alarm and the fix: why one reference score per case cannot tell a regression from a resample, how a min/max band over K=5 separates them on a public fixture, the three changes it judged the week after, and the three things it still cannot see