Every example on this page is constructed by a test, and the metadata columns
come out of a program that runs all of them. In the examples, judge is your
Judge function, claim_judge your ClaimJudge, and scorer an autoevals
scorer — see api.md for the protocols.
Choosing one¶
Do the cases carry a human mark (ground truth)?
├── yes → the gate is a verdict about the run: Precision · Recall · Accuracy · F1
│ and the per-case check under it answers "did the system agree?"
└── no → gate per case, and pick by what "correct" means here:
├── an exact string, or a fragment that must (not) be there ... Equals · Contains · NotContains · Affix
├── a shape rather than a value ............................... Regex · IsJson · JsonSchema · Length
├── close enough to an expected text .......................... Levenshtein
├── a judgement that only words can state ..................... LlmRubric → wrap in Repeated
├── supported by the retrieved context ........................ Faithfulness
├── no personal data on the way out ........................... PiiAbsent
├── which tools it called on the way there .................... ToolsCalled
├── and with what arguments ................................... ToolCalledWith
└── a scorer you already own .................................. FromAutoevals
Always, next to whichever you picked: CostBudget · LatencyBudget
What each one puts in the verdict¶
# showcase.py
"""Every assertion, run once, printing what it puts in the verdict.
The `produces` line on each card comes from here, so a card cannot describe a
key that no longer exists.
"""
from digline.core import (
F1,
Accuracy,
Affix,
CaseOutcome,
ClaimReply,
Contains,
CostBudget,
Equals,
EvaluatorInputs,
Faithfulness,
IsJson,
JsonSchema,
JudgeReply,
LatencyBudget,
Length,
Levenshtein,
LlmRubric,
NotContains,
PiiAbsent,
Precision,
Recall,
Regex,
Repeated,
Verdict,
)
TEXT = "Hello Rome 42"
def show(assertion: object, **kwargs: object) -> None:
inputs = EvaluatorInputs(output=kwargs.pop("output", TEXT), **kwargs) # type: ignore[arg-type]
verdict = assertion(inputs) # type: ignore[operator]
keys = ", ".join(verdict.score.metadata) or "—"
score = "—" if verdict.score.score is None else f"{verdict.score.score:.3f}"
print(f"{verdict.score.name:<16} {score:>7} {verdict.status:<5} {keys}")
print(f"{'check':<16} {'score':>7} {'status':<5} metadata keys")
show(Equals(), expected=TEXT)
show(Contains(needle="Rome"))
show(NotContains(needle="Milan"))
show(Affix(affix="Hello"))
show(Regex(pattern=r"\d+"))
show(IsJson(), output='{"city": "Rome"}')
show(JsonSchema(schema={"type": "object"}), output='{"city": "Rome"}')
show(Length(minimum=1, maximum=50))
show(Levenshtein(), expected="Hello Rome 43")
show(
LlmRubric(
rubric="Is it civil?",
judge=lambda p: JudgeReply(0.8, "civil"),
threshold=0.7,
tolerance=0.05,
)
)
show(
Faithfulness(
judge=lambda p: ClaimReply(supported=3, total=4, reason="one unsupported"),
threshold=0.7,
tolerance=0.05,
),
context=("Rome is in Italy",),
)
show(PiiAbsent(), output="write to a@b.com")
show(CostBudget(max_usd=0.10, tolerance=0.02), cost_usd=0.05)
show(LatencyBudget(max_ms=800.0, tolerance=0.10), latency_ms=200.0)
show(Repeated(inner=Contains(needle="Rome"), samples=3, min_agreement="2/3"))
print()
kept = Contains(needle="MATCH")
outcomes = [
CaseOutcome(case_id=f"c{i}", label=label, verdict=kept(EvaluatorInputs(output=out)))
for i, (label, out) in enumerate(
[
("positive", "MATCH"),
("positive", "MATCH"),
("positive", "MISS"),
("negative", "MATCH"),
("negative", "MISS"),
("negative", "MISS"),
]
)
]
print(f"{'aggregate':<16} {'score':>7} {'status':<5} metadata keys")
for aggregate in (
Precision(over="contains", threshold="1/2", tolerance="1/6"),
Recall(over="contains", threshold="1/2", tolerance="1/6"),
Accuracy(over="contains", threshold="1/2", tolerance="1/6"),
F1(over="contains", threshold="1/2", tolerance="1/6"),
):
verdict: Verdict = aggregate(outcomes)
keys = ", ".join(verdict.score.metadata) or "—"
score = "—" if verdict.score.score is None else f"{verdict.score.score:.3f}"
print(f"{verdict.score.name:<16} {score:>7} {verdict.status:<5} {keys}")
$ python showcase.py
check score status metadata keys
equals 1.000 pass —
contains 1.000 pass —
not_contains 1.000 pass —
starts_with 1.000 pass —
regex 1.000 pass —
is_json 1.000 pass json_kind
json_schema 1.000 pass —
length 1.000 pass length, unit, minimum, maximum
levenshtein 0.923 pass distance, length
llm_rubric 0.800 pass —
faithfulness 0.750 pass claims_total, claims_supported
pii_absent 0.000 fail pii_iban, pii_codice_fiscale, pii_partita_iva, pii_email, pii_phone_it, pii_total
cost_budget 0.667 pass cost_usd, max_usd, ratio
latency_budget 0.800 pass latency_ms, max_ms, ratio
contains 1.000 pass samples, agreement, spread, errored_samples, scores
aggregate score status metadata keys
precision 0.500 pass true_positive, false_positive, true_negative, false_negative, considered, suspended_excluded, errored_excluded, unlabelled_excluded, canary_excluded
recall 0.667 pass true_positive, false_positive, true_negative, false_negative, considered, suspended_excluded, errored_excluded, unlabelled_excluded, canary_excluded
accuracy 0.500 pass true_positive, false_positive, true_negative, false_negative, considered, suspended_excluded, errored_excluded, unlabelled_excluded, canary_excluded
f1 0.571 pass true_positive, false_positive, true_negative, false_negative, considered, suspended_excluded, errored_excluded, unlabelled_excluded, canary_excluded
Two things to read off that table. starts_with and contains appear where
Affix and Repeated were asked for: the name is the identity, Affix
names itself after the end it checks, and Repeated keeps the name of the check
it wraps — so wrapping one does not produce a new plus a missing. And every
metadata value is a number or a short label: that is what lets the counts cross
a perimeter when the text they were measured on may not.
Per case¶
Equals¶
Use it when the answer is a fixed string or a fixed structure — a classification label, an enum, a canned refusal.
Takes nothing but the defaults; compares against Case.expected.
threshold=1.0, tolerance=0.0. Accepts text, structured and conversation.
Produces 1.0 or 0.0, no metadata.
Watch out a case with no expected is error, not fail: "expected is
missing: equals cannot judge without an expected value".
Contains¶
Use it when the answer must always carry a fragment — a sign-off, a disclaimer, a required field name.
Takes needle, case_sensitive=True. Text only.
Produces 1.0 or 0.0, no metadata.
Watch out Contains("") is refused at construction: it would pass on
everything. Handed structured output it is error, never a silent str().
NotContains¶
Use it when something must never appear: a competitor's name, an internal code, the phrase legal struck out.
Takes needle, case_sensitive=True. Text only.
Produces 1.0 or 0.0, no metadata.
Watch out the negation lives in the assertion, not in a threshold you invert
— that keeps the reason readable in the report. NotContains("") is refused: it
would fail on everything.
Affix¶
Use it when position matters and presence does not: a reply that must open
with a greeting, a JSON payload that must end with }.
Takes affix, at="start"|"end", case_sensitive=True. Text only.
Produces 1.0 or 0.0, no metadata. Names itself starts_with or
ends_with.
Watch out Affix("") is refused: every string starts and ends with it.
Regex¶
Use it when the shape is describable and the value is not — an order number, a date format, a citation marker.
Takes pattern. Text only.
Produces 1.0 or 0.0, no metadata.
Watch out a pattern is not a parser. If you are matching braces or nesting,
the question is IsJson or JsonSchema.
IsJson¶
Use it when the only question is whether it parses — the first check on any tool-calling or structured-output path.
Takes top_level="any"|"object"|"array". Text only.
Produces 1.0 or 0.0, and json_kind — object, array, invalid, and
so on, which is what tells you how it was wrong.
Watch out it says nothing about the contents. Pair it with JsonSchema.
JsonSchema¶
Use it when the structure has rules: required keys, types, enumerations.
JsonSchema(
schema={
"type": "object",
"required": ["intent", "confidence"],
"properties": {"confidence": {"type": "number"}},
}
)
Takes schema (a JSON Schema document). Text and structured.
Produces 1.0 or 0.0, no metadata.
Watch out it validates, it does not parse-and-explain: for "did the model
even return JSON" the answer is IsJson, and the two are different questions on
purpose. This is the one assertion with a runtime dependency (jsonschema), for
the reason in ADR 0001: a half-written validator is a vacuously green check.
Length¶
Use it when answers are drifting longer, or must fit a channel — an SMS, a card, a summary field.
Takes minimum, maximum (at least one), unit="characters"|"words".
Text only.
Produces 1.0 or 0.0, and length, unit, minimum, maximum — so the
report says how far out it was, not merely that it was.
Watch out Length() with neither bound is refused: it passes on everything.
Levenshtein¶
Use it when you have an expected text and "nearly right" is a real answer — a translation, a normalisation, a rewrite.
Takes nothing required; compares against Case.expected. threshold=0.9 by
default. Text only.
Produces a graded score, plus distance and length.
Watch out graded is the whole reason to have it — a binary near-match is
just Equals with extra steps. No expected is error.
LlmRubric¶
Use it when the criterion is a judgement that only words can state: is it polite, does it stay on policy, does it answer this question.
LlmRubric(
rubric="Answers the question without inventing policy",
judge=judge,
threshold=0.8,
tolerance=0.05,
)
Takes rubric, judge, and threshold and tolerance with no
defaults — an LLM judge is not reproducible, and an implicit tolerance over a
noisy value is a green light nobody gave. Text and conversation.
Produces the judge's score; no metadata of its own (the judge's reason
becomes Verdict.reason, which is payload and does not travel).
Watch out three things are error, never fail: a judge that raises, a
score outside [0, 1], and a reply with no reason. And a bare LlmRubric is
almost always the wrong shape — see Repeated, and chapter 2 of the
guide.
Faithfulness¶
Use it when the system retrieves and then answers, and the question is whether the answer stayed inside what was retrieved.
Takes judge (a ClaimJudge), threshold, tolerance. Reads
EvaluatorInputs.context. Text only.
Produces a graded score, plus claims_total and claims_supported.
Watch out the judge returns two counts and the core divides. A model
asked for a ratio returns a number nobody can check; asked for counts it returns
something arithmetic can contradict. No claims found at all is error, not
1.0.
FromAutoevals¶
Use it when you already have an autoevals scorer and want it under a
baseline rather than under a threshold.
Takes scorer, threshold, tolerance. Text only.
Produces whatever the scorer's Score carried: its name, its value, its
metadata, passed through.
Watch out it needs the autoevals package, which digline does not depend
on. The adapter exists so migrating in costs one line, not a rewrite.
PiiAbsent¶
Use it when the output reaches a person, or leaves the perimeter, or is about to be pasted into a ticket.
Takes patterns, defaulting to the Italian set: IBAN, codice fiscale,
partita IVA, email, phone — the first three checksum-verified, so a number that
merely looks like an IBAN is not one. Text only.
Produces 1.0 or 0.0, plus a count per pattern and pii_total.
Watch out binary on purpose: "a bit of PII" is not a degree of quality, and
a graded score would invite a threshold meaning "some leakage is fine". The
counts travel and the text found never does — the number of matches crosses a
boundary, the matched string is not even put in the metadata.
CostBudget¶
Use it when always. A budget is a ceiling that fails the run, not a metric on a dashboard.
Takes max_usd, tolerance; threshold=0.5. All output kinds.
Produces a graded score — 1.0 at zero spend, falling as the ceiling
approaches — plus cost_usd, max_usd, ratio.
Watch out graded rather than binary is the point: a cost that creeps from
40% to 80% of budget is visible long before it breaches. tolerance has no
default because cost is noisy by nature — tokens, retries, a slow provider — and
an implicit zero would call ordinary noise a regression every run.
LatencyBudget¶
Use it when always, alongside the cost. The same reasoning, in milliseconds.
Takes max_ms, tolerance; threshold=0.5. All output kinds.
Produces a graded score plus latency_ms, max_ms, ratio.
Watch out measure what you can control. If your target includes a network
you do not own, tolerance is doing more work than the threshold.
ToolsCalled¶
Use it when the target is an agent and the answer being right is not the whole question. A model that was supposed to look something up and answered from memory produces a well-formed answer that happens to be invented, and every other assertion here is blind to it.
Takes expected — the tool names, in the order they should be called;
threshold=1.0. All output kinds. Binary: the sequence matches or it does not.
Produces 1.0 or 0.0, plus tool_calls (the count) and called (the
names). The count crosses a boundary on its own merit; the names are strings
and need Disclosure(score_metadata={"called"}) to travel in a redacted
document. called can hold null, at the position of a call the provider
reported without naming its tool: ["search", null, "cite"]. It means this
call was not named. It is never a name, and never a shorter list either,
because dropping the hole would move every later name to a position it did
not hold. Do not str() an entry, and do not take the list's length as the
number of named calls. A suite that discloses called meets this null in
--json exactly as written.
Watch out it needs a target that reports its tool calls — a provider
plugin on a provider that names them. Anywhere else it is an error, not a
failure: nobody reported, so what the model called is not knowable, and
"called nothing" would be a finding nobody established. It errors too when a
provider contradicts itself, ending the turn on a tool call and then naming
none. A call whose tool nobody named never passes: it fails where the reply
settles the mismatch without it (a different number of calls, or a named call
out of place), and errors where every named call matches, because then the
missing name is what decides.
ToolCalledWith¶
Use it when the trajectory is right and you need to know it asked the right
question. An agent that called lookup with the wrong identifier called exactly
the tools it was supposed to, in exactly the order it was supposed to, and
answered about somebody else — which ToolsCalled cannot see and no assertion
on the text can reach.
Takes tool — the name, and arguments — the mapping that must
have been sent; match="exact" (the same set of keys) or "subset" (every
declared key present and equal, others ignored); threshold=1.0. All output
kinds. Binary.
Produces 1.0 or 0.0, plus arguments_matched and arguments_expected.
Both are counts, so both cross a boundary on their own merit — no argument
value ever enters the verdict, because an argument is the end company's data
by construction and Score.metadata is projected onto the wire.
Watch out it does not check the order: ToolsCalled owns that, and where
the model called the same tool more than once, any one matching call satisfies
this. Use subset when a framework adds keys of its own — under exact a
request_id nobody declared is a failure. Arguments that do not decode are an
error, not a failure, for the reason json_schema reports an undecodable
output as one.
Repeated¶
Use it when the check underneath it is a judge. This is judge noise made into a measurement instead of a surprise.
Repeated(
inner=LlmRubric(
rubric="Stays on policy", judge=judge, threshold=0.8, tolerance=0.10
),
samples=5,
min_agreement="3/5",
)
Takes inner, samples, min_agreement (a fraction). Threshold and
tolerance come from inner, and so does the name.
Produces the folded score, plus samples, agreement, spread,
errored_samples, and scores — the raw votes, which is the column digline
view shows under the combined number.
Watch out below the agreement floor the verdict is error, never a
pass: "the samples did not agree: 0.67 of them share the majority verdict,
below the required 1.00". min_agreement must be a fraction samples can
actually produce — 0.67 with three samples is refused at construction with the
list of the ones that exist. And this is the judge wobbling; for the target
answering differently, the answer is Suite(samples=…).
Per run¶
All four take over — the name of the one per-case assertion whose verdicts get
counted — plus a mandatory threshold and tolerance, both readable as
fractions. All four produce the same metadata: the four cells of the confusion
matrix, considered, and the exclusions of What is left out of a
denominator, below. All four are error, never 1.0, on an empty
denominator.
Every case must carry a label the moment one of these is declared, and Suite
refuses over that names no assertion or two. Where to put the threshold:
where the system measurably is — chapter 5 of the guide.
All four also take by_group. Set it and the whole-run figure stays, with
one aggregate added per class the cases declare — precision[group=refunds]
beside precision — each inheriting the same threshold, the same tolerance and
the same noise floor, computed over that class's cases. Every class or none:
there is no Precision(group="x"), because the class that degrades is the one
you were not watching. It costs a group on the cases and nothing else
(ADR 0010).
Precision¶
Use it when false positives are what your users see: the item that should not have been kept, the ticket that should not have been escalated.
TP / (TP + FP). Rises when the system gets choosier — including when it gets
choosier by keeping almost nothing, which is what Recall is for.
by_group — false positives rarely spread evenly. One class the system is
too eager about can sit under a whole-run figure that clears its bar, and this
is what makes it a gate rather than a footnote.
Recall¶
Use it when what is missed is what hurts: the fraud not flagged, the outage not escalated.
TP / (TP + FN). Rises when the system gets more generous, and a system that
keeps everything scores 1.0 — which is what Precision is for.
by_group — what is missed is missed unevenly, and a class the system never
catches is invisible in a run-level recall that the other classes carry.
Accuracy¶
Use it when the two classes matter equally and are roughly balanced.
(TP + TN) / considered. Watch out on a skewed suite it flatters: with one
positive in twenty, answering "no" to everything scores 0.95.
by_group — the sharpest of the four here, because the flattery is exactly
what a class boundary undoes: a group cannot be carried by another group's
balance. The classifier example ships four red per-class figures under two
green whole-run ones, which is the case this parameter exists for.
F1¶
Use it when you need one number and precision and recall are trading against each other — a stricter prompt keeps fewer items and gets more of them right, raising one and lowering the other.
2TP / (2TP + FP + FN). Written that way and not as 2PR / (P + R): the same
number with one denominator to check instead of three, and no decision to make
about what F1 means once precision has already gone to error.
by_group — one number per class, with the same warning as the whole-run
one: it says a class moved, never which of the two halves moved it. Read
precision[group=…] and recall[group=…] beside it when it does.
What is left out of a denominator¶
Five exclusions, and every one of them is counted in the verdict's metadata so that a number can be reconciled with the case file:
suspended_excluded |
somebody set the case aside; it never ran |
errored_excluded |
the check could not be judged |
unlabelled_excluded |
the case carries no human mark to count against |
canary_excluded |
the case watches the model rather than measuring it — see api.md |
calibration_excluded |
the case calibrates the judge on a fixed answer — see api.md |
suspended_excluded is the one figure here that can be improved by doing
less work, which is why it is never printed on its own. The clause naming the
canary appears only when there is one: rendered always, it would have rewritten
the recorded reason of every aggregate verdict in every baseline already
committed. The calibration clause, and the calibration_excluded key itself,
appear only when there is one too, for the same reason.
Targets¶
Not a metric, but the other half of what a suite declares: the thing the
assertions are pointed at. A target is any callable (Case) -> Response, and
most are ten lines of your own. digline.targets exists for the case that is
always the same shape — a prompt file, a model, a bill.
ProviderTarget¶
Use it when the system under test is a prompt sent to a provider, which is the ordinary case.
AnthropicTarget(
prompt_file=Path(__file__).parent / "prompts/answer.md",
system_file=Path(__file__).parent / "prompts/system.md",
model="claude-sonnet-5",
max_tokens=1024,
)
Takes prompt_file, model, and whatever the plugin needs;
system or system_file, never both. Produces a Response with the
composed prompt in input, the cost in cost_usd, the measured duration in
latency_ms, and the model and token counts in metadata.
Watch out both files are recorded in every run, so the baseline carries the
prompt that produced it — which is the point, and which means a redacted run
withholds them unless the suite says Disclosure(artifacts=True).
A declared file outside the perimeter is refused when the run is made; one under
.digline or .git is refused by the command that would let it cross. Neither is
dropped in silence.
HttpTarget¶
Use it when the application is not Python, or is not importable — a JVM service, a Go binary, something behind a gateway.
HttpTarget(
"http://localhost:8080/classify",
request=lambda case: {"text": case.vars["text"]},
output_path="data",
cost_path="usage.cost_usd",
latency_from_response="usage.elapsed_ms",
)
Takes a URL, a callable that builds the body from the case, and dotted paths
to the answer, the cost and the time the service reports. Produces a
Response like any other target: what made it is not digline's business.
Watch out preflight asks whether anything is listening before the first
case, so a service that is down fails once with a sentence rather than once per
case with a stack trace. Without latency_from_response the round trip is
measured here, network included — a different number measuring a different
thing. urllib only: for retries, pooling or auth, pass your own callable.
PromptTemplate¶
Use it when you want the prompt in a file rather than in a Python string — which you do, because a file is diffable and a run records it.
Takes a path, read at construction. Produces text, sha, variables,
and render(vars).
Watch out substitution is {identifier} by regex, never str.format, so
JSON in a prompt survives. Values render deterministically or are refused: an
object whose str() carries a memory address would give two runs two prompts
and no record of why.
Pricing¶
Use it when the plugin's list is stale, which it will be.
Takes USD per million tokens. Produces cost(model, usage) -> float.
Watch out an unknown model raises. A model priced at zero passes every
CostBudget there is, and does it quietly.
See also¶
guide.md— how these get calibrated, in the order the problems arriveapi.md— the full parameter reference, the judge protocols, custom assertionsadr/0001-verdict-not-score.md— why a verdict has three states