What it reads back: what ran, what moved and by how much, inside or outside
which measured interval, what was set aside, what could not be judged, and
which of the three configurations differed underneath it all. digline report
compresses the same facts the other way, into a document with a customer's
answer in a box, and its whole craft is deciding what fits on the first screen.
It states, and never advises. There is no "you should re-run this", no
"this looks promotable". That boundary is deliberate and is the reason the
command has a decision record of its own,
ADR 0012: whether a red run is a regression or a
wobble, and which run deserves to be the reference, is judgment digline leaves
to a person. Where you want the counsel it is in
AGENTS.md, written
for the purpose and marked as judgment.
Using it¶
One run key, and the presence of a baseline decides the rest:
The numbers below are an example of the shape. No scenario in this repository produces them, so nothing here replays this block.
$ digline explain --suite suite.py --run latest
What ran
5 cases ran.
10 checks ran.
Every case could be judged.
No case is suspended.
The suite changed since the reference, so these numbers compare different rules.
llm_rubric: 0.0% of 15 judged scores at 0 or 1, against 20.0% of 15 in the reference.
What differed underneath
prompts/system.txt changed: +1 −0 lines.
llm_rubric.threshold moved from 0.7 to 0.6, which is a looser bar than the reference was approved under.
The agreement floor is not recorded in a run, so whether it moved cannot be told from these two.
What moved
gift-wrap · llm_rubric got worse: 1.000000 to 0.700000, a drop of 0.300000.
opening-hours · levenshtein got worse: 1.000000 to 0.636364, a drop of 0.363636.
The shape line is there for every judged check, and it reports one thing: what fraction of the judged scores sat at the ends of the scale, here and in the reference. It never says which fraction is better — a rubric that has stopped grading and started voting is something a reader has to weigh, not a verdict (ADR 0024 §6.3). The reading itself is set out under Shape, below.
A comparison states no threshold, and the report is why: its case tables are
about movement against a reference and print no bars, so a reading of one
prints none either. The number is on the fact and in --json, and the run
document shows score / threshold on every row. That rule has a name — every
value this command prints appears in the same run's report — and it is a test,
not a convention.
With no baseline it reads the run alone, and says so by what it does not say — no "reference", no "got worse", no comparison of any kind:
The numbers below are an example of the shape. No scenario in this repository produces them, so nothing here replays this block.
$ digline explain --suite suite.py --run latest
What ran
5 cases ran.
10 checks ran.
Every case could be judged.
No case is suspended.
How it was set up
prompts/system.txt was under test.
The system under test was configured with model claude-haiku-4-5.
What it found
opening-hours · levenshtein is under its bar at 0.636364. The bar is 0.750000.
There is no mode flag, and there is not going to be one. Making you name the scope means making you know, before you type, which of two documents you are entitled to — and the first person to hit that is always somebody on their first run, who has no baseline and does not yet know what one is.
The exit code¶
explain gates, exactly as report does: 0 fine, 1 got worse, 2
could not be judged. It is reading the same comparison, against the same
approved reference, that compare gates on.
With no baseline it can never exit 1 — "worse" is a relation and there is
nothing here to be worse than — but 2 survives, because a case the suite
could not judge is a fact about the harness rather than about a reference.
This is the opposite of digline diff, which always exits 0, and
the difference is the reference: a verdict exists only against one that a
person approved.
A run that does not reconcile¶
Before any aggregate is computed, the driver checks that every question the suite put to the run came back as exactly one verdict. A suspended case is asked nothing, a calibration case is asked its one check, and every other case is asked every assertion. Where the answer is missing, or is an answer nobody asked for, the run records an errored verdict in its place and names the case and the check. The reading then opens with it:
1 check does not reconcile with what the suite asked. This is not a regression: what the run measured is not known. It is named below, among the checks that could not be judged.
It exits 2 and cannot be promoted, and it is not a regression.
ADR 0027 says why,
and what it cannot see: a target that catches its own exception and returns an
empty answer is a case that was scored, and only a check on content can tell.
A verdict named otherwise than its assertion¶
A verdict carries the name of the assertion that produced it. Every shipped
assertion already does. A third-party one that names its verdict otherwise,
for example from the answer (named_Rossi), would put that name into every
document, redacted ones included, because a verdict's name always crosses. So
the driver compares each verdict's name with the one the assertion declared
before the first case, and refuses the mismatch. That check's verdict on that
case becomes an errored verdict under the declared name, marked misnamed. The
case's other checks stand. The reading says so directly before it counts the
cases that could not be judged:
1 check returned a verdict under a name that is not its assertion's, and it was refused. The assertion is at fault, not the system under test. It is named below, among the checks that could not be judged.
The marker survives redaction, which the reason does not. So a redacted
document still tells this apart from an assertion that raised. It exits 2
and cannot be promoted, like every errored verdict.
ADR 0027 §6, as amended, says why, and what it
does not close: an assertion declared with a name taken from client data
matches its verdicts, and passes.
--json: the facts, not the prose¶
emits the fact list the prose is rendered from — typed facts with case and assertion references — and no sentences at all:
The numbers below are an example of the shape. No scenario in this repository produces them, so nothing here replays this block.
{
"baseline_key": "2026-10-21T07-41-16-574186-00-00-8d81ff7740eae5b6",
"exit_code": 1,
"facts": [
{
"about": "run",
"count": 5,
"kind": "cases",
"state": null
},
{
"about": "run",
"count": 10,
"kind": "checks",
"state": null
},
{
"about": "run",
"count": 0,
"kind": "unjudged",
"state": null
},
{
"about": "run",
"count": 0,
"kind": "suspended",
"state": null
},
{
"about": "run",
"count": 0,
"kind": "suite_config",
"state": true
},
{
"about": "run",
"count": 0,
"kind": "shape",
"shape": {
"assertion_id": "4e7ba5d3e226025c",
"check": "llm_rubric",
"reference": {
"claims_unrecorded": 0,
"extremes": 3,
"sample_means": 0,
"scores": 15,
"single_claim": 0
},
"run": {
"claims_unrecorded": 0,
"extremes": 0,
"sample_means": 0,
"scores": 15,
"single_claim": 0
}
},
"state": null
},
{
"about": "setting",
"added": 1,
"after": null,
"before": null,
"kind": "artifact",
"name": "prompts/system.txt",
"outcome": "changed",
"removed": 0,
"withheld": false
},
{
"about": "setting",
"added": 0,
"after": 0.6,
"before": 0.7,
"direction": "loosened",
"kind": "rule",
"name": "llm_rubric.threshold",
"outcome": "changed",
"removed": 0,
"withheld": false
},
{
"about": "setting",
"added": 0,
"after": null,
"before": null,
"kind": "rule",
"name": "min_agreement",
"outcome": "unknown",
"removed": 0,
"withheld": false
},
{
"about": "check",
"after": 0.7,
"assertion": "llm_rubric",
"assertion_id": "4e7ba5d3e226025c",
"before": 1.0,
"case_id": "gift-wrap",
"delta": -0.3,
"denominator_moved": false,
"kind": "regressed",
"noise_judged": [],
"noise_max": null,
"noise_min": null,
"noise_samples": 0,
"noise_sd": null,
"noise_source": "",
"noise_width": null,
"scope": "case",
"threshold": 0.6
},
{
"about": "check",
"after": 0.636364,
"assertion": "levenshtein",
"assertion_id": "9831ae8fdaf2e736",
"before": 1.0,
"case_id": "opening-hours",
"delta": -0.363636,
"denominator_moved": false,
"kind": "regressed",
"noise_judged": [],
"noise_max": null,
"noise_min": null,
"noise_samples": 0,
"noise_sd": null,
"noise_source": "",
"noise_width": null,
"scope": "case",
"threshold": 0.5
}
],
"note": "",
"output_version": 2,
"run_key": "2026-10-21T07-41-16-853431-00-00-c9092e23876f6e60",
"scope": "comparison"
}
note is what resolving latest stepped over, the line printed on stderr, and
empty here because nothing was. It is never a fact about the run: it says which
run was read, and why that may not be the newest one the store remembers.
run_key is the run that was read, and baseline_key the reference it was held
against. Both come from the documents read, not from what you typed, so they
name the run latest picked and confirm a key typed by hand. baseline_key is
there only in the comparison scope: in the run scope no reference was read.
compare --json carries run_key beside its baseline_key for the same reason.
Three shapes, and you discriminate on about first and then on kind: two of
them have a kind called within_noise and they mean different things — one
check that moved inside its interval, and the count of every check that did.
"kind": "rule" is the odd setting, and the discriminator is what keeps it
honest. The other three settings are things under test — how the system was
configured, which instrument graded, which file was the subject. A rule is the
bar they were held to, and it carries direction: "loosened" where this run
is held to less than the reference was approved under, "tightened" where it is
held to more. The key is absent for two different reasons, and neither is
unchanged (ADR 0028 §4). One row moved and has no direction to give: a
samples row on a run-level aggregate, because an aggregate's floor is still
the range of its own samples and nothing here can weigh them — a case check's
samples does carry a direction, since more samples steadies the score and
narrows the floor. The others are not known to have moved at all: a threshold, a
tolerance, a samples or a width comes through as unknown where one side's
own cases do not settle on a single value, by disagreeing or, for samples, by
erroring before it was counted. min_agreement is always one of those, because
a run does not record it; its row is there only where the configuration's
fingerprint moved, and it says that whether the floor changed cannot be read
from these two documents. withheld is always false here: nothing a rule is
made of can be kept back, which is why this is the one part of the reading that
is complete at a boundary.
scope is "comparison" or "run" and is stated rather than left to be
inferred: a run whose comparison found nothing and a run with no reference at
all produce different readings, and the difference must not be something you
deduce from an absence.
exit_code is the number the process exits with, carried as a field for the
same reason it is on compare --json.
Shape: judged scores at the extremes¶
Against a reference, the reading adds one line per judged check — a check whose
class declares KIND = "judged", written into the run as "judged": true:
It reads raw per-sample scores, never a folded mean, over the cases that
count: not a canary, not a calibration case, not a verdict that errored. A
Faithfulness verdict with a single claim can only score 0 or 1, so it is left
out, and the line says how many were. At samples > 1 the run keeps only the
mean claim count, so a sampled verdict averaging two or more is read, and the
line says its per-sample counts were not recorded. Against a reference written
before 0.14.0, which carries no "judged" key, the line says there is nothing to
set beside the share.
Verdicts whose samples are means are left out and counted. A judged check
in Repeated in a sampled suite, a nested Repeated at any sample count, and
a --judge-samples replay of a Repeated check all store the means of
their judgements where the judgements would be. Read as judgements, a judge that
alternates 0 and 1 would show 0% at the extremes, the opposite of what it is.
Since 0.15.0 such a verdict carries "sample_means": true. The line leaves it
out, and says how many were left out and that their per-judgement scores were
not recorded. On the reference side, a sampled verdict with no stamp is left out
as well wherever this run stamped the same check, because a reference written by
0.14.x holds these folds unstamped. When a document cannot say what an absence
means, the rule errs toward leaving a verdict out, never toward misreading one.
The one case this cannot reach. A run that stamps nothing for the check,
such as the same Repeated check now at samples=1, against a reference
promoted before 0.15.0 at a different sample count. Nothing in the run points
at the check, and nothing in the reference says what its samples are, so its
share may be means read as judgements. config_changed is true for that
comparison. Re-promote the reference to read it. ADR 0024 §6.5.
It says nothing about which share is larger, and that is not an omission. More than the reference needs a threshold, and on a suite of twenty cases one verdict moves a share by five points. The threshold is sized on data and added by a dated amendment to ADR 0024 §6.3. Until then this is a measurement, not a verdict. It is never in the headline and never an exit code: the calibration case is the gate, and shape is the diagnosis.
In --json it is a fact with "kind": "shape", carrying its counts under
shape — check, assertion_id, and run and reference, each with
extremes, scores, single_claim and claims_unrecorded (reference is
null where there is none). compare --json full carries the same list as
shape.
A check whose class declares no KIND is not read, and digline run names it.
An autoevals scorer wrapped in FromAutoevals is not read either, and is not
named — the known hole, described in the
API reference.
What it will not tell you¶
The judge's words. No fact carries a reason — the field does not exist on
the types, which is what makes the payload boundary something no later edit can
open by accident. For the judge's words, digline report is one command
away, and it renders them in a Reason column.
The same rule takes the stated reason a case was suspended: that a case was set aside travels, because it is a fact about coverage; why somebody set it aside does not, because a developer writes things like "fails on the Rossi account".
Anything about a second run. A reading is of one run and its reference, so it will never tell you that a drop "did not repeat" or that a check is "drifting". Those need a cycle of runs, and the thing that runs cycles is the operator loop — whose alert keeps its own layers for exactly that, and marks the one a model wrote as an opinion rather than as digline's verdict.
--locale¶
en or it, defaulting to en. Terminal output is for you, so it follows the
runtime rule; a document — which is what digline report --locale writes —
has a recipient who did not choose English, and takes the flag as mandatory.
There is no --out, and its absence is what keeps that distinction honest
rather than an oversight: the moment a reading is written to a file it has a
recipient, and the locale would have to become mandatory too.