ADR 0007 — The declarative suite format

• Status: accepted — shipping next. The document lands first and the code starts after it, in core 0.5.0 with a release of the three plugins, the way ADR 0006 was written before the branch that implemented it
• Date: 2026-09-03
• Refines: fixed decision 6 (CLAUDE.md), "providers as plugins (entry points)" — stated since the first commit and not yet real anywhere in the tree. This is the decision that needs it, and therefore builds it
• Assumes: ADR 0001 (an assertion is a pure function and the core decides the verdict), ADR 0002 §2 (the payload stays where it is born) and §9 (how a suite is loaded), ADR 0003 (declared artifacts), ADR 0004 §1 (a plugin ships a target and both judges) and §5 (no plugin reads the environment), ADR 0005 §4 (a judge's identity is provider/model) and §8 (an HTTP target reports its configuration in the answer), ADR 0006 (the verdict this format inherits without touching)
• Touches: fixed decision 3 (no vacuously green assertion), fixed decision 6 (providers as plugins), fixed decision 9 (what crosses a boundary)

Context

The README says it in three words — "Not YAML" — and the reasoning under them is right: a Judge is an object, a Target is a function, a Disclosure is declared in code by construction, and none of that survives a configuration file without reinventing a language. cli/loader.py opens with the same sentence and the CLI acts on it: it imports the suite, it does not interpret it.

That is not what this ADR reverses. What it addresses is who is left outside.

The CI-gate reader has no Python. Since 0.4.0 the official container image runs the whole cycle with nothing installed on the machine — but the suite it mounts is still a .py file, so the last thing standing between a Java shop with an HttpTarget endpoint and a working gate is a language they do not write. The image completed the tooling half of "no Python anywhere" and left the authoring half exactly where it was.

World 2 has no landing format. ADR 0002 designs the bridge from production failures back to committed cases. A failure that becomes a case has to land as something, and it will be written by a machine and reviewed by a person. A Python file generated by a program is a diff nobody can review; a data file with stable ids is one they can.

And a suite is not, in fact, mostly code. Of the sixteen assertions the package exports, twelve are constructible from data today, with no new parameter and no new class: Equals, Contains, NotContains, Affix, IsJson, Length, Levenshtein, Regex, JsonSchema, PiiAbsent, CostBudget, LatencyBudget. All four aggregates — Precision, Recall, Accuracy, F1 — take over, threshold, tolerance and a name, and nothing else. What is genuinely code is a short list: the judge, the target, FromAutoevals' scorer, and anything a user wrote themselves.

So the shape of the answer is not "a configuration language that can express Python". It is: the things that already have coordinates get named by their coordinates, and the things that do not stay in suite.py — said out loud, in the format's own error messages, rather than discovered.

TOML, not YAML, and not only because the README says so. tomllib is in the standard library from 3.11, so the format costs no runtime dependency — this package has exactly one and intends to keep it. YAML would cost a second, to parse a language whose surprises (no is False, 1.0 is a string in one implementation and a float in another) are a poor trade for block scalars nothing here needs.

Decision

1. Assertions are one ordered list, and the type is a value

[[assertions]]
type = "contains"
needle = "Northwind Support"

[[assertions]]
type = "llm_rubric"
rubric = "Does the reply answer the question in at most three sentences?"
judge = "anthropic/claude-haiku-4-5"
threshold = 0.7
tolerance = 0.05

type selects the class. Every other key is a constructor keyword argument, handed to it with no interpretation of what it means: needle is needle because Contains says so, and this format has no opinion about it. A parameter added to an assertion in the core is available here the day it is added, and a parameter this document does not mention is not a parameter this format refuses.

The list is ordered and the order is kept. It is report order — the sequence a reader meets the checks in, in the terminal and in the HTML — and a format that silently reordered it would be rewriting a document somebody arranged.

The type token is the name the check already carries in every report and every baseline: contains, llm_rubric, cost_budget, precision. So a reader who has seen the output can write the file, which is the cheapest documentation there is. Two classes derive their name rather than declaring one — Affix becomes starts_with or ends_with depending on at, and Repeated takes the name of what it wraps — and for those two the token is the class, affix and repeated, while the resulting check keeps the name it derives.

Three details fall out of the code as it stands, and are decided here rather than met later:

A key whose field is a set is written as an array. accepts is the only one today. It is read as a TOML array and constructed as the frozenset the field declares — a coercion driven by the field's declared type, applied uniformly, never a rule about that key in particular. It matters more than it looks: config_hash fingerprints an assertion's field values through canonical(), and a list where the Python form has a frozenset is a different fingerprint for the same suite. §9 is why that is not allowed to happen.

Repeated nests, and nesting is the same rule applied again.

[[assertions]]
type = "repeated"
samples = 3
min_agreement = "2/3"

  [assertions.inner]
  type = "llm_rubric"
  rubric = "…"
  judge = "anthropic/claude-haiku-4-5"
  threshold = 0.7
  tolerance = 0.05

inner is a constructor argument whose value is an assertion, so it is built by the rule above and passed. This is the one place where "passed through with no interpretation" needed a precise reading, and the reading is that the loader still interprets nothing: it dispatches, recursively. The alternative — leaving Repeated out — would put judge noise, which ADR 0006 exists for, out of reach of every suite that is data. That is not a boundary anybody would defend; it is an omission.

PiiAbsent gets its default patterns and no others. A custom PiiPattern is an object with a checksum function in it. Named coordinates would be a plugin registry for regexes, which nothing has asked for; until something does, a custom pattern is code, and §6's error says so by name.

2. Aggregates are in the same list, and over still names an assertion

Precision, Recall, Accuracy and F1 are written as entries of the same [[assertions]] list, with their own types. The loader knows which types are per-run and puts them where Suite keeps them — run_assertions, which is a separate field for reasons that have nothing to do with authoring and everything to do with the driver evaluating them at a different level.

One list because the alternative asks the author to know an internal distinction before they can write a file. Two lists would be the object model leaking into the format, and the object model is not the audience.

[[assertions]]
type = "precision"
over = "llm_rubric"
threshold = "9/10"
tolerance = "1/10"

over names a check's name, not a type and not a position, exactly as it does in Python — and Suite.__post_init__ already refuses an over that matches nothing, and refuses one that matches two. The second refusal is the one this format makes easier to trip: two contains in one suite is the ordinary case, and in TOML it is four lines. The rule does not change, the error does not change, and the fix is the one that was always right — give one of them a name.

Ratios stay ratios. threshold = "9/10" is read by as_agreement, the same function that already accepts "2/3" for min_agreement, and a float no k/n can produce is refused at construction. A number that is really a count of cases is written as one.

3. A judge is named by coordinates, and this is where entry points become real

judge = "anthropic/claude-haiku-4-5"

provider/model. Not a new naming scheme: it is exactly the identity ADR 0005 §4 already records for every judge that grades a run, and provider is already a class attribute on both halves of every plugin (provider = "anthropic"). A reader who has opened a run file has already read this string.

Resolution goes through entry points, which is fixed decision 6 — "providers as plugins (entry points), not vendored into the repo" — becoming true for the first time. It has been stated since the first commit and implemented nowhere: there is no [project.entry-points] table in any of the three plugins today, and no importlib.metadata call anywhere under src/. Each plugin registers its provider name; a registry in digline.targets maps the name to the factories the plugin ships.

Resolved by name at runtime, never by import. tests/test_layering.py forbids anything under src/ from importing anything under packages/, and that gate is not weakened for this: the registry reads what is installed, it does not know what exists. A provider that is not installed is a load error naming the package to install — judge = "anthropic/…" with no digline-anthropic in the environment is the most predictable mistake this format has, and it deserves the sentence rather than a KeyError.

One coordinate resolves to both judges. ADR 0004 §1 makes every plugin ship a Judge and a ClaimJudge, so LlmRubric is given the first and Faithfulness the second from the same string. Choosing a provider stays one decision, which is the whole point of that ADR.

No credential appears in a suite file, ever. There is no api_key key, and its absence is not an oversight to be fixed by a later revision: ADR 0004 §5 resolves keys through the SDK's own environment lookup precisely so that no digline object holds one, and a suite is a file that gets committed. A credential in a committed data file is the one payload no Disclosure can release.

A custom judge has no coordinates, and stays in suite.py. A BriefJudge-shaped object — a judge with domain rules of its own, the kind that exists in the fixtures — is not a provider and not a model, and nothing about it can be named in eleven characters. Said here, and said again by the loader when a TOML suite asks for one, because a boundary a user discovers by guessing is a boundary that gets called a bug.

4. Cases are a file reference, always

[suite]
cases = "cases.json"

One form. No inline cases and no dual form, and the reason is not tidiness:

• A suite is rules and cases are data, and they move at different rhythms, usually by different hands. A rule changes when somebody decides the bar moved; a case is added every time production produces a new one. Bundled, every case addition rewrites the file the thresholds live in.
• The diff has to say which one moved. A pull request that changes a threshold and a pull request that adds forty cases are two different reviews. In one file they are one blob.
• It is what world 2's bridge will write. Machine-generated cases arriving into a file whose other half is hand-written rules is a merge conflict per incident.

The cost, stated rather than buried: even a two-case toy is two files. That is worse than the Python form, where the quickstart is one file that runs. It is accepted, and the quickstart in the TOML form will be two files from its first line so that nobody meets the split later as a surprise.

And this ADR is where cases.json becomes a format. It has been called one in conversation and it is not one yet: five examples each read their own file with json.loads and build Case objects by hand, in two incompatible shapes — classifier writes {id, vars, expected, label, metadata}, which is Case-shaped, while langchain4j and external-app write {id, question} and map question into vars in the suite. The first shape is the format: a JSON array of objects whose keys are Case's fields, with id mandatory. The second is an application-specific mapping, and a mapping step is code — those examples keep their suite.py, and nothing about them breaks.

5. The target has two forms, and there is no third

[target]
type = "provider"
provider = "anthropic/claude-haiku-4-5"
prompt_file = "prompt.md"
max_tokens = 500
temperature = 0.0

[target]
type = "http"
url = "http://localhost:8080/answer"
output_path = "data.answer"
cost_path = "usage.cost_usd"
latency_from_response = "usage.elapsed_ms"
config_path = "config"
timeout = 30.0

type = "provider" carries the coordinate of §3 plus only the parameters the plugin already exposes as declarative configuration — prompt_file, model (inside the coordinate), system, system_file, temperature, max_tokens, prefill and their peers per provider. Not client, not pricing: those are objects, and a plugin's injection points are not configuration.

type = "http" is HttpTarget's constructor, and here the mapping is literal for every parameter but one — url, output_path, cost_path, latency_from_response, config_path, headers, timeout are already data, and cost_path and config_path are ADR 0005 §8 exactly as it shipped.

The exception is request, and it is a real one. HttpTarget.request is Callable[[Case], Mapping[str, object]], and it is a callable on purpose: both examples that use it carry the same comment — "a callable rather than a template, because a real payload has shapes a template cannot" — which is a decision this ADR has no business quietly reversing. So the data form is not a template and does not become one. It is a body whose leaves name case fields:

  [target.body]
  question = "case.vars.question"
  tenant_id = "case.vars.tenant_id"

A mapping from the request body's shape to the case's, one level of reference and no expressions — no concatenation, no conditionals, no formatting. The nesting, the arrays and the non-string types of a real payload survive, because the table is the payload. What does not survive is a body that has to be computed, and that body is what request= remains for.

This is the one place where "exactly its parameters" cannot hold literally, and it costs HttpTarget a new parameter in 0.5.0 rather than costing the format an escape hatch. There is no escape hatch of any kind: no python =, no import =, no dotted path to a callable. A target that is a function is suite.py territory, which is not a demotion — see §9.

6. The extension chooses the format, and unknown keys are a load error

--suite eval/suite.toml loads the TOML form; --suite eval/suite.py loads the Python form. No new flag. The CLI already dispatches on the shape of what it is given — a trailing :attr, a .py suffix, a path separator — and one more extension is the smallest possible addition to a surface people have already learned.

[suite] mirrors Suite's fields: the same names, the same defaults, the same load-time validations. That last one is free rather than reimplemented, and it is the strongest argument for the whole design: the loader constructs the same frozen dataclasses, so Suite.__post_init__ runs, and a TOML suite that samples without min_agreement, or declares an impossible "2/5" over three samples, or repeats a case id, or declares no assertions at all, fails with the sentence the Python form already fails with. Nothing is validated twice, which means nothing can validate differently.

--target has no meaning here and is refused with a TOML suite rather than ignored: the target is in [target], and a flag pointing at a Python attribute would be the escape hatch §5 refuses, entered through the command line.

An unknown key anywhere is a load error. Never ignored, never warned about:

suite.toml, [[assertions]] #2: `contains` has no parameter `treshold`.
Did you mean `threshold`?

A silently dropped treshold is a check running on its default — and for a threshold, the default that a typo would fall back to is the one that passes. That is fixed decision 3's vacuously green assertion, arriving in configuration form: the check is present, the report is green, and nothing anywhere says the bar was never set. Fixed decision 3 is about defaults that cannot fail; this is the same failure with a spelling mistake in front of it, and it gets the same answer.

Some of that strictness is already mechanical — a wrong keyword to a frozen dataclass raises TypeError — and what 0.5.0 adds is not the refusal but the message: which file, which entry, which key, and the near-miss when there is one.

When a TOML asks for what only Python can give, the error names the boundary and the way out. Not "unknown type", not "invalid value":

suite.toml, [[assertions]] #4: `from_autoevals` needs a scorer, which is a
Python object — custom assertions are code, and this suite needs a suite.py.
See docs/api.md.

The same for a custom judge, a computed request body, and a custom PiiAbsent pattern. A user who hits a wall and is told what is on the other side of it makes a decision; a user who is told "invalid" files a bug.

7. Disclosure is pinned to NOTHING_EXTRA for a data suite

Not a new rule. Suite.disclosure carries the note already: "disclosure lives here for the same reason, and literally: the ADR says it must be declared in the suite's code and never read from data." This section says what that means once suites can be data — the field is not settable from TOML, and a TOML suite discloses nothing beyond the verdict.

In world 3 that is a security property and not a limitation. What a Disclosure widens is what leaves the end company's perimeter, and a data file is the artifact most likely to be generated, templated, copied between customers, or edited by someone who has never read ADR 0002. A suite that is data cannot widen the boundary; widening it requires writing Python, in a repository, under review — which is exactly the ceremony the decision was supposed to carry.

A suite that genuinely needs to disclose more is a suite.py. That is a real cost for world 2's software house, and it is the right one: the case where a payload leaves a perimeter is the case that should be hard to reach by accident.

8. Built for machine generation and human review

The format is designed to be written by a program and read by a person, because that is what world 2's bridge will do with it.

• Case ids are stable and mandatory. They are what a verdict finds its counterpart in the baseline by; a generated id that changes on regeneration would make every run a page of new and missing.
• The suite/cases split is a diffing decision (§4): a generated batch of cases touches one file, and the file holding the thresholds is untouched and visibly so.
• The order is the author's (§1), so a generator that appends leaves the existing entries where the reviewer last saw them.
• Nothing is inferred. No key means what another key implies; there is no form where omitting something changes the meaning of something else. A reviewer reads the entry in front of them and knows what it does.

9. The two forms are equal citizens

No TOML-only feature, ever. And suite.py is not "advanced mode": it is the form that expresses what a data file cannot, and the documentation says that without apologising for either.

Concretely, and testably: the loader builds the same objects. A TOML suite and its Python twin produce equal Suite objects, the same assertion identities, and therefore the same config_hash — which means a suite ported from one form to the other keeps its baseline, and a run made from the TOML form is comparable with a run made before the port. If that ever stops being true, the format has forked the engine, and the fork is the defect.

This is what §1's coercion rule is for, and it is why the loader dispatches and never interprets: every decision about what a check means stays in the class that implements it, and the format's whole job is to say which class and with what arguments.

10. Compatibility

Core 0.5.0, plus a release of all three plugins, which gain their entry-point registration and nothing else. A plugin that is not re-released keeps working in the Python form and is simply not resolvable by coordinate, which is a load error naming the version to install.

No change to the run or baseline schema. SCHEMA_VERSION does not move. The format produces Suite objects and nothing downstream can tell how they were built — deliberately, per §9. Step 1 of this ADR was a read of the loading path specifically to check that claim, and it holds, with two things that are worth naming because they are changes even though the schema is not:

• load_suite() returns (Suite, ModuleType) today, and the module is what load_target() looks the target up in. A TOML suite has no module; the CLI layer gains a shape for that. It is internal to digline.cli, which is the outermost layer and the only one allowed to know about files.
• HttpTarget gains the body parameter of §5. Additive, keyword-only, and the callable form is untouched.

The container image runs TOML suites with no Dockerfile change. That is the follow-on this ADR exists to complete: the image already carries the CLI and the three plugins and installs nothing at runtime, so a suite.toml plus a cases.json mounted at /work is a gate with no Python anywhere — not on the machine, not in the repository. The 0.5.0 image is a rebuild at the new version, not an edit: docker build docker/ --build-arg DIGLINE_VERSION=0.5.0, which the publishing workflow already does from one pinned line.

Consequences

• Fixed decision 6 stops being aspirational. Entry points arrive because a feature needs them, which is the right order, and the plugin contract of ADR 0004 gets its registry.
• The audience widens by one and the engine does not fork. A Java shop with an endpoint can write a gate in TOML; the objects it builds are the objects every test in this repository already covers.
• Two files for a toy suite, and the quickstart in this form will show both from the start.
• A new surface to keep honest. Every assertion added from 0.5.0 on has a type token, and a test has to hold the token table to what the package exports — otherwise the format drifts behind the engine silently, which is the failure mode of every configuration layer ever written.
• The error messages are the feature. Half the sections above are about what happens when a file is wrong, because a format whose failure mode is a silent default is worse than no format. That is the largest share of the implementation and it is not incidental to it.

Alternatives considered

Type as the table key — [assertions.contains], [assertions.llm_rubric]. Reads beautifully and loses two things that are not negotiable. Order is gone: TOML tables are a mapping, and report order would become dictionary order. The same type twice is impossible, and two contains in one suite is the ordinary case, not the exotic one. It is also hostile to a generator, which has to key by something unique and would end up inventing contains_2 — a name nobody chose, in a document meant for review.

A judge as module:class — judge = "digline_anthropic:AnthropicJudge". It is an import written in a string: it reintroduces exactly what §5 refuses, it lets a TOML file execute arbitrary code by naming it, and it hands the format's own audience — a reader who does not write Python — a piece of Python jargon as a required field. anthropic/claude-haiku-4-5 says the same thing in the vocabulary of the person writing it, and it is already the identity the run file records.

Inline cases — [[cases]] in the same file. One file for a toy suite, which is the only thing it is better at, and diff pollution forever after: a production incident becoming a test case would rewrite the file the thresholds live in, and a review would have to separate "we added six cases" from "we moved the bar" by reading. §4 is that argument in full.

YAML. The README says "Not YAML", and it is not stubbornness. It would add a runtime dependency to a package that has one and intends to keep it, where tomllib has been in the standard library since 3.11 and costs nothing. Its flexibility is the wrong kind for a file that gates a release: anchors, multiple document forms, and a type system with famous surprises. Nothing about this format needs a block scalar badly enough to buy the rest.

A general expression language for the request body — Jinja, JMESPath, or a small one of our own. This is the escape hatch under a respectable name. It buys the last few per cent of HTTP bodies and costs a language with a parser, a sandbox question, and its own error messages, inside a file whose value is that it can be read at a glance. The line is drawn where the code already draws it: one level of reference, and a body that must be computed is a function.

Test plan

• Every type token resolves to an exported class, checked against digline.core.__all__, so a new assertion cannot ship without one.
• A TOML suite and its Python twin build equal Suite objects and the same config_hash — the §9 property, as a test rather than as a promise. The quickstart is the fixture, in both forms.
• Every Suite.__post_init__ refusal has a TOML case: no assertions, no cases, duplicate case id, samples without min_agreement, an impossible ratio, an over that matches nothing and an over that matches two.
• An unknown key fails, in each position — top level, [suite], an assertion entry, [target] — and the message names the file, the entry and the key.
• Each Python-only boundary fails with the sentence that names it: a custom judge, from_autoevals, a custom PII pattern, a computed request body, disclosure in [suite], and --target beside a TOML suite.
• A coordinate for a provider that is not installed names the package to install; a coordinate whose provider exists and whose model is unknown fails in preflight, where an unpriced model already fails.