Reach for the TOML form when the suite is plain data — cases, checks, thresholds, an endpoint — and nobody on the team writes Python. Reach for the Python form when it is not, which the next-to-last section names exactly.

The reasoning behind every rule here is ADR 0007.

The two files

A suite is always two files. The rules and the data change at different rhythms, usually by different hands, and a diff has to say which one moved.

# suite.toml
[suite]
tenant = "northwind"
environment = "staging"
name = "support"
cases = "cases.json"

[target]
type = "http"
url = "http://127.0.0.1:8730/answer"
output_path = "data.answer"
cost_path = "usage.cost_usd"

  [target.body]
  question = "case.vars.question"

[[assertions]]
type = "contains"
needle = "Northwind Support"

[[assertions]]
type = "cost_budget"
max_usd = 0.02
tolerance = 0.05

[
  {"id": "where-is-my-order", "vars": {"question": "Where is order 4821?"}},
  {"id": "how-do-i-return",   "vars": {"question": "How do I send it back?"}}
]

A case is a JSON object whose keys are Case's fields, so the file needs no schema of its own: id, vars, expected, label, group, context, metadata, suspended, canary, calibration. Name one that does not exist and the loader lists the ones that do. calibration is the one field that is an object, and its own keys are refused by name the same way:

{"id": "half-supported",
 "context": ["Refunds take 30 days.", "A receipt is required."],
 "calibration": {"output": "Refunds take 30 days and need no receipt.",
                 "check": "faithfulness", "low": 0.3, "high": 0.7,
                 "input": "How do refunds work?"}}

What it is, and what it refuses, is in the API reference.

Then the cycle you already know:

$ digline run     --suite suite.toml
$ digline promote --suite suite.toml --run latest --replacing none
$ digline compare --suite suite.toml --run latest

A complete, runnable version of this is in examples/quickstart-toml/, with the application it talks to.

Every relative path is relative to the suite file, never to the directory you happen to run from: cases.json sits beside suite.toml, and so does the prompt a [target] names.

[suite]

The same names as Suite's own fields, the same defaults, the same refusals.

Key
tenant	mandatory. The perimeter, and a directory: everything lands in .digline/<tenant>/
environment	mandatory, no default. Where inside the perimeter this ran — staging, production
name	mandatory. The suite's name, which is what a baseline is filed under
cases	mandatory. The cases file, always a file
samples	how many times to ask the target per case. Default 1
min_agreement	mandatory as soon as samples > 1. Written as a count: "2/3"
artifacts	files that are the thing under test — a prompt above all

disclosure is not settable here. See "What it cannot say".

A suite that samples without a floor, declares no assertions, repeats a case id or asks for a ratio no sample count can produce is refused when it loads — with the same sentence the Python form gives, because it is the same check running.

[[assertions]]

An ordered list. type selects the check; every other key is passed to it as its own parameter, and the parameters are the ones in docs/metrics.md — this format adds none and renames none.

The order is kept, because it is the order the report shows.

[[assertions]]
type = "regex"
pattern = "^[A-Z]"
name = "starts_capitalised"

name is worth setting whenever a suite has two checks of one kind: it is what the report prints and what a comparison pairs on.

Per case: equals, contains, not_contains, affix, is_json, json_schema, length, levenshtein, regex, llm_rubric, faithfulness, pii_absent, tools_called, tool_called_with, cost_budget, latency_budget, repeated.

Per run: precision, recall, accuracy, f1.

The token is the name the check already carries in a report and in a baseline, so if you have read the output you can write the file.

Aggregates go in the same list

They are written as entries like any other and land where they belong. over names a check's name, not a type:

[[assertions]]
type = "precision"
over = "contains"
threshold = "9/10"
tolerance = "1/10"

A threshold that is really a count of cases is written as one. An over that matches nothing is refused; so is one that matches two, and the fix is to give one of them a name.

by_group splits an aggregate by class

An aggregate over the whole run is an average, and an average carries a class that is broken. Give the cases a group and set by_group on the aggregate, and you keep the whole-run figure and gain one per class:

[
  {"id": "art-01", "label": "positive", "group": "refunds", "vars": {"question": "…"}},
  {"id": "art-02", "label": "negative", "group": "travel",  "vars": {"question": "…"}}
]

[[assertions]]
type = "precision"
over = "contains"
threshold = "1/2"
tolerance = "1/2"
by_group = true

That suite gates on three names rather than one — precision, precision[group=refunds], precision[group=travel] — with the threshold, the tolerance and the ADR 0006 noise floor inherited and computed over each group's own cases. The groups come from the cases and never from a declaration, so a class appears the moment a case names it.

Nothing to learn beyond those two words: group is a Case field like any other and by_group is a parameter like any other, which is why neither needed a rule of its own here.

There is no group on an assertion. You get every class or none — the class that degrades is the one you were not watching — and the near miss is caught by name:

suite.toml, [[assertions]] #2: `precision` has no parameter `group`.
Did you mean `by_group`?

The reasoning is in ADR 0010.

repeated wraps another check

For judge noise: the same output graded several times.

[[assertions]]
type = "repeated"
samples = 3
min_agreement = "2/3"

  [assertions.inner]
  type = "llm_rubric"
  rubric = "Does the reply answer the question in at most three sentences?"
  judge = "anthropic/claude-haiku-4-5"
  threshold = 0.7
  tolerance = 0.05

[assertions.inner] is a check written the same way as any other — the rules above, applied again.

The judge, by coordinates

judge = "anthropic/claude-haiku-4-5"

provider/model. The provider half is the name a plugin registers under; the model half is handed to that plugin untouched, so an identifier with slashes of its own — a Bedrock inference profile ARN, say — arrives whole.

The plugin has to be installed: uv add digline-anthropic, or the equivalent for digline-openai and digline-bedrock. Naming one that is not installed tells you which package to install.

One coordinate answers for both kinds of judge. llm_rubric is given the plugin's scoring judge and faithfulness its claim judge, because every plugin ships both.

No key appears in a suite file. There is no api_key and there will not be: a suite is a file that gets committed, and each provider's own SDK reads the key from the environment. Set ANTHROPIC_API_KEY (or the equivalent) where the run happens.

A coordinate carries the instrument, not its settings. The judge grades with the plugin's own defaults. A judge that needs a different max_tokens, a different temperature, or rules of its own is an object, and objects live in a suite.py.

[target]

Two forms, and there is no third.

type = "http"

Your application, on the other end of a URL. It can be written in anything.

Key
url	mandatory
output_path	mandatory. Where the answer lives in the response, dotted
cost_path	where the application reports what the call cost
latency_from_response	where it reports its own time. Left out, digline times the round trip, which includes the network
config_path	where it says which model answered and how it was set up. Worth writing: without it a run records nothing about the system under test
expect_config	the configuration you expect it to report. A reported value that contradicts it errors the case. See below
tools_path	where it lists the tools the model called, by name and in order
tool_calls_path	where it lists those calls with their arguments. See below
usage_path	where it reports the token counts. See below
headers	sent with every request
timeout	seconds. Default 30
[target.body]	mandatory: the payload

The trajectory: tools_path and tool_calls_path

Without one of these, tools_called and tool_called_with load and then error on every case — a check that cannot be answered rather than one that refuses, which is the worse of the two.

Declare tool_calls_path alone and you are done: the names come from the calls, so the two cannot disagree. Declaring both is allowed, and they must agree in the same order — digline refuses the answer rather than choosing a reading.

{"trajectory": {"calls": [
  {"tool": "activate_skill",
   "arguments": {"skill": "invoice"},
   "status": "success",
   "result": "ok"}
]}}

status is mandatory on every call and is one of success, error or not_reported. There is no default: a tool that ran and failed must not be able to report as one that worked by saying nothing. Report not_reported — with result_absence beside it — where your application does not know.

A call your application could not name is "tool": null, which keeps its position; "" is refused rather than read as that absence. An argument never leaves your perimeter: tool_called_with records how much of what you declared matched, never what the model sent.

The counts: usage_path

An object, by digline's own names — input_tokens and output_tokens are mandatory, cache_read_tokens, cache_write_tokens and thinking_tokens optional. Any other key is refused, because a count under another name is a count nothing adds up.

{"usage": {"input_tokens": 120, "output_tokens": 44, "cache_read_tokens": 9}}

Leave usage_path out and the run records no counts, which is an honest answer. It does not record zeros.

The review: expect_config

A configuration your application reports is written by your application. digline checks its shape — the keys are closed, the values are scalars — but not its truth, so provider and model reach a run, and cross a boundary, as values nobody reviewed. That is what this key repairs: you declare the system you expect, your application reports, and a contradiction errors the case naming both values. The value in the record is then one that went through a pull request.

[target]
type = "http"
url = "http://localhost:8080/answer"
output_path = "data"
config_path = "config"
expect_config = { provider = "gemini", model = "gemini-2.5-flash" }

Partial on purpose: the keys you name are the keys checked. Declare provider and model and nothing else is constrained — a temperature you did not name is recorded and not reviewed. Nothing is mandatory, including region. What is refused, at load time rather than mid-run:

• expect_config without config_path, because then no configuration is ever read, the check never runs, and the run is green whatever answered;
• an empty expect_config, which accepts everything — exactly what leaving the key out does, except that it looks like a review;
• a key outside the closed table, or a value that is not a scalar, because an application could never report one and the expectation could never be met.

A key you declare and your application never reports is a mismatch too. Absence is not agreement: you asked, and got none of an answer.

What this does not fix

A suite that declares no expect_config is in exactly the position it was before, which is every suite that has not adopted the key: provider and model are values your application chose, they travel in clear, and nobody checked them. Worse, a reader of the run cannot tell the two apart — the fact that a configuration was reviewed is not yet recorded in the document. ADR 0030 §5.1 owes that field to a schema bump and §6 says plainly what stays open until then. Declaring the paths above does not close it; declaring expect_config closes it for your suite only.

[target.body] is shaped like the payload itself. A leaf string that starts with case. reads from the case; anything else is a literal:

  [target.body]
  question = "case.vars.question"
  channel  = "email"

Readable: case.id, case.expected, case.context, case.vars.<key>, case.metadata.<key>. One level of reference and no expressions — no concatenation, no conditionals, no formatting — so the nesting, the arrays and the types of a real payload survive, because the table is the payload. A reference that names no case field is refused when the suite loads, not once per case half way through a run.

The one shape this gives up is a body that has to be computed. That is what HttpTarget(request=…) remains for, in a suite.py.

type = "provider"

A model, called directly. This one needs a key in the environment.

[target]
type = "provider"
provider = "anthropic/claude-haiku-4-5"
prompt_file = "prompt.md"
max_tokens = 500
temperature = 0.0

provider is the coordinate. Everything else is a parameter that plugin names — prompt_file, system, system_file, temperature, max_tokens, prefill and its peers; a misspelling is refused with what the plugin does accept. model is not among them: it is the coordinate's second half, and writing it twice would give one fact two places to be wrong.

client is not configuration either — it is where a test injects a stand-in, and a data file has nothing to put there.

What the endpoint charges: [target.pricing]

A plugin prices calls from its provider's own list. Pointed at an OpenAI-compatible aggregator, a corporate gateway or a model you host, that list is somebody else's prices, and a CostBudget over it measures a cost your calls never incurred. Declare what the endpoint actually charges:

[target]
type = "provider"
provider = "openai/gpt-5.6-sol"
base_url = "https://api.aggregator.example/v1"
prompt_file = "prompt.md"

  [target.pricing]
  input_per_mtok = 1.10
  output_per_mtok = 4.40
  cache_read_per_mtok = 0.11     # optional: absent means no cached tier
  cache_write_per_mtok = 1.375   # optional

USD per million tokens. The declared price replaces the plugin's entry for this model, and the run records pricing = "declared" beside the rates. A model you host is four zeros, written down — a zero nobody wrote would be a budget that cannot fail. An http target reads its cost out of the response and refuses the table.

A declared price is part of the suite's identity. It enters config_hash, because the rate is the ruler a CostBudget reads cost on: declaring one, or changing one, makes the baseline comparable and no longer promotable, and the comparison says the rules changed.

At a named endpoint the rates are withheld — a negotiated rate is the customer's commercial fact — and the withholding has a limit worth stating exactly:

Withholding a declared rate is a latch, not a constraint — the value never prints, but the hash narrows it; a rate you cannot afford to narrow belongs in a Python suite, or at an unnamed endpoint.

ADR 0022 is the reasoning.

What it cannot say

Deliberately. What a data file cannot express is what a data file should not be able to smuggle past a review.

A judge with rules of its own	it has no coordinates
A judge set up differently from the plugin's defaults	the coordinate carries the instrument, not its settings
A target that is a function	including an HTTP body that must be computed
A custom assertion, and from_autoevals	a scorer is a Python object
Custom pii_absent patterns	they carry a checksum function
disclosure	see below

Each of these fails when the suite loads, with a sentence that names the wall and where to go — never "invalid value".

disclosure deserves its own line. It is not settable from a data file at all, and that is a security property rather than a limitation: what a Disclosure widens is what leaves the end company's perimeter, and a data file is the artifact most likely to be generated, templated, or copied between customers. A suite that is data cannot widen the boundary. Widening it means writing Python, in a repository, under review.

Moving between the two forms

Neither form is the advanced one. They build the same objects, so the same suite written both ways has the same assertion identities and the same config_hash — which is what a stored baseline is matched on.

So a suite can be ported in either direction without re-promoting: rewrite it, run it, and compare against the baseline you already had. If the comparison reports the suite as changed, the two are not the same suite, and the difference is a real one worth finding.

When something is wrong

Every refusal names the file, the entry inside it, and the key:

suite.toml, [[assertions]] #2: `contains` has no parameter `treshold`. Did you mean `threshold`?

An unknown key is always an error, never a warning and never ignored. A silently dropped treshold would be a check running on its default, and for a threshold the default a typo falls back to is the one that passes — a check that is present, green, and never set.