- Status: accepted
- Shipped: 0.2.0
- Date: 2026-08-31
- Amended: 2026-09-01 — §8, HTTP targets. Added rather than a new ADR: it
changes no decision above it, and reads as a correction to §6's aside about
HttpTargethaving no model - Amended: 2026-09-10 — §9's perimeter rule corrects itself, the same day.
The delta-pass run over 0.8.0's new surface before the announcement found
that §9 applied two different tests to two fields arriving in the same
reply from the same server:
fingerprintwas withheld because on a compatible endpoint its value is chosen by software nobody here reviews, andresolved_modeltravelled in clear as "a public product name". That last claim holds only where nobase_urlwas set. Reproduced on the document that actually travels: withbase_urlwithheld as perimeter,resolved_model = "acme-legal-assistant-prod-eu-west-v3"went with it. The rule is now conditional on the fact redaction already uses. No decision above §9 moves. Ships in 0.8.1 - Amended: 2026-09-10 — §9, the observed identity. Same test as §8: it widens §1 and §6 and overturns nothing. §3 above all is untouched, and that is what makes it an amendment rather than a new ADR — this is the record, never the hash
- Amended: 2026-09-23 — §9,
resolved_modelover HTTP: a ruling, not a widening. The Not decided here entry waited for an application with an identity to report;floweris that application and the answer is still no, because the perimeter rule partitions the two model names by provenance and over HTTP §8 gives both the same one. Added rather than a new ADR on the 2026-09-01 test: it refuses what §1's table already refuses and overturns nothing. It also states the consequence it does not repair —modelis mandatory over HTTP and equally unreviewed — which is ADR 0030 - Amended: 2026-09-27 — §9, what survives a projection, by ADR 0034 §4 at its acceptance. One sentence owed as a consequence of that record's enumeration: in a projected document every reported configuration key and value is a token or absent. Nothing is recorded differently, and the rule for a redacted document is unchanged
- Supersedes: the proposed — open question draft of 2026-08-28, whose five open points are the five sections below
- Assumes: ADR 0003 §3
(what is under test does not enter
config_hash) and §4–5 (withheld rather than absent,unknownrather than a guess), ADR 0004 §1 (a plugin ships a target and a judge) and §5 (a credential never appears anywhere), ADR 0002 §2 (the payload stays where it is born) - Touches: fixed decision 9 (
CLAUDE.md), which lists what crosses a boundary
Context¶
A run records the verdicts, the fingerprint of the configuration that judges them, and — since ADR 0003 — the files under test. It does not record which model answered, at what temperature, under what token cap, in which region or against which endpoint.
config_hash is built from the assertions, their thresholds and tolerances,
samples and the aggregates. Nothing the target does reaches it.
Response.metadata carries the model and the token counts to the assertions,
and CaseResult keeps only case_id, verdicts and suspended — so none of
it is persisted either.
Three consequences, and they are not equally serious.
The report can say something true that reads as false. Two runs that differ
only in temperature produce the same config_hash, and compare() reports
"The configuration is the same as the reference." Under the current definition
of "configuration" that sentence is exact: the rules that judge did not move. To
a reader in world 3 it says something else, namely that nothing about the system
moved. The word is doing two jobs.
A baseline cannot say what produced it. ADR 0003 closed this for the prompt
and left it open for everything around the prompt. A run from a dirty tree at
temperature=0.7 and one at temperature=0.3 are the same document.
The gap is invisible. Nothing warns; the numbers simply mean less than they
look like they mean. digline-bedrock documented it on the argument itself,
which is a patch on one plugin's docstring, not a decision.
This is deliberately not a case for folding these values into
config_hash. ADR 0003 §3 decided that a change to the system under test must
leave two runs comparable — that comparison, old against new with the score
deltas beside it, is the experiment digline exists for. Fingerprinting the
decoding parameters would make every such experiment un-promotable.
Decision¶
1. What "the configuration" is, and what it is not¶
The configuration of the system under test is the set of parameters that decide how the model answers, declared by the target itself. Named, closed, and small:
provider |
which plugin answered — always present |
model |
the model id, or the inference profile id — always present |
max_tokens |
the output cap |
temperature |
where set |
top_p, top_k |
where set, on a provider that names them |
seed |
where the provider exposes one |
region |
Bedrock: what was called is what was priced |
base_url |
the host of a custom OpenAI-compatible endpoint |
response_format, json_mode |
the shape the answer was asked for, where the provider takes one |
The last row is not a decoding parameter, and it is here for the same reason
the others are: asking for a JSON object changes what comes back, so a
regression can coincide with it. It is recorded reduced to a scalar — the
type of the request, json_mode as the boolean it already is — because a
json_schema carries a whole schema, and a schema is structure rather than a
value to diff.
Each plugin declares exactly what it actually sends. Nothing is invented: a
provider without a seed records no seed, and a parameter left unset is
absent, not None — "we did not send it, the provider's own default
applied" is a different fact from "we sent nothing for it", and only absence
says the first one honestly.
base_url records the host and port, never the scheme, never the path and
never the userinfo: a URL is one of the places a credential is written by
accident, and ADR 0004 §5 makes a credential the one category with no
Disclosure that can release it.
additional_request_fields and extra_body stay out. They are the escape
hatch — whatever Converse or a compatible endpoint takes and the plugin's own
signature does not name. ADR 0004 already keeps them out of config_hash, and
the end-to-end test in each plugin pins that; this ADR adds its sibling and
leaves them out of the recorded configuration for the same reason. What is
outside the contract is outside the record. A mapping of unknown keys with
unknown values cannot be diffed into a sentence a reader can act on, and it is
exactly where an account-specific identifier or a customer's own tuning would
sit — the argument of §2, in the box where it cannot be checked. A suite that
needs those values to show keeps them in a file and declares it in
Suite.artifacts, where ADR 0003 already carries them into the run and into
the diff.
2. The perimeter rule, by type: one special field, one existing rule¶
A model id and a number carry nobody's data, and travel in clear.
claude-sonnet-5, 0.7, 1024, eu-west-1 are public product names and
decoding parameters. They are measurements of the system, which is precisely
what decision 9 lets cross a boundary. Withholding them would cost the whole
feature and protect nothing.
base_url is the one field that describes the client's perimeter.
https://llm-gw.internal.acme-bank.it/v1 is topology: it names an internal
gateway, and often the customer. So under redaction it gets exactly the ADR
0003 artifact treatment — the value is discarded, the key is recorded as
withheld rather than dropped, and a comparison across it reports unknown
rather than same. One special field, one existing rule, no new mechanism: in
particular no new Disclosure member, because the prudent default of ADR 0003
§4 is the answer here too and an opt-in nobody has asked for is a widening
nobody reviewed.
Amended 2026-09-15 by ADR 0022 §6: the
per-million rates a suite declares for its target are withheld at a named
endpoint too, beside resolved_model — a negotiated rate is the customer's
commercial fact. Because config_hash is computed from the rate and travels in
clear, that withholding is declared a latch, not a constraint.
unknown and not same is the same choice ADR 0003 §5 made and for the same
cause: with no value on one side, same would be a guess wearing the clothes
of a finding.
3. Where it lands: target_config, beside the hash and not inside it¶
Run.target_config, a SystemConfig — a flat mapping of scalars plus the set
of keys withheld — recorded in every run, and in the baseline at promote time
because a baseline is a promoted run file.
It is not folded into config_hash, and that is the point of the ADR
rather than a detail of it. config_hash is the identity of the suite: the
rules that judge. This is the identity of the system: the thing judged. Two
runs at two temperatures must stay comparable and promotable, exactly as two
runs of two prompts do (ADR 0003 §3).
And the feature is the named delta. An opaque hash difference is what this
ADR exists to replace: "the configuration differs" sends a reader to reconstruct
what differed, while temperature 0.3 → 0.7 is a sentence they can act on.
SCHEMA_VERSION goes to 8, additively, and is migrated in place like 6 → 7:
a run written before this ADR recorded no configuration, which is exactly what
an empty one says.
4. The judge is recorded too, and its change is louder¶
Run.judge_config, same shape, collected from the judges the suite's
assertions hold.
A judge is not part of the system under test — it is the measuring instrument. When the target moves, the thing being measured moved and the scores are a finding. When the judge moves, the scale itself moved, and the scores are not comparable with the baseline regardless of what the target did: a rubric graded by one model and then by another is two measurements of one output by two instruments nobody calibrated against each other. That is a stronger statement than a target change and the report makes it as one — "reduced comparability", not "here is what changed".
Collected rather than declared, because a judge is bound where it is used: the
target is one object passed to execute(), while a judge is a field of the
assertion that asks it (§6). A suite may therefore hold several, and the record
is in two halves that answer two questions.
Which instruments graded is always recorded, as the set of distinct
provider/model identities bound in the run, however many there are. This is
the half that cannot be allowed to fall silent. The first shape of this decision
recorded nothing at all once two judges disagreed on their identity — and a
suite grading with two instruments that replaces one of them is precisely the
event this section exists to catch, so the record went blind exactly where it
was most needed. A rule that holds for one judge and abandons the case for two
is not a rule.
How it was set up is recorded only when there is one of them. With a single
identity the scalars are merged, and any scalar the judges disagree on is
absent rather than reconciled: a ScoreJudge capped at 400 tokens beside a
ClaimCountJudge capped at 800 is two set-ups, and writing one down would be a
fact nobody established. With two identities there is no single set-up at all —
inventing a merged max_tokens would describe a judge nobody built — and the
identity list carries the whole answer.
A judge added or removed is reported as added or removed, never as a value
that moved. model a → b is what replacing a lone judge looks like and it stops
being true the moment a suite grades with three; one identity gone and one
arrived stays true at every count.
5. What the reader sees¶
compare() reports and never fails. Two headline facts, on the pattern
artifacts_changed already set:
target_config_changed, with the named deltas — field, before, after;judge_config_changed, reported more strongly, flagging the comparison as of reduced comparability.
A field withheld shows as withheld rather than absent, and a baseline that
recorded no configuration at all yields unknown for every field rather than a
column of fabricated news (§7).
And the sentence this ADR exists for: where a regression coincides with a
configuration delta, the report says so next to the regression — "this drop
coincides with temperature 0.3 → 0.7". Not a claim of cause: coincides is
the strongest word the data supports, and it is the word that makes a reviewer
check the config before blaming the prompt. Both facts render in the HTML
report in both locales, in the terminal summary, and in --json.
The configuration section sits above the score deltas, beside the artifact diff, for the reason ADR 0003 §5 put it there: what changed comes before what it did.
And the word is given back. With this ADR the report prints two sentences in a row, and until they were read next to each other they both said configuration while meaning different things:
The configuration is the same as the reference. The system under test answered under a different configuration: temperature 0.3 → 0.7.
The first one is about the rules that judge, and the rules have a name of their
own — the suite. So it becomes "The suite is unchanged from the
reference.", in both locales and in the terminal, and configuration is left
to mean one thing: how the system under test was set up. The field names
(config_hash, Comparison.config_changed) do not move, because they are a
parsed contract and this was a reading problem; the view's OLDER SUITE marker
follows the copy, since it was the same word about the same fact.
6. Granularity: recorded where the thing is bound¶
A target is bound once per run — execute(suite, target, …) takes one, and
the prompt × provider matrix is a loop above the driver, one run per cell.
So one target_config per run, which is the same level Run.artifacts sits at
and the same level the question is asked at.
A judge is bound per assertion: LlmRubric(judge=…), Faithfulness(judge=…),
and a Repeated wrapping either. The driver walks the declared assertions,
follows a wrapper through to what it wraps, and records the identity of every
judge it finds — plus the merged set-up when they are all one instrument, which
is the ordinary case even with two of them, since ADR 0004 §1 makes ScoreJudge
and ClaimCountJudge two objects normally built against one model.
So SystemConfig carries identities beside values: the set of instruments,
and the settings of the one when there is one. On the target side it is empty —
a target is bound once per run, so the set could only ever hold a single element
and would repeat what values already says. A judge that declares no instrument
at all is passed over the way a plain-function target is.
A target that declares nothing records nothing. A Target is any callable
and most are plain functions; HttpTarget calls an application that has no
model at all. config is therefore an optional protocol asked for, the same
family as Preflight and HasArtifacts and asked for the same way — not a
mandatory member of Target or of Judge, which would break every function
target and every fake judge in every test suite, ours included. Absent stays
absent, and absent is not a change.
7. Compatibility: keys added, nothing re-promoted¶
SCHEMA_VERSION 7 → 8, additive, with its migration step. Old files stay
readable through digline migrate, and a baseline promoted before this ADR
needs no re-promotion: compared against a run that does record its
configuration, every field reports unknown — "this reference predates the
record" — rather than an error or a wall of new. Deltas appear from the first
pair of runs that both have one.
OUTPUT_VERSION follows its existing rule and does not move: --json
gains keys, and a consumer that does not read them is unaffected.
8. HTTP targets: the configuration arrives in the answer¶
Addendum, 2026-09-01. §6 settled that a target which declares nothing records
nothing, and named HttpTarget as the case — "an application that has no model
at all". That was wrong about the ordinary case. The application behind the
endpoint usually does have a model; what it has is no way to say so.
The Java path (examples/langchain4j/) is HttpTarget against a service
digline cannot import, and it loses precisely what §1 was written to record.
The prompt is covered — ADR 0003 carries it as an artifact, because the file
sits in the repository the suite sits in — but the model, the temperature and
the token cap are chosen on the other side of HTTP and are invisible. A team
that switches their LangChain4j app from one model to another gets a run that
compares clean on the configuration and says nothing, which is the sentence §5
exists to make impossible.
So HttpTarget gains config_path, symmetric with cost_path. It names a
JSON object in the response, and the target implements HasConfig from it:
HttpTarget(
url,
request=...,
output_path="data",
cost_path="usage.cost_usd",
config_path="config",
)
{"data": "...",
"usage": {"cost_usd": 0.0009, "elapsed_ms": 41.0},
"config": {"provider": "openai", "model": "gpt-4o-mini",
"temperature": 0.0, "max_tokens": 512}}
Cost is the precedent and the argument: when the model call happens elsewhere, the only party who can price it is the one who made it, so digline reads a number the application computed rather than pretending to know. The configuration is the same fact one field over. Nothing about the mechanism is new — a dotted path, a value read out of the answer.
Absent config_path, absent configuration. The parameter is optional, the
recorded object is empty, and §6 holds unchanged: absent is not a change. No
existing suite behaves differently.
The contract is enforced here, not followed¶
Every other config in this repository is written by a plugin, in Python, and
reviewed as code. This one is written by an application nobody here reviews, so
the rules of §1 and §2 stop being conventions the author follows and become
checks the reader performs. Five, and each refuses rather than repairs:
- The closed key table of §1 and nothing else. An unknown key is refused by
name, with the allowed set in the message. Not dropped: silently discarding a
field is how a team believes they recorded something they did not. This is
§1's
additional_request_fieldsargument arriving from the other direction — an open mapping of unknown keys is exactly where an account identifier or a customer's own tuning would sit, and here it would arrive over the wire. - Scalars only, the same check
SystemConfigalready makes, made earlier so the message names the path in the answer rather than the field in the record. nullmeans not sent, exactly assent()reads an unset parameter: the key is absent, and the provider's own default applied.base_urlis reduced to host and port here, not trusted. A plugin passes a URL it constructed; an application reporting its own endpoint is far more likely to send the whole thing, userinfo included, and ADR 0004 §5 makes a credential the one category noDisclosurecan release. Under redaction it then gets the treatment of §2 unchanged — withheld, not dropped,unknownrather thansame— because by then it is an ordinarybase_urlandPERIMETER_FIELDSdoes not care where it came from.- A configuration that cannot say who answered is refused, per §1: no
provider, nomodel, no record. Report the object completely or leave it out.
One run measures one system¶
A plugin is constructed once and answers the same way all run. An endpoint can answer case 1 on one model and case 7 on another, and §6 has no reading under which that is one configuration. Merging would describe a set-up nobody built; recording the first silently would report a system that was not the one measured throughout.
So the first configuration reported is the run's, and a later answer that disagrees errors its own case, naming the field and both values. The run is still written, the deltas are still there, and the case that broke the premise is visible as an error rather than absorbed. A team that wants to compare two set-ups runs two runs, which is what §6 already says a matrix is.
target_config is asked twice¶
The driver read the target's configuration before the first case, so that a
malformed one failed before the suite was paid for. An HttpTarget has nothing
to declare at that point — it learns by answering. execute() therefore asks
before the first case and after the last, and records the second answer. A
target that declares statically gives the same answer both times, so nothing
about a plugin's behaviour moves; what the early call still buys is the early
failure it was added for.
What this does not do¶
It does not make the reported configuration true. An application can report a model it did not call, and digline has no way to know — it has no way to know what a plugin sends either, and the recorded value is a declaration in both cases. What changes is that there is now something to declare, and that a change in it becomes the named delta of §3 and the coinciding sentence of §5 for a team whose application is not Python.
It does not touch config_hash: §3 holds, and two runs of one suite against two
models stay comparable and promotable.
It does not extend to the judge. A suite whose target is an HttpTarget grades
with whatever Judge it holds, on this side, and judge_config collects it
unchanged. An application that judges its own output is not a judge digline can
record, and nothing here pretends otherwise.
9. What the provider said answered¶
Amendment, 2026-09-10. Widens §1 and §6. §2, §3, §4, §5, §7 and §8 hold, and
§3 above all: what follows is recorded beside config_hash and never inside
it, so two runs across a model rotation stay comparable and promotable.
§1 records what the target sent. model="claude-sonnet-5" is an alias, and
an alias is a promise about a family, not a name of a system: the provider
decides which snapshot behind it answers, and rolls that decision on its own
schedule without anyone touching the suite, the prompt or the parameters.
So the run records claude-sonnet-5 on Monday and claude-sonnet-5 on Friday,
compare() reports the configuration as unchanged, and a drop between the two
sends a reviewer to the prompt. This is §Context's first consequence — the
report can say something true that reads as false — one level down and harder
to see, because here the sentence is not merely narrow, it is the sentence a
reader would have written themselves from the same evidence.
The record admits what the provider observed, beside what the target sent.
resolved_model |
the model id the provider said answered |
fingerprint |
the backend build, where a provider names one |
Two rows added to §1's closed table, and nothing removed from it. They arrive
through the reply, on the record ADR 0004 §6 widened _complete to return;
they are that amendment's passengers, and this section is where they are
unloaded.
Observed is not sent, and absence means something else¶
§1's rule is sent(): an unset parameter is absent, because "we did not
send it, the provider's own default applied" is a different fact from "we
sent nothing for it", and only absence states the first one honestly.
An observed field is absent for a different reason: the provider did not
say. Both are honest absences and both stay absent — no None, no
placeholder — but they are not the same fact, and one of the two is the whole
answer for one of the three plugins. sent() therefore gains a sibling and not
a flag, so that a plugin author reading either name knows which category they
are in.
The third column is the argument for keeping the distinction visible:
| Anthropic | OpenAI | Bedrock Converse | |
|---|---|---|---|
| resolved model in the reply | message.model |
completion.model |
nothing |
| backend build | — | system_fingerprint |
nothing |
Converse returns output, stopReason, usage, metrics,
additionalModelResponseFields, trace, performanceConfig and
serviceTier. There is no model id anywhere in it: the caller passed a
modelId and the API does not echo one back, resolved or otherwise. So on
Bedrock — where an inference profile is precisely a name that stands for a
model chosen elsewhere, and where the gap is therefore widest — the record
degrades to nothing stated, and nothing stated is what it records. We do not
fill the gap from the request. Copying the requested modelId into
resolved_model would manufacture the one fact this section exists to obtain,
and it would manufacture it identically whether or not the profile had moved.
How strong the signal actually is has to be measured, not assumed. Whether
a given API echoes the alias back or resolves it to a snapshot is that API's
behaviour on the day, and this repository already has the precedent and the
machinery for finding out: CACHE_READS_ARE_INSIDE_INPUT_TOKENS is a constant
that exists because the cache-token convention was measured against the API
rather than inferred from the field names (friction 25), and it is re-measured
by a @pytest.mark.live test that fails loudly if the answer changes. The two
first-party plugins that can report an identity carry the same obligation: a
live test that asserts the reply's model id against the alias that was sent,
and says which it got. If an API turns out to echo the alias, resolved_model
equals model, the delta stays silent, and the record has cost a field and
told no lie.
The perimeter rule, by type — and one field joins PERIMETER_FIELDS¶
§2's test is what the value describes.
resolved_model travels in clear where no base_url was set, and is
withheld where one was. claude-sonnet-5-20260115 carries nobody's data; it
is a measurement of the system, which is what fixed decision 9 lets cross. But
that is a fact about a first-party endpoint, and base_url is what tells the
two apart.
Corrected 2026-09-10, hours after this section was written, by the
release delta-pass RELEASING.md now requires over new surface. The paragraph
above originally read "resolved_model is a public
product name and travels in clear, exactly as model does" — flatly, with no
condition — and the paragraph below then withheld fingerprint on the ground
that a compatible endpoint's values "arrive from software nobody here reviews".
Both fields arrive in the same reply from that same server. The test was right
and was applied to only one of the two.
On a customer's own vLLM, Ollama or gateway, the model the reply names is
whatever that server chose to put there: acme-legal-assistant-prod-eu-west-v3
is a project codename, an environment and a region, and it travelled beside a
base_url withheld for describing exactly that. So where base_url is set,
resolved_model is in fingerprint's trust category and gets fingerprint's
treatment.
The condition is the fact redaction already holds, which is why this needs
no new mechanism and no new Disclosure member: base_url present means a
compatible endpoint, and the redaction is already looking at it. Present in
values or in withheld — a run redacted twice must not widen on the
second pass, and after the first pass base_url has moved.
The sent model keeps travelling, and the asymmetry is the point. It is
written in the suite, and the suite goes through a review — the same argument
ADR 0003 §4 makes for opting artifacts in. The observed one is written by the
server and nobody reviewed it. Two model names, two provenances, two answers.
The test is on the configuration, not on the side it describes.
perimeter() reads base_url from the object it is called on, and
Run.judge_config is a SystemConfig exactly as Run.target_config is. So a
judge behind a customer's gateway withholds its resolved_model, and a
target that declares no base_url — a plugin on the official endpoint, an
application reporting its own configuration — carries it in clear. Written
down because the rule is easy to carry away as a target's rule, and it is
not: the two sides differ only in how often each names an endpoint while
reporting a resolved id, which a judge rarely does and a target behind a
gateway usually does. Both happen in one document as soon as the target names
an endpoint and the judge does not. (Added 2026-09-26, after the two were
measured side by side in one redacted document.)
Inside the perimeter it always travels. World 1 is the developer, who sees
everything; this is a boundary rule, not a storage rule, and an unredacted run
records resolved_model whatever the endpoint. What is lost is narrow and
worth stating: on a compatible endpoint world 2 stops seeing which snapshot
answered and keeps seeing that it changed — the field is withheld, so
compare() answers unknown rather than same, and the alias-rolled delta
that §9 exists for still fires on a first-party endpoint, where it was the
motivating case.
fingerprint is withheld under redaction. (This read "and this is the
one genuinely new ruling here" until the correction above gave resolved_model
the same treatment on the same ground. There are two, and they are one ruling
applied twice.) On the official endpoint it is an opaque backend id and
harmless. But base_url makes one plugin cover every OpenAI-compatible server,
and on a customer's own vLLM or Ollama the value of system_fingerprint is
whatever that server chose to put there — a build path, a container tag, a
hostname. That is the argument §2 made for base_url word for word: it
describes the client's perimeter rather than the model, and it arrives from
software nobody here reviews.
So it gets §2's existing treatment and no new mechanism: it joins
PERIMETER_FIELDS, the value is discarded at a boundary, the key is recorded
as withheld rather than dropped, and a comparison across it answers
unknown rather than same. No new Disclosure member — the prudent default
of ADR 0003 §4 is the answer here too, and an opt-in nobody has asked for is a
widening nobody reviewed.
The cost is stated rather than hidden: world 3 loses an opaque string that told them little on its own, and world 1 — the developer, who sees everything — keeps it. That is the right side to lose it on.
Where it lands, and why no version moves¶
Run.target_config.values, beside the sent parameters, in the record and not
in the hash (§3, unchanged).
SCHEMA_VERSION does not move, and this is by construction rather than by
luck. target_config is serialised as an open mapping of scalars, so a new
key inside values is the same event as a plugin declaring a top_p it did
not declare before: no field is added to any document, no reader has to learn a
shape, and digline migrate has nothing to do. A baseline promoted before this
release needs no re-promotion. OUTPUT_VERSION does not move either, by its own
existing rule: --json gains keys and a consumer that does not read them is
unaffected.
The rendering is free for the same reason: config_deltas iterates over the
keys that are there, and config_changes prints field, before and after. The
sentence §5 exists for arrives with no new code —
The system under test answered under a different configuration: resolved_model claude-sonnet-5-20260115 → claude-sonnet-5-20260301.
— and where a regression coincides with it, §5's coincides sentence names it next to the regression. That is the whole feature: an alias that rolled under a suite nobody edited is now a named delta a reviewer reads instead of a silence they cannot see past.
One sentence has to change, because it says sent¶
Two of the phrases §5 introduced are written for parameters that are sent:
config.change.new {field} {after}, not sent for the reference
config.change.missing {field} {before}, no longer sent
Applied to an observed field they state something false. resolved_model was
never sent — not by this run and not by the reference — and a reference
promoted last month recorded no observed identity at all, so this is not a
corner case: it is what every first comparison against an existing baseline
will print.
So the observed fields are a named set in the same file that already holds
PERIMETER_FIELDS, and they get two phrases of their own in both locales —
"not reported for the reference", "no longer reported". The existing
sentences are not reworded. ADR 0005's own consequences noted the cost of
changing report text a pipeline may match on; that cost is worth paying to give
a word back (§5) and is not worth paying to avoid a second table entry.
The judge is asked once, and once is too early¶
§4 records the instrument, and the instrument has the strongest version of this problem: when a judge's alias rolls, the scale moves under a suite nobody edited, and §4's ruling is that a changed instrument makes the scores less comparable regardless of what the target did. An alias rotation is that event with nobody to notice it.
But execute() asks judge_config(suite) before the first case, where a
target's configuration is asked twice (§8) and a judge's is not. Before the
first case a judge has observed nothing, so the record would be empty by
construction.
So the judge is asked twice too, on §8's pattern and for §8's reason. Read before the first case, so a malformed configuration still fails before the suite is paid for, and read again after the last, and the second answer is the one recorded. A judge that declares statically gives the same answer both times, so nothing about any existing plugin, fake or suite moves.
One limitation, stated because §4 was written against exactly this shape: with two judges on one alias whose observations disagree, §4's merge rule drops the disagreeing scalar, so the record falls silent on the loudest possible event. That is §4's rule applied honestly — any scalar the judges disagree on is absent rather than reconciled — and the identity list still carries who graded. Making a disagreeing observation louder than a silent one is a change to §4's merge, and it needs a frictions entry before it needs a rule.
Rotation: §8's rule, extended, and split by field¶
§8 decided this event once already, for the target digline cannot import:
A plugin is constructed once and answers the same way all run. An endpoint can answer case 1 on one model and case 7 on another, and §6 has no reading under which that is one configuration.
The premise's first half is now wrong. A plugin is constructed once, but what
it observes is not its own — a provider can roll a model between case 1 and
case N, and the plugin has no more control over that than an HttpTarget has
over the application behind it. The event §8 governs has arrived on the side
§8 assumed was safe, and two rules for one event is one too many.
So §8's rule is extended, and split, because the two new fields are not the same kind of fact:
resolved_model— first wins, and a later disagreement errors its own case. Word for word §8: the run is still written, the deltas are still there, and the case that broke the premise is visible as an error rather than absorbed. A model that changed under an alias mid-run is the system under test changing mid-measurement, which is the thing §8 refuses to average away, and a suite that wants to compare two systems runs two runs.fingerprint— a later disagreement records nothing. The field is absent from the run, under the observed-absence rule above: the provider did not give one answer for this run. It does not error the case. OpenAI documentssystem_fingerprintas changing whenever they change the backend configuration, so erroring on it would paint runs red for an event with no bearing on which model answered — the false red that teaches a team to stop reading the colour.
The asymmetry is not a softening of §8. It is §8's own question asked of each field: does a change here mean a different system was measured? For a model id it does. For a backend build it does not, and claiming it did would be the manufactured fact this ADR spends its length refusing.
resolved_model stays closed over HTTP, and the reason is the partition¶
Amendment, 2026-09-23. Rules the entry Not decided here left open, and does not widen §1's table. §8 and the perimeter rule above are unchanged; what follows is a refusal, stated with its reason, because the reason is the part that transfers.
The open entry set a condition and named its price: "§8's closed key table
refuses what it does not know, so an application cannot report one until the
table says it may. Adding it there is a line in CONTRACT_FIELDS and a line
here, and it should wait for an application that has one to report."
An application with one to report now exists, and the answer is still no. That is the whole of this section, and it is a ruling rather than a deferral: the condition has been met, and what refuses the field is not the absence of a caller but the absence of the test the perimeter rule is built on.
The partition is not available over HTTP. The rule above separates the two
model names by provenance, and says so in as many words — the sent model
travels because "it is written in the suite, and the suite goes through a
review", while the observed one is withheld at a named endpoint because "the
observed one is written by the server and nobody reviewed it." Over HTTP
neither name is written in a suite. §8 is explicit about where both come from:
the configuration arrives in the answer, and declared_config reads
provider, model and every other key out of the application's own reply. One
provenance, two fields. A rule that distinguishes them by who wrote them has
nothing to distinguish here, and admitting a field under a premise that does not
hold at its boundary is how a perimeter is widened by accident rather than by
decision.
The mechanical repair exists and is refused on its own merits, not as too
expensive. An HttpTarget knows its endpoint — endpoint_host(url) is
computed at construction and the constructor raises when there is no host to
reduce to — so it could supply base_url itself and make every HTTP run a named
endpoint by construction. That is the correct mechanism and it needs no new
machinery. It also buys nothing: sighting() and config_deltas() redact
before they read, for world 1 as much as for anyone, so a withheld
resolved_model over HTTP would be recorded in the run file and rendered
nowhere at all — no alias-roll delta, no echoed row, no headline. The field
would be write-only, and the question above would be settled by a mechanism
instead of by a decision.
And the reading that a pilot asks for is the one that has to be refused in
writing. Leaving the condition exactly as it stands is the cheapest path: an
application that declares no base_url is read as first-party, and its
resolved_model travels in clear. The withholding would then turn on the
presence of a key the measured party controls — and it is worse than
controls, because declared_config deletes base_url when its value has no
parsable host, silently and with no refusal. An application that reports its
endpoint as anything unparseable is thereby read as having no endpoint, and
nothing tells it so. A boundary the subject can switch off is not a boundary.
ADR 0029 §9b declared the document version of this shape and deliberately did not fix it: nothing is signed, "a party who writes a run document can make it say anything, and the party who writes it is exactly the party a pin measures." That is a control over drift nobody noticed, and it is declared not to be a security boundary. This would be the same shape one level worse — there the trusted thing is a document, here it would be the perimeter that decides what leaves the end company at all. A forged score misreports an experiment; a disarmed perimeter sends the data.
What the reader gets instead, and it is not nothing. §8's record already
names the system: provider and model as the application declares them, the
endpoint reduced to a host, the parameters. What is absent is the observed
identity, and the reading says which absence it is rather than inventing a
value — an HTTP run whose document records a writer reads as not_reported,
row 6, and never as row 7. The seventh absence is unreachable over HTTP for a
reason worth stating: it says an echoed id is not a verified identity, and
over HTTP there is no id to echo. An application that echoed its own alias back
would be reporting the field this section closes.
What this implies, and does not repair¶
The paragraph above establishes that over HTTP the provenance partition does not
hold. It does not hold for model either, and model is mandatory and
travels in clear today. An application declares it, declared_config requires
it as a non-empty string, it is in no perimeter set, and it reaches the
projected document — model = "kayak-skills-prod-eu-west-v3" crosses beside a
base_url withheld for describing exactly that, which is the failure the
2026-09-10 correction found on the field next to it and did not look for here.
This section does not repair it, and says so rather than leaving the premise
fallen and the consequence unstated: which of §8's reported fields are
measurements and which are the end company's perimeter is a question about fixed
decision 9, it changes what an existing document carries, and it has its own
record in ADR 0030. Opening
resolved_model here would have shipped half of that decision inside a release
built to let a pilot read its tool calls.
What survives a projection¶
Amendment, 2026-09-27, by ADR 0034 §4, at its acceptance. It changes what crosses one boundary and nothing about what is recorded: §3 holds, and the fields above are still recorded beside the hash and never inside it.
In a projected document, target_config and judge_config are ADR 0034's
class (c): every key and every value is a token or absent, whatever the
endpoint. That includes resolved_model and fingerprint. The perimeter
rule above decides what a redacted document carries, and it is unchanged.
A projection carries less, because its rule is an enumeration: a field that is
neither digline's own vocabulary nor the committing party's own string does
not cross as text. A reported value is neither. The provider wrote it, or the
end company's own application did.
Inside the perimeter nothing moves. The run at the data owner records every field this section admits, and the alias-roll sentence §9 exists for is read there, where the store is. What the software house loses is that sentence in the committed file. Under ADR 0034 §15 that absence is declared in words and never filled in silence.
Consequences¶
- A baseline is now self-contained evidence of the whole experiment: the verdicts, the rules that judged them, the prompt that produced them, and the system that answered.
- The sentence "The configuration is the same as the reference" stops being able to mislead — it is now "The suite is unchanged from the reference", and the rules and the system are two facts with two names.
- A suite that grades with several judges is comparable on the thing that matters most about it: replacing one of two instruments is reported, and reported as the stronger fact.
- A software house can hand world 3 a report saying that a drop coincided with a model change without handing over the endpoint its customer's gateway sits on.
- Every plugin gains one property. The protocol did not gain a mandatory
member, so nothing anyone has written stops satisfying
TargetorJudge. - A run file grows by two small objects. A suite whose target declares nothing
grows by
"target_config": {}. - An application digline cannot import can now be recorded as completely as a plugin, by reporting one object in its answer. The Java path stops being the one where a model change is invisible (§8).
- The configuration contract acquires an enforced form as well as a followed one. A field added to a plugin is a code review; the same field arriving over HTTP is refused until it is added to the table here (§8).
-
Existing report text changed in both locales. A pipeline matching on the English sentence rather than on
--jsonhas to be updated; the JSON keys did not move. -
A baseline records not only which model was asked for but, where the provider says so, which one answered. An alias that rolled under a suite nobody edited becomes a named delta beside the score it coincides with, instead of a silence a reviewer has to already suspect.
- Nothing is re-promoted, nothing is migrated, and no version number moves. The record was already a record of scalars, so admitting two more is the same event as a plugin declaring a parameter it did not declare before.
- Bedrock is honest about having nothing to say, and stays that way. The one provider where an inference profile makes the gap widest is the one that cannot close it, and the record says so rather than echoing the request back.
fingerprintis the second field redaction keeps back, joiningbase_urlfor the same reason: on a custom endpoint its value is written by software nobody here reviews.- The judge's configuration is read after the run as well as before it. No existing plugin, fake or suite behaves differently, and a judge whose alias rolls stops being the one instrument change §4 could not see.
digline-anthropicanddigline-openaigain a live test each whose job is to keep this section honest about how strong the signal is. If an API starts echoing the alias, the test says so before a reader has to infer it from a delta that never fires.
Not decided here¶
Whether a rotated resolved_model should also fail the run, as opposed to
erroring the cases that observed the second one. It errors today, which is
§8's answer, and §8's answer is enough until somebody's run says otherwise.
Recording the observed identity per case rather than per run. It is the
shape that would describe a rotation completely — every case says who answered
it — and it is a CaseResult field, which is a SCHEMA_VERSION bump and a
migration for a fact nothing reads yet. The run-level record plus an errored
case says the same thing in the cases that matter, at no cost to any document.
A resolved_model for HttpTarget — ruled, 2026-09-23, and no longer open.
The condition this entry set was met and the answer was still no: see §9's
closing subsections. The reason is not that nobody had asked, it is that the
perimeter rule's test cannot be applied at that boundary.
prefill. Anthropic's assistant-prefill is text put in the model's mouth,
so it is prompt — the thing under test rather than a parameter of the system
that answers it — and prompt is ADR 0003's subject: a suite that needs it
recorded declares the file it lives in as an artifact, and gets a diff rather
than a scalar. It is also, of the arguments in this family, the one most likely
to carry a customer's own phrasing.
token_param. It decides which argument carries the output cap —
max_tokens or max_completion_tokens — for endpoints that disagree about the
name. It is API plumbing: the cap it delivers is already recorded, and the model
answers the same way whichever spelling reached it.
Both are deliberate exclusions rather than open questions. Reversing either is
an edit to a plugin's config and a line here, not another ADR.
A keyed digest for base_url, which is the same question ADR 0003 left
open for artifacts and gets the same answer: revisit when the bridge exists and
there is somewhere for the key to live.
Whether a judge change should be able to fail a run. It reports today.
Making "reduced comparability" a gate would need a policy about who decides it
and where that policy is declared — a Suite field, presumably — and no
frictions log entry has asked for it yet.