The problem, in one story¶
I run a small script that reads a few hundred AI articles every morning and asks a language model to pick the five worth reading. It worked. Then one day it started skipping the good ones.
Nothing had crashed. No test had failed. The code was the same as the week before. What had changed was the behaviour of the model — a prompt tweak here, a model update there — and behaviour is not something a unit test checks. The script still returned five articles. They were just the wrong five.
That is an LLM regression: your system still runs, still answers, still passes its tests, and is quietly worse than it was.
Why your tests don't see it¶
Ordinary tests compare an output with an expected value. A language model doesn't give you an expected value. Ask it the same question twice and you may get two different answers, both acceptable. So teams either stop testing the model's behaviour, or they write a test that passes today and can't tell you anything about tomorrow.
The practical questions turn out to be:
- Compared to what? You need a reference — the behaviour you last accepted.
- Worse by how much? A score that moves from 0.88 to 0.86 may be noise. From 0.88 to 0.73 is a regression. Without knowing how much the system wobbles on its own, you can't tell the two apart.
- Who decides what "right" means, and where is that decision written down?
What digline does¶
digline is a gate you put in front of that question. You write down, once, what you expect from your system as a set of checks — deterministic ones (the answer is valid JSON, contains no phone numbers, mentions the customer's name) and judged ones (a second model rates the answer against a rubric). You run them against your system. When you're happy with the result, you promote it: it becomes the baseline, a file committed in your repository next to the code.
From then on, every run is compared with that baseline, and the report answers the only question that matters. Here is a run where nothing got worse, though two checks moved and one case could not be settled:
Nothing got worse compared with the reference. 2 checks moved within noise. 3 checks are on the line: the bands they measured cover their thresholds. Every case could be judged. 1 case is suspended. The suite is unchanged from the reference. 1 file under test changed since the reference.
and here is one where six checks did:
6 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference. 1 file under test changed since the reference.
Both are what digline compare printed on the example suite of the Guide's first chapter, captured from the release the docs come from. The model and the judge in that suite are canned, and no provider is called, so the answers and their costs are fixed by the example and reproduce exactly; the numbers are what digline computed from them.
The baseline is a file in git. It has history, it can be diffed, and nobody promotes it but you. There is no dashboard to log into and no service that receives your data — digline runs where your code runs.
Thirty seconds, with pictures¶
Episode 1 of a short series on LLM regression. Served from this site — no third-party player, no cookies.
Where to go next¶
- Why — the reasoning behind the design, if you want to argue with it.
- Guide — install it and get your first
comparein a few minutes. - The GitHub Action — the same gate on every pull request, and what each exit code means.
- How digline compares — if you're already using an eval tool and want to know what's different.
Ten words you'll meet along the way: run, baseline, compare, promote, check, judge, tolerance, noise floor, regression, gate. Each is explained the first time it appears in the Guide.