digline

Regression testing for LLM applications

with the baseline in your repository, not on someone’s server.

A run below the reference and above the threshold The approved reference sits at 0.91. A dashed threshold sits at 0.70. The latest run leaves the reference, falls to 0.74, comes back up and settles at 0.78 — under the reference, still over the threshold. reference 0.91 threshold 0.70 0.78
was 0.91, now 0.78, still above the threshold
pip install digline

Requires Python 3.12+.

What it looks like

$ digline compare --suite support.py --run latest1 check got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference. The system under test answered under a different configuration: temperature 0.3 → 0.7.system · temperature 0.3 → 0.7where-is-my-order · llm_rubric · Score fell from 1.000000 to 0.700000. This drop coincides with temperature 0.3 → 0.7.

why a hash wasn’t enough → see it on your own suite →

The failure has no error message

Your prompt worked on Tuesday. On Thursday it works a little less, not enough to break, enough for a user to notice in two weeks. No ordinary test catches it: there is no correct output to compare against, only a better or a worse one.

An approved reference, and the distance from it

digline gives you an approved reference, the baseline, and on every change tells you whether you are below it: which case, which check, by how much.

The baseline is a file you own

The baseline is a JSON file in your repository, so it goes through code review and it rolls back with git. No server, no account, no network call you have not configured yourself.

Roadmap

Where it’s headed

Being built now — verdict credibility. Repeated runs scored as a distribution rather than one sample, and thresholds set against measured variance: a drop is a regression only when it clears the noise floor of the case.

Two things will never be on the roadmap. A hosted service that receives your payloads — prompts, outputs and the judge stay in your perimeter. And usage data collection, ever.

the whole roadmap, tracks and gates rather than dates →

Pilots

Building LLM features for clients?

I’m looking for a few pilot teams to run digline on real projects — especially teams that build and maintain AI solutions for third parties. In exchange you get direct support and a say in what gets built next.

Write me: hello@digline.dev or open a GitHub issue