digline
EN

Open source, Apache-2.0v0.31.0, pre-1.0

You changed one prompt.What else got worse?

A new prompt, a model version, a library upgrade. The app still runs, and one answer your client relied on is now wrong. digline tells you which case, by how much, and whether the drop is bigger than the judge’s own noise.

pip install digline

Requires Python 3.12 or newer

Start here Why a reference in the repo

  • Quickstart needs no API key
  • No server, no telemetry
  • Exit code for CI
digline comparesupport · exit 1

how-do-i-returngot worse

llm_rubric
1.00 → 0.40
contains
1.00 → 0.00
Also worse
is-it-waterproof, where-is-my-order
prompt.md · +1 −1 lines, changed since the reference
-Sign every reply as "— Northwind Support".+Sign every reply as "— the Northwind team".

6 checks got worse. CI stops here.exit 1

Canned model and judge, 1 sample per case: judge noise is not measured in this demo. Captured with digline 0.31.0, run 2026-10-12T14-02-23-833592… · home.json

The problem

An LLM app can get worse without anything breaking.

The tests pass, the app answers, the demo works. Then a prompt edit, a model version or a library upgrade changes one answer your client depended on, and nothing in the pipeline says so.

How it usually goes

  • Re-run the evals and look at the average
  • No way to tell a real drop from the judge having a bad day
  • What was approved, and when, lives in someone’s notebook
  • The client finds the regression first

With an approved reference in the repo

  • Every change is compared case by case
  • The judge’s noise is measured before a drop is called
  • The reference is a file in a PR, with a commit and a name on it
  • CI stops before the release does

How it fits

It lives in your repo, not on a server.

Nothing to sign up for and nothing to host. The reference is a file, and the tool is a package.

The reference is a file in your repo

Scores, prompt and model config, committed next to the code. It changes through a PR, so there is a commit and a name on every approval.

One dependency, no provider SDK

digline installs jsonschema and nothing else of its own. No server, no telemetry, nothing hosted. In CI the exit code is the whole interface.

Providers

One plugin each, installed only if you use it.

  • Anthropicdigline-anthropic
  • OpenAIdigline-openai and any OpenAI-compatible endpoint
  • Amazon Bedrockdigline-bedrock

Worked examples

A real app on each framework, with its suite.

Other ways to run it

In a container, or for an agent.

Commands

Thirteen commands, one job each.

Every command has its own page or a place in the guide.

  • Record and approve
  • Compare
  • History
  • Maintenance

Checks

Deterministic first, judge last.

22 checks today, in five kinds. The list grows; the kinds stay.

Without a reference, and with one

The same eval run, read two ways.

One line of the prompt changed between two runs. On the left, the scores you get without a reference. On the right, what digline prints against the approved one.

The new run on its own no reference

case               llm_rubric  containshow-do-i-return    0.40        0.00is-it-waterproof   0.40        0.00where-is-my-order  0.40        0.00Were these 1.00 yesterday? Did anything change?Nothing here says.
no baselineno change on recordno exit code

digline compare against the reference

6 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference. 1 file under test changed since the reference.tenant 'northwind', against baseline 2026-10-12T14-02-23-521445-00-00-282b0c02d6511fb4  prompt.md · +1 −1 lineshow-do-i-return · llm_rubric · Went from passing to failing (1.000000 → 0.400000).how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).… 4 more linesexit 1
each check namedprompt change detectedexit 1

Both columns come from the same captured run, digline 0.31.0. The left one shows only the new scores; the right one is the compare output, shortened where marked.

Try it

The quickstart runs without an API key.

You need Python 3.12 or newer. The model call in the guide is canned, so you can see the whole loop before wiring a real provider.

  1. digline run

    Runs the suite and writes a run file.

  2. digline promote

    A person approves that run as the reference.

  3. digline compare

    Every later run is checked against it.

you type
digline prints
$ pip install digline==0.31.0

      
$ digline run --suite support.py

      
2026-10-12T14-02-22-249098-00-00-282b0c02d6511fb4

      
$ digline promote --suite support.py --run latest --replacing none

      
tenant 'northwind' · support baseline set to 2026-10-12T14-02-22-249098-00-00-282b0c02d6511fb4

      
$ digline compare --suite support.py --run latest

      
Nothing got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.tenant 'northwind', against baseline 2026-10-12T14-02-22-249098-00-00-282b0c02d6511fb4

    

    
Output captured from digline 0.31.0, run 2026-10-12T14-02-22-249098…. Full guide

  






  

    
In an agentic flow

    
Where the check sits when an agent is making the changes.

    
The agent edits, digline decides whether the edit can go further.

    Animated diagram of digline in an agentic flow: an agent's output passes through digline, a gate inside your repository, which measures it against a noise floor and the approved baseline. Drift within the floor is absorbed silently; a real drift goes to a human as a dossier, and only the human's signature updates the baseline. When the agent reaches for promote, the server answers that no such tool exists.
    Animated diagram of digline in an agentic flow: an agent's output passes through digline, a gate inside your repository, which measures it against a noise floor and the approved baseline. Drift within the floor is absorbed silently; a real drift goes to a human as a dossier, and only the human's signature updates the baseline. When the agent reaches for promote, the server answers that no such tool exists.
  

  






  

  
Roadmap

  
Where it’s headed

  
Shipped — verdict credibility: a drop is a regression only when
  it leaves the interval the baseline measured across its own samples, and the
  open work now is adoption.

  
Two things will never be on the roadmap. A hosted service that
  receives your payloads — prompts, outputs and the judge stay in your
  perimeter. And usage data collection, ever.

  
the whole
  roadmap, tracks and gates rather than dates →

  






  

  
Pilots

  
Building LLM features for clients?

  
I’m looking for a few pilot teams to run digline on
  real projects — especially teams that build and maintain AI solutions for
  third parties. In exchange you get direct support and a say in what gets built
  next.

  

    Write me:
    hello@digline.dev
    
    or open a GitHub issue
  

  






  

    

      
Start with one suite and one approved run.

      
The guide needs no key.

    

    

      pip install digline
    

    

      why a hash wasn’t enough →
      see it on your own suite →
      what an agent can and cannot do →
      how it differs from the tools you know →