There is a feed: /feed.xml. Full posts stay here.

2026

16 October 2026 — The more samples I took, the less my noise floor could see

A noise floor built from the range of K single samples, compared against a mean: why it widens as K grows while the measurement behind it gets better, what it adds on two real suites, and what should replace it.

29 September 2026 — The check was written well, and it could not fail

A faithfulness check on an LLM call scored 1.000 because the prompt already forbade what it measured. What a green check says, how to make one red on purpose, what that costs, and what it cannot find.

25 September 2026 — Four ways an agent walked past my command hook

Four times in two days, a guard of mine named the act it guarded, and the act arrived under another name. In the order it happened, with three checks to run.

23 September 2026 — The field you added changed the answer you already had

Asking an LLM call for one more field changed how it scored everything else, and moving the field after the score didn't undo it. Measured on one judge, eight runs, $0.88.

18 September 2026 — The case that errors is usually the case that was failing

A case that errors drops out of the denominator, so precision and recall get measured over a smaller suite than the run you are comparing against. The score goes up.

13 September 2026 — Bad evals, my own: five exercises from two LLM judges

I applied the reading Dan Luu applies to other people's benchmarks to my own two LLM judges. Numbers first, explanations after.

9 September 2026 — My LLM eval cried wolf. Here's what I measured.

A case went from 5/5 to 2/5 with nothing changed. How I measured the noise floor of an LLM-judged eval, what it caught the week after, and where it still can't see.