truescore
Statistics for LLM-judged evaluations: bias-corrected scores, valid confidence intervals, and model comparisons that survive review.
pip install truescore truescore doctor your_eval_results.csv
On real data
MT-Bench published GPT-4's pairwise judgments and human judgments over the same 1814 comparisons. GPT-4 agrees with the humans 88% of the time, and still reports its own win rate 12.7 points above what they give it.
| model | GPT-4 as judge | human | gap |
|---|---|---|---|
| gpt-4 | 0.8521 | 0.7249 | +0.1272 |
| claude-v1 | 0.7146 | 0.6776 | +0.0370 |
| gpt-3.5-turbo | 0.6158 | 0.6431 | -0.0273 |
| vicuna-13b-v1.2 | 0.4615 | 0.4735 | -0.0120 |
| alpaca-13b | 0.2388 | 0.2938 | -0.0550 |
| llama-13b | 0.1154 | 0.1776 | -0.0622 |
A 9.3-point self-preference survives controlling for the judge exaggerating the quality spread (p = 7.5e-07). Full write-up, with a script that downloads the data and prints every number.
Background
A support assistant, 4000 questions graded by an LLM judge, 600 of them also graded by a
person. The data is simulated, so the real pass rate is known: 0.7140.
| estimate | value | 95% interval | contains the truth? |
|---|---|---|---|
| judge only | 0.8383 | [0.8265, 0.8493] | no |
| corrected | 0.7150 | [0.6836, 0.7463] | yes |
| human labels only | 0.7183 | [0.6810, 0.7528] | yes, but wider |
The judge reports 12 points high and its interval stops short of the true value. The corrected interval contains it, and is narrower than the one from the 600 human labels alone.
Comparisons carry more error
| v4 minus v3 | |
|---|---|
| judge says | +0.1700 |
| corrected | +0.0950 [+0.0505, +0.1394] |
| actual | +0.0900 |
v4 is better, by about half what the judge reports. The gap traces to answer length:
v4's median answer runs 197 tokens against v3's 99, and the judge's measured length bias is
+0.069 per 100 tokens (p = 1e-05).
Split the same evaluation by support segment and the judge reports an improvement on
all three. One regressed by 17 points, the one where answers grew longest. Per-segment
correction returns -0.1659 against a planted truth of -0.1663.
Repeated checks invalidate a fixed-sample interval
A 95% interval is valid at a sample size chosen in advance. Inspected repeatedly as data arrives, it is not. Measured over 300 simulated healthy streams:
| checked after every observation | false alarms |
|---|---|
| fixed-sample interval | 47.7% |
| truescore confidence sequence | 0.3% |
Quick start
doctor reads your file, works out what your columns are, and tells you what
it can compute and what's blocked:
$ truescore doctor results.csv
results.csv: 4000 rows, 5 columns
column kind coverage detail
judge_passed verdict 100% pass/fail on every row
human_passed sparse_verdict 15% pass/fail on 600 of 4000 rows
response_tokens numeric 100% range 35 to 1016
what this file supports today:
- correct the score for judge error
- monitor a release for regressions, no human labels needed
what the judge appears to be biased by:
- response_tokens: the judge gets more generous as it rises
(+0.00069 per unit, adjusted p=2.23e-05)
In CI
- uses: SaifPunjwani/truescore@v0.7.2
with:
command: drift
args: anchor.csv --baseline judge_pinned --current judge_today --gold human
Exit codes are 0 for no finding, 2 for a finding, and 1 for a failure to run. A CI job can fail on judge drift without also failing on a malformed file.
Human labels
The correction requires trusted labels on a subset of examples. Calibrating a measurement instrument requires a reference to calibrate against.
Roughly half the library runs without any: regression monitoring, contamination testing, position-bias detection and label planning use judge output alone. Where labels are needed it is a few hundred, they serve every future release against that evaluation set, and many teams already hold implicit human verdicts (escalations, thumbs-down, refunds, repeat contacts).
The library prices it before you spend anything:
+/-0.05: 232 labels (without the judge: 317) +/-0.03: 670 labels (without the judge: 879) +/-0.02: 1644 labels (without the judge: 1978)
Validation
Every interval estimator carries a coverage test: simulate from a known ground truth several hundred times and confirm the 95% interval covers about 95% of the time. Confidence sequences are checked over whole trajectories.
The suite fuzzes every public function under one rule: any input produces a finite result or raises. That pass found three defects before the first release, including a near-deterministic slice whose interval covered 7.7% of the time.
663 tests. mypy --strict. numpy and scipy are the only dependencies. truescore never calls a model.