Refeed Raptor - a local RAG lab

Is the decision model calibrated?

The loop asks d1-3B one question: is the evidence enough? The model returns a probability, the loop trusts it in the high band and delegates in the low band. This page checks whether the probability means what it says.

questions

0.940

citation hit rate

0.055

Brier score

delegated to LFM

Place raptor-reserve. Mean predicted probability 0.912, observed accuracy 0.940. A perfectly calibrated model sits on the dashed diagonal: when it says 0.8, it is right eight times out of ten.

The reliability curve

Each circle is a 0.1 probability bin. The number in the circle is how many questions landed in that bin; the circle size follows the same count. The diagonal is perfect calibration.

The curve is the whole argument for a calibration band. A decision model that says 0.9 and is right half the time is not a small model with a cheap opinion, it is a random number with a latency cost. A model that says 0.9 and is right 0.9 of the time can gate a loop.

Where the questions land

Most questions sit high, which is what a 100-document corpus with 50 curated questions should look like.

Bin	Questions	Mean predicted	Observed	Gap
0.0-0.1	0	-	-	-
0.1-0.2	0	-	-	-
0.2-0.3	0	-	-	-
0.3-0.4	0	-	-	-
0.4-0.5	1	0.452	0.000	-0.452
0.5-0.6	1	0.526	1.000	+0.474
0.6-0.7	2	0.643	1.000	+0.357
0.7-0.8	0	-	-	-
0.8-0.9	5	0.864	1.000	+0.136
0.9-1.0	41	0.952	0.951	-0.001

The bands

The loop keeps a d1 answer in the high band, delegates a low confidence answer to LFM2.5, and treats the middle band as its own. This table shows what each band was worth on this set.

Band	Questions	Delegated	Accuracy
low	4	4	0.750
middle	3	0	1.000
high	43	0	0.953

45 of the ten probability bins are empty. A gap in the curve is not a failure; it is the corpus saying that the questions are either clearly answerable or clearly not.

What correctness means here

Correct means the answer cites a document that holds the gold answer. That is a cheap automatic proxy, and it is not a faithfulness score. A sentence can be true and cite the wrong record, and a record can be right while the sentence around it is wrong. The recorded turns are checked by hand for exactly this reason.

The Brier score is the mean squared distance between the probability and the outcome. Lower is better. A model that always says 0.5 on a balanced set scores 0.25; this one scores 0.055.

Reproduce it

The input is the committed loop report:

pixi run eval -- --place raptor-reserve --modes --unknown \
  --output lab/demos/raptor-reserve/evals/modes.json
pixi run calibrate

The report keeps every per-question row, including the probability, the band, and the citations, so each number on this page can be checked against the raw file.