refeed.raptor.today github

Watch the loop

A 23 second silent walkthrough: recorded replay, the refeed turn, the live question, and the retrieval eval on the same 50 questions.

The captions are burned in. The replay turns need no model, so this clip works on a machine without a GPU.

Turn 7 is the beat the eval report measures. One-shot retrieval answers 67 osprey chicks and cites two unrelated records. The loop splits the question, looks again, and answers 21 chicks with the season report and the drone log, after one refeed.

What this is

Most RAG demos stop at embed, retrieve, answer. This lab asks what happens when the first retrieval is not enough. A small decision model checks the evidence, sends the query back when it is thin, and says when it is unsure. The writer only writes after the evidence passes.

The goal is not another chatbot. The goal is to make the pipeline visible: the chunks, the tree, the decisions, the rewrite, the citations, and the cost.

The first corpus: Raptor Reserve

Raptor Reserve is a fictional island reserve with real bird species and 100 original documents. The facts are not in the model's pretraining, so the answers have to come from the corpus.

85 documents are reserve records: governance, species, field operations, research, visitors, infrastructure, and history. 15 are data and methods notes: RAPTOR, SearchJev, EmbeddingGemma 2, LFM2.5, d1-3B, and the metrics.

The eval set has 50 questions: 30 reserve-specific, 10 methods, and 10 multi-hop. The drone wind question is the key demo. It needs the 28 km/h limit from the checklist and the flight approval from the patrol log. One-shot retrieval often misses one of them.

The loop

1. Embed the question with EmbeddingGemma 2.
2. Retrieve the top 20 chunks and tree summaries with cosine similarity.
3. Score each result with d1-3B and keep the top 8. A passage is judged against the query that found it, so a rewrite does not throw away the passage it found. Leaf chunks are preferred as evidence; a summary is used only when no chunk passes the floor.
4. Ask d1 whether the evidence is sufficient. On a low-confidence answer, delegate the call to LFM2.5. If the evidence is still thin, d1 picks an action, LFM rewrites the query, and the loop retries. The limit is two refeeds.
5. Give the writer up to four leaf chunks whose titles match the question. LFM2.5 writes at temperature 0, and every sentence without a citation gets one, or a sources footer when nothing matches.
6. Ask d1 whether the answer is grounded. One repair pass is allowed, then the answer ships with a warning if needed.

Every decision is a typed question with a probability. The trace shows the probability, the confidence band, and whether a low-confidence call was delegated. Replay can run the stored trace with no model at all.

The stack

One llama.cpp router, three presets, one CUDA context, one port.

About 8.3 GB of model weights. The test card is a 12 GB GPU and the three loaded presets use about 11.1 GB.

What we measure

Retrieval, answer quality, calibration, cost, and refusal. The lab compares keyword search, one-shot RAPTOR retrieval, and RAPTOR plus the loop on the same 50 questions. The report keeps the failures.

RAPTOR retrieval on its own: Recall@5 0.96, Recall@10 0.98, Recall@20 1.00, MRR 0.861. The keyword floor wins Recall@5 by one question. RAPTOR ranks the right passage better and finds every gold document by rank 20.

The loop is slower and, on this curated set, one citation hit worse. That is the trade. It fixes q05, where the one-shot writer cites a survival analysis instead of the season report. It breaks q08 and q29, where the answer is supported by the record it cites but that record is not on the gold list. Both modes miss q47, which cites a record that holds the ferry fact while the gold list names two other records. The metric counts gold documents, and the report says so.

The loop earns its keep on turn 7 of the demo: one-shot says 67 osprey chicks and cites unrelated records, the loop says 21 with the season report and the drone log after one refeed. Both answers are in evals/script.json.

The refusal set is the other side of the contract. 12 questions the corpus cannot answer, all 12 returned as "not enough evidence", at a mean of 1.1 s.

Calibration: the first sufficiency probability has a Brier score of 0.055 over the 50 questions, with 4 decisions delegated to LFM. The reliability curve is a page of its own: calibration.html.

Every eval artifact is committed under lab/demos/raptor-reserve/evals/. The full reference lives in mds/metrics.md.

Why local

Because the point is to see the whole pipeline. A hosted model hides the retrieval, the prompt, the judge, and the cost. A local stack lets you change one piece and rerun the same 50 questions. It also keeps the corpus on the island, which is the right default for a reserve's records.

Notes

• No cloud model and no API key. The only network call is the local llama.cpp router.
• The reader supports live RAPTOR, committed recorded runs, and a model-free keyword replay. The recorded runs live in lab/demos/raptor-reserve/runs/.
• The corpus generator is the source of truth. The Markdown in corpus/ is the build output.
• The current demo video is a silent walkthrough with burned-in captions. A narrated version is the next media artifact.

Deep dives

One file per technology, written by hand, with the numbers from this lab.

• Metrics - retrieval, answer, calibration, and cost metrics, and how to read them
• Calibration - the reliability curve of the d1 sufficiency decision
• System 1 decisions - typed questions, the confidence band, and when the writer takes over
• The refeed loop - retrieve, judge, rewrite, retry, cite, check
• The RAPTOR tree - the build, the summaries, and what the tree really buys
• UMAP and GMM - the clustering behind the clusters
• The local router - one llama.cpp server, three presets, a pinned build
• The model stack - EmbeddingGemma 2, LFM2.5, and d1-3B
• SQLite and numpy - the database, the vectors, and the fingerprint
• The app - the four tabs, the recorded replay, and the tests
• The place contract - the demo format and how an agent adds a place

Run it

The lab expects a pixi workspace named rptr, a CUDA host, and about 12 GB of VRAM.

1. pixi install
2. pixi run models to download the three GGUFs, then pixi run llm-up
3. pixi run index, pixi run embed, and pixi run build-tree
4. pixi run serve

Optional: pixi run phoenix-up starts the local trace collector, and RPTR_OTLP_ENDPOINT=http://127.0.0.1:6006/v1/traces sends every run to it as OpenTelemetry spans.

Open http://localhost:8000. Ask and live replay need the router. Recorded replay and the keyword floor do not. The corpus can be regenerated with pixi run build-corpus.