Benchmark
Every figure here is computed deterministically — real tokenizer, real primitives, no model and no estimation. The one thing a model is required for, end-to-end answer accuracy, is reported as unmeasured rather than filled in with a plausible number.
This is not a reproduction of the paper. The paper reports a 92.0% mean (95.3% best case) over a 240-document corpus that is not public. This is an independent measurement on 6 documents authored for this project, with 52 questions. The gap is large and the reason is measurable — see below.
Why the gap: reduction scales with document size
Traversal's cost is roughly flat — an index, a few dense summaries, two resolved nodes. Injection's cost grows linearly with the document. So reduction is not a fixed property of the format; it is a function of how much document you are not reading. Across questions, the correlation between document size and reduction is r = 0.52.
| Document | Class | Size | Injection | Traversal | Reduction | |
|---|---|---|---|---|---|---|
| skillfile | Skill File | 667 | 684 | 481 | 29.7% | |
| kb | Knowledge Base | 806 | 823 | 301 | 63.4% | |
| plan | Execution Plan | 819 | 836 | 321 | 61.6% | |
| runbook | Operational Runbook | 924 | 943 | 443 | 53.1% | |
| techdoc | Technical Documentation | 957 | 975 | 231 | 76.3% | |
| handbook | Knowledge Base (large) | 2,160 | 2,177 | 339 | 84.4% |
The largest document here is still only 2,160 tokens. Real handbooks and API references run 10–50k. The trend suggests the paper's 92% is reachable, on documents larger than any in this corpus — but this benchmark does not demonstrate that, it only points at it.
By task type
| Task | n | Mean reduction |
|---|---|---|
| information-lookup | 23 | 73.7% |
| procedure-execution | 4 | 57.6% |
| multi-step-planning | 5 | 46.1% |
| role-conditional-access | 1 | 68.1% |
| cross-node-reasoning | 10 | 58.4% |
| update-detection | 2 | 44.1% |
| assertion-verification | 5 | 60.7% |
| multi-agent-handoff | 2 | 79.4% |
Cost of the defence
Two earlier versions of this gate were discarded for firing on 38% and 28% of a clean corpus. A defence that fires constantly spends the token saving it exists to protect, so false-positive rate is the binding constraint here, not detection rate.
Answer accuracy
not measured
End-to-end answer accuracy requires a language model. No ANTHROPIC_API_KEY was available when this report was generated. The harness exists (scripts/accuracy.ts) and will populate this section when run with a key. It is deliberately left empty rather than estimated.
This matters. A token reduction that quietly broke correctness would be worthless, and nothing on this page rules that out. It is the first thing I would measure with a key.
Limitations
- 01n = 52 questions over 6 documents, all authored for this project. This is not the paper's 240-document corpus, which is not public, so these figures are an independent measurement and NOT a reproduction of the paper's 92.0% mean.
- 02Token counts use cl100k_base, not Claude's tokenizer, which Anthropic does not publish. Both lanes are measured with the identical tokenizer, so the ratio is sound even though absolute counts drift a few percent.
- 03Traversal cost assumes the agent resolves its top two index candidates. A model that explores more pays more; a model that guesses better pays less. The figure is a model-independent floor, not a prediction of live behaviour.
- 04Reduction is computed against single-turn injection. A real agent re-injects the document on every turn, so multi-turn savings would be larger. Single-turn is the conservative comparison.
- 05Answer accuracy is unmeasured. A token reduction that broke correctness would be worthless, and this report cannot yet rule that out.
- 06Attack effectiveness is measured on routing, not on final answers. A changed route usually means a wrong answer, but the link is inferred rather than observed.
method
- tokenizer
- cl100k_base (gpt-tokenizer) as a proxy for Claude's tokenizer, which is not public
- model
- none — every figure below is computed deterministically
- corpus
- 6 documents authored for this project
- questions
- 52
- taskTypes
- 8
Regenerate with npm run bench