Agents that need basic repository facts — what a project is, how to build and test it, where the docs and security policy live — usually re-scrape and re-read each repository from scratch. This benchmark measures the alternative: answering a fixed research workload of 2452 tasks across all 613 indexed repositories from the dotrepo public surface alone.
Report generated 2026-07-09 05:40 UTC · regenerated with each release gate · raw JSON
The workload asks the same four questions of every repository, chosen before looking at which answers exist — so the numbers cannot flatter the index by only asking questions it can answer.
| Intent | Tasks | Task hit rate | Field hit rate | Abstention |
|---|---|---|---|---|
| Overview | 613 | 74.1% | 93.2% | 6.9% |
| Execution (build & test) | 613 | 56.8% | 63.5% | 36.5% |
| Documentation | 613 | 71.8% | 71.8% | 28.2% |
| Security stewardship | 613 | 35.9% | 35.9% | 64.1% |
The request reduction is the headline: one agent research pass over the whole index needs 13 cacheable GET requests instead of 1226+ per-repository fetches, before counting the many requests a real scrape spends on READMEs, manifests, and CI files per repository.
The payload comparison is deliberately conservative. The compact dotrepo payload for the full workload is 3.4 MB; the scrape proxy it is compared against is 1.5 MB of already-extracted local records — not the far larger cost of fetching and model-reading raw repository material. dotrepo's structured payload includes trust, provenance, evidence pointers, and freshness context that raw scraping does not produce at any cost.
Abstention is counted as a feature, not padded over: when the index does not know a build command or security contact, it says so. A fabricated answer would score better here and be worse everywhere it matters.
The workload builder and measurement harness are deterministic and ship in the repository. The release gate re-runs them against a versioned baseline on every release.
uv run python scripts/build_public_lookup_workload.py \ --public-root public --mode research --limit 0 \ --output /tmp/workload.json uv run python scripts/measure_public_lookup_efficiency.py \ --public-root public --index-root index \ --workload /tmp/workload.json \ --output-json /tmp/lookup-efficiency.json
Methodology details: docs/public-lookup-efficiency-benchmark.md