What a lookup can answer
Start with usable metadata coverage and the need for upstream fallback. Then compare modeled request counts and measured payload costs. These reports do not establish end-to-end agent savings.
Lookup report generated 2026-10-28 22:08 UTC · Lookup report (JSON)
Fields present vs. policy acceptable
22.1% pass the build-and-test policy. 479 of 615 profiles (77.9%) require upstream fallback for those two commands.
Build and test
Repository description
Documentation
Build command
Test command
Fields present means every field needed for that task contains a value. Policy acceptable means the reference consumer accepts the profile without source fallback at the evaluation time. Commands require explicit, high-confidence extraction, a source, and a matching check timestamp. Inferred or weaker assessments and candidate commands do not qualify.
Independent correctness and task completion: Not measured. Policy acceptance is metadata for planning, not permission to execute commands.
Policy, per-repository decisions, and fallback reasons (JSON)
Complete instructions by ecosystem
Contextual acceptance additionally requires assessed directory, scope, prerequisites, and source. This comparison requests repository scope at the root; component instructions remain separate. It is still not a correctness or execution-success measurement.
| Dominant ecosystem | Profiles | Accepted scalar pairs | Accepted contextual pairs |
|---|---|---|---|
| Go | 70 | 30 | 0 |
| Other | 243 | 15 | 0 |
| Python | 116 | 6 | 0 |
| Rust | 55 | 15 | 0 |
| TypeScript / JavaScript | 131 | 70 | 0 |
View coverage as a data table
| Task | Fields present | Policy acceptable | Independently correct | Task completed |
|---|---|---|---|---|
| Build and test | 325 (52.8%) | 136 (22.1%) | Not measured | Not measured |
| Repository description | 615 (100.0%) | 614 (99.8%) | Not measured | Not measured |
| Documentation | 151 (24.6%) | 151 (24.6%) | Not measured | Not measured |
| Build command | 361 (58.7%) | 178 (28.9%) | Not measured | Not measured |
| Test command | 365 (59.3%) | 205 (33.3%) | Not measured | Not measured |
Field presence across the research workload
The deterministic research workload asks four fixed questions of each repository: 2460 tasks across 615 repositories. This counts populated values, without applying the consumer policy above.
- Tasks with all fields present
- 47.6%
- Requested fields present
- 71.9%
- Requested fields missing
- 28.1%
Missing fields alone do not establish correct abstention. None of these presence rates establish factual accuracy or successful task completion.
| Intent | Tasks | All fields present | Fields present | Fields missing |
|---|---|---|---|---|
| Overview | 615 | 75.0% | 93.3% | 6.7% |
| Execution (build & test) | 615 | 52.8% | 59.0% | 41.0% |
| Documentation | 615 | 24.6% | 24.6% | 75.4% |
| Security stewardship | 615 | 38.0% | 38.0% | 62.0% |
Modeled requests and measured payloads
Batching reduces the modeled lookup count. It does not remove fallback work, and fewer requests do not necessarily mean fewer bytes.
Requests · modeled
- dotrepo batch GETs
- 13
- Local proxy file fetches
- 1230
98.9% modeled request reduction. Excludes source fallback, refresh work, cache behavior, retries, and model inference.
Artifact size · measured
- dotrepo profiles + query input
- 6.7 MiB
- Local record + evidence proxy
- 2.8 MiB
The dotrepo artifacts are 2.37× the proxy size. These are local artifact sizes, not live GitHub traffic or measured batch-response bytes. 1 MiB = 1,048,576 bytes.
How the request model and byte counts work
For 615 repositories and
9 distinct requested fields,
the harness batches up to 50 repositories,
25 paths, and 500 results per request.
The proxy counts one fetch for each unique checked-in record.toml
and evidence.md file needed for the same repositories.
Byte totals count each unique public profile.json and
query-input artifact once, then compare them with the local proxy files.
Evidence and freshness metadata have a payload cost; other extraction systems
can preserve them too.
Correct abstention and factual accuracy require independently sourced expected answers. The head-to-head benchmark retains favorable and unfavorable results. Historical fixes do not establish current out-of-sample accuracy.
Actual task cost must include fallback requests, source bytes, latency, model usage, and an allocated share of index maintenance. No measured end-to-end savings claim is made here.
Reproduce it
The workload builder and measurement harness are deterministic and ship in the repository. The release gate re-runs them against a versioned baseline on every release.
uv run python scripts/build_public_lookup_workload.py \ --public-root public --mode research --limit 0 \ --output /tmp/workload.json uv run python scripts/measure_public_lookup_efficiency.py \ --public-root public --index-root index \ --workload /tmp/workload.json \ --output-json /tmp/lookup-efficiency.json uv run python scripts/measure_public_policy_coverage.py \ --public-root public \ --output-json /tmp/policy-coverage.json
Methodology details: docs/public-lookup-efficiency-benchmark.md