codegraph/docs/benchmarks/agent-eval-feedback-metrics.md

The three explore feedback metrics — start here

The agent-eval harness reports three metrics on every run. They are not three views of one number; each answers a different question, and a retrieval change can move one without moving the others. This page says which is which, which harness to run, and how to read the output. The per-metric docs carry the derivations and the caveats — read the one that matters once a number moves.

Metric	The question it answers	Doc
Residual context occupancy (CG-7)	How much of the window does this arm's retrieval still hold when the run ends — i.e. what does every following turn have to work in?	residual-context-occupancy.md
Explore sufficiency (CG-8)	Was a response enough? Read off what the agent did next: explored again, read a file, or answered.	explore-sufficiency.md
Allocation efficiency (CG-9)	Of the bytes a response spent, what share went to files the answer actually drew on?	explore-allocation-efficiency.md

All three are harness-only: parsed out of transcripts we already write. Nothing is emitted from the product and nothing leaves the machine.

---

Which harness

Pick by the question you are actually asking. All three metrics print in both.

Isolating a retrieval change — ab-new-vs-baseline.sh. New build (HEAD) vs a baseline build (a git ref), both arms codegraph-on, same task. This is the harness the three metrics were built for: with codegraph on in both arms, every number is measuring the change rather than adoption.

RUNS=3 scripts/agent-eval/ab-new-vs-baseline.sh /tmp/codegraph-corpus/express \
  "Add a charset option to res.send and wire it through" main

It builds each arm, indexes a throwaway copy of the target, pre-warms a codegraph daemon per run, runs the task RUNS times per arm, prints the three metric blocks under each run, and ends with the side-by-side table below. The pre-warm is load-bearing and must not be removed: without it the agent dives into Read/grep before codegraph finishes its ~2–3s startup, and the run measures attach latency instead of retrieval.

With vs without codegraph — run-all.sh. Codegraph-on against an empty MCP config. A different question: displacement and adoption, not the effect of a change. Multi-turn is where occupancy is actually charged, so separate turns with ||.

scripts/agent-eval/run-all.sh /tmp/codegraph-corpus/gin \
  "How does gin route requests through its middleware chain?||\
Where is the 404 / no-route case handled in that same chain?"

CG_ARMS=with|without re-runs one arm without redoing the other; the comparison table still renders against whichever arm's logs are already in $AGENT_EVAL_OUT.

A campaign — bench-readme.sh. The 7 README repos, three turns each, RUNS per arm, through run-all.sh — so every run in a campaign carries all three metrics. Aggregate with parse-bench-readme.mjs. One has been run: the 2026-08-05 baseline (sonnet, 3 turns, 4 runs/arm) — read its regime box before comparing anything to it, and note that it is not the regime the README's table was published in.

A log you already have. parse-run.mjs <run.jsonl> [run.tN.jsonl …] prints the three blocks for any stream-json log; --brief drops the numbered call transcript. parse-session.mjs <project-dir> does sufficiency and allocation for an interactive session. compare-arms.mjs <out-dir> <label>… builds the table from logs on disk, at any time, for any labels.

Model policy, both harnesses, not negotiable: --model sonnet --effort high on every arm, both arms the same model. Sonnet is the deliberate floor — an affordance that lands on it generalizes up to every host; one that only works on a stronger model does not generalize down to the agents most users have.

---

Reading the output

Each run prints its three blocks (see the per-metric docs for the shape of each), then one table puts the arms side by side:

====== ARM COMPARISON — /private/tmp/cg22/ab-express ======
                                                           new                baseline
  runs                                                       3                       3

  behavior
    duration (s)                                    24 [18–35]              26 [24–30]
    Read                                                     0                       1
    codegraph calls                                    2 [1–2]                       2

  residual context occupancy (CG-7) — tokens still resident at end of run
    codegraph residual (tok)             11,549 [7,193–12,591]  10,388 [10,373–10,447]
    file-access residual (tok)                     231 [0–242]     1,661 [1,306–1,663]
    → retrieval residual (tok)           11,780 [7,193–12,833]  12,034 [11,753–12,051]
    → share of final context               23.3% [15.8%–24.9%]     23.8% [23.4%–23.9%]

  explore sufficiency (CG-8) — pooled over every answered explore call
    answered explore calls                                   5                       6
    explore again                                       2  40%                  3  50%
    Read a file we returned                              0  0%                  3  50%
    Read a file we did not return                        0  0%                   0  0%
    Grep/Glob                                            0  0%                   0  0%
    moved on / answered                                 3  60%                  0  0%

  explore allocation efficiency (CG-9) — share of returned bytes the answer cited
    pooled efficiency                                    96.9%                   82.0%
    per-run efficiency                     100.0% [92.5%–100.0%]   81.9% [81.9%–82.0%]

  contamination — the CLI must never be how codegraph is reached
    CLI calls that RETURNED output                           0                       0
    CLI attempts blocked                                     0                       0

That is the real CG-22 express pass, and it is a worked example of all three reading together: the baseline spent 18% of its envelope on a file no answer ever cited, so the agent read a file we had already returned in 3 of 6 calls and the run ended at 82% efficiency. The new build ships the right bytes — 0 of 5 in that bucket, 96.9% — for about the same residual. Occupancy alone would have called these arms equivalent.

The table is "did it move?"; the per-run blocks are "why?" Only the blocks name the query that fell short and the file the agent went and read instead, which is usually enough to reproduce a miss with probe-explore.mjs.

Which bucket points at which fix

The sufficiency buckets are chosen so each maps to a distinct fix, and two of them tie directly to the other metrics:

• Read a file we returned → allocation: right file, wrong bytes. Expect allocation efficiency to be soft on the same runs, and note the asymmetry — efficiency scores a cited file at 100% of its section even if the agent then had to read it for the part we clipped. This bucket is what catches that.
• Read a file we did not return / Grep/Glob → recall: the file never surfaced. Allocation efficiency cannot see this at all; the envelope was simply missing something.
• explore again → ambiguous by construction. It says the response did not answer, not whether that was allocation or recall. The follow-up query usually says which.
• moved on / answered → sufficient, which is not the same as correct.

Efficiency is not value, and occupancy is not sufficiency. A response can be 100% efficient and useless — one small file the answer names in passing — and a small residual is only good if the answer was still right. Read all three, which is the point of wiring them into the same run.

---

Caveats that survive the summary view

Each metric's doc has the full list. These are the ones that change how you should read the table itself:

• Allocation efficiency is relative, not absolute. Attribution is by citation, and an agent can use a file without ever naming it — to rule it out, or to build a model it writes up from elsewhere. The error is one-sided. Only compare builds on the same question, and never quote the number as "codegraph wastes N% of what it returns." The corpus median sits in the eighties because these are flow questions whose answers walk the whole chain; the discrimination lives at p25 and below.
• Occupancy shares do not transfer between hosts. These are Claude Code on a nominal 200k window (CG_WINDOW_TOKENS overrides it). Window size, system prompt, and compaction policy all differ elsewhere. The ratio between the arms is the part that travels; the percentages are not a claim about Cursor.
• Compare the right pair. In a with/without A/B that is codegraph's residual against the without-arm's file-access residual (Read + Grep/Glob + Bash) — the two ways an agent gets the same bytes into its head. Counting only the Read tool scores as "read nothing" a run that reached for cat through Bash.
• Sufficient is not correct, and a Read is a vote rather than a proof. The bucket is still the right signal — the agent read because something was missing — but a single call is noisy.
• Small-n, always. Runs make 1–5 explore calls, so one run's percentages are coarse. The table prints median [min–max] for exactly this reason: report the range. Use RUNS>=2, and a campaign for a verdict.
• Subagent contexts are not counted in occupancy. A Task subagent has its own window and only its summary returns. Sufficiency does follow the subagent thread (a delegation is judged by what the subagent did first), so the two metrics treat delegation differently on purpose.
• Deferred tool schemas land in occupancy's base. codegraph_explore is deferred: ToolSearch pulls the schema in later, and that injection is not a tool result. The fixed-overhead line prices the part present from the start.

---

Contamination — read this row first

Both harnesses run every arm with the codegraph CLI blocked: a PATH with the binary symlinked out, plus a PreToolUse hook that blocks absolute-path invocations (no-cli-shim.sh, shared by both). Both layers exist because both were needed — an agent denied codegraph on PATH ran find / -iname "*codegraph*" and invoked it by absolute path.

The contamination row is the detection half, and it is not redundant with the prevention half: prevention fails silently the next time the binary lands somewhere new.

• In a with/without A/B, a CLI call means the without-arm was not without codegraph. 14 of 15 without-arm runs in one 7-repo pass did this before the shim existed; any older result from this harness should be assumed contaminated.
• In a new/baseline A/B, both arms are codegraph-on, so a CLI call is not a leak but an attribution failure that breaks all three metrics at once: output arriving through Bash is charged to Bash in the occupancy table, and an explore issued through the CLI is not a tool call at all, so it never reaches the sufficiency classifier or the allocation parse. The run silently drops calls from every number above it.

CLI attempts blocked is benign — the agent tried, nothing entered the window. CLI calls that RETURNED output is not.

---

Tests

node scripts/agent-eval/parse-run.mjs --selftest     # 68/68

Covers all three metrics over synthetic transcripts with known answers: the occupancy math (calibration, eviction, compaction), every sufficiency bucket plus the same-message / thread / delegation rules, and the allocation citation channels with their guards. See each metric's doc for the case list.