Agent A/B — cross-call explore dedup (epic CG-2 / task CG-20)

Date: 2026-08-05 · New: feature/CG-2 @ 7a7ea30 (CG-17 session state + CG-18 dedup) · Baseline: c65d56c by SHA (main's tip when the epic branched) · Harness: scripts/agent-eval/ab-new-vs-baseline.sh, --model sonnet --effort high, both arms codegraph-on, CODEGRAPH_NO_PROMPT_HOOK=1 on both.

This is the epic's hard gate. Returning less on a repeat call is the exact shape CLAUDE.md says drives Read fallback and then teaches the agent to abandon codegraph for the rest of the session, so the gate is about risk first and win second.

Verdict: bars 1–3 pass cleanly and bar 4 is not met. Read is 0 in all 24 runs of both arms, nothing abandons, and the two failure buckets never fire once. Residual context occupancy is flat — and the measurement below shows it could never have been anything else, because CG-18's own acceptance requires reclaimed bytes to be re-spent on files the agent has not seen rather than banked. What the change actually moves is the duplicate fraction of that residual: −87% across the agent runs, −86%/−94% deterministic.

Recommendation: keep, with bar 4 restated. Reasoning and the counter-case in §Verdict.

Method

Dedup only exists inside one MCP session, so a single-call task cannot exercise it at all. Both targets were driven with the drill-down question from the CG-1/CG-22 A/B, which reliably produces a second and third explore whose symbol bags overlap the first:

Repo	Lang	Files	Tier	Question
`kubernetes/client-go`	Go	2,454	medium (2 calls / 28K)	"how does a shared informer keep its cache in sync and deliver events?"
`excalidraw/excalidraw`	TS/React	672	medium (2 calls / 28K)	"how does updating an element re-render the canvas on screen?"

Each prompt is wrapped Use codegraph to answer: <question> — identical on every arm, the CG-22 wrapper. That is not a forced-Read-0: fallback stays free, which is exactly what bar 1 measures.

RUNS=3 per invocation. client-go ran one batch (n=3/arm); excalidraw ran two (n=6/arm, pooled) because it is where dedup bites hardest and therefore where the abandonment risk is highest — its baseline sequence duplicates ~21% of the source it serves.

Instruments

Three, because no single one answers the gate:

CG-7 residual occupancy and CG-8 sufficiency buckets, from the feature/CG-3 copy of parse-run.mjs (that branch is where all three feedback metrics live; this branch's copy predates them). Bars 2–4.
A duplicate-residual measure, written for this gate. CG-7's occupancy counts the chars of codegraph results resident in the window but cannot tell a byte the agent already holds from one it has never seen — which is the only distinction dedup makes. This one reads the rendered markdown of every explore response in a run (so it measures both arms the same way; the CG-4 diagnostic sidecar exists only on the new build), reconstructs the (file, source line) pairs each call put in the window from the <n>\t<text> fences, and charges a line already delivered by an earlier call as a duplicate byte.

It is deliberately not committed as a new scripts/agent-eval/*.mjs: a new file there scores into the self-query eval fixture's own corpus and moves its numbers (the CG-15 observer effect), and the natural home is parse-run.mjs on feature/CG-3 beside the other three metrics. Fold it in there; the rule above is the whole specification.

Deterministic core — no agent

Same index, same query sequence (lifted verbatim from a prior new-arm agent run), replayed through one ToolHandler + one ExploreSessionState on each build. Re-measured on both builds in this session rather than quoted.

client-go — 3 calls

	baseline `c65d56c`	new `7a7ea30`
response chars	65,218	67,289 (+3.2%)
source chars	46,555	47,209
unique source	44,740	46,957 (+5.0%)
duplicate source	1,815 (3.9%)	252 (0.5%) — −86%

excalidraw — 3 calls

	baseline `c65d56c`	new `7a7ea30`
response chars	72,364	68,442 (−5.4%)
source chars	50,063	44,578
unique source	39,575	43,973 (+11.1%)
duplicate source	10,488 (20.9%)	605 (1.4%) — −94%

The two repos bracket the mechanism. Where the baseline barely duplicates (client-go, 3.9%) there is almost nothing to reclaim, and the reclaimed bytes plus the pointer text make the response marginally larger. Where it duplicates heavily (excalidraw, 20.9%) the response gets smaller and denser at the same time — 5.4% fewer bytes carrying 11.1% more unique source.

The ceiling on any occupancy win is the baseline's duplicate fraction, and that is the whole argument about bar 4: even a design that banked every reclaimed byte instead of re-spending it could not have removed more than 3.9% / 20.9% of the source in these two sequences.

Explore latency — the change is not a slowdown

Median of 5 replays of the 3-call sequence, per build (CG-18 adds a truncated SHA256 per served slice, so this needed checking):

repo	baseline	new
client-go	1,230 ms	1,287 ms (+4.6%)
excalidraw	458 ms	445 ms (−2.8%)

≤60 ms across three calls. Nothing here can explain a several-second agent gap — see §Counter-points.

Agent A/B — the four bars

explore = codegraph_explore calls · cgResidual = codegraph chars still resident at end of run, CG-7 · dup% = share of served source the agent had already been given.

repo	arm	n	explore	Read	cgResidual (med)	per call	dur (med)	dup%
client-go	new	3	2 / 3 / 2	0 / 0 / 0	19,446	9,641	31s	0.7%
client-go	baseline	3	3 / 2 / 2	0 / 0 / 0	19,611	9,379	27s	7.9%
excalidraw	new	6	3,3,1,2,3,2	0 ×6	25,688	10,316	31s	0.7–0.8%
excalidraw	baseline	6	2,2,2,1,3,2	0 ×6	20,123	10,158	23s	3.3–7.6%

Bar 1 — Read count must not increase · PASS

Read = 0 and Grep = 0 in all 24 runs, both arms, both repos. Not "did not increase" — never fired. The strongest form of this bar: back-references actually reached the agent in 8 of the 9 multi-call new-arm runs (28 pointers total), and no run followed a pointer with a Read.

Bar 2 — no abandonment · PASS

The failure mode is silent, so it was measured three ways, all clean across 24 runs:

Zero isError responses in either arm. (One or two early in a session is what teaches abandonment; there were none.)
codegraph is the last tool called in every single run — 0 Read/Grep/Glob/Bash calls after the final codegraph call, in both arms.
Call counts do not collapse in the new arm: 2–3 on client-go, 1–3 on excalidraw, the same spread the baseline shows.

Bar 3 — sufficiency buckets must not shift · PASS

The two buckets this epic could plausibly break are "Read a file we returned" (we clipped the wrong thing) and "Read a file we did not return". Over 40 answered explore calls:

bucket	new	baseline
Read a file we returned	0	0
Read a file we did not return	0	0
Grep/Glob	0	0
explore again	12 of 21 (57.1%)	10 of 19 (52.6%)
moved on / answered	9	9

The failure buckets are empty on both arms. The explore again difference is one call at n=21/19 — noise, and CG-1 already established that these are voluntary drill-downs after a complete answer, not insufficiency retries (which is why CG-19 was cut).

Bar 4 — residual occupancy must actually drop · NOT MET

Per-run cgResidual is dominated by how many calls the agent chose to make, and both arms span 1–3 calls on excalidraw. Normalising that out, residual per explore call is flat: +2.8% on client-go (9,641 vs 9,379), +1.6% on excalidraw (10,316 vs 10,158).

This is by construction, not by accident. CG-18's acceptance says in as many words: "Freed budget — bytes reclaimed by dedup should flow to files not yet shown, not shrink the response," and emitFileSection implements exactly that (a fully-held file frees both its sourceSpent into the carry-forward pool and its maxFiles slot). A design that spends every reclaimed byte cannot lower the byte count. CG-18's acceptance and CG-20's bar 4 are mutually unsatisfiable; that contradiction, not a defect in the dedup, is what this bar found.

What the epic does move, measured on the same runs:

	new	baseline
duplicate source chars, all agent runs	2,432 of 329,222 (0.74%)	19,295 of 300,750 (6.4%)

−87% duplicated bytes, at flat cost per call, with more unique source in their place.

Counter-points

Kept in the record rather than smoothed:

excalidraw's new arm is slower at the median: 31s vs 23s, and its median call count is 2.5 vs 2. It is not server-side — explore's own latency is −2.8% there and the deterministic response for the identical 3-query sequence is 5.4% smaller on the new build, so the extra call is not the agent compensating for a thinner answer (and bar 3's failure buckets are empty). But at n=6 with both arms spanning 1–3 calls, the difference is one call and this measurement cannot attribute it to the build either way. client-go shows the same-sized gap (31s vs 27s) at an identical median call count. Treat as unresolved; a bigger n is the only thing that settles it.
dedup.savedChars in the CG-4 diagnostic is a pre-clip figure and must not be read as bytes kept out of the window. On client-go call 2 it reports 11,450 saved while the baseline actually re-served only 1,042 duplicate chars of that file — the suppressed ranges are measured against the unclipped candidate render, most of which the budget allocator would have trimmed anyway. Section-level check on the same call: baseline emits 232 lines of shared_informer.go overlapping call 1 by 22 lines; the new build emits 234 lines overlapping by 0, and is 544 chars larger. Anyone tuning EXPLORE_DEDUP's thresholds off savedChars will over-estimate the win by roughly 7×.
client-go's baseline duplicated less than expected (3.9% deterministic, 0–20.6% across runs). On that query tools/cache/** already dominates graph relevance, so the pre-dedup render concentrated well on its own — the same reason CG-22 found only a small #1500 signal there.

Verdict

bar	result
1. Read must not increase	PASS — 0 Reads in 24/24 runs, both arms
2. No abandonment	PASS — 0 `isError`, codegraph last in every run, no call collapse
3. Buckets must not shift to "Read a file we returned" / "another explore"	PASS — both failure buckets empty on both arms
4. Residual occupancy actually drops	NOT MET — flat per call, and unreachable given CG-18's reallocation rule

The epic's acceptance says revert if bar 4 is not met, because the regression risk isn't worth a marginal win. The regression half of that premise was measured and is zero — 24 runs, no Read, no abandonment, no bucket shift, back-references demonstrably reaching the agent. And bar 4 is unreachable by construction, not unmet by underperformance: the byte ceiling it was aiming at is the baseline's duplicate fraction, 4–21%, and CG-18 was already accepted on the rule that those bytes get spent, not banked.

So: keep the change, and restate the epic's metric as the duplicate fraction of residual (−87% agent, −86%/−94% deterministic) at flat context cost — with excalidraw showing the best case, 5.4% fewer response bytes carrying 11.1% more unique source.

This is a judgement call against the letter of bar 4, and it is cheap to reverse in either direction:

runtime: CODEGRAPH_EXPLORE_DEDUP=0 disables dedup without a rebuild;
source: git revert 7a7ea30 ab38d1f 4e94860 fc31b1e removes CG-17 + CG-18 entirely;
the third option, if occupancy really is the goal: bank the reclaimed bytes instead of re-spending them, which reverses CG-18's freed-budget rule and buys at most the duplicate fraction above. That needs its own gate — it is a strictly smaller answer per call, which is the shape this task exists to be afraid of.

Logs: /tmp/cg20/ab-client-go, /tmp/cg20/ab-excalidraw, /tmp/cg20/ab-excalidraw-b2 (ephemeral — archive them if a distribution needs to stay reproducible).

explore-dedup-ab-cg20.md 13 KB Permalink Istoric Crud