Przeglądaj źródła

docs(benchmarks): record the CG-31 A/B — no regression, four repos stop truncating

Deterministic (6 repos, clean rebuilds, both builds): four deliver more source
and one more file each, two are byte-identical, none deliver less. Agent A/B
(django n=3, okhttp n=2, gin n=2, sonnet/effort high, both arms codegraph-on,
0 contamination): the new arm is faster on all three, Read at or below
baseline, occupancy lower.

Also records the two corrections the suite forced on the first cut of the
guard, and the two residuals CG-26 inherits — the render loop's 600-char
epilogue margin (a sweep was run and deliberately NOT shipped) and the BUY
arm's source-space-only guard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Colby McHenry 1 miesiąc temu
rodzic
commit
be7c968439
2 zmienionych plików z 141 dodań i 0 usunięć
  1. 1 0
      CHANGELOG.md
  2. 140 0
      docs/benchmarks/explore-displacement-guard-ab-cg31.md

+ 1 - 0
CHANGELOG.md

@@ -29,6 +29,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - When a file changed on disk after its last index sync, `codegraph_node` and `codegraph_explore` could return a different symbol's code under the requested name — current file bytes cut at outdated line positions — while presenting it as verbatim, trustworthy source. This hit hardest on projects queried through `projectPath` (for example, sub-projects of a monorepo), which have no live file watcher to flag pending edits. Both tools now verify each file against the index before showing sliced code: an out-of-date file is either shown whole with its full current source, or its code is withheld with a clear "changed on disk" notice — never served as a wrong slice. A fresh re-index restores normal output automatically. Thanks @inth3shadows for the thorough report and verification passes. (#1474)
 - A file built around one very long function no longer takes the whole `codegraph_explore` answer for itself — or disappears from it. Previously such a file was shown in full however big it was, which used up the room every file after it needed, and when the function was larger than the entire response the file was dropped without a word. These files now come back as a bounded window on whole lines — the signature and the top of the body, plus the call site when the call path runs through it — with the rest one follow-up `codegraph_explore` away.
 - `codegraph_explore` no longer lets the first file in an answer spend the room set aside for the files below it, so the rest of the answer still arrives. Previously a large file near the top could quietly use up everything left, and the files ranked under it — each already judged relevant enough to include — were dropped with no source at all; on one question only one of six made it into the answer. Every file now keeps what it was given, and a question that really is about one file still concentrates on that file.
+- When a `codegraph_explore` answer runs right up against its size limit, it now drops the trailing notes rather than a whole file's source. Previously the last file was cut even though trimming the notes alone would have fit, so a file that had already been read, ranked and rendered was thrown away at the last moment. Across a range of real projects this returns one more file and up to 20% more source per call.
 - The blast-radius section of `codegraph_explore` flagged "no covering tests found" whenever no test called a symbol directly — falsely branding helpers that tests exercise through their callers as untested (about 40% of flagged symbols in a measured sample). The check now follows caller chains up to 3 hops and reports indirect coverage as "tested via callers"; when nothing is found it states exactly what was checked instead of an unconditional warning. Thanks @inth3shadows for measuring the false-positive rate. (#1475)
 
 ## [1.5.0] - 2026-07-21

+ 140 - 0
docs/benchmarks/explore-displacement-guard-ab-cg31.md

@@ -0,0 +1,140 @@
+# Agent A/B — cluster-path displacement guard (task CG-31)
+
+**Date:** 2026-08-06 · **New:** `bugfix/CG-31` · **Baseline:** `bugfix/CG-30` @ `0d014a6` ·
+**Harness:** `scripts/agent-eval/ab-new-vs-baseline.sh`, `--model sonnet --effort high`,
+**both arms codegraph-on**, CLI blocked (0 contamination in every run),
+`CODEGRAPH_NO_PROMPT_HOOK=1`. Every index measured on was **fully rebuilt**, never
+incrementally synced (CG-33).
+
+Baseline is the CG-30 tip, not `main`, so every number here isolates CG-31. CG-30's own A/B
+against `main` is `explore-oversize-member-ab-cg30.md`; read them in sequence for the combined
+picture the two issues asked for.
+
+CG-31 stops a clustered render from spending a reservation still owed to a file the render loop
+has not reached. The whole-file BUY arm has always refused that trade (`owedBelow`); the cluster
+path read what was left before the hard ceiling instead of what was still promised.
+
+**Verdict: no regression, and this one is a straight win on both halves.** Four of six suite
+repos deliver MORE source and one more file each; the other two are byte-identical. The agent
+runs are faster in all three repos measured, with Read at or below baseline.
+
+---
+
+## The two corrections the measurement forced
+
+Worth recording, because the first version of the guard was **wrong in the direction the guard
+itself is about**, and only a suite measurement showed it.
+
+**1. Holding back the full owed sum was too much.** The allocator splits the envelope; the render
+loop spends against a ceiling that also has to hold the response's own prose, so on a saturated
+response the promises are over-subscribed and the tail is going to be dropped whatever happens
+above it. Bytes held for a file that is then dropped are bytes nobody receives. Measured: django
+−2,319 source, tokio −1,298, both handed to a section the ceiling threw away.
+`owedPayableBelow` now holds back only the prefix of what is owed below that the response can
+still pay, in rank order.
+
+**2. The final truncation was eating the guard's work.** It cut at the last file-section header,
+which drops that whole section *and* the trailing notes. Dropping the notes alone is almost
+always enough. The epilogue is a pointer list and two reminders; a section is source the agent
+otherwise has to Read. Cutting the epilogue first is what turned the remaining deficits into
+gains — and it is the same starvation CG-31 is about, arriving one layer below the guard.
+
+A third, smaller fix: `flow.text` is prepended to `lines` to make the final output but was never
+counted in `totalChars`, so the render loop spent against a ceiling it was ~2K under on
+symbol-bag queries.
+
+## Deterministic measurement — the primary evidence
+
+Same clean-rebuilt index, same query, both builds. One `codegraph_explore` per repo.
+
+| repo | base source | new source | Δ | base files | new files |
+|---|---|---|---|---|---|
+| django | 20,033 | **20,791** | +758 | 5 (truncated) | **6** |
+| excalidraw | 18,776 | **20,204** | +1,428 | 7 (truncated) | **8** |
+| okhttp | 15,628 | **19,034** | +3,406 | 4 (truncated) | **5** |
+| tokio | 20,340 | **21,521** | +1,181 | 4 (truncated) | **5** |
+| gin | 10,776 | 10,776 | 0 | 4 | 4 |
+| alamofire | 11,662 | 11,662 | 0 | 2 | 2 |
+
+Queries: django "How does a QuerySet turn into SQL and fetch rows from the database?";
+excalidraw "How does updating an element re-render the canvas on screen?"; gin "How does a
+registered route handler get invoked for an incoming HTTP request?"; alamofire "How does a
+request get built and sent through the session?"; okhttp "How does a call go through the
+interceptor chain to the network?"; tokio "How does a spawned task get scheduled and run by a
+worker?".
+
+No repo delivers less. **Four of six stopped truncating**, which is where the extra file comes
+from: each of those responses had been throwing a fully-rendered section away.
+
+**gin and alamofire are byte-identical between the builds** — nothing in them is oversize enough
+for the guard to engage and neither response was truncated. That is what a control should show,
+and it means every gin number in the agent table below is run-to-run variance.
+
+**Fixture** — `__tests__/fixtures/displacement-ts`, four pipeline stages competing for one
+envelope, the first a single ~20K function. Padded past 500 indexed files on purpose: the
+displacement only exists on the 24K tier, where the reservations plus the preamble genuinely
+saturate the render ceiling.
+
+| | baseline | new |
+|---|---|---|
+| `ingest.ts` | 9,301 chars on a 6,289 spendable, then **dropped whole** by the ceiling — 0 delivered | 4,851, bounded |
+| `types.ts` / `sink.ts` | skipped `budget-whole-file` | delivered |
+| admitted files delivered | **3 of 6** | **6 of 6** |
+| envelope | 14,908 | 22,066 |
+
+Pinned by `__tests__/explore-displacement-guard.test.ts` (11 tests; 3 fail on the baseline).
+
+**Allocation fixtures** — `scripts/agent-eval/allocation-fixtures.json` flips back to
+**BOTH PASS**. Its `afterCG30` verdict blamed an over-RESERVED incidental file; the reservation
+is identical in both arms — the file was over-SPENDING, which is exactly this defect. Recorded
+honestly in `afterCG31`.
+
+## Agent runs
+
+| | django new | django base | okhttp new | okhttp base | gin new | gin base |
+|---|---|---|---|---|---|---|
+| runs | 3 | 3 | 2 | 2 | 2 | 2 |
+| duration (s) | **36** [33–51] | 46 [35–49] | **42** [40–44] | 52 [50–53] | **33** [31–35] | 39 |
+| tool calls | 3 [3–4] | 3 [3–4] | **4** [3–4] | 5 [4–5] | **4** [3–4] | 5 [4–5] |
+| Read | 0 [0–1] | 0 | **0** | 1 [0–2] | 1 [0–1] | 1 [0–2] |
+| Grep/Glob | 0 | 0 | 0 | 0 | 0 | 0 |
+| codegraph calls | 2 | 2 [2–3] | 3 [2–3] | 3 [2–3] | 2 | 3 [2–3] |
+| occupancy share | **32.1%** [31.1%–36.1%] | 33.8% [28.7%–42.1%] | **40.0%** [36.6%–43.3%] | 40.8% [40.3%–41.4%] | **29.7%** [28.3%–31.1%] | 33.5% [29.8%–37.2%] |
+| allocation efficiency | 96.8% | 98.9% | 88.2% | 97.2% | **91.2%** | 85.0% |
+
+Prompts are the deterministic queries above with "Trace the flow end to end." appended.
+
+**Sufficiency, pooled per call.** okhttp: **0** "Read a file we returned" in 5 against the
+baseline's 1 in 5 — the arm that returns 3,406 more chars needs fewer follow-up Reads, which is
+the mechanism working. django: 1 in 6 against 0 in 7. gin: 1 in 4 against 1 in 5, on a repo where
+the builds emit identical bytes. Neither arm produced a single recall miss (a Read of a file we
+did NOT return, or a Grep) on any repo.
+
+**Where the new arm looks worse, and why it is not read as a regression:**
+
+- *okhttp allocation efficiency, 88.2% vs 97.2%.* The new arm's envelope is 85,197 chars against
+  the baseline's 66,014 — it returns substantially more source, and the metric is the share of
+  returned bytes the answer *cited*. A larger, more complete response with a smaller cited share
+  and Read driven to 0 is the trade this tool exists to make. The metric's own documentation says
+  it is relative and must not be read as waste.
+- *django, 1 allocation miss in 6 answered calls against 0 in 7.* One run, n=3, and django is the
+  repo whose duration range overlaps most (33–51 vs 35–49).
+
+## Residual carried forward — for CG-26
+
+Four repos stopped truncating; **okhttp, django, excalidraw and tokio now land at 24,758–24,998
+chars against a 25,000 hard ceiling.** That is deliberate (the ceiling exists so the host never
+externalizes the result) but it means the render loop's 600-char margin for the epilogue is still
+wrong — the epilogue measures 1,064 (gin), 1,788 (django), 2,231 (excalidraw). The response now
+survives that by dropping the epilogue rather than a section, which is strictly better, but the
+honest fix is for the loop to budget for the epilogue in the first place.
+
+A margin sweep was run and deliberately **not** shipped: at 1,200 django stops truncating on its
+own but tokio loses 286 chars; at 2,400 django loses 1,895. Tuning one constant against the suite
+is the trap CG-30's own record warns about. Sizing the margin from the epilogue the response is
+actually going to emit is the real fix and belongs with the end-to-end reservation invariant.
+
+Second residual: the whole-file BUY arm's fit test (`totalChars + fileContent.length +
+FILE_OVERHEAD <= renderCeiling`) has no `owedBelow` term of its own — its displacement guard is
+source-space only. It was left alone here to keep this change attributable; the epilogue-first cut
+removes the failure mode it would have caused.