Просмотр исходного кода

docs(benchmarks): record the CG-36 cluster-starvation measurement

The issue blamed the density tiebreak; both real cases lost on maxImportance,
so ranking was left alone. Full before/after table, the one cost (okhttp's
rank-6 file, squeezed out by reservations that were already structurally
over-subscribed), and what ships to keep it measurable.
Colby McHenry 1 месяц назад
Родитель
Сommit
10f1ac601a

+ 1 - 0
CHANGELOG.md

@@ -34,6 +34,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - When a `codegraph_explore` answer runs right up against its size limit, it now drops the trailing notes rather than a whole file's source. Previously the last file was cut even though trimming the notes alone would have fit, so a file that had already been read, ranked and rendered was thrown away at the last moment. Across a range of real projects this returns one more file and up to 20% more source per call.
 - Every file `codegraph_explore` decides to include now actually arrives. A file shown in full could still spend room set aside for files below it — the fix above covered files shown as excerpts but not files shown whole — and the answer's own size bookkeeping under-counted each file's heading, so the answer ran past its limit and a fully prepared file was discarded at the end. A file that no longer fits whole is now shown as excerpts instead of vanishing, and one that overshoots by a little is trimmed to fit rather than dropped.
 - The list of files an answer could not cover — the "explore these names for their source" pointers — is no longer thrown away when the answer is full. It is now budgeted for and trimmed to fit, so a full answer still tells you what it left out and which names to ask for next, instead of ending with no pointers at all.
+- When a `codegraph_explore` answer shows a file as excerpts, a large excerpt that no longer fit was dropped entirely instead of being shortened. If the file's first excerpt happened to be a trivial one — an import block, a one-line helper next to the code you asked about — the excerpt carrying the actual answer was the one thrown away, and the file came back with a quarter of the room it had been given. On real projects that meant the top-ranked file delivered a fraction of its share while a far less relevant file took the rest. Excerpts are now shortened to fit, whole method by whole method, and only dropped when what is left is too small to hold anything readable.
 - The blast-radius section of `codegraph_explore` flagged "no covering tests found" whenever no test called a symbol directly — falsely branding helpers that tests exercise through their callers as untested (about 40% of flagged symbols in a measured sample). The check now follows caller chains up to 3 hops and reports indirect coverage as "tested via callers"; when nothing is found it states exactly what was checked instead of an unconditional warning. Thanks @inth3shadows for measuring the false-positive rate. (#1475)
 
 ## [1.5.0] - 2026-07-21

+ 164 - 0
docs/benchmarks/explore-cluster-starvation-cg36.md

@@ -0,0 +1,164 @@
+# Deterministic measurement — cluster starvation inside one file (task CG-36)
+
+**Date:** 2026-08-06 · **Baseline:** `feature/CG-24` @ `76ab1fe` (the CG-24 epic tip) ·
+**Harness:** `scripts/agent-eval/probe-file-spend.mjs` and `probe-suite-envelope.mjs` over the
+deterministic 6-repo corpus at `/tmp/codegraph-corpus`, clean full-rebuilt indexes (CG-33), plus
+two hermetic fixtures through `probe-allocation.mjs`. No agent A/B: the claim is which bytes go
+to which file, and the agent runs are far too noisy to see a 2K shift.
+
+**Verdict: the defect is real, the diagnosis in the issue was half wrong, and the fix holds.**
+All 8 starvation flags across the suite clear; net **+1,012 source chars**. One repo (okhttp)
+trades its weakest file for +7,196 chars in the two that answer the question — stated in full
+below rather than smoothed over.
+
+---
+
+## The defect
+
+A file's ranked clusters were all-or-nothing past the first one. The top-ranked cluster was
+always taken — shrunk to the highest-importance whole symbol ranges that fit when it overran —
+and every cluster below it was rendered **whole** and then either fit the remainder or was
+dropped entirely. On a file whose top-ranked cluster is trivial, that discards the answer.
+
+What made it invisible: the response stays FULL. The unspent reservation carries forward exactly
+as CG-31 designed, so a lower-scoring file takes the bytes and every envelope-share measure still
+reads healthy. Measured on the epic tip:
+
+| repo | file | score | reserved | spent | share |
+|---|---|---|---|---|---|
+| django | `db/models/sql/query.py` | 83 | 7,947 | 1,923 | **24%** |
+| django | `contrib/admin/filters.py` | 18 | 2,271 | 8,057 | **355%** |
+| okhttp | `.../RealInterceptorChain.kt` | 86 | 6,058 | 1,474 | **24%** |
+| okhttp | `.../CallServerInterceptor.kt` | 20 | 1,974 | 5,832 | **295%** |
+
+A score-83 file spends a quarter of its reservation while a score-18 file takes 3.5× its own.
+
+## What the issue got wrong
+
+The issue named two candidate fix points and suspected the first: cluster ranking breaks ties on
+**density** (`score / span`) after `hasSpine` → `maxImportance`, which structurally favours a
+small trivial cluster over a large answer-bearing one. Dumping the cluster set says otherwise —
+in **both** real cases the loser lost on `maxImportance`, not on density:
+
+```
+django/db/models/sql/query.py   budget 10,135   spent 1,923
+  KEPT    1379–1400   span   22   score  14   maxImp 6   (check_related_objects — a glue symbol)
+  DROPPED  306– 929   span  624   score 290   maxImp 3   (133 members: the whole Query class)
+
+okhttp .../RealInterceptorChain.kt   budget 6,058   spent 1,474
+  KEPT      16–  44   span   29   score  44   maxImp 6   (package decl + import block)
+  DROPPED  113– 373   span  261   score 171   maxImp 3   (73 members: the chain itself)
+```
+
+`maxImportance` first is deliberate and protective — it is what stops Alamofire's `Session.swift`
+from losing its budget to the top-of-file property list — so **ranking was not touched**. The
+lever is the second fix point: stop dropping the loser whole.
+
+## The change
+
+Two sites, one rule — *hold the remainder while it is still worth a section*, which is CG-26's
+between-FILES lesson applied between CLUSTERS.
+
+1. **Selection.** A later cluster is now shrunk into what is left of the file's budget, by the
+   same whole-member rule the first cluster already used. Below `MIN_CHARS` (700) the remainder
+   cannot hold a readable block, so it stays a drop rather than a stutter of fragments the next
+   call's dedup has to shred around. The never-empty windowing floors may overrun that room; the
+   first cluster is allowed that overshoot, a later one is not.
+2. **The ceiling trim.** When the exact section cost overruns `renderCeiling`, the weakest chosen
+   cluster is re-rendered into the room that remains before being dropped. This one is worth
+   naming on its own: on excalidraw's `typeChecks.ts` the section-cost estimate missed by
+   **13 chars** and a 1,512-char cluster — the file's highest-*scoring* one, last in rank order
+   only because rank breaks ties on density — was thrown away to pay for it. Recovered 1,501 of
+   excalidraw's 1,449-char loss.
+
+## Suite result
+
+`node scripts/agent-eval/probe-file-spend.mjs`, 6 repos, clean rebuilds. Only files whose spend
+moved are listed; score is the candidate's ranking score, reserved its allocation.
+
+| repo | file | score | reserved | before | after |
+|---|---|---|---|---|---|
+| django | `db/models/sql/query.py` | 83 | 7,947 | 1,923 | **10,082** |
+| django | `contrib/admin/filters.py` | 18 | 2,271 | 8,057 | 2,198 |
+| django | `utils/autoreload.py` | 12 | 1,747 | 3,145 | 1,709 |
+| django | `db/models/fields/related_descriptors.py` | 11 | 1,660 | 2,516 | 1,493 |
+| excalidraw | `element/src/typeChecks.ts` | 23 | 2,740 | 3,102 | 2,573 |
+| excalidraw | `excalidraw/types.ts` | 14 | 1,942 | 819 | 1,372 |
+| okhttp | `.../RealInterceptorChain.kt` | 86 | 6,058 | 1,474 | **6,038** |
+| okhttp | `.../Interceptor.kt` | 64 | 4,697 | 2,027 | **4,659** |
+| okhttp | `.../RealCall.kt` | 52 | 3,972 | 3,628 | 3,922 |
+| okhttp | `.../Call.kt` | 54 | 4,097 | 4,097 | 2,073 |
+| okhttp | `.../CallServerInterceptor.kt` | 20 | 1,974 | 5,832 | 1,959 |
+| okhttp | `androidMain/.../AndroidDns.kt` | 21 | 1,999 | 1,812 | **0** |
+| tokio | `task/local.rs` | 40 | 4,361 | 4,599 | 4,798 |
+| tokio | `runtime/task/harness.rs` | 14 | 1,981 | 2,565 | 2,341 |
+| gin | `routergroup.go` | 87 | 5,782 | 3,273 | **5,632** |
+| gin | `tree.go` | 17 | 1,693 | 892 | 1,969 |
+| gin | `ginS/gins.go` | 26 | 2,213 | 4,431 | 2,171 |
+| alamofire | `Source/Core/Session.swift` | 34 | 2,792 | 2,797 | 3,396 |
+| alamofire | `Source/Core/Request.swift` | 148 | 9,100 | 8,865 | 8,453 |
+
+Bytes move up the score order in every repo. Envelope totals:
+
+| repo | source before | after | Δ | files | ceiling |
+|---|---|---|---|---|---|
+| django | 20,878 | 20,719 | −159 | 6 → 6 | 24,963 ≤ 25,000 |
+| excalidraw | 19,652 | 19,704 | +52 | 8 → 8 | 24,813 ≤ 25,000 |
+| okhttp | 18,870 | 18,651 | −219 | 6 → **5** | 24,985 ≤ 25,000 |
+| tokio | 21,607 | 21,582 | −25 | 5 → 5 | 24,777 ≤ 25,000 |
+| gin | 10,776 | 11,952 | **+1,176** | 4 → 4 | 14,655 ≤ 19,500 |
+| alamofire | 11,662 | 11,849 | +187 | 2 → 2 | 12,862 ≤ 19,500 |
+| **total** | **103,445** | **104,457** | **+1,012** | | |
+
+Starvation flags: **8 → 0**.
+
+## The one cost, stated plainly
+
+okhttp drops its rank-6 file, `androidMain/.../AndroidDns.kt` (score 21, a platform DNS helper on
+a question about the interceptor chain), and 219 source chars, in exchange for +4,564 to
+`RealInterceptorChain.kt` and +2,632 to `Interceptor.kt` — the two files that answer the question.
+
+This is not a new defect and it is not the fix over-reaching. okhttp's reservations are
+**structurally over-subscribed**: the allocator splits `maxOutputChars` charging a flat
+`FILE_OVERHEAD` of 200 per file while a real header runs 300–500, so the sum of promises
+(~22,800 source + ~2,100 of real headers) exceeds what the ~24,760-char render ceiling can hold.
+`owedPayableBelow` already refuses to hold bytes back for a file it can see will be dropped, and
+AndroidDns.kt is the file past that line. On the epic tip it survived only because the files above
+it under-spent — by luck, not by design. Closing the over-subscription means charging the
+allocator per-file header estimates rather than the flat 200; that is a wider change than this
+issue, and CG-26 deliberately kept `FILE_OVERHEAD` as the allocator's own constant.
+
+## What did NOT change
+
+- **Cluster ranking.** `hasSpine` → `maxImportance` → density → score → span, untouched.
+- **Alamofire `Session.swift`.** The shape density-first exists for; it *gains* 599 chars.
+- **The factory-closure outcome (CG-27).** `probe-factory-closure.mjs`: 7 of 11 inner closure
+  definitions delivered, identical to the epic tip.
+- **The reservation invariant (CG-31/CG-26).** `explore-reservation-invariant.test.ts` green;
+  every repo stays at or under its hard ceiling.
+- **All four allocation fixtures pass** — `payroll-go`, `self-query`, and the two added here.
+
+## What ships so this stays measurable
+
+- `scripts/agent-eval/probe-file-spend.mjs` — the standing per-file reservation-vs-delivered
+  sweep. It flags a **pair**, never a single file: a large share unspent *while* a materially
+  lower-scoring file overspends. Either alone is legitimate (a small file has less to say;
+  carry-forward is the mechanism that hands its slack down), which is why the envelope probe
+  could never see this. Exit code 1 on any flag, so it gates.
+- `__tests__/fixtures/starved-cluster-ts/` — django's and okhttp's shape reduced to a fixture.
+  Fails on the epic tip (28.8% of reservation, neither `proceed` nor `writeAndRead` delivered),
+  passes with the fix.
+- `__tests__/fixtures/dense-header-ts/` — the `Session.swift` shape, byte-identical on both
+  builds. The counterweight: it fails if a future change lets density outrank importance again.
+- `spendShareAtLeast` in `probe-allocation.mjs`, and `__tests__/explore-cluster-starvation.test.ts`
+  pinning both fixtures in `npm test`.
+
+## Method note
+
+None of this is visible in the rendered markdown. To see it you must dump the cluster set —
+patch `dist/mcp/tools.js` just before `let assembled = assembleSection(chosenIndices);` and log
+`fileBudget` / `projectedChars` / each ranked cluster's span, score, `maxImportance`, chosen flag
+and members. Reading only the response makes member-selection effects look like budget effects.
+Equally, a source-chars diff between builds is not automatically a regression: excalidraw's
+−1,449 on the first cut was the elastic epilogue expanding into room a 13-char accounting error
+had released, not source lost to allocation.

+ 9 - 6
docs/benchmarks/explore-noise-epic-cg24.md

@@ -47,18 +47,21 @@ change at all**.
 | **CG-25** | Recognize `Generated by <tool> by running <command>` banners. Precision held by requiring two `by` clauses, so ordinary prose does not match. |
 | **CG-28** | Damp declaration-only files that nothing in the index depends on. Does not stack with the generated penalty (`Math.min`), and naming a declaration symbol exempts its file. |
 | **CG-33 / CG-35** | Incremental sync converges with a rebuild, plus a regression suite that fails when the fix is disabled. |
+| **CG-36** | A later cluster is shrunk into the remainder rather than dropped whole — at selection, and again in the ceiling trim. All 8 starvation flags across the suite clear; +1,012 source chars net. `explore-cluster-starvation-cg36.md`. |
 
 Deterministic across the 6-repo suite: no repo truncates, none loses a file,
 okhttp gains one, every repo lands at or under the 25,000 hard ceiling.
 
 ## Open
 
-**[CG-36]** — a file's non-first clusters are never shrunk, so a trivial cluster
-starves the answer-bearing one. Found as a byproduct of CG-27's measurement and
-confirmed firing on the epic tip: `query.py` leaves 81% of its budget unspent,
-keeping a score-14 cluster and dropping a score-290 one. Two candidate fix
-points — the density tiebreak in selection, and the never-shrink rule — and
-density-first must keep working (the `Session.swift` case).
+Nothing. CG-36, the last one, shipped 2026-08-06.
+
+Its own measurement is worth carrying forward, because the issue named the wrong
+fix point: both real cases (`query.py`, `RealInterceptorChain.kt`) lost on
+`maxImportance`, **not** on the density tiebreak the issue suspected. Ranking was
+left alone; the never-shrink rule was the lever. Full numbers and the one cost
+(okhttp trades its rank-6 file for +7,196 chars in the two that answer the
+question) in `explore-cluster-starvation-cg36.md`.
 
 ## Closed because measurement contradicted them