瀏覽代碼

docs(benchmarks): record the CG-24 epic resolution

Four shipped fixes, one open defect (CG-36), and five issues closed because
measurement contradicted them. The headline is that the reported symptom was
not an explore bug at all — it was a degraded index (CG-33), and the reported
query answers correctly on a clean rebuild with no explore change.

Records the two traps that cost real time and are now guarded in tooling: the
nonexistent .codegraph/graph.db path that sqlite3 silently creates, and
ab-new-vs-baseline.sh swapping src/ mid-run so a commit captures baseline
sources.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Colby McHenry 1 月之前
父節點
當前提交
76ab1fe130
共有 1 個文件被更改,包括 95 次插入0 次删除
  1. 95 0
      docs/benchmarks/explore-noise-epic-cg24.md

+ 95 - 0
docs/benchmarks/explore-noise-epic-cg24.md

@@ -0,0 +1,95 @@
+# Epic resolution — explore response noise (CG-24)
+
+Worked 2026-08-05 → 2026-08-06. Started from one bad `codegraph_explore` response
+in a real session and ended with four shipped fixes, one open defect, and five
+issues closed because measurement contradicted them.
+
+**The headline: the reported symptom was not an explore bug.** It was a degraded
+index. The explore defects the investigation found are real and were fixed, but
+none of them caused the report.
+
+## The report
+
+A prose flow query returned an unusable response: the symbol the agent had named
+never rendered, and a 12k-line generated Cloudflare ambient-types file took 60.7%
+of the output envelope.
+
+```
+#  deliv%   bytes  reserved  score  pen  flags                file
+1    1.0%     251    10,970   87.0  1.00 named entry central  <the named file>
+2   60.7%  15,043     6,484   49.0  1.00 entry central        worker-configuration.d.ts
+4      —        —         —   19.9  1.00 dropped: budget      <a third file>
+```
+
+## Root cause
+
+**Index drift ([CG-33](index-drift-cg33.md)).** The live incrementally-synced index
+diverged from a clean rebuild by 4.3% of distinct edges, bidirectionally,
+overwhelmingly `calls`. RWR graph mass is relative and normalized, so call edges
+missing elsewhere inflate an unaffected file's share — the `.d.ts` carried mass
+0.24750 drifted vs 0.13119 rebuilt (~1.9×), score 49.0 vs 27.0.
+
+Two causes, both fixed: incremental sync re-resolved only references *in* changed
+files, and `getNodesByName` had no `ORDER BY`, so ties broke by rowid — i.e. by
+the order files happened to be **written**. The second is why scope alone could
+never converge. Stale edges dropped 671 → 2 across an 80-commit replay.
+
+On a freshly rebuilt index the reported query answers correctly **with no explore
+change at all**.
+
+## Shipped
+
+| | what |
+|---|---|
+| **CG-30** | Bounded how far an oversize cluster member may overshoot; windows on whole lines past 1.5× instead of emitting whole — or, when larger than the response ceiling, dropping the file silently. |
+| **CG-31** | Gave the cluster path the `owedBelow` displacement guard the whole-file BUY arm always had, holding back only the prefix of what is owed below that the response can actually pay. |
+| **CG-26** | Closed the remaining holes: whole-file arms had no displacement guard at all, section overhead was charged at a flat 200 against a real 300–500, and `owedPayableBelow` held all-or-nothing. |
+| **CG-25** | Recognize `Generated by <tool> by running <command>` banners. Precision held by requiring two `by` clauses, so ordinary prose does not match. |
+| **CG-28** | Damp declaration-only files that nothing in the index depends on. Does not stack with the generated penalty (`Math.min`), and naming a declaration symbol exempts its file. |
+| **CG-33 / CG-35** | Incremental sync converges with a rebuild, plus a regression suite that fails when the fix is disabled. |
+
+Deterministic across the 6-repo suite: no repo truncates, none loses a file,
+okhttp gains one, every repo lands at or under the 25,000 hard ceiling.
+
+## Open
+
+**[CG-36]** — a file's non-first clusters are never shrunk, so a trivial cluster
+starves the answer-bearing one. Found as a byproduct of CG-27's measurement and
+confirmed firing on the epic tip: `query.py` leaves 81% of its budget unspent,
+keeping a score-14 cluster and dropping a score-290 one. Two candidate fix
+points — the density tiebreak in selection, and the never-shrink rule — and
+density-first must keep working (the `Session.swift` case).
+
+## Closed because measurement contradicted them
+
+Five, which is the story of this epic as much as the fixes are.
+
+| | why |
+|---|---|
+| **CG-32** | Named file "didn't render first." Drift artifact; on a clean index it renders first and takes 89%. |
+| **CG-34** | "Allocator over-reserves for low-scoring files." Filed on a runner's diagnosis without checking the numbers. The file was never over-reserved (4,314 in both arms) — it was over-*spending*, which is CG-31. |
+| **CG-27** | Adding `function`/`method` to `ENVELOPE_KINDS` measured as a **large regression** — rank #1 fell from 7,539 delivered chars to 397, 7 of 11 inner closures to 0. The enclosing range was holding the file together as one cluster, inside which `shrinkCluster` already did the per-symbol ranking the issue wanted. A careful version was noise (69 vs 68 across nine queries). |
+| **CG-29** | Prose-vs-symbol query gap. Inverted on measurement: prose matches symbol on django and delivers 63% more source on okhttp. The founding observation was drift. |
+| **CG-37** | Duplicate of CG-36, filed without seeing it. |
+
+## What this epic is actually a lesson in
+
+**A confident diagnosis is worth less than a cheap measurement.** Every issue
+above was filed by someone — human or agent — who had read the code and had a
+plausible mechanism. Five were wrong. The ones that survived did so because a
+deterministic probe disagreed with them and the probe won.
+
+Two specific traps this cost real time on, both now guarded in tooling:
+
+- **`.codegraph/graph.db` does not exist** — the index is `codegraph.db`, and
+  `sqlite3` against a mistyped path *creates* an empty database rather than
+  failing. An empty schema reads exactly like a stale pre-migration index. This
+  produced a wrong root cause. `diff-index-drift.mjs` refuses a missing path.
+- **`ab-new-vs-baseline.sh` checks the engine out at the baseline ref mid-run.**
+  A commit made while it runs captures baseline sources and silently reverts the
+  fix under test. This happened during CG-30. Check the `changed:` line before
+  believing any A/B result.
+
+And one measurement discipline worth keeping: **compare sets, not totals.** The
+drift that started all of this shows up as +0.7% on raw edge counts, because it
+is bidirectional and nets out. On distinct edge triples it is 4.3%.