perf(resolution): adaptive pool engagement — projected-settle bar replaces the fixed 150k-ref gate for mid-run boot; tokio −23% (#1390)

The 9-language competitor matrix exposed tokio as the worst fresh-index
gap: 77% of its wall was resolution running SEQUENTIALLY — 56k Rust refs
sit under the fixed 150k pool gate while costing 36µs each (9× Go's
4µs/ref on prometheus). A ref-count gate can't see per-ref cost.

After each sequential batch the loop now projects the remaining
sequential settle from the measured rate and boots the pool mid-run when
it clears 400ms. The switch rides machinery that already existed: pool
boot is async and fan-out engages only when ready, admission order is
mode-independent, and the #1320 edges-before-fanout invariant holds at
every batch boundary regardless of when the pool arrives. Up-front
engagement at >=150k refs is unchanged; 2-core/low-memory hosts still
decline inside tryCreate's sizing; CODEGRAPH_NO_PARALLEL_RESOLVE still
disables; downgrade permanence is preserved (one engage attempt per run).

Measured (n=3, interleaved, caffeinated): tokio 3.06-3.12 → 2.40-2.57s
(resolution 2,443→~1,330ms); express (tiny control) unchanged with zero
engagements; dubbo unchanged (ref-count path). Gates: tokio + excalidraw
adaptive-vs-sequential dumps byte-identical (87,302 / 89,903 rows),
dubbo dump identical to the session baseline, suite 2,689 ×2 with
CODEGRAPH_KERNEL_EXPECT=1. Known follow-up: sampling the rate mid-first-
batch would close the remaining ~0.3s to the forced-engage ceiling.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -24,6 +24,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines.
 - Indexing on macOS now uses the machine's real memory headroom when sizing its parallel-resolution workers. macOS deliberately keeps RAM filled with reclaimable cache, so the previous free-memory reading came back tiny (~1GB on an otherwise idle machine) and silently halved the worker pool — a medium Java project's fresh index ran about 15–20% slower than the hardware allowed. Graphs remain byte-for-byte identical; the same fix also lets a memory-driven analysis cache engage fully on macOS for large C codebases.
 - Fresh indexing got a sizeable across-the-board speedup: during the initial build, the database's secondary lookup indexes are set aside and rebuilt once after parsing instead of being maintained row by row — the same proven trick the later linking phase already used, now applied to the whole parse lane — and the reference-resolution loop likewise stops maintaining lookup indexes it never reads, rebuilding them at the end when almost nothing is left in the table. A medium Java project's parse phase runs about 58% faster and its full fresh index about 19% faster end-to-end; a Linux-kernel-scale index that took ~15 minutes on an 8-core machine now completes in about 11, with the resolution phase alone dropping by a third. Graphs remain byte-for-byte identical, and incremental syncs are unaffected.
+- Parallel reference resolution now engages adaptively instead of by a fixed project-size cutoff: the indexer measures the actual per-reference resolution rate on the first batch and spins up the worker pool mid-run whenever the remaining work justifies it. Languages whose references are expensive to resolve benefit most — Rust especially: a fresh index of tokio runs about 23% faster, with the graph byte-for-byte identical. Small projects and low-core machines (2-core CI runners) keep the single-threaded path exactly as before.
 - The dynamic-dispatch analysis at the end of indexing now skips passes that provably can't produce anything for the project at hand: React re-render bridging when no class has a `render` method, React Native and Expo cross-platform pairing when the required languages aren't present, and MyBatis mapper linking when there's no mapper XML. Previously each of these scanned the whole graph before coming up empty — on a 4,000-file Java project that was about 0.9 seconds of wasted analysis per fresh index. The interface-implementation bridging pass also got cheaper on real work: it no longer re-fetches a hub interface's method list once per implementer, and classes that extend or implement nothing are skipped before any per-class lookups. Graphs remain byte-for-byte identical.
 - Indexing large C and C++ codebases spends much less time in the function-pointer dispatch analysis (the pass that connects handler tables like a command table or an ops struct to their call sites): each source file is now read and prepared once instead of four times, files that can't contribute any dispatch wiring are skipped outright in the later linking steps, and on platforms with the native engine the per-file scanning itself now runs natively too. On a Linux-kernel-scale tree the pass runs about a third faster end-to-end, with graphs byte-for-byte identical; platforms without a native binary keep the same results on the previous path.
 

src/resolution/index.ts

@@ -1473,17 +1473,38 @@ export class ReferenceResolver {
     // costs zero wall-clock. Any failure downgrades to sequential permanently.
     let pool: ResolverPool | null = null;
     let poolReady = false;
-    const tPoolStart = Date.now();
-    if (parallel && total >= minRefsForPool()) {
-      pool = ResolverPool.tryCreate(parallel.dbPath, this.projectRoot);
-      pool?.ready().then(
+    // True once pool creation has been attempted by EITHER engage site (the
+    // up-front ref-count gate or the adaptive projection below) — a pool that
+    // failed or was destroyed must stay down (downgrade is permanent), and
+    // tryCreate's sizing probes shouldn't re-run every batch on hosts that
+    // declined.
+    let poolEngageTried = false;
+    const createPool = (t0: number, why: string): ResolverPool | null => {
+      poolEngageTried = true;
+      if (!parallel) return null;
+      const p = ResolverPool.tryCreate(parallel.dbPath, this.projectRoot);
+      p?.ready().then(
         () => {
           poolReady = true;
-          if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[pool-timing] pool ready after ${Date.now() - tPoolStart}ms`);
+          if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[pool-timing] pool ready after ${Date.now() - t0}ms (${why})`);
         },
-        () => { void pool?.destroy().catch(() => undefined); pool = null; }
+        () => {
+          void p.destroy().catch(() => undefined);
+          if (pool === p) pool = null;
+        }
       );
+      return p;
+    };
+    if (parallel && total >= minRefsForPool()) {
+      pool = createPool(Date.now(), 'ref-count');
     }
+    // Adaptive engagement bar (see the batch-loop hook): projected remaining
+    // sequential settle above this boots the pool mid-loop. Boot is async and
+    // fan-out waits for ready, so a marginal engage costs background boot
+    // only; the bar just needs to clear the fan-out's own overhead class.
+    const ADAPTIVE_ENGAGE_SETTLE_MS = 400;
+    let adaptiveSeqMs = 0;
+    let adaptiveSeqRefs = 0;
 
     // Process in PIPELINED batches (double-buffer). The enumeration is the
     // head of the pending set in rowid order; every ref a persisted batch
@@ -1606,6 +1627,28 @@ export class ReferenceResolver {
       if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[pool-timing] batch ${inFlight.mode}: ${batch.length} refs in ${Date.now() - tBatch}ms`);
       lp('settle', tBatch);
 
+      // Adaptive pool engagement: the fixed ref-count gate can't see PER-REF
+      // cost, and settle rates differ ~9× by language (56k Rust refs cost
+      // more sequential settle than 154k Go refs — 36µs vs 4µs measured on
+      // tokio/prometheus). After each sequential batch, project the remaining
+      // settle from the observed rate and boot the pool mid-loop when it
+      // clears the bar. The loop already switches to fan-out only when the
+      // async boot reports ready, admission order is mode-independent, and
+      // 2-core/low-memory hosts still decline inside tryCreate's sizing —
+      // so the switch changes wall-clock, never the graph.
+      if (inFlight.mode === 'seq' && parallel && pool === null && !poolEngageTried) {
+        adaptiveSeqMs += Date.now() - tBatch;
+        adaptiveSeqRefs += batch.length;
+        const remaining = total - processed - batch.length;
+        const projectedMs = (adaptiveSeqMs / Math.max(1, adaptiveSeqRefs)) * Math.max(0, remaining);
+        if (projectedMs >= ADAPTIVE_ENGAGE_SETTLE_MS) {
+          if (process.env.CODEGRAPH_SYNTH_TIMINGS) {
+            console.error(`[pool-timing] adaptive engage: projected ${Math.round(projectedMs)}ms sequential settle over ${remaining} remaining refs`);
+          }
+          pool = createPool(Date.now(), 'adaptive');
+        }
+      }
+
       // WAL-valve backstop at the ONE pool-idle boundary of the double-buffer
       // (this batch settled, the next not yet fanned out): past the hard cap
       // the writer parks for a full backfill here, where the pool's readers