perf(resolution): worker connection recycling — WAL-depth writes-under-readers fix, superphase −11.4% at 8c

The §7a.6 anomaly probed to its mechanism with five discriminating runs
(§7a.7 table): main-thread B-tree writes triple under attached readers
because READERS PIN WAL checkpoint progress — the deep WAL taxes every
writer page operation (deletes 42.6s pool-off vs 118.8s pool-4 on
identical hardware; an aggressive 64MB valve recovers the writes but
overpays +129s in full-park folds; the v2 cache resurrection was
falsified — long-tail name traffic is uncacheable at any capacity).

Fix: workers close and reopen their read-only connections every 8
batches at the double-buffer's worker-idle boundary
(ResolverPool.recycleWorkers + QueryBuilder.rebind + a cadence call).
Reopens are sub-millisecond, resolver caches survive (only prepared
statements re-prepare), and the existing checkpoints advance instead of
parking. Failed recycle downgrades to sequential, same as a failed
fan-out.

Measured (8c pool-4, linux v7.2-rc2, cadence 25 → 8 iterated):
resolution superphase 715.0 → 633.6s (−11.4%), envelope best 14.8min,
recreate 59.7 → 45.3s. Byte-neutral everywhere: git dumps byte-identical
old-vs-new, linux dump sha 6dd1185b reproduced (10,446,478 lines),
counts 2,049,153/6,413,518, suite 2517 green. 2c unchanged by
construction (no pool → no recycling).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -21,6 +21,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - `codegraph install` and `codegraph upgrade` now offer CodeGraph Pro beta access after finishing — answer yes, type your email, and you join the same waitlist as the getcodegraph.com homepage form. Strictly opt-in and asked at most once per machine total: nothing is sent unless you say yes and enter an email, either answer is remembered so no later install or upgrade ever re-asks, and non-interactive runs (`--yes`, scripts, CI) never see the question.
 - Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`.
 - Indexing inside CPU- or memory-limited containers (Docker, CI runners) now sizes its worker pools from the container's actual allowance instead of the host machine's, and giant codebases no longer balloon temporary database files during indexing (previously tens of GB of transient disk on Linux-kernel-scale projects). Together these prevent out-of-memory and out-of-disk failures on constrained machines; set `CODEGRAPH_RESOLVE_WORKERS` to override the resolution worker count explicitly.
+- Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines.
 
 ### Fixes
 

docs/design/rust-kernel-migration-plan.md

@@ -90,10 +90,15 @@ them are the ORIGINAL plan and carry expectations that measurement later correct
       double-buffer WORKS (8c settle 3.6s — "core-invariant" superseded);
       2c record now 16.5min, 8c range 15.0–16.4 (n=2). Two cache
       experiments killed by measurement same-day (nameCache scaling, lazy
-      candidates — §7a.6 has the numbers; code reverted). Queue now:
-      **writes-under-readers probe** (8c deletes+inserts +102s under the
-      pool — the biggest attributed delta) > **cFnPtr native site
-      extraction** (synthesis ~230s) > backpressure byte volume > recreate.
+      candidates — §7a.6 has the numbers; code reverted).
+      Writes-under-readers PROBED + FIXED 2026-07-18 (§7a.7): mechanism =
+      WAL read-through depth under reader pins (proven by pool-off/dose/
+      valve discrimination); fix = worker connection recycling at the
+      pool-idle boundary (cadence 8); superphase **715 → 633.6s (−11.4%)**,
+      8c envelope best **14.8min**, byte-neutral at every gate. Queue now:
+      **cFnPtr native site extraction** (synthesis ~230s) >
+      continuous-shallow WAL (the remaining ~45s to the valve floor) >
+      backpressure byte volume > recreate.
 - [x] **R7a. C/C++ port** — DONE 2026-07-17, same-day walker+gates after the
       survey (#1344) and grammar vendoring (#1345). One dual-language walker
       (`codegraph-kernel/src/ccpp/`), preParse HOISTED to the route point
@@ -850,6 +855,55 @@ morning's run; report ranges, never single runs on this box.** 8c parse
   (~80s paid in-container that a 24GB target-class box gets back free) —
   container numbers UNDERSTATE the true 8-core-class target.
 
+#### 7a.7 Writes-under-readers probe + fix (2026-07-18) — WAL depth named, worker connection recycling shipped
+
+Five discriminating runs (all 8c pool-4 unless noted, same tree/build family;
+each ~16min), then the fix in two cadence iterations:
+
+| run | deletes | insertEdges | read | backpressure | settle | superphase |
+|---|---|---|---|---|---|---|
+| pool-4 baseline | 118.8 | 55.3 | 33.9 | 121.9 | 3.6 | 715.0 |
+| pool-OFF | 42.6 | 38.3 | 32.6 | 120.1 | 108.5 (main) | 685.9 |
+| workers=2 (dose) | 58.9 | 54.0 | 23.9 | 174.7 | 10.3 | 670.0 |
+| v2 caches @8c | 108.0 | 50.3 | 31.4 | 168.0 | 3.9 | 687.3 |
+| valve 64MB | **56.5** | **27.8** | **16.8** | 297.2 | 3.8 | 739.2 |
+| **recycle c25** | 99.8 | 42.9 | 32.7 | 148.8 | 3.7 | 663.3 |
+| **recycle c8 (SHIPPED)** | 104.3 | 47.8 | 32.5 | 127.6 | 4.0 | **633.6** |
+
+- **Mechanism proven: WAL read-through depth under reader pins.** Pool-off
+  restores writes on identical hardware (deletes 118.8 → 42.6); the dose
+  scales with reader count (knee between 2 and 4); the aggressive valve —
+  which forces a shallow WAL — recovers deletes/inserts/read to their floors
+  (56.5/27.8/16.8) but overpays +129s in full-park folds. Reconciliation:
+  checkpoints run in every topology, but READERS PIN their progress — the
+  WAL runs deep exactly when workers are attached, and deep-WAL page
+  operations tax the writer everywhere (including the unattributed
+  between-stage spans, recreate, and synthesis reads).
+- **v2-at-8c falsified the cache resurrection** (deletes 108.0 ≈ baseline):
+  name-lookup traffic is long-tail-dominated — uncacheable at any capacity —
+  so reader traffic can't be reduced by caching. The caching family is
+  triple-dead (2c settle, 8c writes).
+- **The fix: worker connection recycling at the pool-idle boundary**
+  (`ResolverPool.recycleWorkers` + `QueryBuilder.rebind` + a cadence call at
+  the double-buffer's worker-idle point). Workers close/reopen their
+  read-only connections every 8 batches (~40k refs) — reopens are
+  sub-millisecond, resolver caches survive (only prepared statements
+  re-prepare), and the existing checkpoints advance instead of parking.
+  Cadence 25 → 8 iterated by measurement; 8 wins via diffuse gains
+  (backpressure −21, recreate 59.7 → 45.3). Attributed deletes stay ~100
+  (the WAL still re-deepens between recycles — the valve's 56.5 floor
+  needs continuous shallowness), but the SUPERPHASE captures the true win:
+  **715.0 → 633.6s (−11.4%)**; envelope best-of **14.8min at 8c**
+  (890s; band across the day's runs 14.8–16.4). Byte-neutral: git dumps
+  byte-identical old-vs-new, linux dump sha `6dd1185b…` reproduced, counts
+  2,049,153/6,413,518 every run, suite 2517 green. 2c unaffected by
+  construction (no pool → no recycling).
+- Levers after this round: cFnPtr native site extraction (~230s synthesis,
+  §7a.6 ranking stands) > continuous-shallow WAL (close the remaining
+  ~45s gap between recycling's ~100s deletes and the valve's 56.5 floor
+  without full-park folds — e.g. passive-checkpoint nudges at the recycle
+  boundary) > backpressure byte volume > recreate.
+
 ### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation)
 Priority order, each gated by the standard A/B + node-explosion probes:
 1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering

src/db/queries.ts

@@ -291,6 +291,24 @@ export class QueryBuilder {
     this.db = db;
   }
 
+  /**
+   * Swap the underlying connection in place. Used by pool workers'
+   * connection recycling (plan §7a.6, writes-under-readers): a long-lived
+   * read connection pins WAL checkpoint progress, and the deep WAL that
+   * accumulates behind it taxes every main-thread B-tree page operation
+   * (deletes measured 42.6s → 118.8s from 0 to 4 attached readers on
+   * identical hardware). Workers therefore close and reopen their read-only
+   * connection at the pool-idle boundary; everything above the connection —
+   * this QueryBuilder, the resolver and its warm caches — survives, and only
+   * connection-derived state (prepared statements) resets, re-preparing
+   * lazily on next use.
+   */
+  rebind(db: SqliteDatabase): void {
+    this.db = db;
+    this.stmts = {};
+    this.batchStmts.clear();
+  }
+
   /** Set the normalized project-name tokens used to down-weight non-discriminative
    * query words in path scoring (#720). Called once when the project opens. */
   setProjectNameTokens(tokens: Set<string>): void {

src/resolution/index.ts

@@ -1408,7 +1408,7 @@ export class ReferenceResolver {
     // these counters name where the other ~340s goes (reads, edge build+insert,
     // deletes/marks, the per-batch count guard).
     const loopProf: Record<string, number> | null = process.env.CODEGRAPH_RESOLVE_PROFILE
-      ? { read: 0, settle: 0, backpressure: 0, createEdges: 0, insertEdges: 0, deletes: 0, marks: 0, countGuard: 0 }
+      ? { read: 0, settle: 0, backpressure: 0, recycle: 0, createEdges: 0, insertEdges: 0, deletes: 0, marks: 0, countGuard: 0 }
       : null;
     const lp = (k: string, t0: number): void => { if (loopProf) loopProf[k] = (loopProf[k] ?? 0) + (Date.now() - t0); };
     let tLp = 0;
@@ -1450,6 +1450,14 @@ export class ReferenceResolver {
     // pending rows forward.
     let prevRemaining = Number.POSITIVE_INFINITY;
 
+    // Cadence for the worker connection recycling below — ~8 batches
+    // ≈ 40k refs between recycles keeps the WAL shallow at kernel scale
+    // while a small sync never recycles at all. (25 recovered only half
+    // the write tax — the WAL re-deepened between recycles; reopens are
+    // sub-millisecond so the shorter cadence is ~free.)
+    const RECYCLE_EVERY_BATCHES = 8;
+    let batchesSinceRecycle = 0;
+
     // Fan-out result of ResolverPool.resolveBatch, settled (never rejecting)
     // so a fan-out begun before the previous batch's persist can't produce an
     // unhandled rejection while it waits to be awaited.
@@ -1557,6 +1565,31 @@ export class ReferenceResolver {
       if (bp) await bp;
       lp('backpressure', tLp);
 
+      // Recycle the workers' read connections periodically at this same
+      // worker-idle boundary (batch k settled, batch k+1 not yet fanned
+      // out): a long-lived reader pins WAL checkpoint progress, and the
+      // deep WAL that accumulates behind it taxes the writer's OWN page
+      // operations — the §7a.6 writes-under-readers finding (deletes
+      // 42.6s → 118.8s from 0 to 4 attached readers; an aggressive valve
+      // recovered the writes but paid +129s in full-park folds). Releasing
+      // the read marks every ~25 batches lets the existing checkpoints
+      // advance instead, at ~milliseconds of reopen cost. A failed recycle
+      // downgrades to sequential permanently, same as a failed fan-out.
+      if (pool && poolReady && ++batchesSinceRecycle >= RECYCLE_EVERY_BATCHES) {
+        batchesSinceRecycle = 0;
+        tLp = Date.now();
+        try {
+          await pool.recycleWorkers();
+        } catch (err) {
+          logDebug('Worker connection recycle failed; falling back to sequential', {
+            error: err instanceof Error ? err.message : String(err),
+          });
+          await pool.destroy().catch(() => undefined);
+          pool = null;
+        }
+        lp('recycle', tLp);
+      }
+
       // Persist in bounded sub-transactions with yields between: a whole
       // batch's edge insert / keyed deletes are otherwise one solid
       // synchronous span each on a multi-GB index, sitting BETWEEN the

src/resolution/resolver-pool.ts

@@ -63,6 +63,7 @@ export class ResolverPool {
   private nextId = 0;
   private waiters = new Map<number, { resolve: (r: ChunkResult) => void; reject: (e: Error) => void }>();
   private synthWaiters = new Map<number, { resolve: (r: SynthPassResult) => void; reject: (e: Error) => void }>();
+  private recycleWaiters = new Map<number, () => void>();
   private failed: Error | null = null;
 
   /**
@@ -174,6 +175,10 @@ export class ResolverPool {
           const waiter = this.synthWaiters.get(msg.id);
           this.synthWaiters.delete(msg.id);
           waiter?.resolve({ edges: msg.edges ?? [], ms: msg.ms ?? 0 });
+        } else if (msg.type === 'recycled' && msg.id !== undefined) {
+          const waiter = this.recycleWaiters.get(msg.id);
+          this.recycleWaiters.delete(msg.id);
+          waiter?.();
         } else if (msg.type === 'error') {
           pw.busy--;
           const err = new Error(`resolver worker: ${msg.message}`);
@@ -211,6 +216,10 @@ export class ResolverPool {
     this.waiters.clear();
     for (const [, waiter] of this.synthWaiters) waiter.reject(this.failed);
     this.synthWaiters.clear();
+    // Pending recycles resolve rather than reject: their per-call timeout
+    // owns rejection, and the recycle caller checks this.failed next round.
+    for (const [, done] of this.recycleWaiters) done();
+    this.recycleWaiters.clear();
   }
 
   /** Whether this batch is worth fanning out. */
@@ -273,6 +282,43 @@ export class ResolverPool {
     });
   }
 
+  /**
+   * Ask every worker to close and reopen its read-only connection, and wait
+   * for all acks. MUST be called only at the pool-idle boundary (all fanned
+   * chunks settled, next batch not yet dispatched) — the workers close their
+   * connections in place. Why: a long-lived reader pins WAL checkpoint
+   * progress, and the deep WAL behind it taxes every main-thread B-tree
+   * page operation (writes-under-readers, plan §7a.6 — deletes 42.6→118.8s
+   * from 0 to 4 attached readers). Releasing the read marks periodically
+   * lets the existing checkpoints advance, keeping the WAL shallow WITHOUT
+   * the full-park folds an aggressive valve pays (+129s measured at 64MB).
+   * A recycle failure fails the pool — the caller's sequential fallback
+   * covers the rest of the run.
+   */
+  async recycleWorkers(): Promise<void> {
+    if (this.failed) throw this.failed;
+    await Promise.all(
+      this.workers.map(
+        (pw) =>
+          new Promise<void>((resolve, reject) => {
+            const id = this.nextId++;
+            const t = setTimeout(() => {
+              if (this.recycleWaiters.delete(id)) {
+                const err = new Error('resolver worker recycle timed out');
+                this.fail(err);
+                reject(err);
+              }
+            }, 10_000);
+            this.recycleWaiters.set(id, () => {
+              clearTimeout(t);
+              resolve();
+            });
+            pw.worker.postMessage({ type: 'recycle', id });
+          })
+      )
+    );
+  }
+
   async destroy(): Promise<void> {
     await Promise.all(
       this.workers.map(

src/resolution/resolver-worker.ts

@@ -39,15 +39,19 @@ let resolver: ReferenceResolver | null = null;
 
 type InMessage =
   | { type: 'open'; dbPath: string; projectRoot: string }
+  | { type: 'recycle'; id: number }
   | { type: 'resolve'; id: number; refs: UnresolvedReference[] }
   | { type: 'synth'; id: number; pass: string }
   | { type: 'close' };
 
+let dbPath: string | null = null;
+
 port.on('message', (msg: InMessage) => {
   try {
     switch (msg.type) {
       case 'open': {
         const tOpen = Date.now();
+        dbPath = msg.dbPath;
         const created = createDatabase(msg.dbPath, { readOnly: true });
         db = created.db;
         db.pragma('busy_timeout = 5000');
@@ -60,6 +64,25 @@ port.on('message', (msg: InMessage) => {
         port.postMessage({ type: 'ready' });
         break;
       }
+      case 'recycle': {
+        // Close and reopen the read-only connection so the WAL checkpoints
+        // the writer runs can advance past this reader's mark (see
+        // QueryBuilder.rebind). Everything above the connection survives —
+        // the resolver keeps its warm caches; prepared statements re-prepare
+        // lazily. Runs only at the pool-idle boundary, so no query is in
+        // flight on this connection.
+        if (!queries || !dbPath) throw new Error('resolver-worker: recycle before open');
+        try {
+          db?.close();
+        } catch { /* already closed */ }
+        const reopened = createDatabase(dbPath, { readOnly: true });
+        db = reopened.db;
+        db.pragma('busy_timeout = 5000');
+        db.pragma('cache_size = -32000');
+        queries.rebind(db);
+        port.postMessage({ type: 'recycled', id: msg.id });
+        break;
+      }
       case 'resolve': {
         if (!resolver) throw new Error('resolver-worker: resolve before open');
         const tRes = Date.now();