perf(store): resolution ref-index window — kernel-scale resolution 423→276s, 8c envelope ≈11min (§4d round 2) (#1369)

Store-architecture arc round 2. The batched resolution loop reads
unresolved_refs ONLY through the status index + the PK keyset pager;
the other five ref indexes (from_node, name, file_path, from_name,
failed_tail) serve sync-time paths — yet every per-batch DELETE of
resolved refs maintained all of them, the biggest single main-thread
stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase)
and 50-81s at kernel scale.

beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as
refIndexLoad hooks next to the existing bulkEdgeLoad pair with the
same minRefsForPool gate (small syncs never pay): drop the five for
the loop, rebuild each in one scan at the end — where the table holds
only the surviving FAILED refs (resolved rows are deleted by then),
so the recreate is near-free. Crash inside the window heals on the
next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS).

Measured:
- dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall
  ~8.5s flat — the freed main-lane time shifts into settle (the worker
  lane now binds the double-buffer at medium scale).
- Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s,
  backpressure 16.8 → 7.4s — fewer index writes mean less WAL and
  cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the
  14.8min pre-arc best; <10min-on-8c now needs ~1 more minute.

Gates: dubbo/gson dumps byte-identical; linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -22,7 +22,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`.
 - Indexing inside CPU- or memory-limited containers (Docker, CI runners) now sizes its worker pools from the container's actual allowance instead of the host machine's, and giant codebases no longer balloon temporary database files during indexing (previously tens of GB of transient disk on Linux-kernel-scale projects). Together these prevent out-of-memory and out-of-disk failures on constrained machines; set `CODEGRAPH_RESOLVE_WORKERS` to override the resolution worker count explicitly.
 - Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines.
-- Fresh indexing got a sizeable across-the-board speedup: during the initial build, the database's secondary lookup indexes are set aside and rebuilt once after parsing instead of being maintained row by row — the same proven trick the later linking phase already used, now applied to the whole parse lane. A medium Java project's parse phase runs about 58% faster and its full fresh index about 19% faster end-to-end; on Linux-kernel-scale projects the whole index drops several percent further because the rebuilt indexes are more compact and every later phase reads them faster. Graphs remain byte-for-byte identical, and incremental syncs are unaffected.
+- Fresh indexing got a sizeable across-the-board speedup: during the initial build, the database's secondary lookup indexes are set aside and rebuilt once after parsing instead of being maintained row by row — the same proven trick the later linking phase already used, now applied to the whole parse lane — and the reference-resolution loop likewise stops maintaining lookup indexes it never reads, rebuilding them at the end when almost nothing is left in the table. A medium Java project's parse phase runs about 58% faster and its full fresh index about 19% faster end-to-end; a Linux-kernel-scale index that took ~15 minutes on an 8-core machine now completes in about 11, with the resolution phase alone dropping by a third. Graphs remain byte-for-byte identical, and incremental syncs are unaffected.
 - Indexing large C and C++ codebases spends much less time in the function-pointer dispatch analysis (the pass that connects handler tables like a command table or an ops struct to their call sites): each source file is now read and prepared once instead of four times, files that can't contribute any dispatch wiring are skipped outright in the later linking steps, and on platforms with the native engine the per-file scanning itself now runs natively too. On a Linux-kernel-scale tree the pass runs about a third faster end-to-end, with graphs byte-for-byte identical; platforms without a native binary keep the same results on the previous path.
 
 ### Fixes

docs/design/rust-kernel-migration-plan.md

@@ -543,6 +543,30 @@ gap. Multi-file write transactions are likely ~zero on the fastInit path
 batching); buffer→bind remains a CPU-side option if the writer re-emerges as
 the wall.
 
+**Store-arc round 2 SHIPPED (2026-07-19): resolution ref-index window.** The
+batched resolution loop reads unresolved_refs ONLY through the status index
++ PK keyset pager; the other five ref indexes (from_node, name, file_path,
+from_name, failed_tail) serve sync-time paths — yet every per-batch DELETE
+maintained all of them. `beginBulkRefLoad`/`endBulkRefLoad` (same
+`minRefsForPool` gate as the edge window) drop the five for the loop and
+rebuild at the end, where the table holds only the surviving failed refs
+(resolved rows are deleted by then), making the recreate near-free.
+
+- **dubbo**: deletes 1.2 → 0.2s, marks 0.6 → 0.3s (recreate 219ms) — but the
+  wall stayed ~8.5s: the freed main-lane time moved into `settle` (the
+  worker lane now binds the double-buffer). The honest read: at medium
+  scale, resolution's floor is now the WORKER lane + pool spin-up, not the
+  writer.
+- **Linux kernel 8c: resolution 423.4 → 275.9s** — deletes 50-81 → **3.2s**,
+  backpressure 16.8 → 7.4s (fewer index writes → less WAL → cheaper folds,
+  compounding), ref recreate 10.3s, edge recreate 26.8s. Synthesis 149.0s.
+  **Envelope ≈ 11.0min** (phase sum 659.7s) — from 14.8min best-ever before
+  this arc's rounds. **The <10min-on-8c target needs ~1 more minute**; the
+  remaining mass is parse-loop 190s (extraction-bound on linux — R7b's
+  wasm-deferred C tail + the store lane) and synthesis 149s.
+- Gates: dubbo/gson dumps byte-identical, linux counts exact + dump sha
+  `6dd1185b…` reproduced, suite green ×2.
+
 ## 4. Per-language tracker
 
 Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept

src/db/index.ts

@@ -221,6 +221,50 @@ export class DatabaseConnection {
     await this.endBulkEdgeLoad();
   }
 
+  /**
+   * unresolved_refs secondary indexes NOT read by the batched resolution
+   * loop. The loop pages pending refs by keyset (`status='pending' AND id>?`
+   * — the status index + PK), deletes resolved rows by id, and parks failures
+   * with a status UPDATE; every other ref index serves SYNC-time paths
+   * (per-file re-index deletes, name-keyed retry, failed-tail heal). Each
+   * per-batch DELETE maintains all of them — the biggest single main-thread
+   * stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase) —
+   * so the batched loop drops them and rebuilds at the end, where the table
+   * holds only the surviving FAILED refs (resolved rows are gone), making
+   * the recreate near-free.
+   */
+  private static readonly BULK_REF_INDEX_NAMES = [
+    'idx_unresolved_from_node',
+    'idx_unresolved_name',
+    'idx_unresolved_file_path',
+    'idx_unresolved_from_name',
+    'idx_unresolved_failed_tail',
+  ] as const;
+
+  /**
+   * Enter bulk-ref mode for the batched resolution loop — see
+   * BULK_REF_INDEX_NAMES. MUST be paired with endBulkRefLoad(); a crash
+   * inside the window heals on the next open (schema.sql re-applies
+   * CREATE INDEX IF NOT EXISTS).
+   */
+  beginBulkRefLoad(): void {
+    for (const idx of DatabaseConnection.BULK_REF_INDEX_NAMES) {
+      this.db.exec(`DROP INDEX IF EXISTS ${idx}`);
+    }
+  }
+
+  /** Leave bulk-ref mode: recreate each index in one scan (yield between). */
+  async endBulkRefLoad(): Promise<void> {
+    const schemaPath = path.join(__dirname, 'schema.sql');
+    const schema = fs.readFileSync(schemaPath, 'utf-8');
+    for (const idx of DatabaseConnection.BULK_REF_INDEX_NAMES) {
+      const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`));
+      if (!m) throw new Error(`schema.sql: ref index ${idx} not found for bulk-load recreation`);
+      this.db.exec(m[0]);
+      await new Promise((resolve) => setImmediate(resolve));
+    }
+  }
+
   /**
    * Names of the NON-UNIQUE edge indexes dropped for a bulk edge load.
    * idx_edges_identity deliberately stays: INSERT OR IGNORE's dedup conflicts

src/index.ts

@@ -1175,6 +1175,10 @@ export class CodeGraph {
         begin: () => this.db.beginBulkEdgeLoad(),
         end: () => this.db.endBulkEdgeLoad(),
       },
+      refIndexLoad: {
+        begin: () => this.db.beginBulkRefLoad(),
+        end: () => this.db.endBulkRefLoad(),
+      },
       backpressure,
     });
   }

src/resolution/index.ts

@@ -1388,6 +1388,10 @@ export class ReferenceResolver {
     parallel?: {
       dbPath: string;
       bulkEdgeLoad?: { begin: () => void; end: () => void | Promise<void> };
+      /** unresolved_refs index window for the batched loop — the loop only
+       *  reads the status index + PK; dropping the sync-path ref indexes cuts
+       *  each per-batch DELETE's B-tree work (DatabaseConnection.beginBulkRefLoad). */
+      refIndexLoad?: { begin: () => void; end: () => void | Promise<void> };
       backpressure?: () => Promise<void> | null;
     }
   ): Promise<ResolutionResult> {
@@ -1532,6 +1536,16 @@ export class ReferenceResolver {
         bulkEdgesActive = true;
       } catch { /* keep the indexes; inserts just pay the per-row maintenance */ }
     }
+    // Same gate for the ref-index window: the loop's deletes stop maintaining
+    // the five sync-path unresolved_refs indexes, and the end-of-loop rebuild
+    // is near-free (only failed refs survive the loop).
+    let bulkRefsActive = false;
+    if (parallel?.refIndexLoad && total >= minRefsForPool()) {
+      try {
+        parallel.refIndexLoad.begin();
+        bulkRefsActive = true;
+      } catch { /* keep the indexes; deletes just pay the per-row maintenance */ }
+    }
 
     try {
     try {
@@ -1717,6 +1731,11 @@ export class ReferenceResolver {
       // Recreate the edge indexes BEFORE synthesis (kind-keyed reads) and on
       // any error path. A crash before this line is healed by the next
       // DatabaseConnection open (schema.sql re-applies IF NOT EXISTS).
+      if (bulkRefsActive) {
+        const tRef = Date.now();
+        await parallel!.refIndexLoad!.end();
+        if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] ref-index-recreate: ${Date.now() - tRef}ms`);
+      }
       if (bulkEdgesActive) {
         const tIdx = Date.now();
         await parallel!.bulkEdgeLoad!.end();