perf(store): parse-lane index deferral — dubbo fresh init −19%, kernel-scale envelope best-ever 14.2min (§4d round 1) (#1368)

Store-architecture arc round 1 (the cbm speed bar: dubbo warm wall
10.7-11.2s vs their ~7.5). §4d measured dubbo's parse-loop as 94%
store-writer busy with B-tree maintenance as the floor (statement
batching and sorted inserts already killed at ~zero). This applies the
resolution phase's proven edge-index window to the whole parse lane:

beginBulkParseLoad/endBulkParseLoad on DatabaseConnection — FRESH-INIT
ONLY (incremental runs delete per-file rows through the file_path
indexes) — drop all 15 nodes/unresolved_refs/files secondary indexes
plus the 4 non-unique edge indexes for the parse phase's mass insert
(the UNIQUE edge identity index stays: OR-IGNORE dedup conflicts on it,
and its source prefix keeps mid-window reads indexed), then rebuild
each in one table scan before resolution, with a yield between builds
(the endBulkEdgeLoad watchdog rationale). A crash inside the window
heals on the next open — schema.sql re-applies CREATE INDEX IF NOT
EXISTS.

Measured:
- dubbo (cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild
  665ms, warm fresh-init wall 10.5-11.3 → 8.46-9.39s (−19%); the bar
  gap vs cbm shrinks from ~3s to ~1.1s.
- Linux kernel 8c: envelope ≈ 14.2min, best ever (prior 14.8). Parse
  itself flat (linux parse is extraction-bound, not writer-bound) and
  the rebuild costs 21.6s — but every downstream phase dropped
  (resolution 517-589 → 423.4s, edge-recreate 36.5s, synthesis 157.1s,
  maintenance 16.3s): bulk-rebuilt B-trees are densely packed where
  incrementally-grown ones are fragmented, so every index-mediated read
  for the rest of the run pays fewer pages.

Gates: dubbo/gson/express/excalidraw full dumps byte-identical
(dubbo's canonical 441,270 lines reproduced); linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests). Incremental
sync paths untouched by construction (freshDb gate).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -22,6 +22,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`.
 - Indexing inside CPU- or memory-limited containers (Docker, CI runners) now sizes its worker pools from the container's actual allowance instead of the host machine's, and giant codebases no longer balloon temporary database files during indexing (previously tens of GB of transient disk on Linux-kernel-scale projects). Together these prevent out-of-memory and out-of-disk failures on constrained machines; set `CODEGRAPH_RESOLVE_WORKERS` to override the resolution worker count explicitly.
 - Indexing very large projects on multi-core machines got faster again: the parallel-resolution workers now periodically refresh their read-only database connections, which lets database housekeeping advance instead of silently building up a backlog behind long-lived readers — a backlog that was taxing the indexer's own writes. Graphs remain byte-for-byte identical; the win is largest at Linux-kernel scale on many-core machines.
+- Fresh indexing got a sizeable across-the-board speedup: during the initial build, the database's secondary lookup indexes are set aside and rebuilt once after parsing instead of being maintained row by row — the same proven trick the later linking phase already used, now applied to the whole parse lane. A medium Java project's parse phase runs about 58% faster and its full fresh index about 19% faster end-to-end; on Linux-kernel-scale projects the whole index drops several percent further because the rebuilt indexes are more compact and every later phase reads them faster. Graphs remain byte-for-byte identical, and incremental syncs are unaffected.
 - Indexing large C and C++ codebases spends much less time in the function-pointer dispatch analysis (the pass that connects handler tables like a command table or an ops struct to their call sites): each source file is now read and prepared once instead of four times, files that can't contribute any dispatch wiring are skipped outright in the later linking steps, and on platforms with the native engine the per-file scanning itself now runs natively too. On a Linux-kernel-scale tree the pass runs about a third faster end-to-end, with graphs byte-for-byte identical; platforms without a native binary keep the same results on the previous path.
 
 ### Fixes

docs/design/rust-kernel-migration-plan.md

@@ -513,6 +513,36 @@ during the parse phase, multi-file write transactions, buffer→bind without obj
 materialization. Note the #1320-arc post-mortem already measured statement batching
 and sorted inserts as ~zero on this path — B-tree maintenance is the floor.
 
+**Store-arc round 1 SHIPPED (2026-07-19): parse-lane index deferral.** The
+first named lever landed as `beginBulkParseLoad`/`endBulkParseLoad`
+(fresh-init only — incremental runs delete per-file rows through the
+file_path indexes): the parse window drops all 15 nodes/unresolved_refs/files
+secondary indexes plus the 4 non-unique edge indexes (identity stays for
+OR-IGNORE dedup), and rebuilds each in one scan before resolution — the edge
+window's measured trade applied to the whole parse lane. Results:
+
+- **dubbo (the cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild
+  665ms, warm wall 10.5-11.3 → 8.46-9.39s** — the bar gap shrank from ~3s to
+  ~1.1s vs cbm's ~7.5s.
+- **Linux kernel 8c: envelope ≈ 14.2min — best ever** (prior best 14.8). The
+  parse-loop itself stayed ~189s (linux parse is EXTRACTION-bound — the
+  wasm-deferred C tail — unlike writer-bound dubbo) and the rebuild costs
+  21.6s, but every downstream phase dropped: resolution 517-589 → 423.4s,
+  edge-recreate → 36.5s, synthesis → 157.1s, maintenance → 16.3s. Mechanism:
+  bulk-rebuilt B-trees are densely packed where incrementally-grown ones are
+  fragmented, so every index-mediated read for the rest of the run pays fewer
+  pages. The rebuild is the gift that keeps giving downstream.
+- Gates: dubbo/gson/express/excalidraw dumps byte-identical (441,270-line
+  dubbo dump reproduced), linux counts exact + dump sha `6dd1185b…`
+  reproduced, suite green ×2.
+
+Remaining store levers, re-ranked: dubbo's residual vs cbm is now resolution
+(~5.3s of the 8.5s wall) + boot (~1s) — the parse lane is no longer the
+gap. Multi-file write transactions are likely ~zero on the fastInit path
+(memory journal, synchronous OFF — same class as the killed statement
+batching); buffer→bind remains a CPU-side option if the writer re-emerges as
+the wall.
+
 ## 4. Per-language tracker
 
 Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept

src/db/index.ts

@@ -151,6 +151,76 @@ export class DatabaseConnection {
     this.recreateFtsTriggers();
   }
 
+  /**
+   * NON-UNIQUE secondary indexes maintained per-row during the parse phase's
+   * bulk inserts — the store-architecture arc's first lever (plan §4d: dubbo's
+   * parse-loop wall is 94% store-writer busy, and the #1320 post-mortem showed
+   * statement batching and sorted inserts are ~zero on this path because
+   * B-TREE MAINTENANCE is the floor). A fresh init writes every row of
+   * nodes/unresolved_refs/files exactly once and reads none of them until
+   * resolution, so the parse window can drop all of these and rebuild each in
+   * one table scan afterwards — the same measured trade as the resolution
+   * phase's edge-index window (2.8s → 1.1s inserting, ~0.3s recreating).
+   * Primary keys and UNIQUE constraints stay (upserts and OR-IGNORE dedup
+   * conflict on them).
+   */
+  private static readonly BULK_PARSE_INDEX_NAMES = [
+    'idx_nodes_kind',
+    'idx_nodes_name',
+    'idx_nodes_qualified_name',
+    'idx_nodes_file_path',
+    'idx_nodes_language',
+    'idx_nodes_file_line',
+    'idx_nodes_lower_name',
+    'idx_unresolved_from_node',
+    'idx_unresolved_name',
+    'idx_unresolved_file_path',
+    'idx_unresolved_from_name',
+    'idx_unresolved_status',
+    'idx_unresolved_failed_tail',
+    'idx_files_language',
+    'idx_files_modified_at',
+  ] as const;
+
+  /**
+   * Enter bulk-parse-load mode (FRESH-INIT ONLY — the caller gates on a fresh
+   * DB, because an incremental index deletes per-file rows mid-phase and needs
+   * the file_path indexes): drop every parse-lane secondary index, including
+   * the four non-unique edge indexes (parse inserts contains-edges too; the
+   * UNIQUE identity index stays for INSERT OR IGNORE dedup, and its `source`
+   * prefix keeps source-keyed reads indexed, as in the edge window). MUST be
+   * paired with endBulkParseLoad(); a crash inside the window is healed on the
+   * next DatabaseConnection open (schema.sql re-applies CREATE INDEX IF NOT
+   * EXISTS).
+   */
+  beginBulkParseLoad(): void {
+    for (const idx of DatabaseConnection.BULK_PARSE_INDEX_NAMES) {
+      this.db.exec(`DROP INDEX IF EXISTS ${idx}`);
+    }
+    this.beginBulkEdgeLoad();
+  }
+
+  /**
+   * Leave bulk-parse-load mode: recreate everything the window dropped, one
+   * table scan per index, with a yield between statements (same
+   * liveness-watchdog rationale as endBulkEdgeLoad — at kernel scale each
+   * build is a long synchronous scan). The edge indexes are rebuilt here too,
+   * so paths that never enter the resolution phase's own bulk-edge window
+   * (small runs) are left with a complete schema; the batched resolver's
+   * beginBulkEdgeLoad simply re-drops them (DROP IF EXISTS — idempotent).
+   */
+  async endBulkParseLoad(): Promise<void> {
+    const schemaPath = path.join(__dirname, 'schema.sql');
+    const schema = fs.readFileSync(schemaPath, 'utf-8');
+    for (const idx of DatabaseConnection.BULK_PARSE_INDEX_NAMES) {
+      const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`));
+      if (!m) throw new Error(`schema.sql: parse index ${idx} not found for bulk-load recreation`);
+      this.db.exec(m[0]);
+      await new Promise((resolve) => setImmediate(resolve));
+    }
+    await this.endBulkEdgeLoad();
+  }
+
   /**
    * Names of the NON-UNIQUE edge indexes dropped for a bulk edge load.
    * idx_edges_identity deliberately stays: INSERT OR IGNORE's dedup conflicts

src/index.ts

@@ -493,6 +493,11 @@ export class CodeGraph {
         // triggers, rebuild nodes_fts once from the nodes table afterwards.
         // Crash inside the window is healed on the next DatabaseConnection.open.
         this.db.beginBulkNodeLoad();
+        // Fresh-init only: also drop the parse-lane secondary indexes for the
+        // mass insert (the store-writer's B-tree-maintenance floor, plan §4d)
+        // and rebuild each in one scan afterwards. Incremental runs keep them
+        // — they delete per-file rows mid-phase through the file_path indexes.
+        if (freshDb) this.db.beginBulkParseLoad();
         let result: IndexResult;
         try {
           result = await this.orchestrator.indexAll(
@@ -506,6 +511,11 @@ export class CodeGraph {
             freshDb ? { dbPath: this.db.getPath(), fastInit } : null
           );
         } finally {
+          if (freshDb) {
+            const tIdx = Date.now();
+            await this.db.endBulkParseLoad();
+            if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] parse-index-rebuild: ${Date.now() - tIdx}ms`);
+          }
           const tFts = Date.now();
           this.db.endBulkNodeLoad();
           if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] fts-rebuild: ${Date.now() - tFts}ms`);