fix(scale): kernel-scale hardening — OOM-safe pass skipping + watchdog-safe index recreate (#1323)

Two hazards found by running today's full stack against the Linux kernel
(70,129 files) in the cg1212 repro container:

1. The parallel-synthesis fallback retried a worker-failed pass on the MAIN
   thread. At multi-million-node scale a worker failure is usually a memory
   ceiling, so the retry would OOM the process and take the whole index with
   it. Above 1.5M nodes a failed pass is now skipped with a clear stderr
   message (its synthesized edges are absent; the index completes). Below
   that, the main-thread retry stays — small-scale worker crashes are
   transient and the retry keeps coverage.

2. endBulkEdgeLoad rebuilt all four edge indexes in one synchronous span —
   measured 79s at kernel scale, past the #850 liveness watchdog's 60s
   stall window. A daemon-triggered re-index would have been SIGKILLed right
   after doing the work. Now async with an event-loop yield between builds,
   keeping each stall to a single index (~20s at kernel scale).

Validation: full Linux kernel index to completion in the repro container —
2,048,674 nodes / 6,405,964 edges, EXIT 0, zero passes skipped, on a 2-CPU
VM (worst case: pool disabled, sequential resolution + synthesis) in ~27min.
Phase walls: parse 6.0m, resolution 19.5m (incl. synthesis 6.3m, recreate
79s), maintenance 74s. Suite green (2444).

Also adds docs/design/native-extraction-kernel.md — the spike-validated
design for the native extraction kernel (Rust parse+walk over dubbo's Java:
202ms rayon / 1.07s single-thread vs 4.7s for the current wasm pipeline).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -15,6 +15,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Indexing large projects got another sizeable speedup — about a quarter less wall-clock on the same 4,000-file Java project, with the graph still byte-for-byte identical. Two changes: the database no longer interleaves expensive checkpoint housekeeping into the middle of resolution on a fresh index (it's folded once at the end instead), and while one batch's results are being written out, the worker threads are already resolving the next batch instead of sitting idle.
 - The dynamic-dispatch analysis that runs at the end of indexing (callback, event, and framework wiring) now runs its passes in parallel on large projects, cutting that stage roughly in half there — and a pass that crashes now retries safely instead of failing the whole index, which also makes very large codebases that previously died in this stage more likely to index to completion. Graphs remain byte-for-byte identical.
 - On large projects, indexing writes its relationship data noticeably faster: secondary database indexes are set aside during the bulk of reference resolution and rebuilt once at the end, instead of being maintained row by row. Graphs remain byte-for-byte identical, and small projects are unaffected.
+- Very-large-codebase reliability: on multi-million-symbol projects, an analysis pass that fails on a worker thread is now skipped with a clear message instead of being retried in a way that could take down the whole index, and the end-of-indexing index rebuild no longer risks tripping the liveness watchdog on huge graphs. Validated end-to-end on the Linux kernel (70k files, 2M symbols, 6.4M relationships) — it now indexes to completion even on a 2-core machine.
 - Indexing is significantly faster — a fresh `codegraph init` on a medium TypeScript project takes about a third less wall-clock time, with the same graph produced byte-for-byte. The gains come from batching database writes, storing files on a dedicated writer thread, memoizing repeated import-resolution lookups, skipping per-row search-index maintenance during the bulk build (rebuilt once at the end), and — on completely fresh databases only — deferring disk durability until the index completes, since an interrupted first index is simply re-run. Set `CODEGRAPH_NO_FAST_INIT=1` to keep full crash-durability during the initial build, or `CODEGRAPH_NO_STORE_WORKER=1` to store on the main thread.
 - `codegraph install` and `codegraph upgrade` now offer CodeGraph Pro beta access after finishing — answer yes, type your email, and you join the same waitlist as the getcodegraph.com homepage form. Strictly opt-in and asked at most once per machine total: nothing is sent unless you say yes and enter an email, either answer is remembered so no later install or upgrade ever re-asks, and non-interactive runs (`--yes`, scripts, CI) never see the question.
 - Every release is now cryptographically verifiable: npm packages publish with npm provenance (the "Provenance" badge on npmjs.com, proving each version was built by this repository's release workflow from a specific commit), and the GitHub Release bundles carry signed build attestations you can check with `gh attestation verify <file> -R colbymchenry/codegraph`.

docs/design/native-extraction-kernel.md

@@ -0,0 +1,87 @@
+# Native extraction kernel — design + spike results
+
+**Status:** spike validated 2026-07-16; project approved, not yet started.
+**Owner context:** the last structural lever for fresh-index wall clock after the
+2026-07-16 arc (#1305, #1320, #1321, #1322) exhausted Node-side scheduling.
+
+## Why
+
+Post-arc, the fresh-index profile on dubbo (4,402 Java files) is:
+parse-loop ~4.7s, resolution ~5.5s (persist-bound), synthesis ~0.9s, total ~11.1s
+vs codebase-memory-mcp v0.9.0 at 7.1s. Two levers were measured dead:
+
+- **RAM-backed DB** (parse-loop 6.9s on a ramdisk vs 4.6–4.8s on SSD, n=2
+  interleaved): fast-init `synchronous=OFF` already writes at page-cache speed.
+  The parse phase is CPU-bound.
+- **TreeCursor rewrite** (earlier arc): web-tree-sitter's traversal is not the
+  cost; the floor is per-node JS↔WASM **marshaling** — every `node.kind`,
+  `.childForFieldName`, `.text` crosses the boundary.
+
+The only remaining parse lever is doing the walk on the native side and
+crossing the boundary **once per file** instead of once per node.
+
+## Spike (2026-07-16)
+
+Minimal Rust binary (`tree-sitter` 0.25 + `tree-sitter-java`, TreeCursor walk
+touching every node's kind/range + `name`-field text, emitting flat
+`(kind_id, start, end, name_len)` rows — the extraction access pattern).
+Dubbo's 4,048 `.java` files, 17MB, 3.59M AST nodes, Apple M3 Pro:
+
+| | wall |
+|---|---|
+| Current pipeline parse-loop (7 wasm workers, incl. extraction + store dispatch) | 4,700ms |
+| Rust parse+walk, rayon | **202ms** |
+| Rust parse+walk, single thread | 1,067ms |
+
+One native thread beats the whole 7-worker wasm pool 4.4×; at equal
+parallelism the walk is ~14× faster. Even charging the kernel for the
+extraction logic it must still perform, parse-loop 4.7s → ~1.0–1.5s is
+realistic, putting dubbo ≈ 7.5–8s total (parity with cbm).
+
+## Architecture
+
+- **Crate:** `codegraph-kernel`, napi-rs, links tree-sitter's C library and
+  vendored grammars natively. Input: `(filePath, content, language)`. Output:
+  flat typed buffers (nodes, edges, unresolved refs) — one boundary crossing
+  per file.
+- **Per-language logic:** migrate extractors to tree-sitter **query files**
+  (`.scm`) executed by a generic Rust emitter; bespoke TS logic that queries
+  can't express (macro salvage, dialect sniffing, content-gated `.h`
+  detection) stays as TS pre/post passes over the returned buffers.
+- **Distribution:** prebuilt `.node` per platform through the existing
+  release-bundle pipeline (same per-platform packages as the Node runtime).
+  The wasm path remains as the universal fallback — same crate compiled to
+  wasm keeps one implementation.
+- **Rollout:** per-language, funnel languages first (TS/JS → Java → Python →
+  Go). A language ships only when its equivalence gate passes.
+
+## Equivalence gate (per language)
+
+Byte-identity against hand-written extractors is NOT expected (bespoke logic
+ports approximately). The gate is:
+
+1. Node/edge/ref **counts** within ±0.5% on 3 real repos (small/medium/large),
+   with every diff category eyeballed.
+2. The retrieval invariants hold: explore-flow connects the language's
+   canonical flows end-to-end (`docs/design/dynamic-dispatch-coverage-playbook.md`),
+   agent A/B shows no regression per the standard methodology.
+3. Fresh-index wall improves on the language's repos; no regression on a
+   control repo of a non-migrated language.
+
+## Non-goals
+
+- Porting resolution, synthesis, frameworks, MCP, or the installer — they are
+  pool-parallel and not marshal-bound. The measured native advantage there is
+  ~1.4× CPU, not worth the correctness moat (2,444 tests, byte-identical
+  determinism, years of invariants).
+- A single static binary (distribution polish, orthogonal to speed).
+
+## Risks
+
+- ABI drift between vendored native grammars and the wasm fallback grammars
+  (keep both built from the same grammar source revs; CI asserts).
+- `.scm` expressiveness ceilings — budget for a per-language "escape hatch"
+  callback in the emitter before declaring a language blocked.
+- napi-rs threading vs the parse-pool: the kernel replaces the wasm workers'
+  parse+extract; the pool orchestration (file-order commit, retry, recycle)
+  stays in TS and drives the kernel synchronously per file.

src/db/index.ts

@@ -185,14 +185,23 @@ export class DatabaseConnection {
    * Leave bulk-edge-load mode: recreate the dropped indexes in one pass each
    * over the (now fully loaded) edges table — far cheaper than maintaining
    * them per-insert. DDL is extracted from schema.sql so it cannot drift.
+   *
+   * Async with a yield BETWEEN the four CREATE INDEX statements: each build is
+   * a synchronous scan of the whole edges table (~20s apiece at Linux-kernel
+   * scale, 79s total measured), and running them back-to-back is a single
+   * event-loop stall longer than the #850 liveness watchdog's 60s window — a
+   * daemon-triggered re-index would be SIGKILLed right after doing the work.
+   * One yield per statement keeps every stall to a single index build, which
+   * stays inside the window.
    */
-  endBulkEdgeLoad(): void {
+  async endBulkEdgeLoad(): Promise<void> {
     const schemaPath = path.join(__dirname, 'schema.sql');
     const schema = fs.readFileSync(schemaPath, 'utf-8');
     for (const idx of DatabaseConnection.BULK_EDGE_INDEX_NAMES) {
       const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`));
       if (!m) throw new Error(`schema.sql: edge index ${idx} not found for bulk-load recreation`);
       this.db.exec(m[0]);
+      await new Promise((resolve) => setImmediate(resolve));
     }
   }
 

src/resolution/callback-synthesizer.ts

@@ -3666,6 +3666,15 @@ export async function synthesizeCallbackEdges(
     else markPass(SYNTH_PASSES[i]!.name, 0);
   }
 
+  // Above this node count, a pass that OOM-killed its worker must NOT be
+  // retried on the main thread — the retry would OOM the whole process and
+  // take the index with it (the #1212 failure class). Below it, a worker
+  // failure is more likely a transient crash than a memory ceiling, and the
+  // main-thread retry keeps coverage. Skipping loses only that pass's
+  // synthesized edges; the index still completes.
+  const MAIN_RETRY_MAX_NODES = 1_500_000;
+  const graphNodes = queries.getNodeAndEdgeCount().nodes;
+
   if (pool && gatedIn.length > 1) {
     await Promise.all(
       gatedIn.map(async (i) => {
@@ -3674,7 +3683,16 @@ export async function synthesizeCallbackEdges(
           const out = await pool.runSynthPass(pass.name);
           passEdges[i] = out.edges;
           markPass(pass.name, out.ms);
-        } catch {
+        } catch (err) {
+          if (graphNodes > MAIN_RETRY_MAX_NODES) {
+            // Worker died at a scale where the main-thread retry is a process
+            // OOM risk: skip the pass, keep the index alive, and say so.
+            console.error(
+              `[synthesis] pass '${pass.name}' failed on a worker at ${graphNodes} nodes — skipped (edges from this pass are absent): ${err instanceof Error ? err.message : String(err)}`
+            );
+            markPass(`${pass.name} (skipped at scale)`, 0);
+            return;
+          }
           // Worker-side failure (crash, OOM, unknown pass after a version
           // mismatch): retry this one pass on the main thread.
           await runPassOnMain(i);

src/resolution/index.ts

@@ -1339,7 +1339,7 @@ export class ReferenceResolver {
     // disables entirely. bulkEdgeLoad hooks (when provided) bracket the batch
     // loop with drop/recreate of the non-unique edge indexes on big runs —
     // see DatabaseConnection.beginBulkEdgeLoad.
-    parallel?: { dbPath: string; bulkEdgeLoad?: { begin: () => void; end: () => void } }
+    parallel?: { dbPath: string; bulkEdgeLoad?: { begin: () => void; end: () => void | Promise<void> } }
   ): Promise<ResolutionResult> {
     // Resolution runs on the indexer's MAIN thread, and the #850 liveness
     // watchdog SIGKILLs a process whose event loop stalls past its window (60s
@@ -1583,7 +1583,7 @@ export class ReferenceResolver {
       // DatabaseConnection open (schema.sql re-applies IF NOT EXISTS).
       if (bulkEdgesActive) {
         const tIdx = Date.now();
-        parallel!.bulkEdgeLoad!.end();
+        await parallel!.bulkEdgeLoad!.end();
         if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] edge-index-recreate: ${Date.now() - tIdx}ms`);
       }
     }