Rust extraction-kernel migration plan + post-kernel roadmap

Audience: the agent/engineer executing the native-kernel project. Self-contained handoff: context, current state, per-language tracker, gates, and the follow-on roadmap. Companion: docs/design/native-extraction-kernel.md (architecture + spike detail). Written: 2026-06-12 planning → executed 2026-07-16/17. R1–R6 ARE DONE. The shipped records live in §3a and §4a–§4f; the per-language tracker is current; §0a is the cold-start handoff for the next session. Read §0 + §0a first — parts of §1/§6 below them are the ORIGINAL plan and carry expectations that measurement later corrected (each is annotated where superseded).

0. Status checklist (R1–R6 done; what remains)

R1. Scaffold the napi-rs crate — done 2026-07-16, §3a. Buffer contract v1, routing + per-file wasm fallback, kill switch, build/release wiring, grammar-source-parity CI.
R2. Port TypeScript/JavaScript (tsx/jsx) — done 2026-07-16, §4a. The generic .scm emitter was SUPERSEDED by bespoke per-language walkers (queries can't express extraction parity); byte-parity from day one of the harness.
R3. TS/JS equivalence gate → DEFAULT-ON — done 2026-07-16, §4b. Dumps byte-identical (express/excalidraw/vscode + flask control). Found + fixed: encoding-dependent error recovery → per-file defer: policy.
R4. Java (incl. Lombok synthesis) → DEFAULT-ON — done 2026-07-16, §4c. dubbo 441k-row dump byte-identical. Found + fixed: node-ID-collision dedupe (cross-language). Found: many-core parse-loop wall is NOT extraction (→ §4d).
Direct-to-store decode — done 2026-07-16, §4d. Main thread never materializes nodes; measured: the many-core fresh-index wall is single-writer SQLite ingest (94% of dubbo's parse-loop) — a store-architecture arc, out of scope here.
R5. Python + Go → DEFAULT-ON — done 2026-07-16, §4e. django 360.8k / prometheus 213.8k row dumps byte-identical; 2-CPU envelope 1.32× / 1.46×.
R6. Kernel-scale re-validation (cg1212) — done 2026-07-17, §4f. No regression (26.4min vs ~27min, identical 2.05M-node graph). The "parse 6m→2m" premise was wrong for THIS repo: the Linux tree is ~99% C (unported T2) — the expectation transfers to the C/C++ port.

Open, in recommended order (rationale in §0a):

O1. Merge the rust-kernel branch — DONE 2026-07-17: PR #1326, merge commit (the integration-branch exception — 9 milestone commits preserved), main tip c1dc78d. Suite green pre-merge (2,472 passed / 4 skipped, CODEGRAPH_KERNEL_EXPECT=1).
O2. Windows VM validation — DONE 2026-07-17. Guest (ARM64 Win11): rustc 1.97.1 aarch64-pc-windows-msvc + MSVC Build Tools (VCTools workload + VC.Tools.ARM64 + Win11 SDK; installed via scheduled task — Windows sshd kills detached children on session close, schtasks is the survival pattern). build-kernel.sh --target aarch64-pc-windows-msvc builds native win32-arm64 in ~2min; all three kernel suites green with CODEGRAPH_KERNEL_EXPECT=1 (33/33). The leg EARNED ITS KEEP: the guest's autocrlf checkout exposed a real CRLF parity bug (docstring cleaning; JS multiline ^ anchors after \r — §0a traps) — fixed + CRLF fixtures pinned cross-platform in #1329. Every prebuild target platform is now validated.
P1. Kernel-scale resolution speed (§7a) — measurement round RUN 2026-07-17 and it RESHAPED the arc (full record §7a.1); items (1) WAL containment and (2) memory-aware/cgroup-honest sizing SHIPPED same day (#1332–#1335, §7a.2) after an implementation arc whose three failed/diagnostic kernel-scale runs each corrected the design (WAL file ≠ WAL backlog; cgroup cache credit; pool net-negative at 2 cores; parse floor). The 2c/6GB envelope already improved 26.4 → 21.6min with counts byte-exact. Record runs DONE (§7a.2): 2c/6GB 20.4min, 8c/7GB 18.3min NO-OOM — byte-exact. Batch-loop profile round DONE (§7a.3, #1339): countGuard quadratic killed, 19.3min. cFnPtr round DONE (§7a.4, #1341): 2.07× standalone, edge set hash-identical, envelope 17.6min (R6 −33%). R7a landed 2026-07-17: envelope now 19.1min on a substantially RICHER graph (the new preParse blanks recover previously-error-swallowed code; wasm-arm on the same graph is 22.9min — the 17.6 record was the old smaller graph and isn't directly comparable). 8c re-run DONE post-R7a (§7a.5): 16.4min (pre-R7a record 18.3min), EXIT 0, counts == both 2c arms, WAL 1.34GB. The <10min-on-8c target remains open, and the re-run re-ranked the levers honestly: at 8c the parse-loop (202.6s) is already AT the single-writer floor, so the target gap is ~entirely the core-invariant resolution superphase (715s ≈ 12 of the 16.4min) — the per-ref path is THE 8c lever. C/C++ deferral round 2 DONE 2026-07-18 (full record: checklist doc): eight new C-only preParse passes + word-list extensions took kernel/+mm/ deferral 58.6% → 33.9% (git 16.1 → 12.2%, redis 25.3 → 24.1%, fmt/protobuf unchanged — cpp-dominant, correct no-op), five-repo sweeps 0-diff, linux full-tree both arms 2,049,153 / 6,413,518 with byte-identical dumps (10,446,478 lines, sha 6dd1185b…); kernel-arm parse-loop 356 → 306s at 2c, envelope ~17.1min (host-contaminated, indicative). Honesty note: full-graph node deltas are small (+858) — wasm error recovery was already salvaging most SYMBOLS on deferred files; the real win is EDGES (+6,585), phantom cleanup, and native-path coverage. Remaining deferral is policy-skips (CONFIG interleaves, TP_PROTO DSL, module_init-no-semi)
- small buckets — this lever is largely SPENT. Per-ref measurement round DONE 2026-07-18 (§7a.6): fresh 2c/8c stage tables; the pool double-buffer WORKS (8c settle 3.6s — "core-invariant" superseded); 2c record now 16.5min, 8c range 15.0–16.4 (n=2). Two cache experiments killed by measurement same-day (nameCache scaling, lazy candidates — §7a.6 has the numbers; code reverted). Writes-under-readers PROBED + FIXED 2026-07-18 (§7a.7): mechanism = WAL read-through depth under reader pins (proven by pool-off/dose/ valve discrimination); fix = worker connection recycling at the pool-idle boundary (cadence 8); superphase 715 → 633.6s (−11.4%), 8c envelope best 14.8min, byte-neutral at every gate. Queue now: cFnPtr native site extraction (synthesis ~230s) > continuous-shallow WAL (the remaining ~45s to the valve floor) > backpressure byte volume > recreate.
R7a. C/C++ port — DONE 2026-07-17, same-day walker+gates after the survey (#1344) and grammar vendoring (#1345). One dual-language walker (codegraph-kernel/src/ccpp/), preParse HOISTED to the route point (both tryKernelExtract and the raw bulk path — no blanking ported to Rust; Metal/CUDA ride the cpp route through the same hoist). Gates: parity sweeps 0 diffs on redis/git/fmt/protobuf/ALS (2,389 files compared); full-init dump-diffs byte-identical on all five; DEFAULT_ROUTED += c, cpp. Three measurement corrections recorded in the checklist doc: (1) C/C++ parse-error incidence is 9–42% per repo (vs 0–0.42% for prior languages), so erroring-file deferral is routine, not a broken-kernel signal — the sweep gained --max-deferral (0.5 for c/cpp) after confirming recovery-divergence is real with the sweep-only no-defer hatch; (2) seven new/extended TS-side preParse blanks (extern-C guard bodies, lone macro lines, statement iterator macros, trailing UNUSED params, the curated Linux/sparse __init-family annotations + container_of type args, cpp leading-attr, directive-line restore) cut real incidence (linux subtrees 79% → 58%) AND grew the wasm path's own graphs (git 7.1k → 13.3k nodes) — so cg1212's "counts must stay 2,048,664/6,405,964" expectation is superseded: the graph legitimately changes with the blanks; the invariant is kernel-arm == wasm-arm at every scale (held: five byte-identical dumps + the linux dump-hash pair); (3) at high deferral the kernel arm initially LOST arm-vs-arm on linux (deferred files ran the pipeline 3×) — fixed with the one-slot defer memo + blanked-source reuse; final cg1212 envelope 19.1 min kernel-arm (parse-loop 560 → 356s; R6 26.4 → P1 17.6 on the old smaller graph → 19.1 on the new richer one: 2,048,295 nodes / 6,406,933 edges, two runs byte-same).
R7b. Remaining long tail per the tracker (§4) — ruby/php/csharp/rust/… T1s are now ~1-day-each with the walker pattern; T3 may stay TS forever (fine).
P2. Arc 3, graph richness (§7b) — product-priority call, standard gates.
P3. Parked items (§7c) — only with explicit maintainer approval.

0a. Cold-start handoff (state as of 2026-07-17)

Where the work lives: MERGED to main 2026-07-17 (PR #1326, merge commit c1dc78d; the 9 milestone commits c5eebe6 R1 → 2a79432 R6 are preserved in history). All scratchpad clones (excalidraw/vscode/dubbo/django/…) were throwaway; re-clone fresh for new gate runs. The cg1212 docker container (Linux kernel, 2 CPU/6GB) is long-lived on the dev Mac and has the current build deployed at /app (tree at /work/linux).

What exists:

codegraph-kernel/ — napi-rs crate. One WALKER MODULE per language (tsjs/, java.rs, python.rs, go.rs, ccpp/ for c+cpp) mirroring TreeSitterExtractor's per-language paths bug-for-bug; shared buffers.rs (wire contract — twin of src/extraction/kernel/layout.ts, byte-matched, ABI-versioned), ids.rs (sha node ids, test-pinned to generateNodeId), docstring.rs, textutil.rs (UTF-16 columns/slices, generated-file patterns, shared regexes), langs.rs (grammar registry).
src/extraction/kernel/ — loader (contract-verifies before routing; a stale .node silently degrades to wasm; CODEGRAPH_KERNEL_DEBUG=1 explains), decode, routing (DEFAULT_ROUTED = ts/tsx/js/jsx/java/python/go/c/cpp; CODEGRAPH_KERNEL_LANGS REPLACES the set; CODEGRAPH_KERNEL=0 kills), the deferred-decode transport (tryKernelExtractRaw → buffers ride to the store worker; files with applicable framework extract() hooks keep the decoded path), and the preParse hoist (preParsedSource — a language's offset-preserving preParse hook runs before BOTH kernel entry points, so c/cpp/metal/cuda blanking stays TS-side and both arms parse identical bytes).
Gates in-repo: scripts/kernel-parity.mjs (per-file kernel↔wasm diff, ORDER-sensitive, full-object; deferral-rate guard), scripts/dump-graph.mjs (natural-key full-DB dump for the byte-identical diff), __tests__/kernel-{scaffold,grammar-parity,tsjs-parity}.test.ts (+ torture fixtures under __tests__/fixtures/kernel-parity/) — all in npm test; the release workflow builds a 6-target prebuild matrix (continue-on-error; kernel is optional everywhere) and runs the suites with CODEGRAPH_KERNEL_EXPECT=1.

Build/run: npm run build:kernel (needs rustup; stages codegraph-kernel/prebuilds/<plat>-<arch>/codegraph-kernel.node) → npm run build → npm test. Parity sweep: node scripts/kernel-parity.mjs <dir>. Dump gate: init twice (kernel arm vs CODEGRAPH_KERNEL=0), dump-graph.mjs each, cmp.

Adding a language (the proven recipe, ~a day for a T1):

Read its languages/<lang>.ts config AND every branch of tree-sitter.ts it exercises (visitNode dispatch, extractCall's language branch, inheritance clauses, fn-ref spec in function-ref.ts, value-ref prune cases). Port bug-for-bug — quirks included (each walker's header comments list its own).
Add the crates.io grammar; vendor the wasm from the SAME tag (clone tag, sha-match parser.c against the cargo registry copy, tree-sitter-cli 0.25.10 build --wasm from CHECKED-IN parser.c, drop into src/extraction/wasm/, add to VENDORED_WASM_LANGS) — tree-sitter-wasms is 2023-era for most languages.
Torture fixture + parity sweeps (small/medium/large real repos) → full-init dump-diffs byte-identical → add to DEFAULT_ROUTED + tests + changelog.

Traps already paid for (do not relearn):

Error recovery is ENCODING-dependent (UTF-8 native vs UTF-16 web-tree-sitter, same grammar bytes + same core) → every walker defers has_error() files via the defer: signal. Incidence 0–0.42%; the harness fails >10% deferral.
Node IDs collide for same-(kind,name,line) — routine in minified one-liners. Any dedupe/self-check that the TS side keys on node IDs must compare ID STRINGS, not table rows (node_ids vec in every walker).
Positions and JS string slices are UTF-16 (textutil::col16/slice_utf16) — that's what web-tree-sitter reports and what .slice(0,100) means.
The extraction seam contract is exactly what extractFromSource returns — e.g. refs carry NO denormalized filePath/language (the store fills them). The strict full-object parity compare exists because a loose one masked precisely this.
Grammar bumps: crate + vendored wasm move TOGETHER or kernel-grammar-parity fails.
JS multiline ^ anchors after \r (and U+2028/U+2029); the regex crate's (?m)^ is \n-only — on CRLF checkouts (Windows autocrlf) the JS reference's greedy \s* eats the \n of a CRLF pair and the cleaned docstring keeps a bare \r. Caught by the O2 Windows leg (6 parity failures), fixed via js_multiline_strip in docstring.rs; CRLF variants of every torture fixture are pinned in kernel-tsjs-parity (derived in-memory — normalization-proof). Any future walker regex with (?m) needs the same scrutiny.
Perf claims: measure before believing — the plan's own §1/§6 expectations were corrected twice (many-core parse-loop wall = store-writer, §4d; cg1212 parse = C-bound, §4f).

1. Mission and the numbers that motivate it

CodeGraph's remaining fresh-index gap vs codebase-memory-mcp (cbm) is the parse+extract phase, and its floor is per-node JS↔WASM marshaling — proven, not suspected:

Measurement (2026-07-16, M3 Pro)	Result
dubbo (4,402 Java files) parse-loop, current 7-wasm-worker pipeline	4,700ms
Same files, Rust tree-sitter parse+walk, rayon (spike)	202ms
Same, single Rust thread	1,067ms
dubbo fresh init today / cbm	11.1s / 7.1s (1.55×)
Linux kernel, same 2-CPU/6GB container	we complete 27min; cbm dies at 0.16%, twice

Spike source: session scratchpad cg-kernel-spike/ (tree-sitter 0.25 + tree-sitter-java, TreeCursor walk touching kind/range/name-field, flat-row output). Reproduce before starting — it's ~80 lines and doubles as the emitter's seed.

Expected end state: parse-loop 4.7s → ~1.0–1.5s on dubbo-class repos → total ≈ 7.5s, parity with cbm on their best surface, while keeping every win we already hold (sync 2.4–2.8×, agent A/B decisive, call-graph density 1.3–2.3×, byte-identical determinism, constrained-hardware envelope).

SUPERSEDED BY MEASUREMENT (§4c/§4d): the many-core parse-loop wall turned out to be the single-writer SQLite ingest (94% of it on dubbo), not extraction — 8 wasm workers already hid extraction CPU behind the main thread on big-core machines. So the Mac dubbo total stays ~11s and closing the remaining cbm gap there is a STORE-ARCHITECTURE arc, not a kernel task. The kernel's wins are real where worker CPU binds: the 2-CPU/6GB CI envelope (excalidraw ~1.5×, dubbo ~1.25×, django 1.32×, prometheus 1.46×) and vscode-scale-on-Mac (1.28×). Every "keep" item held — byte-identical determinism is now enforced per language by the dump gate.

2. What the kernel is — and the boundary that makes it safe

One napi-rs crate (codegraph-kernel) linking tree-sitter's C library and native grammars. Input (filePath, content, language) per file; output flat typed buffers (nodes, edges, unresolved refs) — one boundary crossing per file. It replaces ONLY the parse+extract walk inside the parse workers, behind the existing ExtractionResult contract.

Never ported (works unchanged for all languages from day one): name-matcher + import-resolver, all framework resolvers (src/resolution/frameworks/), all 36 synthesis passes, MCP/explore, sync/watcher, installer. They consume the graph and raw source, not the parse tree.

Coexistence is permanent: a language routes to the kernel only after its gate passes; everything else stays on the wasm path forever if need be. No flag-day. Rollback per language = flipping the route.

Distribution: prebuilt .node per platform through the existing release-bundle pipeline (scripts/build-bundle.sh + per-platform npm packages); the same crate compiled to wasm is the universal fallback. Zero-native-build-on-install stays true.

3. Phase 0 — scaffold (do first, ~days)

codegraph-kernel/ crate: napi-rs, tree-sitter C, rayon optional (workers already parallelize per-file — start synchronous per call, one kernel call per file from the existing ParseWorkerPool workers; do NOT rebuild the pool).
Buffer contract: decide the flat encoding (suggest: one Buffer per table, fixed-width rows + a string arena; version byte first). Write the TS decoder next to parse-worker.ts.
Generic emitter driven by per-language .scm query files + a small per-language Rust config (node-kind → NodeKind mapping, name-field conventions). Escape hatch: a per-language post(buffers, source) TS hook for logic queries can't express.
Build integration: napi prebuilds wired into the release workflow next to the Node bundles; CODEGRAPH_KERNEL=0 kill switch; wasm fallback auto-selected when the .node is absent (source runs, unsupported platforms).
CI: assert native grammars and wasm grammars are built from the SAME grammar source revisions (ABI drift between paths would make per-language routing non-deterministic).

3a. Phase 0 — SHIPPED 2026-07-16 (what exists and the decisions made)

Crate: codegraph-kernel/ (napi 3, tree-sitter 0.25, no CLI dependency — scripts/build-kernel.sh does cargo build + stage into codegraph-kernel/prebuilds/<platform>-<arch>/codegraph-kernel.node; npm run build:kernel). Exports extractFile, contractInfo, grammarInfo.
Buffer contract v1: five Buffers (meta/nodes/edges/refs/arena), fixed-width LE rows, string arena with (offset,len) refs, 0xFFFFFFFF = absent, version byte first, node IDs computed Rust-side (sha256, byte-identical to generateNodeId — pinned by test), tri-state bool flags, extraJson escape slot per node row, and a RESERVED u32 metrics slot (Arc 3.2). Layout doc lives twice and must match: codegraph-kernel/src/buffers.rs ↔ src/extraction/kernel/layout.ts. NODE_KINDS/EDGE_KINDS array ORDER in src/types.ts is wire contract now (EDGE_KINDS became a runtime array for this).
Emitter: generic, .scm-driven (@def.<NodeKind> + @name + @ref.<EdgeKind> capture convention), scope stack by byte-range nesting → ::-joined qualifiedNames, contains edges, refs attached to innermost enclosing def (file node fallback) — the TreeSitterExtractor conventions. Seed TS/JS queries are SMOKE-level only; R2 replaces.
Routing: inside extractFromSource (tree-sitter.ts) — tryKernelExtract first, wasm TreeSitterExtractor as fallback (also per-FILE fallback on any kernel error). DEFAULT_ROUTED is EMPTY; dev opt-in via CODEGRAPH_KERNEL_LANGS=<langs|all>; global kill switch CODEGRAPH_KERNEL=0; loader verifies ABI + kind tables before routing (stale .node → silent wasm, CODEGRAPH_KERNEL_DEBUG=1 to see why). The escape hatch landed as post(result, source) over the DECODED result (not raw buffers) — decoded is what TS logic wants; see POST_PASSES in src/extraction/kernel/index.ts.
Grammar parity (the §3.5 CI) — and a decision that changed the wasm path: the parity test (__tests__/kernel-grammar-parity.test.ts, behavioral: ABI + node-kind + field tables compared id-by-id) caught on day one that tree-sitter-wasms ships 2023-era TS/JS grammars (^0.20.x) vs crates.io current. Resolution: vendored fresh wasm into src/extraction/wasm/ built from the exact crate revisions — tree-sitter-typescript v0.23.2 (f975a62) for typescript+tsx, tree-sitter-javascript v0.25.0 (44c892e) for javascript+jsx — from each repo's CHECKED-IN parser.c (no generate), tree-sitter-cli 0.25.10, emcc. So the production wasm TS/JS grammars are UPGRADED as of this change (full suite green, 2456 tests) and R2/R3 parity diffs are grammar-neutral. Bump crate + vendored wasm together, or the parity test fails.
Release wiring: kernel matrix job in release.yml (macos-14 ×2 targets, ubuntu-22.04, ubuntu-22.04-arm, windows-latest ×2 — all continue-on-error: kernel is optional, a toolchain flake never blocks a release) → artifacts → release/kernel/ → build-bundle.sh stages lib/kernel/codegraph-kernel.node when present. The release job runs the kernel tests with CODEGRAPH_KERNEL_EXPECT=1 (missing binary = FAILURE there, skip elsewhere).
Loader search order: CODEGRAPH_KERNEL_PATH → <pkgroot>/kernel/ (bundle) → <pkgroot>/codegraph-kernel/prebuilds/<plat>-<arch>/ (source runs).
Known R2 gate item: native columns are UTF-8 byte offsets; web-tree-sitter's are UTF-16-derived — column NUMBERS on non-ASCII lines will differ in parity dumps (text, lines, IDs unaffected). Classify or normalize when it shows up. RESOLVED in R2: the walker emits UTF-16 columns natively (util::col16), and JS string-slicing semantics (signature truncation at 100/80/120 units) are reproduced in UTF-16 units too — no column/slice diff class exists.

4a. R2 — TS/JS port SHIPPED 2026-07-16 (and a §3 design revision)

The generic .scm emitter is superseded. Real TS/JS parity needs logic queries can't express (extractCall's receiver-qualified callees, store/RTK/component recognition, fn-ref capture+gating, value-ref shadow pruning, docstring wrapper climbs) — so R2 replaced the R1 query emitter with a bespoke per-language walker (codegraph-kernel/src/tsjs/, ~1,900 lines) that mirrors TreeSitterExtractor's TS/JS paths function-for-function, bug-for-bug. emitter.rs + queries/ are deleted (git has them); expect T1 languages (java/python/go) to be walkers too. The post(result, source) TS escape hatch remains available but TS/JS needed none.
Parity evidence (macOS): scripts/kernel-parity.mjs (multiset diff of canonicalized nodes/edges/refs per file, FULL objects) — this repo 353/353 files, excalidraw 643/643 files (10,650 nodes / 10,726 edges / 68,307 refs), plus checked-in torture fixtures (__tests__/fixtures/kernel-parity/) covering components/HOCs/styled, zustand-through-middleware, RTK endpoints+hooks, vuex/pinia, fn-refs (incl this.x + shadowing gates), value-refs (incl the shadow prune), decorators, enums, type-alias members + tuple contracts, re-exports, JSX. Kept alive in npm test by __tests__/kernel-tsjs-parity.test.ts (strict full-object compare).
One decoder bug found by the strict compare: decode.ts pre-filled filePath/language on refs; wasm extractors leave them unset (the store denormalizes via ?? filePath). Fixed — the seam contract is "exactly what extractFromSource returns", not "what the store makes of it".
Perf (M3 Pro, excalidraw 643 files / 7MB): extraction single-thread 487ms kernel vs 1,255ms wasm (2.6×, identical outputs). End-to-end init on an 11-core host moves only ~3.4s → ~3.2s — parse is a small, already-pool-parallelized slice there; the win concentrates on constrained hardware (2-core CI class) and kernel-scale parse (R6). Headroom if R4's dubbo target needs it: arena interning, memoized UTF-16 line prefixes, and skipping wasm-grammar loads in workers for kernel-routed languages (worker cold-start).
Not yet done (R3 gate): large-repo parity (vscode-class), full-repo dump-diff through the DB, retrieval invariants, agent A/B, Linux docker + Windows VM parity runs, control-repo perf. Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until then. → Done same day, §4b.

4b. R3 — gate PASSED, TS/JS DEFAULT-ON (2026-07-16)

Evidence (tools: scripts/kernel-parity.mjs now ORDER-sensitive — identical multisets in a different emission order would shift rowids and change resolution — and scripts/dump-graph.mjs, natural-key full-DB dumps):

Graph parity — byte-identical, not ≤0.5%: full init dump-diff kernel-vs-wasm: express (13,712 rows), excalidraw (89,898), vscode (2,378,238 rows) — all byte-identical. Control repo (flask, Python) byte-identical + timing unchanged. Extraction-level order-sensitive sweeps: repo 352/354 (+2 deferred), express 141/141, excalidraw 643/643, vscode 12,055/12,106 (+51 deferred), 0 diffs.
The one real find — encoding-dependent error recovery: same grammar bytes (sha-verified parser.c/scanner.h), same tree-sitter core (0.25.10), but error RECOVERY on files with parse errors differs between UTF-8 (native) and UTF-16 (web-tree-sitter) parsing — proven by parsing the divergent vscode file natively in UTF-16, which reproduced the wasm tree exactly. Incidence: 0% (express) / 0.31% (excalidraw) / 0.42% (vscode) of files. Policy: the kernel defers any file whose tree has_error() to the wasm extractor (defer: signal, silent, per-file) — parity by construction on erroring files, 99.6%+ keep the fast path, and the harness fails if deferrals exceed 10% (a broken kernel can't hide).
Retrieval invariants: kernel-indexed excalidraw — mutateElement → renderStaticScene connects end-to-end via explore (callback + react-render + jsx hops shown); synthesized-edge families present (408 jsx-render / 46 react-render / 14 interface-impl / 1 callback); byte-identical DB ⇒ counts equal by construction.
Agent A/B: byte-identical DBs make the A/B vacuous (identical graph, identical MCP server) — same justification as the #1320–#1322 perf PRs, which shipped on the dump-diff gate. Not burned.
Perf: vscode init 105.4s → 82.1s (1.28×) on the 11-core Mac; excalidraw on a 2-CPU/6GB Linux container (the CI-runner envelope) 6.2–7.1s → 4.3–4.8s (~1.5×, n=2 interleaved); Mac excalidraw ≈ neutral-to-slightly-better (parse already a small pool-parallelized slice at 11 cores). Control unchanged.
Platforms: Linux (arm64 bookworm container, in-container cargo build): all 22 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. Windows VM: deferred — VM stopped and prlctl start needs Parallels Pro; benign because a missing/broken .node falls back to wasm, and the release workflow builds + gates win32 prebuilds. Run the kernel suites on the VM when it's next up.
Suite: 2,465 tests pass WITH default-on routing — the entire extraction test corpus now exercises the kernel for TS/JS on machines with a staged .node.

Default routing: DEFAULT_ROUTED = {typescript, tsx, javascript, jsx} in src/extraction/kernel/index.ts. CODEGRAPH_KERNEL_LANGS REPLACES the set; CODEGRAPH_KERNEL=0 kills. Changelog entry added under [Unreleased].

4c. R4 — Java PORTED + gate PASSED + DEFAULT-ON (2026-07-16)

Walker: codegraph-kernel/src/java.rs (self-contained, sharing the crate-level docstring/textutil modules) — package namespaces, imports, javadoc, annotations → decorates, type_list inheritance, fields/constants (static-final → constant), enum_constant members, anonymous classes (<T$anon@line> incl. the TS side's 0-based-line quirk, mirrored bug-for-bug), method_invocation calls with the this.field unwrap + the Foo.getInstance().bar() chain encoding, static-member value reads, method_reference fn-refs (this::x / Type::m), value refs, and the full Lombok member synthesizer (#912: @Getter/@Setter/@Data/@Value/@Builder/ @ToString/@EqualsAndHashCode/@Slf4j-family, taken-member dedup by exact classQN::name). Grammar: tree-sitter-java crate 0.23.5; wasm vendored from the SAME tag (94703d5, parser.c sha-matched), replacing tree-sitter-wasms' ^0.20.2 build.
Parity: extraction sweeps — gson 262/262, retrofit 341/341, dubbo 4,048/4,048, torture fixture (__tests__/fixtures/kernel-parity/Torture.java, in npm test). Full-init dump-diffs byte-identical: gson (49,766 rows), retrofit (62,735), dubbo (441,266 rows). All R2/R3 repos re-verified after the fix below.
The gate caught a REAL cross-language bug: retrofit's minified website JS exposed that fn-ref dedupe and value-ref self-checks must compare node ID strings, not table rows — IDs collide for same-(kind,name,line) nodes (routine in minified one-liners: many function e on line 3) and the TS side keys on ${fromNodeId}|${name}. Fixed in BOTH walkers (node_ids per row); this affected tsjs too (latent since R2, never released).
Benchmark honesty (the §6 expectation was wrong about WHERE the win lands): dubbo fresh-init on the 11-core M3 Pro is ~FLAT end-to-end (11.3–11.5 wasm → 11.0–11.6 kernel; parse-loop wall 5,020→4,394ms) because that phase's wall is main-thread-bound (file reads + result store + SQLite), not worker-CPU-bound — 8 wasm workers already hide extraction CPU behind the main thread on big-core machines. Where worker CPU binds, the kernel delivers: dubbo on 2-CPU/6GB Linux 27.8–28.6s → 22.3–22.8s (~1.25×); excalidraw same envelope ~1.5×; vscode-on-Mac 1.28×. The cbm-parity Mac headline therefore needs the next lever: decode the kernel's buffers DIRECTLY into store rows (skip per-node JS object materialization on the main thread) — buffer contract already carries everything; tracked as the top §7a-adjacent follow-up.
Platforms: Linux container (arm64): all 23 kernel tests green EXPECT=1; Windows VM still deferred (same fallback rationale as §4b).
Default routing now includes java.

4e. R5 — Python + Go PORTED + gates PASSED + DEFAULT-ON (2026-07-16)

Walkers: codegraph-kernel/src/python.rs + src/go.rs (the java.rs pattern). Python: decorated_definition docstring/decorator handling (decorates only for bare-identifier decorators — the call-kind quirk mirrored), fn-in-class → method, module assignments always variable (no isConst hook), from-import binding refs, self.x fn-ref candidates as BARE names, attribute callees via the namedChild(1) fallback. Go: receiver methods with Recv::name QNs + first-earlier-struct contains edges, type_spec → struct/interface classification (embedding → extends; interface method_elems → method nodes), composite-literal instantiates keeping the package qualifier, top-level var/const initializer walks attributed to the symbol (#693), 2-hop field chains (#1276), New().Method() re-encode (#645/#608), GO_SPEC fn-ref layers (literal_element/expression_list fan-out).
Grammars: crates tree-sitter-python 0.23.6 (bffb65a) + tree-sitter-go 0.23.4 (3c3775f); wasm vendored from the same tags, parser.c sha-matched (both were 2023-era in tree-sitter-wasms).
Parity: extraction sweeps 100% — flask 83/83, django 3,035/3,038 (+3 error-file deferrals), gin 99/99, prometheus 978/979 (+1). Full-init dumps byte-identical: flask (10,833 rows), gin (17,540), django (360,794), prometheus (213,758). Torture fixtures in npm test. Even Mac-side init already moves where extraction matters: prometheus 5.7→4.5s, django 9.0→8.7s. On the 2-CPU/6GB envelope (the CI-runner class): django 22.0→16.7s (1.32×), prometheus 15.0→10.3s (1.46×).
Default routing now: typescript, tsx, javascript, jsx, java, python, go.

4f. R6 — kernel-scale re-validation (2026-07-17)

Fresh init of the Linux kernel in the cg1212 container (2 CPUs / 6GB), current build (R5 kernel + direct-to-store active), CODEGRAPH_SYNTH_TIMINGS:

Completes, exit 0: 1,586s (26.4min) vs the ~27min #1212/#1323 baseline — no regression with per-file routing checks, error-file deferral, and the d2s store path live. Graph scale identical: 2,048,664 nodes / 6,405,964 edges (baseline 2.05M/6.4M).
Phase walls: scan 1.3s (70,239 files), parse-loop 371.9s (6.2m — unchanged), fts-rebuild 6.4s, edge-index-recreate 78.9s, callback-synthesis 350.3s, resolution 1,149.7s (19.2m — the wall, P1's territory), maintenance 47.2s.
The §6 parse expectation (6m → ~2m) was mis-premised: the Linux tree is 63,810 C/H files vs 422 Python — ~99% C, an UNPORTED T2 language, so the kernel can't touch its parse time. The expectation transfers to the C/C++ port (R7, with the blanking pre-passes staying TS-side per §4). The tree's own Python tooling: 99/99 files byte-parity.
Kernel-scale priority order after this run: P1 resolution (73% of the wall) > C/C++ port (23%) > everything else.

4d. Direct-to-store decode (2026-07-16) — and where the wall ACTUALLY is

Kernel-routed files now ship their flat buffers from the parse worker all the way to the STORE WORKER, which decodes + finalizes them there (tryKernelExtractRaw → ExtractionResult.kernelBuffers → KernelStoreBundle → decodeKernelBundle; filter semantics shared via finalizeStoreBundle). The main thread's per-file work drops to O(1) + the content hash — it never materializes per-node objects, and both postMessage hops move flat bytes instead of object graphs. Files whose applicable frameworks carry an extract() hook keep the decoded path (hooks merge into decoded results); non-writer paths (main-thread store, tests) materialize via materializeKernelResult. Byte-identical dumps re-verified on dubbo, excalidraw, express, gson.

Measurement that closes the §4c question: with the store worker instrumented, dubbo's parse-loop wall is 94% store-writer busy time (4,202ms of 4,493ms on the kernel arm). The many-core fresh-index wall is the single-writer SQLite ingest — not extraction, not main-thread work. d2s still improves the writer lane ~11% (4,726→4,202ms: buffers skip structured-clone deserialization ON the writer) and frees the main thread, but the remaining cbm gap on many-core medium repos is a STORE-ARCHITECTURE question (their RAM-first design defers all durability). Next levers there (a separate perf arc, not this project): deferred/bulk index builds during the parse phase, multi-file write transactions, buffer→bind without object materialization. Note the #1320-arc post-mortem already measured statement batching and sorted inserts as ~zero on this path — B-tree maintenance is the floor.

4. Per-language tracker

Tiers: T1 = mostly .scm + mapping config. T2 = needs bespoke pre/post passes kept in TS (listed). T3 = not a plain tree-sitter walk (standalone/multi-grammar extractor) — migrate last or never; wasm/TS path is a fine permanent home.

The user-facing language contract is README.md → Language Support (34 logos incl. Metal, CUDA, Terraform/OpenTofu, Pascal/Delphi). Keep this tracker in sync with it — every README language must have a row here, even the ones that only ride another language's port.

Grammar column: crates.io = mainstream native grammar crate exists; vendored = we ship a rebuilt/patched wasm (ABI-15) and the kernel must compile OUR fork natively — verify parity before porting the language.

Language(s)	Today	Tier	Grammar source	Migration notes / known traps	Status
typescript, tsx, javascript, jsx	`languages/typescript.ts`, `javascript.ts` + shared branches	T1	crates.io	First target. Value-reference edges (#895/#897) and component recognition (#841 forwardRef/memo/styled) must survive — they're extraction-side. Largest test surface; gate is strictest here. PORTED + GATE PASSED + DEFAULT-ON (§4a/§4b); erroring files defer to wasm per-file.	✅
java	`languages/java.ts`	T1	crates.io	Second target; unlocks the dubbo-parity claim. Lombok member synthesis (#912) is a NODE synthesizer hook in extraction (`synthesizeMembers`) — port or keep as TS post-pass. PORTED incl. Lombok + gate passed + DEFAULT-ON (§4c).	✅
python	`languages/python.ts`	T1	crates.io	Third. Decorator extraction feeds framework route detection — parity required. PORTED + DEFAULT-ON (§4e).	✅
go	`languages/go.ts`	T1	crates.io	Third (tie). Value-reference edges ship here too (#897). PORTED + DEFAULT-ON (§4e).	✅
ruby, php	dedicated files	T1	crates.io	Straightforward; PHP property-receiver shapes (#1220/#1251) are RESOLUTION-side, unaffected.	☐
csharp	`languages/csharp.ts`	T1	crates.io	Plain.	☐
rust, dart, scala, lua, luau, r	dedicated files	T1	crates.io (luau/r/scala: verify crate freshness vs our wasm)	Long-tail T1; port opportunistically after the big five.	☐
kotlin	`languages/kotlin.ts`	T1½	crates.io	Expect/actual pairing is synthesis-side (fine); extraction is clean but validate against a KMP repo.	☐
swift	shared + dedicated branch	T1½	crates.io	Trap: in-class property extraction lives in `tree-sitter.ts`'s DEDICATED branch, not `swift.ts` (#1020 — Alamofire went 0→348 props). Gate on Alamofire.	☐
c, cpp	`languages/c-cpp.ts`	T2	crates.io	DONE (R7a, 2026-07-17) — `ccpp/` walker; ALL pre-passes stayed TS-side via the route-point preParse hoist (+6 new blanks added during gating — see the checklist doc); content-based `.h` C-vs-C++ detection stays upstream at detectLanguage. Parity 0-diff + dump byte-identical on redis/git/fmt/protobuf/ALS.	☑
metal, cuda	dialects over the cpp grammar	T2 (rides c/cpp)	crates.io (cpp)	DONE (rides R7a) — `.metal`/`.cu`/`.cuh` map to 'cpp' and their blanks run in the hoisted preParse (filePath rides along for the extension gates); hoist-parity pinned in kernel-ccpp-parity.test.ts + the metal/cuda suites.	☑
objc	`languages/objc.ts`	T2	crates.io	Rides the c-cpp trap family; RN bridge extraction feeds `rnCrossPlatformEdges` (synthesis-side, fine).	☐
arkts	`languages/arkts.ts`	T2	vendored (harmony-contrib)	Dot-prefixed refs + decorator-gated matching fixed 36,840 wrong edges — that logic must port exactly or stay TS-side. Compile our grammar fork natively.	☐
pascal	`languages/pascal.ts`	T2	vendored	Paired with dfm-extractor (T3); `extractPascalDefProc` indexed lookups.	☐
vbnet	`languages/vbnet.ts`	T2	vendored, patched + external scanner	Our wasm is a patched grammar WITH a C external scanner — the kernel must build that scanner; ts-cli 0.24 dropped `\p{...}` classes during the original build (#1164). Highest grammar-build risk of any language.	☐
cobol	`languages/cobol.ts`	T2	vendored fork	Paragraph-extent reconstruction + copybook resolution are extraction logic (#1161, CardDemo 43/44). Port carefully or keep TS post-pass.	☐
erlang	`languages/erlang.ts`	T2	vendored (WhatsApp/ELP)	npm `tree-sitter-erlang` is HIJACKED — never source from it (#1165). gen_server dispatch is synthesis-side (fine).	☐
nix	`languages/nix.ts`	T2	vendored (ABI-15 rebuild)	Option-path synthesizer is synthesis-side; the `===`-always-false → `.equals()` lesson (#1190) is wasm-binding-specific and disappears natively — still gate on nixpkgs (44k files).	☐
solidity	`languages/solidity.ts`	T2	vendored	`modifier_invocation` outside body walk (#1170) is extraction-side; port it.	☐
terraform	`languages/terraform.ts`	T2	vendored	`:`-scoped refs for module-boundary bridging (#1173); metadata does NOT persist — re-read source (#1174).	☐
cfml, cfscript, cfquery	`cfml-extractor.ts` + 3 grammar files	T3	vendored ×3	3-grammar family with BOM-sensitive dialect sniffing (#1118/#1153–55). Leave on wasm until the very end, possibly forever.	☐
svelte, vue, astro, liquid	standalone extractors	T3	n/a (custom/embedded parsing)	Not tree-sitter walks. Permanent TS home is acceptable — file counts are small and these repos are small.	☐
dfm (Delphi forms), razor, mybatis XML	standalone extractors	T3	n/a	Same as above. mybatis pairs with a synthesis pass (fine).	☐

Do-not-regress invariants during any port (extraction-side, will show up in the gate): node metadata is re-read from source, never persisted; parse commits stay in FILE ORDER (#1015); MAX_FILE_SIZE skip; generated-file detection; CODEGRAPH_PARSE_WORKERS semantics; framework extract() hooks keep running TS-side per file after the kernel pass.

5. Equivalence gate (run per language, no exceptions)

Byte-identity vs hand-written extractors is NOT expected — the gate is behavioral parity:

Graph parity: fresh-index 3 real repos (small/medium/large for the language) on wasm-path vs kernel-path builds. Dump with the dump-graph.mjs pattern (natural keys). Node/edge/ref deltas ≤0.5% AND every diff category manually classified (the 13-edge supertype-visibility bug this week was caught exactly this way — small diffs are real).
Retrieval invariants: the language's canonical flows still connect end-to-end in codegraph_explore (playbook: docs/design/dynamic-dispatch-coverage-playbook.md); node counts stable; synthesized-edge spot-check.
Agent A/B non-regression per the standard methodology (CLAUDE.md): --model sonnet --effort high ALWAYS, ≥2 runs/arm, pre-warmed daemon, CODEGRAPH_NO_PROMPT_HOOK=1, forbid subagent delegation in the prompt.
Perf: fresh-index improves on the language's repos; a NON-migrated control repo is unchanged; suite green; Linux docker + Windows VM passes for platform-sensitive bits.

6. Rollout order and expected wins

Executed 2026-07-16/17; outcomes vs these expectations are in §4a–§4f. Two expectations below were corrected by measurement: (2) the dubbo-on-Mac headline is store-writer-bound, not extraction-bound (§4c/§4d — the win lands on the low-core envelope instead); (4) cg1212 is ~99% C, an unported T2 language, so its parse expectation belongs to the C/C++ port (§4f).

TS/JS/TSX/JSX — most indexed files in the funnel; excalidraw 3.3s → ~2.3s expected.
Java — dubbo 11.1s → ~7.5s expected (the cbm-parity headline).
Python, Go — rounds out ~90% of real-world indexed files.
Kernel-scale re-run in the cg1212 container after (2): parse 6.0m → ~1.5–2m expected.
Long tail opportunistically; T3 possibly never — that's fine by design.

Measurement discipline (hard-won this week — do NOT relearn these):

Profile first. Ideas killed by measurement this week: sorted-chunk inserts (zero), statement-batching the persist (zero — B-tree maintenance is the cost), RAM-disk/ in-memory DB build (SLOWER — fastInit already writes at page-cache speed).
CODEGRAPH_SYNTH_TIMINGS=1 now emits full phase walls ([phase-timing]) + pool/batch timings. UI distorts phase walls — pipe stdout away.
Check host load before timing (iOS simulators inflated every phase ~30%); the Monitor-on-loadavg pattern (fire <3.5) gives clean windows.
grep is aliased to ugrep and silently treats callback-synthesizer.ts as binary — use grep -a.

7. AFTER the kernel: the follow-on roadmap (in order)

7a. Kernel-scale resolution speed — NOW THE TOP OPEN PERF ITEM

Confirmed by the R6 run (§4f): resolution is 19.2min of the 26.4min Linux-kernel wall (73%) — ~~sequential BY DESIGN in the 2-CPU container (the resolver pool requires ≥4 cores to engage)~~ (premise corrected in §7a.1: it was pooled all along). Parse is 6.2min (23%) and belongs to the C/C++ port (R7a). Steps: re-run cg1212 validation on ≥4-core allocation (pool + parallel synthesis #1321/#1322 engage — this first measurement is cheap and may reshape the whole problem); profile; likely levers: worker count scaling, batch size at scale, warmCachesYielding on multi-GB DBs. Target: kernel <10min on a normal 8-core host.

7a.1 First measurement round (2026-07-17) — the arc reshaped

The cheap first measurement was run and did exactly what it was for: it invalidated the premise and surfaced two structural defects that now gate any speed work.

Premise correction — resolution was NEVER sequential in cg1212. Pool sizing (resolver-pool.ts tryCreate: min(os.cpus().length − 2, 6), engage at ≥2) uses os.cpus(), which is cpuset-blind — inside the 2-CPU container it saw the Docker VM's 8 CPUs and ran 6 workers time-slicing 2 cores (r6 log: 6× worker open, 14k pool-timing lines). A real <4-CPU host (os.cpus() < 4) gets no pool at all. This also explains why the earlier "19.5m sequential" and R6's 19.2m match: same 6-on-2 topology. os.availableParallelism() (cgroup/affinity- aware) is the honest sizing input — candidate fix rides item (2) below.
Failure 1 — container at 8 real cores / 7GB: cgroup OOM (oom_kill=5, OOMKilled=true), silent EXIT=1 (SIGKILL inside the liftoff re-exec surfaces as code 1, no output). Died mid-parallel-synthesis, 4 passes in. At 8 real cores all 6 workers hold peak anon memory simultaneously — the 2-core runs survived only because time-slicing kept concurrent peak lower. The pool sizes by cores only; there is no memory-aware term and no size knob (CODEGRAPH_NO_PARALLEL_ RESOLVE is all-or-nothing).
Failure 2 — WAL blowup at kernel scale: 22.2GB WAL on a 4.6GB DB (container, at death; the Mac-native attempt was watchdog-killed at 5GB free disk with the WAL already 2.8GB at resolution start). Mechanism: the pooled resolution/ synthesis superphase writes continuously while 6 workers hold overlapping read snapshots — checkpointing can never truncate past the oldest reader, so the WAL accretes ~the phase's entire write volume. Invisible on medium repos (writes are ~100s of MB); at kernel scale it is a ~5× disk blowup and a page-cache pressure source that feeds Failure 1. The #1231 WAL valve doesn't contain it (valve checkpoints can't truncate past pinned readers either). Fix direction: workers recycle their DB connection between batch rounds (release snapshots at a checkpoint barrier), or an equivalent writer-coordinated wal_checkpoint(RESTART) window; instrument to confirm the starvation point before building.
What did move: parse-loop 371.9s → 199.2s (1.87×) at 2→8 cores (container, clean window) — already riding the single-writer store floor (§4d), so the C/C++ port (R7a) will cut worker CPU but the parse wall won't drop below the writer lane on many-core hosts.
Parity spot-check: Mac-native post-parse node count 2,048,675 vs R6's final 2,048,664 (+11 across 2.05M; post-parse vs post-maintenance and a contended run — not a parity gate, just no red flag; the dump-diff gates remain the authority).
Host truth for the target claim: the dev Mac is 8 logical cores / 24GB — literally the P1 target class. Its constraint is transient DISK (fresh kernel- scale init currently needs ~25GB+ free for tree+DB+WAL until the WAL fix lands).

Revised P1 order: (1) WAL containment → (2) memory-aware, cgroup-honest pool sizing → (3) re-run the 8-core measurement (container at ≥12GB or the Mac with disk headroom) → then profile what remains. The <10min-on-8-cores target stands.

7a.2 P1 items (1)+(2) SHIPPED 2026-07-17 — the implementation arc (#1332–#1335)

Four PRs, each carrying its measurement; the arc took three failed/diagnostic kernel-scale runs to get right, and every failure taught a design fact:

Run (2c/6GB unless noted)	Build	Outcome
R6 baseline	pre-P1	26.4min, EXIT 0; WAL unbounded (mid-run peak unmeasured); pooled 6-on-2 (cpuset-blind)
run 1	#1332 hook	EXIT 137 (OOM) — WAL 22.2GB, 0 of 5.4M frames ever backfilled; futile 20-pass parks amplified memory churn
diagnostic	+latch/debug	EXIT 0, ~24min; pool KILLED by mis-measured 57MB cgroup budget → exposed sequential resolution 853s vs 1,150s pooled and cFnPtrEdges = 306s of synthesis's 358s
instrumented	+sizing fixes	EXIT 0, 21.6min (R6 −18%); parse floor restores 373.5s; passives complete but the FILE marched 361→721MB → named the wrap-never-happens gap; peak 17.2GB
record (first attempt)	#1335	EXIT 1: "database is locked" — the timer-path truncate won the lock race after the recreate's multi-GB burst and stalled the writer past its 5s busy_timeout → truncate is barrier-only now (#1336). Bonus data: recreate 7.9s (vs 68–95s) once the WAL stays folded
record	#1336	EXIT 0, 20.4min (R6 −23%); WAL peak 1.57GB (−14×); counts byte-exact 2,048,664/6,405,964. parse 354.9s · resolution 812.5s · synthesis 329.0s · recreate 57.5s · maintenance 43.5s
8-core retry (8c/7GB)	#1336	EXIT 0, NO OOM — 18.3min; WAL peak 1.09GB; pool sized 4 by the memory term (ap=8, budget 5.1GB, db 4.1GB); counts byte-exact. parse 208.7s · resolution 835.9s · synthesis 338.7s

The 8-core verdict (the question P1 set out to ask): 18.3min vs the <10min target — infrastructure fixed, speed target NOT met, and the gap is now precisely characterized. Resolution is CORE-INVARIANT at kernel scale: 835.9s pooled-4-on-8 ≈ 812.5s sequential-on-2 — worker parallelism buys nothing, so the bottleneck is the per-ref main-thread path (admission + persist + per-ref resolver work), not topology. Of the 18.3min, ~14min is core-invariant resolution+synthesis. Next levers, in order: (a) profile the per-ref path inside resolution (the 812–836s floor), (b) cFnPtrEdges (306s, 86% of synthesis — parallelize/window WITHIN the pass), (c) the R7a C/C++ port (parse 209s → kernel-native). 4× cores currently buys only 2min end-to-end (20.4 → 18.3) because parse is the only core-scaling phase left.

Design facts these runs established (each now enforced in code + tests):

WAL backlog and WAL file are different resources. Passive backfills bound the backlog; the FILE only stops growing when a commit finds zero reader marks — observed never in practice. Containment = backfill + TRUNCATE at a parked barrier (the one guaranteed no-reader window) + a raw file-size trigger at 4× the soft cap (#1334/#1335). Dubbo: 251MB → 69MB peak, dumps byte-identical under aggressive fold cycling.
cgroup v2 memory.current counts reclaimable page cache — post-parse it read 57MB free on a 6GB box and silently disabled the pool. inactive_file is credited back (#1335); the same box reads 4.4GB.
The pool loses to sequential at 2 real cores (853s vs 1,150s resolution; cold worker caches + serialization + time-slicing exceed the parallelism), and pooled synthesis is Amdahl-bound by cFnPtrEdges (306s of 358s) at kernel scale. Sizing: min(availableParallelism − 1, 6) + memory term + CODEGRAPH_RESOLVE_WORKERS knob (#1333/#1335); ap=2 → sequential by choice.
Parse needs ≥2 workers even on 2 cores (1 worker = +34%; main + store worker don't fill the second core). Floored (#1335): 373.5s ≈ the 369s oversubscribed baseline, at a fraction of the memory.
Silent failure modes burned three 25-minute cycles: give-ups were verbose-gated, sizing's null path logged nothing, the timer path logged nothing. All valve/sizing decisions now print under CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG — the armed line answers "is it even alive" in one glance.

New synthesis lever surfaced: cFnPtrEdges is 86% of kernel-scale synthesis wall — parallelizing WITHIN that one pass (or windowing its scan) is worth more than pooling all 36 passes. Filed under the next P1 profiling round.

7a.3 Batch-loop profile + de-quadratic round (2026-07-17, #1339)

CODEGRAPH_RESOLVE_PROFILE (shipped in #1339: per-outcome resolveOne histogram

loop-stage attribution) overturned the arc's founding assumption — resolveOne owns only ~93s of the ~433s kernel-scale batch loop. Stage attribution and what happened to each:

Stage	Before	After #1339	Note
countGuard	93.9s	0.0s	per-batch COUNT(*) was O(remaining) — replaced by summed SQLite `changes` (zero-removals IS the runaway signal; real COUNT only arbitrates the suspicious path)
read	54.6s	57.2s	keyset replaced OFFSET, but the cost is row MAPPING (5000-row materialization + candidates JSON), not prefix-walking — theory falsified, keyset kept as hygiene; lever = leaner row mapping
backpressure	111.2s	121.2s	DB-scaled caps didn't help: the fold tax is TOTAL checkpoint I/O (write set is cold pages, not re-dirtied hot ones) — a disk-I/O floor ≈ WAL bytes written
settle (resolveOne)	85.7s	88.0s	the real work; exact-match 3.17M×13µs=41s is the biggest legit class
inserts/deletes/marks	~84s	~84s	B-tree floor (#1320 post-mortem)

2c/6GB envelope: 26.4min (R6) → 20.4 (#1336) → 19.3min (#1339), counts byte-exact every run; dubbo dump byte-identical. The 8-core re-run post-#1339 is pending (est. ~17.5min from the stage arithmetic). Remaining levers by size: parse 351s→R7a C/C++ port; cFnPtrEdges 306s; backpressure 121s (I/O floor — shrinks only by writing fewer bytes); settle 88s; read-mapping 57s.

7a.4 cFnPtrEdges round (2026-07-17, #1341) — 2.07× standalone, probe-driven

Iterated with a STANDALONE in-container probe against the live kernel DB (readonly; ~4min/cycle instead of 25-min inits) with per-sweep sub-timings (CODEGRAPH_SYNTH_TIMINGS prints the cFnPtr sub: line):

Iteration	Standalone total	What moved
baseline	278.8s	attribution: E 112s, D 92s, strip 71.8s (4.4×/file), C 41s
regex hoists + D field-name pre-gate + incremental line count	249.3s	D −24s
budget-aware strip cache (first cut)	—	thrash lesson: a partial LRU on cyclic sweeps ≈ 0% cross-sweep hits — cap ~61k AND cap == files.length both lost (includes push the working set over)
all-or-nothing cache + 5% slack	187.8s	strips exactly 1.0/file; `getNodesInFile` theory killed (10s, not 127s)
`sliceLines` → split-once-per-file	134.8s	~1.6M full-file splits eliminated (D 46→20.5s, E 94.5→69.1s)

Identity proof at full scale: optimized edge set (merge-dedup + canonical sort, 274,762 edges) SHA256 21c2a971… == the pre-optimization edges extracted from the live kernel DB — this pass never runs on the dubbo gate repo (C-gated), so the DB comparison is the right gate. Suite 2,491.

In-run validation (2c/6GB): total 17.6min (from 19.3; −33% cumulative vs R6), synthesis 336→251s, counts byte-exact, WAL 1.09GB. Honest caveat: the full strip cache did NOT engage in-run at 6GB (mid-run memory budget below the 2×-cache safety threshold → deliberate fallback to the 128 floor; strips 283k, costing ~60s vs the probe) — the memory-safe degradation working as designed. Boxes with headroom get the full 2.07×; the 6GB envelope gets the algorithmic wins only.

Levers remaining, re-ranked: parse 338s (R7a C/C++ port — the last big rock) > backpressure ~120s (checkpoint I/O floor) > E-scan 69–93s (approaching honest regex work over 1.5GB) > settle 88s > read-mapping 57s.

7a.5 8-core re-run, post-R7a (2026-07-17) — 16.4min; the 8c gap is now all resolution

Same provisioning as the §7a.2 retry (cg1212 at cpuset 0-7 / 7GB), the deployed R7a build, fresh init of the v7.2-rc2 tree: EXIT 0, envelope 981s = 16.4min (pre-R7a 8c record: 18.3min — and that was the smaller pre-blank graph). Counts 2,048,295 / 6,406,933 == both 2c arms; WAL peak 1.34GB (same contained regime as 1.09–1.57GB records). Phases: parse-loop 202.6s (pre-R7a all-wasm 8c: 208.7s — both sit ON the single-writer store floor, so 8c parse is writer-bound, not extraction-bound) · resolution superphase 715.0s (was 835.9s) containing callback-synthesis 257.4s (was 338.7s) and edge-index-recreate 52.0s · maintenance 47.6s. The −1.9min vs the record is the post-#1336 rounds (#1339 countGuard, #1341 cFnPtr, R7a native parse + defer-reuse) landing at 8c for the first time.

Consequence for the <10min target: ~12 of the 16.4 minutes are the core-invariant resolution superphase. Deferral cuts can't materially move the 8c envelope (parse is already at the writer lane); they remain queued for graph richness + the 2c/low-core envelope. The 8c target now lives or dies on the per-ref resolution path (§7a.2's lever (a)).

7a.6 Per-ref path measurement round (2026-07-18) — fresh tables, two falsifications, two live levers

Fresh CODEGRAPH_RESOLVE_PROFILE tables on the round-2 build (v7.2-rc2 tree, cg1212), then two cache experiments run against them — both killed by measurement, code reverted same-day; this section is what survives.

stage (batch loop)	2c sequential (clean host)	8c pool-4
read	37.0s	33.9s
settle (resolveOne)	79.7s	3.6s
backpressure	138.3s	121.9s
createEdges	3.4s	7.6s
insertEdges	33.8s	55.3s
deletes	37.7s	118.8s
marks	5.3s	6.5s
loop total	339s	357s

2c: superphase 645s (loop + synth 251.6s [cFnPtr ~230: E 95.0 + strip 78.5 at n=283k, budget-declined again + C/D 88.8] + recreate 54.5); envelope 16.5min — the new 2c record (the 17.1 r2-gate figure carried host contamination). Settle decomposition: exact-match 35.1s @ 11µs × 3.17M, import 16.9s, fail:calls 9.2s × 1.63M. 8c: superphase 655.5s (+ synth 242.6

recreate 56.1), envelope 14.95min — n=2 range 15.0–16.4min with the morning's run; report ranges, never single runs on this box. 8c parse 178.5s: round 2's deferral cuts DID move the 8c parse wall (202.6 → 178.5) — §7a.5's "writer-floor won't move" prediction was partly wrong.
The pool double-buffer WORKS. settle 3.6s at 8c — the workers absorb the entire 3.17M exact-match population (12–17s per worker, parallel). §7a.2's "resolution is core-invariant" framing is superseded: the 8c cost was never resolveOne.
THE 8c anomaly — writes-under-readers: deletes 37.7 → 118.8s (+81) and insertEdges 33.8 → 55.3s (+22) with 4 readonly workers attached. Main-thread B-tree writes run ~3× slower under the pool. Mechanism UNPROVEN — candidates: page-cache competition (4 × 32MB worker caches + reads), WAL read-through depth while readers hold positions, wal-index lock contention. Next probe: instrument (per-op delete timing vs worker activity windows), then either shorten reader hold-times (worker connection recycling at the barrier — §7a.1's original fix direction, never built) or cut delete volume. Potential ≈ −100s at 8c.
Killed by measurement #1 — nameCache scaling (the 5k-thrash theory). v1: budget-scaled classic LRU (~478k entries) → settle 102.7s, exact-match 52.7s @ 17µs — WORSE; delete+set-per-get churn on a huge Map plus resident-array GC ate more than the SQLite statements saved. v2: mutation-free second-chance cache at 250k → exact-match 37.9s @ 12µs ≈ the 35.1s baseline. Verdict: the 11µs is NOT refetch overhead — the 5k cache already holds the true Zipf head, the tail doesn't repeat enough to cache at any size, and the floor is the per-ref JS around one indexed lookup. Both variants byte-correct (counts 2,049,153/6,413,518; git dumps byte-identical) — correctness was never the issue. Code reverted; the second-chance design lives in this entry if a big-RAM-validated attempt ever wants it.
Killed by measurement #2 — lazy candidates JSON parse: read stage 37.0 → 38–40s across variants (flat). The eager parse was never the read cost; row materialization + the statement walk is. Reverted.
Levers, re-ranked: writes-under-readers probe (+102s at 8c — the single biggest attributed delta) > cFnPtr NATIVE SITE EXTRACTION (synthesis ~230s: emit fn-ptr assignment sites from the C walker at parse time for the now-66% kernel-routed population — E-scan 95s + reads + much of strip 78s die; needs bug-for-bug regex-semantics parity in Rust and the raw-vs-preParsed scan-text question settled first) > backpressure byte volume (~122–138s I/O floor; value-neutral schema interning is migration-wide — parked) > recreate 54–70s.
Box note: cg1212's 6–7GB deliberately degrades the cFnPtr strip cache (~80s paid in-container that a 24GB target-class box gets back free) — container numbers UNDERSTATE the true 8-core-class target.

7a.7 Writes-under-readers probe + fix (2026-07-18) — WAL depth named, worker connection recycling shipped

Five discriminating runs (all 8c pool-4 unless noted, same tree/build family; each ~16min), then the fix in two cadence iterations:

run	deletes	insertEdges	read	backpressure	settle	superphase
pool-4 baseline	118.8	55.3	33.9	121.9	3.6	715.0
pool-OFF	42.6	38.3	32.6	120.1	108.5 (main)	685.9
workers=2 (dose)	58.9	54.0	23.9	174.7	10.3	670.0
v2 caches @8c	108.0	50.3	31.4	168.0	3.9	687.3
valve 64MB	56.5	27.8	16.8	297.2	3.8	739.2
recycle c25	99.8	42.9	32.7	148.8	3.7	663.3
recycle c8 (SHIPPED)	104.3	47.8	32.5	127.6	4.0	633.6

Mechanism proven: WAL read-through depth under reader pins. Pool-off restores writes on identical hardware (deletes 118.8 → 42.6); the dose scales with reader count (knee between 2 and 4); the aggressive valve — which forces a shallow WAL — recovers deletes/inserts/read to their floors (56.5/27.8/16.8) but overpays +129s in full-park folds. Reconciliation: checkpoints run in every topology, but READERS PIN their progress — the WAL runs deep exactly when workers are attached, and deep-WAL page operations tax the writer everywhere (including the unattributed between-stage spans, recreate, and synthesis reads).
v2-at-8c falsified the cache resurrection (deletes 108.0 ≈ baseline): name-lookup traffic is long-tail-dominated — uncacheable at any capacity — so reader traffic can't be reduced by caching. The caching family is triple-dead (2c settle, 8c writes).
The fix: worker connection recycling at the pool-idle boundary (ResolverPool.recycleWorkers + QueryBuilder.rebind + a cadence call at the double-buffer's worker-idle point). Workers close/reopen their read-only connections every 8 batches (~40k refs) — reopens are sub-millisecond, resolver caches survive (only prepared statements re-prepare), and the existing checkpoints advance instead of parking. Cadence 25 → 8 iterated by measurement; 8 wins via diffuse gains (backpressure −21, recreate 59.7 → 45.3). Attributed deletes stay ~100 (the WAL still re-deepens between recycles — the valve's 56.5 floor needs continuous shallowness), but the SUPERPHASE captures the true win: 715.0 → 633.6s (−11.4%); envelope best-of 14.8min at 8c (890s; band across the day's runs 14.8–16.4). Byte-neutral: git dumps byte-identical old-vs-new, linux dump sha 6dd1185b… reproduced, counts 2,049,153/6,413,518 every run, suite 2517 green. 2c unaffected by construction (no pool → no recycling).
Levers after this round: cFnPtr native site extraction (~230s synthesis, §7a.6 ranking stands) > continuous-shallow WAL (close the remaining ~45s gap between recycling's ~100s deletes and the valve's 56.5 floor without full-park folds — e.g. passive-checkpoint nudges at the recycle boundary) > backpressure byte volume > recreate.

7a.8 cFnPtr calibration round (2026-07-18) — the 230s decomposed; fuse-then-link is step 1

Three quick measurements before any port, two of them killing assumptions:

JS strip rewrite: killed by measurement. stripCStyle's split('') looked like allocator pathology; a segment-builder rewrite (byte-identical, pinned by __tests__/strip-cstyle-differential.test.ts — kept as the oracle for any future rewrite) measured 1.0× on 15.1M chars of linux C. V8's scan rate is the honest cost: ~73MB/s, and 283k strips ≈ 4 strips/file × ~20KB × that rate ≈ the observed 78s. The strip lever is the 4× redundancy (all-or-nothing cache declined at 6–7GB → every sweep re-strips), not the scanner.
E-stage regexes alone: ~46MB/s → ~30s of E's 95s. DISPATCH_RE + ARRAY_DISPATCH_RE over stripped kernel/ text yield 1,112 matches / 15.1M chars. The other ~65s is per-match logic, body slicing, lineAt, and getNodesInFile. A native regex scan alone caps at −30s.
Calibrated attack for the ~230s, re-ordered:
1. Fuse-then-link refactor (TS, step 1): one per-file extraction pass computes strip ONCE and collects {function macros, object macros, defined sets, struct fields, raw registration matches, raw dispatch matches, per-function declared-receiver types}; a text-free global linking pass then builds registries and edges. Kills the 4× strip (−~58s) + repeated reads (−~8s) + part of E's slicing overhead. Parity discipline: collectors insert in the same file order the global passes iterate today (Map insertion order = current registry order), FANOUT_CAP and match-evaluation order preserved per function; gate = edge-set hash vs the live kernel DB (§7a.4 probe) + linux dump sha. The chain/receiver resolution must be pre-collected as per-file declared-type tables so linking never touches text.
2. Native per-file extractor (step 2): the same boundary then accepts a Rust implementation of the per-file pass (raw text in, collected records out — no preParse interaction; the synthesizer reads raw disk text). Bug-for-bug regex semantics required; worth it only for the remaining ~100s of per-file scan+logic after step 1 lands.
Note for step 2 sizing: strip at native memchr rates (~500MB/s+) would be ~6-10s for the full corpus even before redundancy cuts — but marshal (UTF-16↔UTF-8 across napi) eats seconds at GB scale; batch the calls.

7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation)

Priority order, each gated by the standard A/B + node-explosion probes:

Test→subject edges (first-class tests edges at index time; we compute covering tests at query time today; cbm materializes 14.8k on dubbo). Feeds test-gap detection (Lite headline) + Pro risk signals. Cheapest, do first.
Per-node code metrics (complexity, cognitive, is_test, is_entry_point, param counts) — computed during extraction (the kernel makes this nearly free — design the buffer contract with a metrics slot!). Feeds Pro risk-ranking verdicts + explore ranking de-noise.
Read/write distinction on references (USAGE vs WRITES). The measured agent frontier ("who mutates this state" — the canvasNonce class). HIGHEST value, HIGHEST risk: scope to exported/state-relevant symbols; the tracking-every-local explosion is the known failure mode (#999/#1212 class). Full validation methodology.
Exception-flow edges (raises) — throw→handler; moderate.
Doc Section nodes (markdown headings as nodes, linked to code) — maps onto Pro's synced-business-docs story.
IaC nodes (k8s/docker/kustomize as graph nodes with cross-references). NOT worth chasing (verified in their cache schema): per-variable node inflation (85% of their node count), DB size parity (theirs is ~60% allocation slack), similarity vectors in the core engine.

7c. Deferred/parked (needs explicit approval before starting)

Single-file SEA binary (distribution polish; zero speed).
Team-shared graph artifact (cbm's graph.db.zst idea — good, but design it for the Pro shared-worker story, not as an OSS clone).
Full native rewrite: rejected with data — the moat (2,444 tests, byte-identical determinism, this week's two caught-by-gate bugs) lives in the TS reference.

8. Context for the executing agent

House rules live in CLAUDE.md (repo root) — the retrieval invariants, A/B model policy, release rules (never npm publish/push tags), changelog format.
This week's PR trail tells the story and the style: #1305, #1320 (checkpoint deferral + double-buffered persist; THE invariant: batch k+1 READS batch k's edges — supertype walks — so edges insert before fan-out), #1321 (parallel synthesis via pool reuse, registry order = merge order), #1322 (bulk edge load, identity index stays), #1323 (kernel-scale hardening: skip-don't-retry-on-main >1.5M nodes, yielding index recreate).
Every perf PR shipped byte-identical with the dump-diff gate; keep that bar.
Competitive context (validated 2026-07-16): cbm wins medium-repo fresh index 1.55–1.8× (their RAM-first design); we win sync 2.4–2.8×, agent A/B (their 14 tools drew ZERO calls in 8/8 runs), call-graph density 1.3–2.3×, and the constrained-hardware envelope (Linux kernel on 2-CPU/6GB: we complete in 27min, they die at 0.16% — their speed IS their memory floor). The kernel project closes their last number without giving up any of ours.

rust-kernel-migration-plan.md 69 KB Пермалинк Историја Датотека