fix(indexing): HDD-class storage — false parse timeouts, dropped files, and WAL checkpoint write-back (#1231) (#1242)

Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.

Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.

Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

.dockerignore

@@ -0,0 +1,7 @@
+node_modules
+dist
+.git
+.codegraph
+.kommandr
+docs
+assets

CHANGELOG.md

@@ -9,6 +9,15 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+### New Features
+
+- Indexing is dramatically faster on slow storage — mechanical HDDs, network folders, and virtualized disks. The database no longer folds its write journal back into the main file thousands of times during a bulk index (that folding was ~95% of all disk activity); it now streams writes sequentially and folds them back in a few large, coalesced passes that run off the main thread. In a disk-throttled benchmark matching the reported hardware, a mid-size Java project went from over 25 minutes to under a minute, and there is no change on fast disks. Opt out with `CODEGRAPH_NO_WAL_DEFER=1`; tune the fold-back threshold with `CODEGRAPH_WAL_VALVE_MB`. (#1231)
+- New `CODEGRAPH_PARSE_TIMEOUT_MS` environment variable to raise the per-file parse budget on unusually slow storage, the same way `CODEGRAPH_PARSE_WORKERS` already tunes the worker count. (#1231)
+
+### Fixes
+
+- Indexing on slow storage (mechanical HDDs, network folders) no longer collapses into false "parse timeout" failures. When disk writes stalled the coordinating thread, parses that had already finished — including empty files — were being misjudged as hung, their workers killed, and the files silently dropped from the index. A parse result is now judged by the worker's own clock, so a stalled coordinator accepts the finished result instead of killing the worker; only a genuinely hung parse is terminated (after a wider grace window). Files that do hit the timeout are retried at the end of indexing instead of being silently lost. Thanks @KnifeOfLife for the exceptional report. (#1231)
+- Parse workers now receive their grammar files from memory instead of each re-reading them from disk on spawn, eliminating a feedback loop on slow disks where every worker restart added more disk contention — and making worker restarts cheaper everywhere. (#1231)
 
 ## [1.3.1] - 2026-07-09
 

__tests__/grammar-wasm-bytes.test.ts

@@ -0,0 +1,43 @@
+/**
+ * readGrammarWasmBytes + bytes-based grammar loading (#1231, Phase 2.1).
+ *
+ * The orchestrator pre-reads each needed grammar's WASM once on the main
+ * thread and hands the bytes to every parse worker, so a worker respawn loads
+ * grammars from memory instead of re-reading them from a (possibly slow) disk.
+ * These tests pin that the byte reader resolves the same artifacts the loader
+ * would, and that web-tree-sitter genuinely accepts the bytes.
+ */
+import { describe, it, expect } from 'vitest';
+import { Parser, Language as WasmLanguage } from 'web-tree-sitter';
+import { readGrammarWasmBytes } from '../src/extraction/grammars';
+
+describe('readGrammarWasmBytes', () => {
+  it('reads bytes for a tree-sitter-wasms grammar and a vendored grammar', async () => {
+    const bytes = await readGrammarWasmBytes(['typescript', 'lua']);
+    expect(bytes.typescript).toBeInstanceOf(Uint8Array); // from tree-sitter-wasms
+    expect(bytes.typescript.byteLength).toBeGreaterThan(10_000);
+    expect(bytes.lua).toBeInstanceOf(Uint8Array); // vendored under src/extraction/wasm/
+    expect(bytes.lua.byteLength).toBeGreaterThan(10_000);
+  });
+
+  it('expands delegating languages to the grammars they need (svelte → ts/js)', async () => {
+    const bytes = await readGrammarWasmBytes(['svelte']);
+    expect(Object.keys(bytes).sort()).toEqual(['javascript', 'typescript']);
+  });
+
+  it('omits languages without a WASM grammar instead of failing', async () => {
+    const bytes = await readGrammarWasmBytes(['yaml', 'unknown']);
+    expect(Object.keys(bytes)).toEqual([]);
+  });
+
+  it('produces bytes web-tree-sitter can load into a working parser', async () => {
+    await Parser.init();
+    const bytes = await readGrammarWasmBytes(['javascript']);
+    const language = await WasmLanguage.load(bytes.javascript);
+    const parser = new Parser();
+    parser.setLanguage(language);
+    const tree = parser.parse('function hello() { return 1; }');
+    expect(tree!.rootNode.hasError).toBe(false);
+    expect(tree!.rootNode.toString()).toContain('function_declaration');
+  });
+});

__tests__/parse-pool.test.ts

@@ -11,7 +11,7 @@
  * parallelism safe.
  */
 import { describe, it, expect } from 'vitest';
-import { ParseWorkerPool, resolveParsePoolSize, type ParsePoolWorker, type ParseTask } from '../src/extraction/parse-pool';
+import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs, type ParsePoolWorker, type ParseTask } from '../src/extraction/parse-pool';
 import type { Language, ExtractionResult } from '../src/types';
 
 const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
@@ -80,6 +80,20 @@ function makePool(
   return { pool, counts: () => ({ spawned, terminated }) };
 }
 
+describe('resolveParseTimeoutMs', () => {
+  it('honors a positive numeric override (CODEGRAPH_PARSE_TIMEOUT_MS)', () => {
+    expect(resolveParseTimeoutMs('30000')).toBe(30000);
+    expect(resolveParseTimeoutMs('1500.9')).toBe(1500);
+  });
+  it('falls back to the 10s default when unset/blank/non-numeric/non-positive', () => {
+    expect(resolveParseTimeoutMs(undefined)).toBe(10_000);
+    expect(resolveParseTimeoutMs('')).toBe(10_000);
+    expect(resolveParseTimeoutMs('abc')).toBe(10_000);
+    expect(resolveParseTimeoutMs('0')).toBe(10_000);
+    expect(resolveParseTimeoutMs('-5')).toBe(10_000);
+  });
+});
+
 describe('resolveParsePoolSize', () => {
   it('treats explicit 0 and 1 as a single worker (the rollback path)', () => {
     expect(resolveParsePoolSize('0', 8)).toBe(1);
@@ -150,14 +164,61 @@ describe('ParseWorkerPool', () => {
     await pool.destroy();
   });
 
-  it('times out a hung parse and stays usable', async () => {
+  it('times out a hung parse (at the hard-kill backstop) and stays usable', async () => {
     const { pool } = makePool(1, (m) => (m.filePath === 'hang.ts' ? { hang: true } : { result: result(9) }), { parseTimeoutMs: 30 });
-    await expect(pool.requestParse(task('hang.ts'))).rejects.toThrow(/timed out/i);
+    const t0 = Date.now();
+    // The base timer (30ms) only marks the job late; the kill happens at the
+    // 3× backstop (90ms), and the message carries the full window.
+    await expect(pool.requestParse(task('hang.ts'))).rejects.toThrow(/timed out after 90ms/i);
+    expect(Date.now() - t0).toBeGreaterThanOrEqual(80);
     const ok = await pool.requestParse(task('ok.ts'));
     expect(ok.durationMs).toBe(9);
     await pool.destroy();
   });
 
+  it('accepts a result that arrives after the base timeout instead of killing the worker (#1231 false-timeout fix)', async () => {
+    // Simulates the HDD stall: the parse "finished" but its result is only
+    // delivered after the base timer fired. Old behaviour killed the worker and
+    // rejected; now the late result is accepted and the worker keeps serving.
+    const { pool, counts } = makePool(
+      1,
+      (m) => (m.filePath === 'late.ts' ? { wait: sleep(80).then(() => result(11)) } : { result: result(9) }),
+      { parseTimeoutMs: 50 }
+    );
+    const res = await pool.requestParse(task('late.ts')); // base timer 50ms < delivery 80ms < backstop 150ms
+    expect(res.durationMs).toBe(11);
+    expect(counts().terminated).toBe(0); // no kill…
+    expect(counts().spawned).toBe(1);    // …and no respawn churn
+    const ok = await pool.requestParse(task('next.ts'));
+    expect(ok.durationMs).toBe(9); // same worker still serving
+    await pool.destroy();
+  });
+
+  it('forwards pre-read grammar WASM bytes to every spawned worker (#1231 respawn I/O fix)', async () => {
+    const grammarBuffers = { typescript: new Uint8Array([1, 2, 3]) };
+    const loadMsgs: Array<{ grammarBuffers?: Record<string, Uint8Array> }> = [];
+    let worker!: FakeWorker;
+    const pool = new ParseWorkerPool({
+      languages: ['typescript'] as Language[],
+      size: 1,
+      grammarBuffers,
+      createWorker: () => {
+        worker = new FakeWorker(() => ({ result: result() }));
+        const orig = worker.postMessage.bind(worker);
+        worker.postMessage = (msg: unknown) => {
+          const m = msg as { type: string; grammarBuffers?: Record<string, Uint8Array> };
+          if (m.type === 'load-grammars') loadMsgs.push(m);
+          orig(msg);
+        };
+        return worker;
+      },
+    });
+    await pool.requestParse(task('a.ts'));
+    expect(loadMsgs).toHaveLength(1);
+    expect(loadMsgs[0].grammarBuffers).toBe(grammarBuffers);
+    await pool.destroy();
+  });
+
   it('serves a queue larger than the pool size', async () => {
     const { pool } = makePool(2, (m) => ({ result: result(Number(m.filePath.replace(/\D/g, ''))) }));
     const ps = Array.from({ length: 10 }, (_, i) => pool.requestParse(task(`${i}.ts`)));

__tests__/wal-deferral.test.ts

@@ -0,0 +1,217 @@
+/**
+ * WAL checkpoint deferral during bulk indexing (#1231).
+ *
+ * The default 1000-page wal_autocheckpoint re-writes hot pages into the main
+ * DB over and over during a bulk index (~95% of all disk I/O on slow
+ * storage). indexAll defers auto-checkpointing for the whole run, a
+ * WalCheckpointValve bounds WAL growth via off-thread PASSIVE checkpoints,
+ * and the interval is restored afterwards. These tests pin the DB helpers,
+ * the valve's trigger/dedupe/backpressure logic, and the end-to-end indexAll
+ * behavior (identical graph with and without deferral; interval restored).
+ */
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import * as fs from 'fs';
+import * as os from 'os';
+import * as path from 'path';
+import { DatabaseConnection } from '../src/db';
+import { WalCheckpointValve, resolveWalValveMb } from '../src/db/wal-valve';
+import CodeGraph from '../src/index';
+
+let tmpDir: string;
+
+beforeEach(() => {
+  tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-wal-deferral-'));
+});
+
+afterEach(() => {
+  fs.rmSync(tmpDir, { recursive: true, force: true });
+});
+
+function openDb(): DatabaseConnection {
+  return DatabaseConnection.initialize(path.join(tmpDir, 'test.db'));
+}
+
+/** Grow the WAL: with autocheckpoint off, every commit appends and nothing folds back. */
+function writeRows(db: DatabaseConnection, rows: number): void {
+  const raw = db.getDb();
+  raw.exec('CREATE TABLE IF NOT EXISTS t (id INTEGER PRIMARY KEY, blob TEXT)');
+  const stmt = raw.prepare('INSERT INTO t (blob) VALUES (?)');
+  for (let i = 0; i < rows; i++) stmt.run('x'.repeat(4096));
+}
+
+describe('resolveWalValveMb', () => {
+  it('honors a positive numeric override and falls back otherwise', () => {
+    expect(resolveWalValveMb('64')).toBe(64);
+    expect(resolveWalValveMb('64.9')).toBe(64);
+    expect(resolveWalValveMb(undefined)).toBe(256);
+    expect(resolveWalValveMb('')).toBe(256);
+    expect(resolveWalValveMb('abc')).toBe(256);
+    expect(resolveWalValveMb('0')).toBe(256);
+    expect(resolveWalValveMb('-5')).toBe(256);
+  });
+});
+
+describe('DatabaseConnection WAL helpers', () => {
+  it('reads and writes the wal_autocheckpoint interval', () => {
+    const db = openDb();
+    expect(db.getWalAutocheckpoint()).toBe(1000); // SQLite default
+    db.setWalAutocheckpoint(0);
+    expect(db.getWalAutocheckpoint()).toBe(0);
+    db.setWalAutocheckpoint(1000);
+    expect(db.getWalAutocheckpoint()).toBe(1000);
+    db.close();
+  });
+
+  it('reports WAL size that grows with deferred commits', () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    const before = db.getWalSizeBytes();
+    writeRows(db, 200);
+    expect(db.getWalSizeBytes()).toBeGreaterThan(before);
+    db.close();
+  });
+
+  it('checkpointWalPassive backfills the WAL from a worker connection and reports the result', async () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    writeRows(db, 500);
+    const dbFile = path.join(tmpDir, 'test.db');
+    const mainSizeBefore = fs.statSync(dbFile).size;
+    const res = await db.checkpointWalPassive();
+    // Backfill moves the committed pages into the main DB file…
+    expect(fs.statSync(dbFile).size).toBeGreaterThan(mainSizeBefore);
+    // …and reports a full backfill (idle DB: every WAL frame checkpointed).
+    expect(res).not.toBeNull();
+    expect(res!.busy).toBe(0);
+    expect(res!.log).toBeGreaterThan(0);
+    expect(res!.checkpointed).toBe(res!.log);
+    db.close();
+  });
+});
+
+describe('WalCheckpointValve', () => {
+  it('check() fires an off-thread checkpoint once growth passes the soft threshold', async () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    writeRows(db, 500); // WAL well past a ~10-byte threshold
+    const valve = new WalCheckpointValve(db, 0.00001); // ~10 bytes soft
+    const dbFile = path.join(tmpDir, 'test.db');
+    const mainSizeBefore = fs.statSync(dbFile).size;
+    valve.check();
+    await valve.drain();
+    expect(fs.statSync(dbFile).size).toBeGreaterThan(mainSizeBefore);
+    db.close();
+  });
+
+  it('advances its baseline on a full backfill — a wrapped WAL does not retrigger it', async () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    writeRows(db, 500);
+    const valve = new WalCheckpointValve(db, 0.00001);
+    valve.check();
+    await valve.drain(); // full backfill on an idle DB → baseline = current file size
+    // The WAL file keeps its high-water size, but growth is now 0: neither
+    // the timer path nor backpressure may fire again (the pre-fix bug fired
+    // on raw size forever and serialized every store behind a checkpoint).
+    expect(valve.backpressure()).toBeNull();
+    valve.check();
+    await valve.drain(); // no-op drain: nothing in flight
+    // New commits recycle wrapped frames — file size is flat, still no trigger.
+    writeRows(db, 5);
+    expect(valve.backpressure()).toBeNull();
+    db.close();
+  });
+
+  it('does not fire below the soft threshold', async () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    writeRows(db, 5);
+    const valve = new WalCheckpointValve(db, 1024); // 1GB soft — never reached
+    const dbFile = path.join(tmpDir, 'test.db');
+    const mainSizeBefore = fs.statSync(dbFile).size;
+    valve.check();
+    await valve.drain();
+    expect(fs.statSync(dbFile).size).toBe(mainSizeBefore);
+    db.close();
+  });
+
+  it('backpressure() is null under the hard cap and a promise above it', async () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    writeRows(db, 500);
+    const relaxed = new WalCheckpointValve(db, 1024);
+    expect(relaxed.backpressure()).toBeNull();
+    const strict = new WalCheckpointValve(db, 0.0000001); // hard cap ~0.4 bytes
+    const bp = strict.backpressure();
+    expect(bp).toBeInstanceOf(Promise);
+    await bp;
+    await strict.drain();
+    db.close();
+  });
+
+  it('foldNow() backfills everything at a phase boundary and resets growth', async () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    writeRows(db, 500);
+    const valve = new WalCheckpointValve(db, 1024); // thresholds never reached on their own
+    const dbFile = path.join(tmpDir, 'test.db');
+    const mainSizeBefore = fs.statSync(dbFile).size;
+    await valve.foldNow();
+    expect(fs.statSync(dbFile).size).toBeGreaterThan(mainSizeBefore); // pages backfilled
+    expect(valve.backpressure()).toBeNull(); // baseline advanced — growth is zero
+    await valve.foldNow(); // second fold is a no-op (growth 0), must not spin
+    db.close();
+  });
+
+  it('dedupes concurrent fires into one in-flight checkpoint', () => {
+    const db = openDb();
+    db.setWalAutocheckpoint(0);
+    writeRows(db, 500);
+    const valve = new WalCheckpointValve(db, 0.00001);
+    valve.check();
+    const first = valve.backpressure();
+    const second = valve.backpressure();
+    expect(second).toBe(first); // same in-flight promise, not a second worker
+    db.close();
+    return first ?? undefined;
+  });
+});
+
+describe('indexAll WAL deferral end-to-end', () => {
+  function writeFixtureProject(): void {
+    fs.mkdirSync(path.join(tmpDir, 'src'), { recursive: true });
+    for (let i = 0; i < 8; i++) {
+      fs.writeFileSync(
+        path.join(tmpDir, 'src', `mod${i}.ts`),
+        `export function fn${i}(x: number): number { return helper${i}(x) + ${i}; }\n` +
+        `function helper${i}(x: number): number { return x * ${i}; }\n`
+      );
+    }
+  }
+
+  it('produces the same graph with and without deferral, and restores the interval', async () => {
+    writeFixtureProject();
+
+    const cg1 = CodeGraph.initSync(tmpDir);
+    const r1 = await cg1.indexAll();
+    expect(r1.success).toBe(true);
+    // Deferral is scoped to the run: the connection is back on the default.
+    const conn1 = (cg1 as unknown as { db: DatabaseConnection }).db;
+    expect(conn1.getWalAutocheckpoint()).toBe(1000);
+    const counts1 = { nodes: r1.nodesCreated, edges: r1.edgesCreated };
+    await cg1.close();
+
+    fs.rmSync(path.join(tmpDir, '.codegraph'), { recursive: true, force: true });
+
+    process.env.CODEGRAPH_NO_WAL_DEFER = '1';
+    try {
+      const cg2 = CodeGraph.initSync(tmpDir);
+      const r2 = await cg2.indexAll();
+      expect(r2.success).toBe(true);
+      expect({ nodes: r2.nodesCreated, edges: r2.edgesCreated }).toEqual(counts1);
+      await cg2.close();
+    } finally {
+      delete process.env.CODEGRAPH_NO_WAL_DEFER;
+    }
+  });
+});

src/db/index.ts

@@ -189,6 +189,98 @@ export class DatabaseConnection {
     return stats.size;
   }
 
+  /**
+   * Size of the `-wal` sidecar file in bytes. 0 when it doesn't exist (non-WAL
+   * journal mode, in-memory DB, or no write since the last checkpoint+reset).
+   */
+  getWalSizeBytes(): number {
+    if (!this.dbPath || this.dbPath === ':memory:') return 0;
+    try {
+      return fs.statSync(`${this.dbPath}-wal`).size;
+    } catch {
+      return 0;
+    }
+  }
+
+  /** Current `wal_autocheckpoint` interval in pages (0 = disabled). */
+  getWalAutocheckpoint(): number {
+    const v = this.db.pragma('wal_autocheckpoint', { simple: true });
+    const n = Number(v);
+    return Number.isFinite(n) ? n : 0;
+  }
+
+  /**
+   * Set the connection's `wal_autocheckpoint` interval (pages; 0 disables).
+   * Bulk indexing defers checkpoints entirely (#1231): the default 1000-page
+   * auto-checkpoint re-writes hot B-tree/FTS pages into the main DB file over
+   * and over — measured at ~95% of ALL disk I/O during a bulk index, and the
+   * difference between 45s and 19+ minutes on HDD-class storage. During
+   * deferral a {@link WalCheckpointValve} bounds WAL growth off-thread.
+   */
+  setWalAutocheckpoint(pages: number): void {
+    this.db.pragma(`wal_autocheckpoint = ${Math.max(0, Math.floor(pages))}`);
+  }
+
+  /**
+   * `PRAGMA wal_checkpoint(PASSIVE)` on a worker thread with its own
+   * connection. PASSIVE never blocks the writer, and running it off-thread
+   * means the main thread — and the #850 watchdog heartbeat — keep turning
+   * even when the backfill is minutes of I/O on slow storage (a synchronous
+   * checkpoint that exceeds the watchdog's 60s window gets a healthy index
+   * SIGKILLed — observed in the #1231 repro).
+   *
+   * Returns SQLite's checkpoint result row — `log === checkpointed` with
+   * `busy === 0` means the ENTIRE WAL was backfilled, so the writer's next
+   * commit restarts the WAL from the top and the file stops growing. The
+   * WAL valve needs that signal because a WAL file's SIZE never shrinks:
+   * after the first wrap, raw file size says nothing about the un-backfilled
+   * backlog. Best-effort: returns null on any failure (including worker
+   * threads being unavailable — a potentially minutes-long checkpoint must
+   * never run inline on the main thread).
+   */
+  async checkpointWalPassive(): Promise<{ busy: number; log: number; checkpointed: number } | null> {
+    if (!this.dbPath || this.dbPath === ':memory:') {
+      try {
+        const row = this.db.prepare('PRAGMA wal_checkpoint(PASSIVE)').get() as Record<string, number> | undefined;
+        return row ? { busy: Number(row.busy), log: Number(row.log), checkpointed: Number(row.checkpointed) } : null;
+      } catch {
+        return null;
+      }
+    }
+    try {
+      const { Worker } = await import('node:worker_threads');
+      const workerSource = `
+        const { workerData, parentPort } = require('node:worker_threads');
+        let row = null;
+        try {
+          const { DatabaseSync } = require('node:sqlite');
+          const db = new DatabaseSync(workerData.dbPath);
+          try { row = db.prepare('PRAGMA wal_checkpoint(PASSIVE)').get(); } catch {}
+          try { db.close(); } catch {}
+        } catch {}
+        parentPort.postMessage({ row });
+      `;
+      return await new Promise((resolve) => {
+        let settled = false;
+        const finish = (row?: Record<string, number> | null): void => {
+          if (settled) return;
+          settled = true;
+          resolve(row ? { busy: Number(row.busy), log: Number(row.log), checkpointed: Number(row.checkpointed) } : null);
+        };
+        try {
+          const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath } });
+          worker.once('message', (m: { row?: Record<string, number> | null }) => { void worker.terminate(); finish(m?.row ?? null); });
+          worker.once('error', () => { void worker.terminate(); finish(null); });
+          worker.once('exit', () => finish(null));
+        } catch {
+          finish(null);
+        }
+      });
+    } catch {
+      return null;
+    }
+  }
+
   /**
    * Optimize database (vacuum and analyze)
    */
@@ -233,6 +325,22 @@ export class DatabaseConnection {
       try { this.db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch { /* ignore */ }
       return;
     }
+    await this.runPragmasOffThread(
+      ['PRAGMA analysis_limit=1000', 'PRAGMA optimize', 'PRAGMA wal_checkpoint(PASSIVE)'],
+      // Worker threads unavailable — bounded in-line fallback, no checkpoint.
+      ['PRAGMA analysis_limit=1000', 'PRAGMA optimize']
+    );
+  }
+
+  /**
+   * Run pragmas on a worker thread against its own connection to this DB
+   * (shared machinery for {@link runMaintenance} and
+   * {@link checkpointWalPassive}). Each pragma is individually best-effort;
+   * the whole call is best-effort. `inlineFallback` (if any) runs on THIS
+   * connection only when worker threads are unavailable — keep it to pragmas
+   * that are safe to run synchronously on the main thread.
+   */
+  private async runPragmasOffThread(pragmas: string[], inlineFallback: string[] = []): Promise<void> {
     try {
       const { Worker } = await import('node:worker_threads');
       const workerSource = `
@@ -240,9 +348,7 @@ export class DatabaseConnection {
         try {
           const { DatabaseSync } = require('node:sqlite');
           const db = new DatabaseSync(workerData.dbPath);
-          try { db.exec('PRAGMA analysis_limit=1000'); } catch {}
-          try { db.exec('PRAGMA optimize'); } catch {}
-          try { db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch {}
+          for (const p of workerData.pragmas) { try { db.exec(p); } catch {} }
           try { db.close(); } catch {}
         } catch {}
         parentPort.postMessage('done');
@@ -253,7 +359,7 @@ export class DatabaseConnection {
           if (!settled) { settled = true; resolve(); }
         };
         try {
-          const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath } });
+          const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath, pragmas } });
           worker.once('message', () => { void worker.terminate(); finish(); });
           worker.once('error', () => { void worker.terminate(); finish(); });
           worker.once('exit', finish);
@@ -262,9 +368,9 @@ export class DatabaseConnection {
         }
       });
     } catch {
-      // Worker threads unavailable — bounded in-line fallback, no checkpoint.
-      try { this.db.exec('PRAGMA analysis_limit=1000'); } catch { /* ignore */ }
-      try { this.db.exec('PRAGMA optimize'); } catch { /* ignore */ }
+      for (const p of inlineFallback) {
+        try { this.db.exec(p); } catch { /* ignore */ }
+      }
     }
   }
 

src/db/wal-valve.ts

@@ -0,0 +1,206 @@
+/**
+ * WAL checkpoint valve — bounds WAL growth while auto-checkpointing is
+ * deferred during a bulk index (#1231).
+ *
+ * Why deferral: SQLite's default `wal_autocheckpoint` (1000 pages) re-writes
+ * hot B-tree/FTS pages into the main DB file over and over during a bulk
+ * index — measured at ~95% of ALL disk I/O, and the difference between 45s
+ * and 19+ minutes on HDD-class storage (150 random IOPS). Deferring
+ * checkpoints turns the store into pure sequential WAL appends; each backfill
+ * pass writes distinct pages once, in page order (≈ sequential).
+ *
+ * Why a valve: unbounded deferral is its own failure mode, both measured in
+ * the #1231 repro. The WAL duplicates hot pages per COMMIT, so it grows far
+ * faster than the DB (5.9GB WAL for a ~340MB DB on a 3.3k-file index) —
+ * filling the disk, and poisoning every subsequent read that must page
+ * through it (the first resolution-phase read blocked the main thread >60s
+ * and the #850 liveness watchdog killed the healthy index). The valve
+ * watches WAL growth on a timer and, past a soft threshold, backfills with
+ * `PRAGMA wal_checkpoint(PASSIVE)` on a worker-thread connection — PASSIVE
+ * never blocks the writer, and off-thread means the main thread (and the
+ * watchdog heartbeat) keep turning regardless of how long a backfill takes.
+ *
+ * The load-bearing subtlety: a WAL file's SIZE never shrinks. After a full
+ * backfill, the writer's next commit RESTARTS the WAL from the top and the
+ * frames recycle inside the same file — so raw size says nothing about the
+ * un-backfilled backlog, and a size-triggered valve degenerates into firing
+ * (and pausing the writer) forever once the file passes its threshold
+ * (measured: guava crawled at ~9min per 160 files). Instead the valve
+ * tracks `sizeAtLastFullBackfill` — refreshed whenever a checkpoint reports
+ * `log === checkpointed` (everything backfilled) — and triggers on GROWTH
+ * beyond that baseline, which only happens when genuinely un-backfilled
+ * frames push past the file's high-water mark.
+ *
+ * Backpressure: if the writer outruns the checkpointer past a hard cap of
+ * growth (2× soft), {@link backpressure} pauses the writer (at a safe,
+ * between-transactions boundary) until a FULL backfill lands. One in-flight
+ * pass is not enough: on a disk saturated by the writer, every concurrent
+ * PASSIVE pass is already stale by the time it finishes (the writer appended
+ * past its snapshot), so neither SQLite's WAL wrap nor the baseline ever
+ * trigger and the WAL grows without bound (measured: 5.9GB on guava at 150
+ * IOPS, then a >60s read stall and a watchdog kill). With the writer parked,
+ * the next pass covers everything, the WAL wraps on the following commit,
+ * and the pause is the disk's honest catch-up cost — the correct terminal
+ * mode when hardware genuinely can't keep up with the append rate.
+ */
+
+import type { DatabaseConnection } from './index';
+
+/** Soft WAL-growth threshold (MB) that triggers an off-thread passive checkpoint. */
+const DEFAULT_WAL_VALVE_MB = 256;
+/** Hard cap = this × soft threshold; past it the writer pauses for a full backfill. */
+const HARD_CAP_MULTIPLIER = 2;
+/** Passes attempted per writer pause before giving up (a pinned reader could stall forever). */
+const MAX_PAUSED_BACKFILL_PASSES = 20;
+/** How often the timer looks at the WAL file size. */
+const CHECK_INTERVAL_MS = 2000;
+
+/**
+ * Resolve the valve's soft threshold from the `CODEGRAPH_WAL_VALVE_MB`
+ * override; non-numeric / non-positive values fall back to the default.
+ */
+export function resolveWalValveMb(envVal: string | undefined): number {
+  if (envVal !== undefined && envVal !== '') {
+    const n = Number(envVal);
+    if (Number.isFinite(n) && n > 0) return Math.floor(n);
+  }
+  return DEFAULT_WAL_VALVE_MB;
+}
+
+export class WalCheckpointValve {
+  private timer: ReturnType<typeof setInterval> | null = null;
+  private inflight: Promise<void> | null = null;
+  /** Writer pause in progress (hard cap breached): passes loop until a full backfill. */
+  private pause: Promise<void> | null = null;
+  /**
+   * WAL file size observed when a checkpoint last reported the ENTIRE WAL
+   * backfilled. Growth is measured against this baseline — see the header
+   * comment for why absolute size cannot be used.
+   */
+  private sizeAtLastFullBackfill = 0;
+  private readonly softBytes: number;
+  private readonly hardBytes: number;
+
+  constructor(
+    private readonly db: DatabaseConnection,
+    softMb: number = resolveWalValveMb(process.env.CODEGRAPH_WAL_VALVE_MB),
+    private readonly intervalMs: number = CHECK_INTERVAL_MS,
+    private readonly log: (msg: string) => void = () => {}
+  ) {
+    this.softBytes = softMb * 1024 * 1024;
+    this.hardBytes = this.softBytes * HARD_CAP_MULTIPLIER;
+  }
+
+  private mb(n: number): string {
+    return `${Math.round(n / 1024 / 1024)}MB`;
+  }
+
+  /** Un-backfilled growth estimate: bytes the WAL has grown past the last full backfill. */
+  private growthBytes(): number {
+    return this.db.getWalSizeBytes() - this.sizeAtLastFullBackfill;
+  }
+
+  /** Begin watching the WAL. Idempotent; the timer never holds the loop open. */
+  start(): void {
+    if (this.timer) return;
+    this.timer = setInterval(() => this.check(), this.intervalMs);
+    this.timer.unref?.();
+  }
+
+  /** Stop watching. Any in-flight checkpoint keeps running — await drain(). */
+  stop(): void {
+    if (this.timer) {
+      clearInterval(this.timer);
+      this.timer = null;
+    }
+  }
+
+  /** One poll: fire an off-thread passive checkpoint when growth passes the soft threshold. */
+  check(): void {
+    if (!this.pause && !this.inflight && this.growthBytes() > this.softBytes) this.fire();
+  }
+
+  /**
+   * Writer-side backstop, called at a between-transactions boundary. Returns
+   * null (no wait) while growth is under the hard cap; past it, returns a
+   * promise that resolves only once a FULL backfill has landed — see the
+   * header comment for why a single pass is not enough on a saturated disk.
+   */
+  backpressure(): Promise<void> | null {
+    if (this.pause) return this.pause;
+    if (this.growthBytes() <= this.hardBytes) return null;
+    this.log(`backpressure: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)} — pausing writer for full backfill`);
+    const t0 = Date.now();
+    this.pause = this.backfillFully().finally(() => {
+      this.pause = null;
+      this.log(`backpressure released after ${Date.now() - t0}ms: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)}`);
+    });
+    return this.pause;
+  }
+
+  /** Await any in-flight checkpoint and writer pause. */
+  async drain(): Promise<void> {
+    while (this.pause || this.inflight) {
+      if (this.pause) await this.pause;
+      if (this.inflight) await this.inflight;
+    }
+  }
+
+  /**
+   * Phase-boundary fold: backfill the ENTIRE WAL now (off-thread, awaited).
+   * Called between bulk phases — e.g. after parsing, before resolution's
+   * first reads — so the next phase never pages a bulk-write-sized WAL on
+   * the main thread (the post-parse read against a multi-GB WAL is what
+   * blew the #850 watchdog's 60s window in the #1231 repro). The await
+   * keeps the event loop (and the watchdog heartbeat) turning.
+   */
+  async foldNow(): Promise<void> {
+    await this.drain();
+    if (this.growthBytes() <= 0) return;
+    this.log(`foldNow: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)}`);
+    this.pause = this.backfillFully().finally(() => { this.pause = null; });
+    await this.pause;
+  }
+
+  /**
+   * With the writer parked on the returned promise, loop passive passes until
+   * one reports the entire WAL backfilled (typically the second: the first
+   * drains the pass that was already running against a stale snapshot). Gives
+   * up after a bounded number of passes — e.g. a reader pinning the WAL —
+   * because unbounded WAL growth degrades; a wedged writer never recovers.
+   */
+  private async backfillFully(): Promise<void> {
+    for (let i = 0; i < MAX_PAUSED_BACKFILL_PASSES; i++) {
+      if (this.inflight) await this.inflight; // fold in the stale in-flight pass first
+      const res = await this.db.checkpointWalPassive();
+      if (!res) return; // checkpoint machinery unavailable — don't spin
+      this.log(`backfill pass ${i + 1}: busy=${res.busy} log=${res.log} checkpointed=${res.checkpointed} wal=${this.mb(this.db.getWalSizeBytes())}`);
+      if (res.busy === 0 && res.log === res.checkpointed) {
+        this.sizeAtLastFullBackfill = this.db.getWalSizeBytes();
+        return;
+      }
+    }
+    this.log(`backfill gave up after ${MAX_PAUSED_BACKFILL_PASSES} passes — WAL stays unbounded this cycle`);
+  }
+
+  private fire(): void {
+    const p = this.db
+      .checkpointWalPassive()
+      .then((res) => {
+        // Full backfill (busy 0, every log frame checkpointed) ⇒ the writer's
+        // next commit wraps the WAL; the file's current size becomes the new
+        // growth baseline. A partial pass (writer appended during it, or a
+        // read transaction pinned frames) leaves the baseline alone, so the
+        // next tick fires again and copies the remainder. In non-WAL mode
+        // SQLite reports log = checkpointed = -1, which is harmless here.
+        if (res && res.busy === 0 && res.log === res.checkpointed) {
+          this.sizeAtLastFullBackfill = this.db.getWalSizeBytes();
+        }
+      })
+      .catch(() => { /* best-effort */ })
+      .finally(() => {
+        if (this.inflight === p) this.inflight = null;
+      });
+    this.inflight = p;
+  }
+}

src/extraction/grammars.ts

@@ -7,6 +7,7 @@
  */
 
 import * as path from 'path';
+import * as fsp from 'fs/promises';
 import { Parser, Language as WasmLanguage } from 'web-tree-sitter';
 import { Language } from '../types';
 
@@ -249,30 +250,100 @@ export async function initGrammars(): Promise<void> {
 }
 
 /**
- * Load grammar WASM files for specific languages only.
- * Skips languages that are already loaded or have no WASM grammar.
- * Must be called after initGrammars().
+ * Grammars that ship their own vendored WASMs under `dist/extraction/wasm/`
+ * (not in tree-sitter-wasms, or the tree-sitter-wasms build is too old).
+ * Lua: tree-sitter-wasms ships an ABI-13 build that corrupts the shared WASM
+ * heap under web-tree-sitter 0.25 (drops nested calls/imports on every file
+ * after the first); we vendor the upstream ABI-15 wasm instead. C#: the
+ * tree-sitter-wasms build (ABI 13) has no primary-constructor support and
+ * parses `class Foo(...)` as an ERROR that swallows the whole class (#237); we
+ * vendor the upstream ABI-15 tree-sitter-c-sharp 0.23.5 wasm, which parses
+ * primary constructors natively. Terraform: tree-sitter-wasms does not ship
+ * HCL/Terraform at all, so we vendor the prebuilt tree-sitter-terraform.wasm
+ * from @tree-sitter-grammars/tree-sitter-hcl 1.2.0 (Apache-2.0) —
+ * byte-identical to the npm package's artifact. ArkTS: tree-sitter-wasms
+ * doesn't ship it either; we vendor the prebuilt tree-sitter-arkts.wasm from
+ * the tree-sitter-arkts 0.2.0 npm package (harmony-contrib/tree-sitter-arkts,
+ * MIT) — byte-identical to the npm tarball's artifact. It extends the
+ * tree-sitter-javascript grammar the same way tree-sitter-typescript does,
+ * adding `struct_declaration` and the `arkui_component_expression` build()
+ * DSL. Nix: tree-sitter-wasms doesn't ship it; we vendor a wasm built from
+ * nix-community/tree-sitter-nix @ 3d0173d (MIT) with tree-sitter-cli 0.25.10
+ * (`generate` + `build --wasm`, ABI 15 — upstream's checked-in parser.c is
+ * still ABI 13; all 54 upstream corpus tests pass on the regenerated parser).
  */
-export async function loadGrammarsForLanguages(languages: Language[]): Promise<void> {
-  if (!parserInitialized) {
-    await initGrammars();
-  }
+const VENDORED_WASM_LANGS: ReadonlySet<GrammarLanguage> = new Set([
+  'pascal', 'scala', 'lua', 'luau', 'csharp', 'r', 'cfml', 'cfscript', 'cfquery',
+  'cobol', 'vbnet', 'erlang', 'terraform', 'arkts', 'nix',
+]);
+
+/** Absolute path of a language's grammar WASM (vendored or tree-sitter-wasms). */
+function resolveWasmPath(lang: GrammarLanguage): string {
+  const wasmFile = WASM_GRAMMAR_FILES[lang];
+  return VENDORED_WASM_LANGS.has(lang)
+    ? path.join(__dirname, 'wasm', wasmFile)
+    : require.resolve(`tree-sitter-wasms/out/${wasmFile}`);
+}
 
-  // SFC languages (svelte/vue/astro) have no grammar of their own — their
-  // extractors delegate <script>/frontmatter content to the TS/JS extractor,
-  // so those grammars must be loaded even when no plain .ts/.js file is in
-  // the index set (e.g. a pure-.astro content site).
+/**
+ * Expand an index set's languages to the grammars actually needed to parse it.
+ * SFC languages (svelte/vue/astro) have no grammar of their own — their
+ * extractors delegate <script>/frontmatter content to the TS/JS extractor, so
+ * those grammars must be loaded even when no plain .ts/.js file is in the index
+ * set (e.g. a pure-.astro content site). CFML (.cfc/.cfm) likewise delegates
+ * bare-script content, <cfscript> tag bodies, and <cfquery> SQL bodies to the
+ * cfscript/cfquery grammars (see injections.scm in tree-sitter-cfml).
+ */
+function expandGrammarLanguages(languages: Language[]): Language[] {
   if (languages.some((l) => l === 'svelte' || l === 'vue' || l === 'astro')) {
     languages = [...languages, 'typescript', 'javascript'];
   }
-
-  // CFML (.cfc/.cfm) delegates bare-script content, <cfscript> tag bodies, and
-  // <cfquery> SQL bodies to the cfscript/cfquery grammars (see injections.scm in
-  // tree-sitter-cfml) — load both even when no standalone .cfs file is in the
-  // index set.
   if (languages.some((l) => l === 'cfml')) {
     languages = [...languages, 'cfscript', 'cfquery'];
   }
+  return languages;
+}
+
+/**
+ * Pre-read the grammar WASM bytes for an index set, keyed by language. The
+ * orchestrator reads each grammar ONCE and hands the bytes to every parse
+ * worker via its `load-grammars` message, so worker spawns/respawns load
+ * grammars from memory instead of re-reading them from disk — on slow storage
+ * (HDD, issue #1231) each respawn's grammar re-read otherwise amplifies the
+ * I/O contention that caused the respawn. Best-effort: a language whose WASM
+ * can't be read here is simply omitted, and the worker falls back to its own
+ * disk load (which surfaces the real error/warning path).
+ */
+export async function readGrammarWasmBytes(languages: Language[]): Promise<Record<string, Uint8Array>> {
+  const out: Record<string, Uint8Array> = {};
+  const toRead = [...new Set(expandGrammarLanguages(languages))].filter(
+    (lang): lang is GrammarLanguage => lang in WASM_GRAMMAR_FILES
+  );
+  for (const lang of toRead) {
+    try {
+      out[lang] = await fsp.readFile(resolveWasmPath(lang));
+    } catch {
+      // fall through — the worker's own load reports the failure
+    }
+  }
+  return out;
+}
+
+/**
+ * Load grammar WASM files for specific languages only.
+ * Skips languages that are already loaded or have no WASM grammar.
+ * Must be called after initGrammars().
+ *
+ * `wasmBytes` (optional) holds pre-read grammar bytes keyed by language (from
+ * {@link readGrammarWasmBytes}, forwarded through the parse pool); when a
+ * language's bytes are present they're loaded from memory instead of disk.
+ */
+export async function loadGrammarsForLanguages(languages: Language[], wasmBytes?: Record<string, Uint8Array>): Promise<void> {
+  if (!parserInitialized) {
+    await initGrammars();
+  }
+
+  languages = expandGrammarLanguages(languages);
 
   // Deduplicate and filter to languages that have WASM grammars and aren't already loaded
   const toLoad = [...new Set(languages)].filter(
@@ -285,35 +356,9 @@ export async function loadGrammarsForLanguages(languages: Language[]): Promise<v
   // Load grammars sequentially to avoid web-tree-sitter WASM race condition on Node 20+
   // See: https://github.com/tree-sitter/tree-sitter/issues/2338
   for (const lang of toLoad) {
-    const wasmFile = WASM_GRAMMAR_FILES[lang];
     try {
-      // Some grammars ship their own WASMs (not in tree-sitter-wasms, or the
-      // tree-sitter-wasms build is too old). Lua: tree-sitter-wasms ships an
-      // ABI-13 build that corrupts the shared WASM heap under web-tree-sitter
-      // 0.25 (drops nested calls/imports on every file after the first); we
-      // vendor the upstream ABI-15 wasm instead. C#: the tree-sitter-wasms
-      // build (ABI 13) has no primary-constructor support and parses
-      // `class Foo(...)` as an ERROR that swallows the whole class (#237); we
-      // vendor the upstream ABI-15 tree-sitter-c-sharp 0.23.5 wasm, which parses
-      // primary constructors natively. Terraform: tree-sitter-wasms does not
-      // ship HCL/Terraform at all, so we vendor the prebuilt
-      // tree-sitter-terraform.wasm from @tree-sitter-grammars/tree-sitter-hcl
-      // 1.2.0 (Apache-2.0) — byte-identical to the npm package's artifact.
-      // ArkTS: tree-sitter-wasms doesn't ship it either; we vendor the prebuilt
-      // tree-sitter-arkts.wasm from the tree-sitter-arkts 0.2.0 npm package
-      // (harmony-contrib/tree-sitter-arkts, MIT) — byte-identical to the npm
-      // tarball's artifact. It extends the tree-sitter-javascript grammar the
-      // same way tree-sitter-typescript does, adding `struct_declaration` and
-      // the `arkui_component_expression` build() DSL.
-      // Nix: tree-sitter-wasms doesn't ship it; we vendor a wasm built from
-      // nix-community/tree-sitter-nix @ 3d0173d (MIT) with tree-sitter-cli
-      // 0.25.10 (`generate` + `build --wasm`, ABI 15 — upstream's checked-in
-      // parser.c is still ABI 13; all 54 upstream corpus tests pass on the
-      // regenerated parser).
-      const wasmPath = (lang === 'pascal' || lang === 'scala' || lang === 'lua' || lang === 'luau' || lang === 'csharp' || lang === 'r' || lang === 'cfml' || lang === 'cfscript' || lang === 'cfquery' || lang === 'cobol' || lang === 'vbnet' || lang === 'erlang' || lang === 'terraform' || lang === 'arkts' || lang === 'nix')
-        ? path.join(__dirname, 'wasm', wasmFile)
-        : require.resolve(`tree-sitter-wasms/out/${wasmFile}`);
-      const language = await WasmLanguage.load(wasmPath);
+      const bytes = wasmBytes?.[lang];
+      const language = await WasmLanguage.load(bytes ?? resolveWasmPath(lang));
       languageCache.set(lang, language);
     } catch (error) {
       const message = error instanceof Error ? error.message : String(error);

src/extraction/index.ts

@@ -19,8 +19,8 @@ import {
 } from '../types';
 import { QueryBuilder } from '../db/queries';
 import { extractFromSource } from './tree-sitter';
-import { ParseWorkerPool, resolveParsePoolSize } from './parse-pool';
-import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages } from './grammars';
+import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs } from './parse-pool';
+import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes } from './grammars';
 import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns } from '../project-config';
 import { isCodeGraphDataDir } from '../directory';
 import { logDebug, logWarn } from '../errors';
@@ -53,9 +53,10 @@ const SYNC_RECONCILE_YIELD_INTERVAL = 1000;
 /**
  * Maximum time (ms) to wait for a single file to parse in the worker thread.
  * If tree-sitter hangs or WASM runs out of memory, this prevents the entire
- * indexing run from freezing. The worker is restarted after a timeout.
+ * indexing run from freezing. The worker is restarted after a (hard) timeout.
+ * Env-overridable via CODEGRAPH_PARSE_TIMEOUT_MS for slow storage (#1231).
  */
-const PARSE_TIMEOUT_MS = 10_000;
+const PARSE_TIMEOUT_MS = resolveParseTimeoutMs(process.env.CODEGRAPH_PARSE_TIMEOUT_MS);
 
 /**
  * Number of files to parse before recycling the worker thread.
@@ -1453,7 +1454,12 @@ export class ExtractionOrchestrator {
   async indexAll(
     onProgress?: (progress: IndexProgress) => void,
     signal?: AbortSignal,
-    verbose?: boolean
+    verbose?: boolean,
+    // Writer-side backstop for deferred WAL checkpointing (#1231): returns
+    // null in the normal case, or a promise to await (at this safe,
+    // between-transactions boundary) when the WAL has outrun the off-thread
+    // checkpointer past its hard cap. See db/wal-valve.ts.
+    walBackpressure?: () => Promise<void> | null
   ): Promise<IndexResult> {
     await initGrammars();
     const startTime = Date.now();
@@ -1549,6 +1555,11 @@ export class ExtractionOrchestrator {
       // CODEGRAPH_PARSE_WORKERS: explicit worker count; 1 = the old single-worker
       // behaviour (the conservative rollback). Unset → clamp(cores-1, 1, 8).
       const poolSize = resolveParsePoolSize(process.env.CODEGRAPH_PARSE_WORKERS, os.cpus().length);
+      // Read each needed grammar's WASM ONCE here and hand the bytes to every
+      // worker, so spawns/respawns load grammars from memory instead of
+      // re-reading them from disk (#1231: on an HDD, respawn re-reads amplify
+      // the very I/O contention that caused the respawn).
+      const grammarBuffers = await readGrammarWasmBytes(neededLanguages);
       pool = new ParseWorkerPool({
         languages: neededLanguages,
         size: poolSize,
@@ -1556,6 +1567,7 @@ export class ExtractionOrchestrator {
         recycleInterval: WORKER_RECYCLE_INTERVAL,
         parseTimeoutMs: PARSE_TIMEOUT_MS,
         log,
+        grammarBuffers,
       });
       log(`Parse worker pool: ${poolSize} worker(s)`);
     } else {
@@ -1603,6 +1615,12 @@ export class ExtractionOrchestrator {
     const storeResult = async (filePath: string, content: string, stats: fs.Stats, result: ExtractionResult): Promise<void> => {
       processed++;
 
+      // WAL hard-cap backstop: between files (never mid-transaction), pause
+      // the store until the off-thread checkpoint catches up. Resolves to
+      // null in the normal case — a single size check, no cost.
+      const bp = walBackpressure?.();
+      if (bp) await bp;
+
       // Store in database on main thread (SQLite is not thread-safe)
       if (result.nodes.length > 0 || result.errors.length === 0) {
         const language = detectLanguage(filePath, content, overrides);
@@ -1815,14 +1833,19 @@ export class ExtractionOrchestrator {
 
     // Retry pass: files that failed due to WASM memory corruption may succeed
     // on a fresh worker with a clean heap. Recycle before each attempt so
-    // every file gets the absolute cleanest WASM state possible.
+    // every file gets the absolute cleanest WASM state possible. Timeouts are
+    // retried too (#1231): most are main-thread-stall artifacts, not slow
+    // parses, and this pass parses one file at a time with the store strictly
+    // after each parse resolves, so the stall window can't recur here.
     const retryableErrors = errors.filter(
       (e) => e.code === 'parse_error' && e.filePath &&
-        (e.message.includes('Worker exited') || e.message.includes('memory access out of bounds'))
+        (e.message.includes('Worker exited') ||
+         e.message.includes('memory access out of bounds') ||
+         e.message.includes('timed out'))
     );
 
     if (retryableErrors.length > 0 && pool) {
-      log(`Retrying ${retryableErrors.length} files that failed due to WASM memory errors...`);
+      log(`Retrying ${retryableErrors.length} files that failed due to WASM memory errors or timeouts...`);
 
       // Fresh WASM heaps for the retry phase. A retry that still crashes its
       // worker makes the pool respawn it, so later retries keep landing on clean

src/extraction/parse-pool.ts

@@ -61,6 +61,18 @@ const MAX_PARSE_POOL_SIZE = 16;
 const DEFAULT_RECYCLE_INTERVAL = 250;
 /** Base per-parse timeout; scaled up for large files by the caller's formula. */
 const DEFAULT_PARSE_TIMEOUT_MS = 10_000;
+/**
+ * A worker is only killed once a parse has gone this many × its budget with no
+ * result. The base timer firing is NOT proof the parse is still running: after
+ * a long synchronous main-thread stretch (the SQLite store on slow disks,
+ * issue #1231) Node runs the timers phase before the poll phase, so the
+ * expired timer fires BEFORE an already-delivered `parse-result` is processed.
+ * Killing at the base timeout therefore produced false timeouts on parses that
+ * finished instantly (even 0-byte files). Instead the base timer only marks
+ * the job late; a result that arrives before this backstop is accepted, and
+ * only a worker that stays silent the whole window is treated as hung.
+ */
+const HARD_KILL_MULTIPLIER = 3;
 /**
  * Max workers cold-starting at once. A worker's cold start is heavy (module load
  * + grammar WASM compile); starting the whole pool simultaneously thrashes CPU.
@@ -84,6 +96,19 @@ const CRASH_BUDGET = 100;
  *   - unset / blank / non-numeric → `clamp(cores - 1, 1, 8)` (leave a core for
  *     the main thread + UI; never zero — parsing always needs a worker).
  */
+/**
+ * Resolve the base per-parse timeout from the `CODEGRAPH_PARSE_TIMEOUT_MS`
+ * override. Slow storage (HDD, network folders) can need a larger budget; a
+ * non-numeric / non-positive value falls back to the default (10s).
+ */
+export function resolveParseTimeoutMs(envVal: string | undefined): number {
+  if (envVal !== undefined && envVal !== '') {
+    const n = Number(envVal);
+    if (Number.isFinite(n) && n > 0) return Math.floor(n);
+  }
+  return DEFAULT_PARSE_TIMEOUT_MS;
+}
+
 export function resolveParsePoolSize(envVal: string | undefined, cpuCount: number): number {
   if (envVal !== undefined && envVal !== '') {
     const n = Number(envVal);
@@ -102,6 +127,11 @@ interface ParseJob {
   reject: (e: Error) => void;
   settled: boolean;
   timer?: ReturnType<typeof setTimeout>;
+  /** Full budget for this parse (base timeout + size scaling), for late-result logging. */
+  budgetMs?: number;
+  /** The base timer fired with no result yet — accept a late result, kill at the backstop. */
+  timerExpired?: boolean;
+  hardKillTimer?: ReturnType<typeof setTimeout>;
 }
 
 /** Shape of a message a worker posts back (grammar-load ack or a parse result). */
@@ -109,6 +139,8 @@ interface ParseWorkerMessage {
   type?: string;
   id?: number;
   result?: ExtractionResult;
+  /** Worker-side parse duration — the worker's own clock, immune to main-thread stalls. */
+  parseMs?: number;
 }
 
 export interface ParseWorkerPoolOptions {
@@ -126,6 +158,15 @@ export interface ParseWorkerPoolOptions {
   createWorker?: () => ParsePoolWorker;
   /** Optional verbose logger (the orchestrator's `[worker] …` logger). */
   log?: (msg: string) => void;
+  /**
+   * Pre-read grammar WASM bytes keyed by language, forwarded to every worker's
+   * `load-grammars` message so a spawn/respawn loads grammars from memory
+   * instead of re-reading them from disk — on slow storage each respawn's
+   * grammar re-read otherwise amplifies the very I/O contention that caused
+   * the respawn (issue #1231). Best-effort: a missing language falls back to
+   * the worker's own disk read.
+   */
+  grammarBuffers?: Record<string, Uint8Array>;
 }
 
 export class ParseWorkerPool {
@@ -147,9 +188,11 @@ export class ParseWorkerPool {
   private readonly parseTimeoutMs: number;
   private readonly createWorker: () => ParsePoolWorker;
   private readonly log: (msg: string) => void;
+  private readonly grammarBuffers?: Record<string, Uint8Array>;
 
   constructor(opts: ParseWorkerPoolOptions) {
     this.languages = opts.languages;
+    this.grammarBuffers = opts.grammarBuffers;
     this.maxSize = Math.max(1, Math.min(opts.size, MAX_PARSE_POOL_SIZE));
     this.recycleInterval = opts.recycleInterval ?? DEFAULT_RECYCLE_INTERVAL;
     this.parseTimeoutMs = opts.parseTimeoutMs ?? DEFAULT_PARSE_TIMEOUT_MS;
@@ -179,7 +222,7 @@ export class ParseWorkerPool {
   /**
    * Parse one file on the pool. Resolves with the extraction result, or REJECTS
    * if the parse times out or its worker crashes — the caller records the error
-   * and (for worker-exit/OOM rejections) re-attempts in its retry pass.
+   * and (for worker-exit/OOM/timeout rejections) re-attempts in its retry pass.
    */
   requestParse(task: ParseTask): Promise<ExtractionResult> {
     if (this.destroyed) return Promise.reject(new Error('Parse pool destroyed'));
@@ -205,7 +248,9 @@ export class ParseWorkerPool {
     w.on('error', (e) => this.onWorkerGone(w, `Worker error: ${e?.message ?? 'unknown'}`));
     w.on('exit', (code) => { if (code !== 0) this.onWorkerGone(w, `Worker exited with code ${code}`); });
     // Load grammars; the worker replies 'grammars-loaded' and only then is idle.
-    w.postMessage({ type: 'load-grammars', languages: this.languages });
+    // Pre-read WASM bytes (when the orchestrator provided them) make this a
+    // memory load instead of a per-spawn disk read.
+    w.postMessage({ type: 'load-grammars', languages: this.languages, grammarBuffers: this.grammarBuffers });
   }
 
   private onMessage(w: ParsePoolWorker, m: ParseWorkerMessage): void {
@@ -220,6 +265,22 @@ export class ParseWorkerPool {
       const job = this.inflight.get(w);
       if (!job || (m.id !== undefined && m.id !== job.id)) return; // stale (post-recycle)
       this.inflight.delete(w);
+      if (job.timerExpired) {
+        // The base timer fired before this result was processed. That almost
+        // always means the MAIN THREAD was stalled (sync SQLite store on slow
+        // disks) while the parse itself finished long ago — the worker's own
+        // clock (parseMs) tells the two apart. Either way the result is here
+        // and valid: accept it instead of the old behaviour (kill worker +
+        // reject), which turned every main-thread stall into false timeouts
+        // and dropped files (issue #1231).
+        const parseMs = typeof m.parseMs === 'number' ? Math.round(m.parseMs) : undefined;
+        const detail = parseMs === undefined
+          ? ''
+          : parseMs < (job.budgetMs ?? this.parseTimeoutMs)
+            ? ` (parse took ${parseMs}ms in-worker — the main thread was stalled, not the parse)`
+            : ` (parse genuinely took ${parseMs}ms)`;
+        this.log(`Late parse-result accepted: ${job.task.filePath}${detail}`);
+      }
       // Recycle the worker once it's done enough parses to have grown its WASM
       // heap; otherwise return it to the idle set for the next job.
       if ((this.parseCounts.get(w) ?? 0) >= this.recycleInterval) {
@@ -269,6 +330,7 @@ export class ParseWorkerPool {
     // Scale the timeout for large files: base + 10s per 100KB (matches the
     // original single-worker formula so pathological-file behaviour is unchanged).
     const timeoutMs = this.parseTimeoutMs + Math.floor(job.task.content.length / 100_000) * 10_000;
+    job.budgetMs = timeoutMs;
     job.timer = setTimeout(() => this.onTimeout(w, job, timeoutMs), timeoutMs);
     job.timer.unref?.();
     w.postMessage({
@@ -281,16 +343,35 @@ export class ParseWorkerPool {
     });
   }
 
+  /**
+   * The base timer fired with no result processed yet. Do NOT kill or settle:
+   * the timer firing doesn't prove the parse is still running — after a long
+   * synchronous main-thread stretch Node services the timers phase before the
+   * poll phase, so an already-delivered `parse-result` is still queued behind
+   * this callback. Mark the job late (onMessage accepts a result that shows up)
+   * and arm the hard-kill backstop for workers that are genuinely hung.
+   */
   private onTimeout(w: ParsePoolWorker, job: ParseJob, ms: number): void {
     if (job.settled || !this.workers.has(w)) return;
-    this.log(`TIMEOUT: ${job.task.filePath} exceeded ${ms}ms — killing worker`);
-    // Kill the (possibly WASM-wedged) worker and reject this parse. A timeout
-    // isn't a crash — don't charge the budget — but the worker is gone, so spawn
-    // a replacement to keep capacity.
+    const graceMs = ms * (HARD_KILL_MULTIPLIER - 1);
+    this.log(`TIMEOUT: ${job.task.filePath} exceeded ${ms}ms with no result — waiting up to ${graceMs}ms more for a late result before killing the worker`);
+    job.timerExpired = true;
+    job.hardKillTimer = setTimeout(() => this.onHardTimeout(w, job, ms * HARD_KILL_MULTIPLIER), graceMs);
+    job.hardKillTimer.unref?.();
+  }
+
+  /** No result after the full hard-kill window — the worker really is hung. */
+  private onHardTimeout(w: ParsePoolWorker, job: ParseJob, totalMs: number): void {
+    if (job.settled || !this.workers.has(w)) return;
+    this.log(`TIMEOUT: ${job.task.filePath} got no result after ${totalMs}ms — killing worker`);
+    // Kill the (WASM-wedged) worker and reject this parse. A timeout isn't a
+    // crash — don't charge the budget — but the worker is gone, so spawn a
+    // replacement to keep capacity. The rejection message contains "timed out"
+    // so the orchestrator's retry pass re-attempts the file.
     this.removeWorker(w);
     this.inflight.delete(w);
     try { void w.terminate(); } catch { /* already gone */ }
-    this.settle(job, undefined, new Error(`Parse timed out after ${ms}ms`));
+    this.settle(job, undefined, new Error(`Parse timed out after ${totalMs}ms`));
     if (this.healthy) this.spawnOne();
     this.drain();
   }
@@ -329,6 +410,7 @@ export class ParseWorkerPool {
     if (job.settled) return;
     job.settled = true;
     if (job.timer) clearTimeout(job.timer);
+    if (job.hardKillTimer) clearTimeout(job.hardKillTimer);
     if (err) job.reject(err);
     else job.resolve(result!);
   }

src/extraction/parse-worker.ts

@@ -55,12 +55,18 @@ import type { Language, ExtractionResult } from '../types';
 const PARSER_RESET_INTERVAL = 5000;
 const parseCounts = new Map<Language, number>();
 
-parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: string; content?: string; languages?: Language[]; frameworkNames?: string[]; language?: Language }) => {
+parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: string; content?: string; languages?: Language[]; frameworkNames?: string[]; language?: Language; grammarBuffers?: Record<string, Uint8Array> }) => {
   if (msg.type === 'load-grammars') {
-    await loadGrammarsForLanguages(msg.languages!);
+    // Grammar WASM bytes pre-read by the main thread (when provided) make this
+    // a memory load instead of a per-spawn disk read — see issue #1231.
+    await loadGrammarsForLanguages(msg.languages!, msg.grammarBuffers);
     parentPort!.postMessage({ type: 'grammars-loaded' });
   } else if (msg.type === 'parse') {
     const { id, filePath, content, frameworkNames } = msg;
+    // Worker-side parse clock: reported back with the result so the pool can
+    // tell a genuinely slow parse from a result whose delivery was delayed by
+    // a stalled main thread (issue #1231 false timeouts).
+    const t0 = performance.now();
     try {
       // The main thread resolves the language (it holds the project's
       // codegraph.json extension overrides) and sends it; fall back to detection
@@ -75,7 +81,7 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st
         resetParser(language);
       }
 
-      parentPort!.postMessage({ type: 'parse-result', id, result });
+      parentPort!.postMessage({ type: 'parse-result', id, result, parseMs: performance.now() - t0 });
     } catch (err) {
       const message = err instanceof Error ? err.message : String(err);
 
@@ -89,6 +95,7 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st
       parentPort!.postMessage({
         type: 'parse-result',
         id,
+        parseMs: performance.now() - t0,
         result: {
           nodes: [],
           edges: [],

src/index.ts

@@ -24,6 +24,7 @@ import {
   FindRelevantContextOptions,
 } from './types';
 import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db';
+import { WalCheckpointValve } from './db/wal-valve';
 import { QueryBuilder } from './db/queries';
 import {
   isInitialized,
@@ -435,6 +436,29 @@ export class CodeGraph {
       } catch {
         return { success: false, filesIndexed: 0, filesSkipped: 0, filesErrored: 0, nodesCreated: 0, edgesCreated: 0, errors: [{ message: 'Could not acquire file lock - another process may be indexing', severity: 'error' as const }], durationMs: 0 };
       }
+      // Defer WAL auto-checkpointing for the whole bulk run (#1231): the
+      // default 1000-page interval re-writes hot pages into the main DB file
+      // over and over — ~95% of all disk I/O during a bulk index, and a
+      // 19+min → 45s difference on HDD-class storage. The valve bounds WAL
+      // growth by backfilling PASSIVEly on a worker thread (never blocking
+      // the writer or the #850 watchdog heartbeat); runMaintenance below does
+      // the final fold-up before the interval is restored in the finally.
+      // Kill switch: CODEGRAPH_NO_WAL_DEFER=1. Non-WAL journal modes (some
+      // network filesystems) have no WAL to defer — skip.
+      const deferWal = process.env.CODEGRAPH_NO_WAL_DEFER !== '1' && this.db.getJournalMode() === 'wal';
+      let walValve: WalCheckpointValve | null = null;
+      let priorAutocheckpoint = 1000;
+      if (deferWal) {
+        priorAutocheckpoint = this.db.getWalAutocheckpoint();
+        this.db.setWalAutocheckpoint(0);
+        walValve = new WalCheckpointValve(
+          this.db,
+          undefined,
+          undefined,
+          options.verbose ? (m) => console.log(`[wal-valve] ${m}`) : undefined
+        );
+        walValve.start();
+      }
       try {
         const before = this.queries.getNodeAndEdgeCount();
         // Mark the index as in-flight BEFORE any writes: a run killed
@@ -446,7 +470,19 @@ export class CodeGraph {
         // path as every file (re-)indexes below — so a full index is also the
         // orphan-cleanup pass for names deleted since the last one.
         try { this.queries.clearNameSegmentVocab(); } catch { /* vocab is advisory — never fail an index over it */ }
-        const result = await this.orchestrator.indexAll(options.onProgress, options.signal, options.verbose);
+        const result = await this.orchestrator.indexAll(
+          options.onProgress,
+          options.signal,
+          options.verbose,
+          walValve ? () => walValve!.backpressure() : undefined
+        );
+
+        // Fold the parse phase's WAL BEFORE the first post-parse reads
+        // (resolver re-init and resolution both read on the main thread):
+        // paging a bulk-write-sized WAL there is what blew the #850
+        // watchdog's 60s window in the #1231 repro. Off-thread + awaited,
+        // so the event loop keeps turning.
+        if (walValve) await walValve.foldNow();
 
         // Re-detect frameworks now that the index is populated. The resolver
         // is constructed with createResolver() before any files exist, so
@@ -501,6 +537,10 @@ export class CodeGraph {
         // successful index. Never load-bearing for correctness.
         if (result.success && result.filesIndexed > 0) {
           const tMaint = Date.now();
+          // Quiesce the valve first so its in-flight checkpoint and the
+          // maintenance checkpoint don't contend for the checkpointer lock
+          // (the loser would silently no-op and leave the WAL unfolded).
+          if (walValve) { walValve.stop(); await walValve.drain(); }
           await this.db.runMaintenance();
           if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] maintenance: ${Date.now() - tMaint}ms`);
         }
@@ -561,6 +601,15 @@ export class CodeGraph {
 
         return result;
       } finally {
+        // Restore the auto-checkpoint interval AFTER the fold-up above so the
+        // next ordinary write doesn't inherit a giant inline checkpoint. On
+        // the error path the WAL may still be large; correctness is unchanged
+        // (SQLite replays the WAL on the next open) and the follow-up write
+        // that folds it is the known cost of a failed run.
+        if (walValve) { walValve.stop(); await walValve.drain(); }
+        if (deferWal) {
+          try { this.db.setWalAutocheckpoint(priorAutocheckpoint); } catch { /* connection may be closing */ }
+        }
         this.fileLock.release();
       }
     });