gstack/docs/designs/SYNC_GBRAIN_BATCH_INGEST.md

/sync-gbrain batch ingest migration

Status: Implemented on garrytan/dublin-v1 (D1-D8 decisions land in this PR) Branch: garrytan/dublin-v1 Owner: Garry Tan Triggered by: /investigate run, 2026-05-09 Estimated effort: human ~3 days / CC+gstack ~2 hr Files touched: 4 source + 1 test = 5 total (under estimate)

Decisions (post-review)

This doc captures the original architecture. Final architecture lands per the 8 review decisions captured in /Users/garrytan/.claude/plans/purrfect-tumbling-quiche.md:

• D1 hierarchical staging dir (mkdir -p per slug segment) — kept
• D2 cut over + delete legacy in same PR (no --legacy-ingest flag) — kept
• D3 scan source-file first, stage only clean — kept
• D4 three-state OK/DEGRADED/ERR verdict COLLAPSED to OK/ERR per Codex finding 7 (gbrain content_hash idempotency makes the third state redundant)
• D5 skip_reason field in state schema DROPPED per Codex finding 7 (re-runs are cheap; no need for permanent skip-tracking)
• D6 trust gbrain's content_hash idempotency; drop bookkeeping scaffolding (skip_reason, three-state, SIGTERM checkpoint)
• D7 per-file failure detection via ~/.gbrain/sync-failures.jsonl (byte-offset snapshot + appended-only read)
• D8 bundle 3 in-scope pre-existing fixes: F6 atomic saveState (tmp+rename), F8 isolated-stage benchmark, F9 full-file sha256 hash (no more 1MB cap)

Verified from gbrain source

Three properties verified by reading ~/git/gbrain/src/:

• Idempotency at core/import-file.ts:242-243, :478 — content_hash check, skip if unchanged, overwrite if changed.
• Frontmatter parity at core/import-file.ts:228, 297, 410-422 — title/type/tags honored; auto-inference only when frontmatter absent.
• Path-authoritative slug at core/sync.ts:260 (slugifyPath), enforced at core/import-file.ts:429.
• Per-file failures surface at commands/import.ts:308-310, comment at :28: "callers can gate state advances" — the intentional API for what D7 uses.

Performance: planned vs measured (post 2026-05-10 perf review)

Metric	Plan target	Measured	Verdict
Prepare phase on 5135 files	—	<10s	FAST
gbrain import on 5135 files	—	>10 min	gbrain-side perf issue, filed
Loop / hang (original bug)	never	never	FIXED
Memory ingest exits null on SIGTERM	no	no — state writes succeed; child gbrain dies with parent	FIXED
FILE_TOO_LARGE blocks last_commit	no	no — failed paths excluded via D7	FIXED

Initial perf miss + correction. The first cold-run measurement (~12 min) was dominated by 1841 sequential gitleaks subprocess spawns at ~256ms each — a redundant security gate. The cross-machine exfiltration boundary is gstack-brain-sync (bin/gstack-brain-sync:78-110, regex-based secret scan on staged diff before git commit). Scanning every source file before ingest into a LOCAL PGLite doesn't change exposure — the secret already lives on disk in plaintext. We made per-file gitleaks opt-in via --scan-secrets. Default is off. That cut the prepare phase from ~12 min to under 10 seconds.

The remaining cold-run cost is gbrain import itself, which scales worse than linear on large staging dirs (10s for 501 files; >10 min for 5031). That's a gbrain-side perf issue, not gstack architecture. Filed as a TODO; the fix likely lives in gbrain's content_hash check loop or auto-link reconciliation phase.

F9 hash migration (one-time cliff)

F9 switched fileSha256 from a 1MB-capped hash to full-file. Existing state entries from before this change carry the old 1MB-capped hash. For any file whose mtime hasn't changed, fileChangedSinceState returns false at the mtime check and the new hash is never computed — so unchanged files behave identically. For any file whose mtime DOES change after upgrade, the full-file hash is recomputed and (correctly) treated as changed, then re-imported. The gbrain doctor probe report's updated_count may show inflated numbers on the first run post-upgrade because every touched file crosses the algorithm boundary. No data loss, but worth knowing.

Follow-ups (filed as TODOs)

1. gbrain import perf on large dirs — investigate why 5031 files take >10 min when 501 takes 10s. Likely culprits: N+1 SQL for getPage(slug) content_hash check, per-page auto-link reconciliation, FTS index updates without batching. Lives in gbrain, not gstack.
2. Optional: source-file changed-detection cache — even with the prepare phase fast, walking 5031 files takes some time. Caching the "no changes since last successful import" state at the batch level (not per-file) would skip the prepare phase entirely on a no-op incremental run.

Problem

/sync-gbrain memory stage takes 35 minutes on a fresh PGLite and exits null, losing all progress. Subsequent runs redo the same 35 minutes. Observed in two consecutive runs (gbrain 0.30.0 broken-postgres run: 712s exit-null; gbrain 0.31.2 PGLite run: 2100s exit-null with 501 pages actually persisted).

Root cause (from /investigate)

Two compounding bugs in bin/gstack-memory-ingest.ts:

1. Subprocess-per-file architecture. The ingest loop at line 911 walks 1,841 files in ~/.gstack/projects/ and spawns two subprocesses per file:

   • gitleaks detect --no-git --source <path> — 46ms cold start (lib/gstack-memory-helpers.ts:157)
   • gbrain put <slug> — 329ms cold start (bin/gstack-memory-ingest.ts:823)
   • Per-file floor: 375ms × 1841 = 690s (11.5 min) of pure subprocess startup before any actual work happens.

2. Kill-no-save timeout. Orchestrator at bin/gstack-gbrain-sync.ts:442 enforces a 35-min timeout. When it fires, spawnSync returns result.status === null, the child gets SIGTERM, and the in-memory ingest state never flushes to ~/.gstack/.transcript-ingest-state.json. Next run starts from the same un-progressed state — explains the redo-everything pattern.

Numbers from the field

Metric	Value	Source
Files in walkAllSources	1,841	find ~/.gstack/projects -type f \( -name "*.md" -o -name "*.jsonl" \)
gbrain put cold start	329ms	time (echo "test" | gbrain put _bench)
gitleaks detect cold start	46ms	time gitleaks detect --no-git --source <small-file>
Theoretical floor (subprocess only)	690s / 11.5 min	375ms × 1841
Observed run time	2100s / 35 min	matches orchestrator timeout exactly
Pages actually persisted	501	gbrain sources list page_count
PGLite growth during run	290 → 386 MB	du -sh ~/.gbrain/brain.pglite

Proposed architecture

Replace the per-file subprocess loop with a prepare-then-batch pipeline:

walkAllSources(ctx)
  → prepareStage (in-process, fast):
       parse transcripts/artifacts
       build PageRecord with custom YAML frontmatter
       gitleaks scan (single subprocess on staging dir)
       write prepared .md to staging dir
  → gbrain import <staging-dir> --no-embed (single subprocess)
  → flush state file with all successes
  → cleanup staging dir

Why gbrain import <dir> is the right batch path

• Already shipped in gbrain CLI (verified: gbrain --help shows import <dir> [--no-embed]).
• Walks dir in-process inside gbrain's own runtime — no subprocess fan-out.
• Honors gbrain's batch-size and embedding-batch tuning.
• gbrain v0.31.2 import did 501 pages + 2906 chunks in 10 seconds during the observed run; the slow part was OUR per-file gbrain put loop above it.

What we keep that the current code does right

• Custom YAML frontmatter injection (title, type, tags) — preserved by writing prepared .md files with frontmatter into the staging dir.
• Secret scanning — preserved, but moved to ONE gitleaks detect --source <staging-dir> call after prepare, before import. Files with findings get redacted or excluded; staging dir guarantees gitleaks sees only the prepared content, not internal gbrain state.
• Partial-transcript detection — preserved in prepare stage; partial files still get a partial: true field in frontmatter.
• Unattributed-transcript filtering — preserved in prepare stage.
• Per-file mtime + sha256 state tracking — preserved; the prepare stage records what got staged, the import-success result records what landed.
• Incremental mode — fileChangedSinceState check stays at the top of the prepare loop.

Migration steps

Step 1: extract preparePages from current ingest loop

Take everything in ingestPass (lines 899-988 of bin/gstack-memory-ingest.ts) between the walk and the gbrainPutPage call. Move into a new function preparePages(args, ctx, state) → { staged: PreparedPage[], skipped, failed }.

Output: list of { slug, body, source_path, mtime_ns, sha256, partial } where body is the full markdown including frontmatter.

Step 2: add staging dir writer

Pure function: writeStaged(prepared, stagingDir) → { written, errors }. Filename: ${slug}.md. Idempotent overwrite.

Staging dir lifecycle:

• Created at ~/.gstack/.staging-ingest-${pid}-${ts}/
• Cleaned in finally block, even on SIGTERM
• One staging dir per ingest pass — never reused across runs

Step 3: single gitleaks pass

Replace per-file secretScanFile(path) calls with one call after prepare: gitleaks detect --no-git --source <staging-dir> --report-format json --report-path -.

Parse JSON output, build Map<slug, findings[]>. Files with findings get removed from staging dir before import (or sanitized in place per existing redaction policy in lib/gstack-memory-helpers.ts).

Step 4: replace gbrainPutPage loop with single import call

const importResult = spawnSync("gbrain", ["import", stagingDir], {
  stdio: ["ignore", "inherit", "inherit"],
  timeout: 30 * 60 * 1000, // generous; whole batch
});

Parse stdout for the Import complete line and the failed count.

Step 5: persist state on partial success

If gbrain import reports imported=N, failed=M, save state for the N successful slugs (not all of them). Failures stay un-state'd so they retry next run, but successes don't redo.

Step 6: SIGTERM handler in gstack-memory-ingest.ts

Wrap main() in:

let interrupted = false;
const flush = () => {
  if (interrupted) return;
  interrupted = true;
  saveState(state); // best-effort flush of whatever's accumulated
  cleanupStagingDir();
  process.exit(143);
};
process.on("SIGTERM", flush);
process.on("SIGINT", flush);

This unblocks the kill-no-save bug independently — even if the batch import runs over the orchestrator timeout, state from the prepare stage survives.

Step 7: orchestrator update

In bin/gstack-gbrain-sync.ts:444:

• Change result.status === 0 to result.status === 0 || (parsedSummary.imported > 0 && parsedSummary.imported >= parsedSummary.skipped + parsedSummary.failed). Treat partial success (most pages imported) as OK, not ERR.
• Surface failed_count and partial_blockers in the stage summary so the user sees Memory ... OK 487/501 imported (14 FILE_TOO_LARGE) instead of ERR exited null.

Step 8: handle FILE_TOO_LARGE specifically

When gbrain reports FILE_TOO_LARGE, log to a new ~/.gstack/.ingest-skip-list.json so the next prepare stage skips that file entirely. Avoids re-staging a file that will always fail. User can review the skip list with a new gstack-memory-ingest --skip-list flag.

Test plan

1. Unit (free, runs in bun test):

   • preparePages against fixture corpus of 50 files: assert YAML correct, partial detection works, unattributed filtered.
   • writeStaged overwrite idempotency.
   • SIGTERM handler flush behavior using a child-process test harness.

2. Integration (free, runs in bun test):

   • End-to-end: prepare → gitleaks → gbrain import on a temp PGLite, assert page_count matches imported count.
   • Partial-success path: inject a deliberate FILE_TOO_LARGE; assert successes still state'd, failure logged to skip list.
   • State preservation across SIGTERM: spawn ingest, kill at midpoint, restart, assert resumed state.

3. Benchmark gate (periodic, paid):

   • Cold run on 1841-file fixture: assert under 8 min.
   • Incremental run (no changes): assert under 60 sec.
   • Test fixture: copy of ~/.gstack/projects/ snapshot for repeatable timing.

Rollback strategy

• New --legacy-ingest flag on gstack-memory-ingest keeps the old per-file path callable for one release cycle.
• If batch path regresses on a real corpus, set gstack-config set memory_ingest_path legacy to revert without redeploy.
• Remove flag + legacy path one minor version after confirming batch is stable.

Risks & open questions for plan-eng-review

1. gbrain import idempotency on overlapping slugs. If a previous run wrote slug X to PGLite with old content, does gbrain import of updated-X overwrite or duplicate? Need to test before relying on it.

2. Frontmatter injection inside gbrain import parser. Current code knows how to inject title/type/tags into existing frontmatter blocks (line 794-821). Does gbrain import honor those fields the same way gbrain put does? Verify in unit test.

3. Staging dir disk pressure. 1841 files × avg ~50KB = ~92MB of staging .md content. Acceptable on dev machines but worth knowing. Alternative: stream prepared content to a tar piped to import (if gbrain supports it) — likely not, ignore for V1.

4. Cross-worktree concurrency. ~/.gstack/.staging-ingest-${pid}-${ts}/ is pid-namespaced so two concurrent /sync-gbrain runs don't collide. But the orchestrator already holds a lock at ~/.gstack/.sync-gbrain.lock so this is belt-and-suspenders. Keep it.

5. The "memory ingest exited null" message. After this change, the orchestrator might still see status=null on real OOM kills or SIGKILL. Should the verdict block be more honest? E.g., ERR memory: killed by signal SIGTERM at 35:00 (timeout).

6. Should we deprecate gbrain put for memory entirely? The legacy path exists for V1.5's put_file migration plan. With batch import working, do we still need single-page put as a fallback for ad-hoc ingestion? Probably yes (for ~/.gstack/.transcript-ingest-state.json updates triggered outside the orchestrator), but worth confirming.

What this isn't

• Not a gbrain CLI change. All work is in gstack.
• Not a CLAUDE.md voice/UX change.
• Not a new user-facing feature. CHANGELOG entry will read: "Memory ingest is ~10× faster on cold runs and survives interruption."

Acceptance criteria

• Cold /sync-gbrain on 1841 files completes in under 8 minutes.
• Incremental /sync-gbrain (no file changes) completes in under 60 seconds.
• SIGTERM mid-run flushes state; next run resumes without redoing successfully-imported files.
• FILE_TOO_LARGE failures don't block sync.last_commit advancement.
• All existing test fixtures (transcripts, learnings, design-docs, ceo-plans) ingest correctly with full frontmatter.
• No regression on partial-transcript or unattributed-transcript handling.