test(agent-eval): measure residual context occupancy, over multi-turn sessions (CG-7)

The A/B arms reported cost, tokens, time and tool counts for one headless
question. They could not report what issue #1500 actually measured: how much of
the context window a tool's responses still occupy once the question is
answered, which every later turn is then charged for.

parse-run.mjs now measures that. Tokens are measured, not estimated: for each
assistant request, input + cache_read + cache_creation is the exact token count
of its whole prompt, so consecutive requests differ by exactly what was appended
between them. That delta is priced against the characters in the gap, calibrated
on gaps that are >=80% tool result. Explore output lands near 2.3 chars/token, so
the usual bytes/4 estimate would have under-counted it by ~40%.

Content also leaves the window, so residual is tracked apart from contributed:
a compact_boundary clears the resident set, and a mid-run context drop is
micro-compaction, which sheds the oldest tool results first and is applied FIFO.

run-all.sh takes "Q1||Q2||Q3" and runs them as one resumed session, one segment
file per turn; parse-run.mjs stitches the segments back together. bench-readme.sh
now runs each README repo as a three-turn session (CG_TURNS=1 restores the
single-question form). parse-bench-readme.mjs reports the arms' retrieval
residual side by side -- codegraph's responses against the without-arm's
Read/Grep/Bash -- in absolute tokens, share of context, and share of window, and
says so explicitly when the rows it aggregated were single-turn.

Two transcript traps are handled and documented at the call site: Claude Code
emits one assistant event per content block, all carrying the same usage (summing
per event double-counts every turn with both thinking and a tool_use), and the
streamed output_tokens is a partial snapshot.

Occupancy lives in parse-run.mjs and is imported by the aggregator rather than
extracted to a module -- a new scripts/agent-eval/*.mjs scores into the
self-query fixture's own corpus and moves its numbers.

scripts/agent-eval/bench-readme.sh

@@ -1,28 +1,47 @@
 #!/usr/bin/env bash
 # Re-run the README "Benchmark Results" A/B (with vs without codegraph) on the
 # current build: the 7 README repos, same queries, RUNS per arm (default 4).
-# Output → /tmp/ab-readme/<repo>/run<n>/run-headless-{with,without}.jsonl
+# Output → /tmp/ab-readme/<repo>/run<n>/run-headless-{with,without}[.tN].jsonl
 # Aggregate with parse-bench-readme.mjs. Repos must be cloned + indexed under
 # $CORPUS (default /tmp/codegraph-corpus) by the build under test.
+#
+# Each row is a THREE-TURN session: the README question, then two follow-ups
+# that stay inside the same flow. Turns 2-3 are where residual context occupancy
+# is actually charged — the first answer's tool output is still in the window,
+# so the arms diverge on how much headroom each left behind. CG_TURNS=1 runs the
+# README question alone (the original single-question A/B).
 set -uo pipefail
 H="$(cd "$(dirname "$0")" && pwd)"
 C="${CORPUS:-/tmp/codegraph-corpus}"
 RUNS="${RUNS:-4}"
+TURNS="${CG_TURNS:-3}"
 ROWS=(
-"vscode|How does the extension host communicate with the main process?"
-"excalidraw|How does Excalidraw render and update canvas elements?"
-"django|How does Django's ORM build and execute a query from a QuerySet?"
-"tokio|How does tokio schedule and run async tasks on its runtime?"
-"okhttp|How does OkHttp process a request through its interceptor chain?"
-"gin|How does gin route requests through its middleware chain?"
-"alamofire|How does Alamofire build, send, and validate a request?"
+"vscode|How does the extension host communicate with the main process?|Where in that path would a message be dropped if the extension host crashes?|What would I need to change to add a new message type to that protocol?"
+"excalidraw|How does Excalidraw render and update canvas elements?|Which part of that path decides whether a full re-render happens or an incremental one?|If I added a new element type, what in that render path would need to change?"
+"django|How does Django's ORM build and execute a query from a QuerySet?|Where in that path is the SQL actually compiled into a string?|What would I change to add a new lookup type to that pipeline?"
+"tokio|How does tokio schedule and run async tasks on its runtime?|Where does a task move between the local and the global queue in that path?|What in that path would I touch to add a per-task instrumentation hook?"
+"okhttp|How does OkHttp process a request through its interceptor chain?|Where in that chain is the connection actually acquired?|What would I change to add a new interceptor stage before the cache?"
+"gin|How does gin route requests through its middleware chain?|Where is the 404 / no-route case handled in that same chain?|What would I change to add a per-route middleware that runs before the global ones?"
+"alamofire|How does Alamofire build, send, and validate a request?|Where does retry / interceptor logic hook into that path?|What would I change to add a new validation step to it?"
 )
-echo "### README A/B START $(date) RUNS=$RUNS"
+echo "### README A/B START $(date) RUNS=$RUNS TURNS=$TURNS"
 for row in "${ROWS[@]}"; do
-  repo="${row%%|*}"; q="${row#*|}"
-  echo "===== $repo ====="
+  repo="${row%%|*}"; rest="${row#*|}"
+  # Take the first $TURNS questions and join them with "||" for run-all.sh.
+  q=""; n=0
+  while [ "$n" -lt "$TURNS" ] && [ -n "$rest" ]; do
+    part="${rest%%|*}"
+    if [ "$rest" = "$part" ]; then rest=""; else rest="${rest#*|}"; fi
+    [ -n "$q" ] && q="$q||"
+    q="$q$part"; n=$((n + 1))
+  done
+  echo "===== $repo ($n turns) ====="
   for run in $(seq 1 "$RUNS"); do
-    AGENT_EVAL_OUT="/tmp/ab-readme/$repo/run$run" bash "$H/run-all.sh" "$C/$repo" "$q" headless 2>&1 | grep -E "exit [0-9]" || echo "  run$run: (no exit line)"
+    out="/tmp/ab-readme/$repo/run$run"
+    mkdir -p "$out"
+    AGENT_EVAL_OUT="$out" bash "$H/run-all.sh" "$C/$repo" "$q" headless > "$out/console.log" 2>&1
+    grep -E "^exit [0-9]" "$out/console.log" | sed 's/^/  /' || echo "  run$run: (no exit line)"
+    grep -E "codegraph +[0-9,]+ tok|→ file-access" "$out/console.log" | sed 's/^/  /' || true
   done
 done
 echo "### README A/B DONE $(date)"

scripts/agent-eval/parse-bench-readme.mjs

@@ -1,71 +1,87 @@
 #!/usr/bin/env node
 // Aggregate the README A/B (bench-readme.sh output): per repo, median of N runs
-// per arm → time, tool calls, tokens, cost, and % saved. Plus an average row.
+// per arm → time, tool calls, tokens, cost, % saved, and RESIDUAL CONTEXT
+// OCCUPANCY. Plus an average row.
 //
 // Tokens = SUM of per-turn assistant `usage` (input + output + cache read +
 // cache creation) — the cumulative "total tokens processed". NOTE: `result.usage`
-// is last-turn-only in current Claude Code, so it under-counts badly; don't use it.
-// `total_cost_usd` and `duration_ms` are already cumulative.
+// is last-turn-only in some Claude Code versions, so reading it alone can
+// under-count badly; parseSession() sums per-segment and dedupes assistant
+// events by message.id (Claude Code emits one event per content block, each
+// carrying the same usage — summing per EVENT double-counts).
+//
+// The occupancy table answers the question "tokens processed" cannot: how much
+// of the window each arm's tool output STILL OCCUPIES when the run ends. Under
+// multi-turn rows that residual is charged against every following turn.
 //
 // Usage: node parse-bench-readme.mjs [/tmp/ab-readme]
-import { readFileSync, existsSync, readdirSync } from 'fs';
+import { existsSync, readdirSync } from 'fs';
 import { join } from 'path';
+import { parseSession } from './parse-run.mjs';
+
 const ROOT = process.argv[2] || '/tmp/ab-readme';
 const REPOS = ['vscode', 'excalidraw', 'django', 'tokio', 'okhttp', 'gin', 'alamofire'];
 
-function parse(file) {
-  if (!existsSync(file)) return null;
-  const L = readFileSync(file, 'utf8').split('\n').filter(Boolean);
-  let tools = 0, reads = 0, grep = 0, cg = 0, tokens = 0, r = null, raced = false;
-  for (const l of L) { let e; try { e = JSON.parse(l); } catch { continue; }
-    if (e.type === 'assistant') {
-      const u = e.message?.usage;
-      if (u) tokens += (u.input_tokens || 0) + (u.output_tokens || 0) + (u.cache_read_input_tokens || 0) + (u.cache_creation_input_tokens || 0);
-      for (const b of (e.message?.content || [])) if (b.type === 'tool_use') {
-        const n = b.name;
-        if (n === 'ToolSearch') continue;
-        tools++;
-        if (n === 'Read') reads++;
-        else if (n === 'Grep' || n === 'Glob') grep++;
-        else if (/codegraph/.test(n)) cg++;
-      }
-    }
-    // MCP cold-start race: the headless agent fired before `codegraph serve --mcp`
-    // finished registering its tools, so early calls returned "No such tool
-    // available" and the agent floundered into grep/Read. That measures CodeGraph's
-    // startup latency, NOT its steady-state value — flag the run so the aggregate
-    // can exclude it (an artifact of headless first-turn timing, not the tool).
-    if (e.type === 'user') for (const b of (Array.isArray(e.message?.content) ? e.message.content : [])) {
-      if (b.type === 'tool_result') {
-        const t = Array.isArray(b.content) ? b.content.map(c => c.text || '').join('') : (b.content || '');
-        if (/No such tool available/.test(t)) raced = true;
-      }
-    }
-    if (e.type === 'result') r = e;
-  }
-  if (!r || r.subtype !== 'success') return null;
-  return { dur: r.duration_ms / 1000, tools, reads, grep, cg, tokens, cost: r.total_cost_usd || 0, raced };
+/** All segment files of one arm's session, in turn order (t1, t2, t3, …). */
+function segments(dir, label) {
+  const first = join(dir, `run-${label}.jsonl`);
+  if (!existsSync(first)) return null;
+  const rest = readdirSync(dir)
+    .map((f) => [f, new RegExp(`^run-${label}\\.t(\\d+)\\.jsonl$`).exec(f)])
+    .filter(([, m]) => m)
+    .sort((a, b) => Number(a[1][1]) - Number(b[1][1]))
+    .map(([f]) => join(dir, f));
+  return [first, ...rest];
+}
+
+function parse(dir, label) {
+  const files = segments(dir, label);
+  if (!files) return null;
+  const s = parseSession(files);
+  if (!s.ok) return null;
+  const o = s.occupancy;
+  return {
+    dur: s.dur, tools: s.tools, reads: s.reads, grep: s.grep, cg: s.cg,
+    tokens: s.processed, cost: s.cost, raced: s.raced, turns: s.turns,
+    segments: files.length,
+    ctx: o.ctxFinal,
+    occCg: o.residual.codegraph,
+    occFile: o.residualFileAccess,
+    // The arm's own retrieval residual: codegraph in the with-arm, Read/Grep/Bash
+    // in the without-arm. Comparing these is the apples-to-apples pair.
+    occSelf: o.residual.codegraph + o.residualFileAccess,
+    occShareCtx: o.ctxFinal > 0 ? ((o.residual.codegraph + o.residualFileAccess) / o.ctxFinal) * 100 : 0,
+    occShareWin: ((o.residual.codegraph + o.residualFileAccess) / o.windowTokens) * 100,
+    window: o.windowTokens,
+  };
 }
+
 const median = (arr) => { const v = [...arr].sort((a, b) => a - b); const n = v.length; return n === 0 ? 0 : n % 2 ? v[(n - 1) / 2] : (v[n / 2 - 1] + v[n / 2]) / 2; };
 const fmtTime = (s) => s >= 60 ? `${Math.floor(s / 60)}m ${Math.round(s % 60)}s` : `${Math.round(s)}s`;
 const fmtTok = (t) => t >= 1e6 ? `${(t / 1e6).toFixed(1)}M` : `${Math.round(t / 1000)}k`;
 const pct = (w, wo) => wo > 0 ? Math.round((1 - w / wo) * 100) : 0;
 
-console.log('repo        n(w/wo)  time WITH→WITHOUT      tools W→WO   tokens W→WO (saved)     cost W→WO (saved)');
-const savings = { cost: [], tokens: [], time: [], tools: [] };
+// Exclude MCP-cold-start-raced WITH runs by default — they measure a startup
+// race, not steady-state value. `CG_INCLUDE_RACED=1` keeps them (to see the raw
+// distribution). The WITHOUT arm has no MCP, so it's never raced.
+const includeRaced = process.env.CG_INCLUDE_RACED === '1';
+const rows = [];
 for (const repo of REPOS) {
   const dir = join(ROOT, repo);
-  const runDirs = existsSync(dir) ? readdirSync(dir).filter(d => /^run\d+$/.test(d)) : [];
-  // Exclude MCP-cold-start-raced WITH runs by default — they measure a startup
-  // race, not steady-state value. `CG_INCLUDE_RACED=1` keeps them (to see the raw
-  // distribution). The WITHOUT arm has no MCP, so it's never raced.
-  const includeRaced = process.env.CG_INCLUDE_RACED === '1';
+  const runDirs = existsSync(dir) ? readdirSync(dir).filter(d => /^run\d+$/.test(d)).sort() : [];
   const W = [], WO = []; let racedExcluded = 0;
   for (const rd of runDirs) {
-    const w = parse(join(dir, rd, 'run-headless-with.jsonl'));
+    const w = parse(join(dir, rd), 'headless-with');
     if (w) { if (w.raced && !includeRaced) racedExcluded++; else W.push(w); }
-    const wo = parse(join(dir, rd, 'run-headless-without.jsonl')); if (wo) WO.push(wo);
+    const wo = parse(join(dir, rd), 'headless-without'); if (wo) WO.push(wo);
   }
+  rows.push({ repo, W, WO, racedExcluded });
+}
+
+// ---- Table 1: the existing throughput view. --------------------------------
+console.log('repo        n(w/wo)  time WITH→WITHOUT      tools W→WO   tokens W→WO (saved)     cost W→WO (saved)');
+const savings = { cost: [], tokens: [], time: [], tools: [] };
+for (const { repo, W, WO, racedExcluded } of rows) {
   if (!W.length || !WO.length) { console.log(`${repo.padEnd(11)} (incomplete: w=${W.length} wo=${WO.length})`); continue; }
   const m = (arr, k) => median(arr.map(x => x[k]));
   const wT = m(W, 'dur'), woT = m(WO, 'dur'), wTok = m(W, 'tokens'), woTok = m(WO, 'tokens');
@@ -82,3 +98,38 @@ for (const repo of REPOS) {
 }
 const avg = (a) => a.length ? Math.round(a.reduce((s, x) => s + x, 0) / a.length) : 0;
 console.log(`\nAVERAGE saved:  cost ${avg(savings.cost)}%  ·  tokens ${avg(savings.tokens)}%  ·  time ${avg(savings.time)}%  ·  tool calls ${avg(savings.tools)}%`);
+
+// ---- Table 2: residual context occupancy. ----------------------------------
+// WITH's retrieval residual is codegraph's tool output; WITHOUT's is Read +
+// Grep/Glob + Bash. Same question, same window — so the pair is comparable.
+const anyMulti = rows.some(({ W, WO }) => [...W, ...WO].some(r => r.segments > 1));
+console.log(`\n\nRESIDUAL CONTEXT OCCUPANCY — retrieval tokens still in the window at end of run`);
+console.log(`(WITH = codegraph responses · WITHOUT = Read + Grep/Glob + Bash responses)`);
+console.log(`${anyMulti ? 'multi-turn sessions' : 'SINGLE-TURN sessions — see the caveat below'}\n`);
+console.log('repo        turns   final ctx W→WO        residual W→WO         % of ctx W→WO     % of window W→WO');
+const occ = { resid: [], shareCtx: [] };
+for (const { repo, W, WO } of rows) {
+  if (!W.length || !WO.length) { console.log(`${repo.padEnd(11)} (incomplete)`); continue; }
+  const m = (arr, k) => median(arr.map(x => x[k]));
+  const wR = m(W, 'occSelf'), woR = m(WO, 'occSelf');
+  const wCtx = m(W, 'ctx'), woCtx = m(WO, 'ctx');
+  const wSc = m(W, 'occShareCtx'), woSc = m(WO, 'occShareCtx');
+  const wSw = m(W, 'occShareWin'), woSw = m(WO, 'occShareWin');
+  occ.resid.push(pct(wR, woR)); occ.shareCtx.push(pct(wSc, woSc));
+  console.log(
+    `${repo.padEnd(11)} ${String(median(W.map(x => x.turns)) + '/' + median(WO.map(x => x.turns))).padEnd(7)} ` +
+    `${(fmtTok(wCtx) + '→' + fmtTok(woCtx)).padEnd(21)}` +
+    `${(fmtTok(wR) + '→' + fmtTok(woR) + ' (' + pct(wR, woR) + '%)').padEnd(22)}` +
+    `${(wSc.toFixed(1) + '%→' + woSc.toFixed(1) + '%').padEnd(18)}` +
+    `${wSw.toFixed(1)}%→${woSw.toFixed(1)}%`
+  );
+}
+console.log(`\nAVERAGE: retrieval residual ${avg(occ.resid)}% lower with codegraph  ·  share-of-context ${avg(occ.shareCtx)}% lower`);
+if (!anyMulti) {
+  console.log(
+    `\nCAVEAT: every row above is a SINGLE-turn session, so the residual is measured at the\n` +
+    `moment the one question is answered. Occupancy is a cost that compounds over the turns\n` +
+    `that FOLLOW; a single-turn number does not settle it. Re-run with "||"-separated\n` +
+    `follow-ups (see run-all.sh) to measure the regime this metric is actually about.`
+  );
+}

scripts/agent-eval/parse-run.mjs

@@ -1,45 +1,321 @@
 #!/usr/bin/env node
-// Parse a Claude Code stream-json run log: tool-call sequence + token usage.
+// Parse Claude Code stream-json run log(s): tool-call sequence, token usage, and
+// RESIDUAL CONTEXT OCCUPANCY — how many tokens of the context window each tool
+// family's responses still occupy when the run ends.
+//
+// Usage: parse-run.mjs <run.jsonl> [run.t2.jsonl ...]
+//   Multiple files = one multi-turn session's segments, IN ORDER (run-all.sh
+//   writes run-<label>.jsonl, run-<label>.t2.jsonl, … for a `Q1||Q2||Q3` set).
+//   `--resume` does not replay prior messages, so the segments concatenate
+//   cleanly and token accounting carries across the boundary.
+//
+// ---------------------------------------------------------------------------
+// Why occupancy, and how it's measured
+// ---------------------------------------------------------------------------
+// A single-question A/B reports cost/tokens/time/tool-calls for ONE answer. It
+// cannot see what issue #1500 measured: a tool response stays in the window for
+// everything that follows, so it is charged against every later turn's headroom.
+// That is a per-session cost our single-question runs structurally miss.
+//
+// Tokens are MEASURED, not estimated at bytes/4. For assistant request k,
+//   ctx_k = usage.input_tokens + cache_read_input_tokens + cache_creation_input_tokens
+// is the exact token count of that request's whole prompt. So
+//   gap_k = ctx_k - ctx_{k-1}
+// is exactly the tokens appended since the previous request: the previous
+// assistant output (thinking + text + tool_use JSON) plus the tool_results and
+// user text that followed it. We split gap_k across those blocks in proportion
+// to their characters, which attributes each tool_result its measured share.
+// (Measured on real runs, explore output lands near 2.3 chars/token — bytes/4
+// under-counts it by ~40%, which is why the estimate isn't good enough.)
+//
+// Two traps this file works around, both verified against real logs:
+//   * Claude Code emits ONE assistant event PER CONTENT BLOCK, all carrying the
+//     same message.id and the same `usage`. Summing usage per event double-counts
+//     every turn that emits both thinking and a tool_use — dedupe by message.id.
+//   * The streamed `output_tokens` is a partial snapshot (observed `out=2` on a
+//     turn that really generated ~1100). Never trust it; the char-proportional
+//     split doesn't need it.
+//
+// Residual ≠ contributed. Content leaves the window two ways, and both are
+// tracked: a `compact_boundary` system event (everything prior is replaced by a
+// summary) and micro-compaction (ctx drops mid-run — oldest tool results are
+// dropped first, so eviction is applied FIFO).
 import { readFileSync } from 'fs';
-const file = process.argv[2];
-const lines = readFileSync(file, 'utf8').split('\n').filter(Boolean);
-
-const toolCalls = [];
-let result = null;
-let initTools = null;
-
-for (const line of lines) {
-  let ev;
-  try { ev = JSON.parse(line); } catch { continue; }
-  if (ev.type === 'system' && ev.subtype === 'init') {
-    initTools = (ev.tools || []).filter(t => /codegraph/.test(t));
+import { pathToFileURL } from 'url';
+
+// Nominal window for the share-of-window column. Override for a [1m] context.
+const WINDOW_TOKENS = Number(process.env.CG_WINDOW_TOKENS || 200_000);
+const CHARS_PER_TOKEN_FALLBACK = 3.0;
+
+/** Which tool family a tool_use belongs to. */
+function familyOf(name) {
+  if (/codegraph/.test(name)) return 'codegraph';
+  if (name === 'Read' || name === 'NotebookRead') return 'read';
+  if (name === 'Grep' || name === 'Glob') return 'search';
+  if (name === 'Bash' || name === 'BashOutput') return 'bash';
+  return 'other';
+}
+const FAMILIES = ['codegraph', 'read', 'search', 'bash', 'other'];
+// The without-arm's way of getting the same bytes: reading and searching files.
+const FILE_ACCESS = ['read', 'search', 'bash'];
+
+const textOf = (content) =>
+  Array.isArray(content) ? content.map((c) => c.text ?? (typeof c === 'string' ? c : JSON.stringify(c))).join('')
+    : typeof content === 'string' ? content
+      : content == null ? '' : JSON.stringify(content);
+
+/** Characters an assistant content block occupies once it is back in the prompt. */
+function assistantBlockChars(b) {
+  if (b.type === 'text') return (b.text || '').length;
+  if (b.type === 'thinking') return (b.thinking || '').length;
+  if (b.type === 'tool_use') return JSON.stringify(b.input ?? {}).length + (b.name || '').length;
+  return JSON.stringify(b).length;
+}
+
+/**
+ * Parse one session (its segment files, in order) into tool + occupancy stats.
+ * Exported so parse-bench-readme.mjs can aggregate without duplicating any of
+ * this — deliberately NOT a separate module file: a new scripts/agent-eval/*.mjs
+ * scores into the self-query eval fixture's own corpus and moves its numbers.
+ */
+export function parseSession(files) {
+  const events = [];
+  for (const f of files) {
+    for (const line of readFileSync(f, 'utf8').split('\n')) {
+      if (!line) continue;
+      try { events.push(JSON.parse(line)); } catch { /* partial line */ }
+    }
   }
-  if (ev.type === 'assistant' && ev.message?.content) {
-    for (const block of ev.message.content) {
-      if (block.type === 'tool_use') {
-        let detail = '';
-        if (block.name === 'Task') detail = ` [subagent_type=${block.input?.subagent_type ?? '?'}] ${(block.input?.description ?? '').slice(0,40)}`;
-        else if (/codegraph/.test(block.name)) detail = ` ${JSON.stringify(block.input?.query ?? block.input?.task ?? block.input?.symbol ?? '').slice(0,60)}`;
-        else if (block.name === 'Bash') detail = ` ${(block.input?.command ?? '').slice(0,50)}`;
-        else if (block.name === 'Read') detail = ` ${(block.input?.file_path ?? '').split('/').slice(-1)[0]}`;
-        toolCalls.push(`${block.name}${detail}`);
+
+  const toolCalls = [];          // display sequence
+  const nameById = new Map();    // tool_use_id -> tool name
+  const counts = {};             // tool name -> calls
+  let initTools = null, result = null, raced = false;
+  const results = [];  // one `result` event per session segment (multi-turn)
+  let compactions = 0;
+
+  // A timeline of everything appended to the context, in order. `req` entries
+  // are assistant requests (carrying that request's ctx); `add` entries are
+  // characters appended (assistant output blocks, tool results, user text).
+  const timeline = [];
+  const seenMsgIds = new Set();
+
+  for (const ev of events) {
+    if (ev.type === 'system' && ev.subtype === 'init') {
+      initTools = (ev.tools || []).filter((t) => /codegraph/.test(t));
+    }
+    if (ev.type === 'system' && (ev.subtype === 'compact_boundary' || ev.subtype === 'compaction')) {
+      compactions++;
+      timeline.push({ kind: 'compact' });
+    }
+    if (ev.type === 'assistant' && ev.message) {
+      const id = ev.message.id;
+      // One event per content block, same id + same usage: count usage once,
+      // but take the content blocks from every event that carries the id.
+      if (id && !seenMsgIds.has(id)) {
+        seenMsgIds.add(id);
+        const u = ev.message.usage || {};
+        const ctx = (u.input_tokens || 0) + (u.cache_read_input_tokens || 0) + (u.cache_creation_input_tokens || 0);
+        timeline.push({ kind: 'req', ctx, out: u.output_tokens || 0 });
+      }
+      for (const b of ev.message.content || []) {
+        timeline.push({ kind: 'add', family: null, chars: assistantBlockChars(b) });
+        if (b.type === 'tool_use') {
+          nameById.set(b.id, b.name);
+          counts[b.name] = (counts[b.name] || 0) + 1;
+          let detail = '';
+          if (b.name === 'Task') detail = ` [subagent_type=${b.input?.subagent_type ?? '?'}] ${(b.input?.description ?? '').slice(0, 40)}`;
+          else if (/codegraph/.test(b.name)) detail = ` ${JSON.stringify(b.input?.query ?? b.input?.task ?? b.input?.symbol ?? '').slice(0, 60)}`;
+          else if (b.name === 'Bash') detail = ` ${(b.input?.command ?? '').slice(0, 50)}`;
+          else if (b.name === 'Read') detail = ` ${(b.input?.file_path ?? '').split('/').slice(-1)[0]}`;
+          toolCalls.push(`${b.name}${detail}`);
+        }
       }
     }
+    if (ev.type === 'user' && ev.message) {
+      const content = ev.message.content;
+      if (Array.isArray(content)) {
+        for (const b of content) {
+          if (b.type === 'tool_result') {
+            const t = textOf(b.content);
+            // MCP cold-start race: the agent fired before `serve --mcp` had
+            // registered its tools, so it floundered into grep/Read. That
+            // measures startup latency, not steady-state value — flag it.
+            if (/No such tool available/.test(t)) raced = true;
+            const name = nameById.get(b.tool_use_id) || '';
+            timeline.push({ kind: 'add', family: familyOf(name), chars: t.length, tool: name });
+          } else {
+            timeline.push({ kind: 'add', family: null, chars: textOf([b]).length });
+          }
+        }
+      } else if (typeof content === 'string') {
+        timeline.push({ kind: 'add', family: null, chars: content.length });
+      }
+    }
+    if (ev.type === 'result') { result = ev; results.push(ev); }
+  }
+
+  // ---- Pass 1: chars/token, calibrated on tool-result-dominated gaps. ------
+  // Splitting a gap in proportion to characters over-attributes to tool results
+  // whenever the assistant's own output is under-represented in the transcript
+  // (redacted/empty thinking blocks are the common case — a gap whose only
+  // visible chars were a 73-char tool_result charged it the whole 830-token
+  // delta, 5.5 tok/char). So calibrate the ratio on gaps that are ≥80% tool
+  // result by characters, then price every result at that ratio.
+  const reqIdx = timeline.map((t, i) => (t.kind === 'req' ? i : -1)).filter((i) => i >= 0);
+  const gaps = [];
+  for (let k = 1; k < reqIdx.length; k++) {
+    const prev = timeline[reqIdx[k - 1]], cur = timeline[reqIdx[k]];
+    let chars = 0, toolChars = 0, compacted = false;
+    const byFamily = {};
+    for (let i = reqIdx[k - 1] + 1; i < reqIdx[k]; i++) {
+      const t = timeline[i];
+      if (t.kind === 'compact') { compacted = true; continue; }
+      if (t.kind !== 'add') continue;
+      chars += t.chars;
+      if (t.family) { toolChars += t.chars; byFamily[t.family] = (byFamily[t.family] || 0) + t.chars; }
+    }
+    gaps.push({ delta: cur.ctx - prev.ctx, chars, toolChars, byFamily, compacted });
+  }
+  let sumD = 0, sumC = 0;
+  for (const g of gaps) {
+    if (g.compacted || g.delta <= 0) continue;
+    if (g.chars > 500 && g.toolChars / g.chars >= 0.8) { sumD += g.delta; sumC += g.toolChars; }
+  }
+  if (sumD === 0) { // no clean gap — fall back to every growing gap, all chars
+    for (const g of gaps) if (!g.compacted && g.delta > 0 && g.chars > 0) { sumD += g.delta; sumC += g.chars; }
+  }
+  const charsPerToken = sumD > 0 ? sumC / sumD : CHARS_PER_TOKEN_FALLBACK;
+  const calibrated = sumD > 0;
+
+  // ---- Pass 2: attribute gap tokens, then apply evictions FIFO. ------------
+  const contributed = Object.fromEntries(FAMILIES.map((f) => [f, 0]));
+  const resultChars = Object.fromEntries(FAMILIES.map((f) => [f, 0]));
+  const resultCount = Object.fromEntries(FAMILIES.map((f) => [f, 0]));
+  for (const t of timeline) if (t.kind === 'add' && t.family) { resultChars[t.family] += t.chars; resultCount[t.family]++; }
+
+  let queue = [];        // resident contributions, oldest first
+  let evicted = 0;
+  const evict = (tokens) => {
+    let left = tokens;
+    while (left > 0 && queue.length) {
+      const head = queue[0];
+      if (head.tokens <= left) { left -= head.tokens; evicted += head.tokens; queue.shift(); }
+      else { head.tokens -= left; evicted += left; left = 0; }
+    }
+  };
+
+  for (const g of gaps) {
+    if (g.compacted) {
+      // Everything before the boundary is gone; the summary replaces it.
+      evicted += queue.reduce((s, q) => s + q.tokens, 0);
+      queue = [];
+    }
+    let toolTokens = 0;
+    for (const [fam, ch] of Object.entries(g.byFamily)) {
+      const tok = ch / charsPerToken;
+      toolTokens += tok;
+      contributed[fam] += tok;
+      queue.push({ family: fam, tokens: tok });
+    }
+    // The gap grew by `delta`; the tool results account for `toolTokens` of it.
+    // A shortfall means the window also shed content — micro-compaction drops
+    // the OLDEST tool results first, so evict FIFO. The tolerance keeps
+    // attribution noise (a run-level ratio priced against one gap's delta,
+    // typically ±2%) from reading as an eviction; real shedding is thousands.
+    const shortfall = toolTokens - g.delta;
+    if (!g.compacted && shortfall > Math.max(200, toolTokens * 0.05)) evict(shortfall);
   }
-  if (ev.type === 'result') result = ev;
+
+  const residual = Object.fromEntries(FAMILIES.map((f) => [f, 0]));
+  for (const q of queue) residual[q.family] += q.tokens;
+
+  const ctxFinal = reqIdx.length ? timeline[reqIdx[reqIdx.length - 1]].ctx : 0;
+  // Multi-turn: duration/cost/tokens are per-segment, so sum them. `result.usage`
+  // is cumulative WITHIN a segment (verified: its in+cache+out equals the sum of
+  // that segment's per-request prompts), so summing segments is correct and does
+  // NOT double-count. It is a "tokens processed" figure — every request re-counts
+  // the whole prefix — which is exactly why it can't answer the occupancy question.
+  const sumUsage = (k) => results.reduce((s, r) => s + (r.usage?.[k] || 0), 0);
+  const processed = sumUsage('input_tokens') + sumUsage('cache_read_input_tokens')
+    + sumUsage('cache_creation_input_tokens') + sumUsage('output_tokens');
+
+  return {
+    files, toolCalls, counts, initTools, result, results, raced,
+    ok: results.length > 0 && results.every((r) => r.subtype === 'success'),
+    turns: reqIdx.length,
+    tools: toolCalls.filter((t) => !t.startsWith('ToolSearch')).length,
+    reads: counts.Read || 0,
+    grep: (counts.Grep || 0) + (counts.Glob || 0),
+    cg: Object.entries(counts).filter(([n]) => /codegraph/.test(n)).reduce((s, [, v]) => s + v, 0),
+    dur: results.reduce((s, r) => s + (r.duration_ms || 0), 0) / 1000,
+    cost: results.reduce((s, r) => s + (r.total_cost_usd || 0), 0),
+    processed,
+    occupancy: {
+      ctxFinal, windowTokens: WINDOW_TOKENS,
+      charsPerToken, calibrated, compactions, evicted: Math.round(evicted),
+      residual: Object.fromEntries(FAMILIES.map((f) => [f, Math.round(residual[f])])),
+      contributed: Object.fromEntries(FAMILIES.map((f) => [f, Math.round(contributed[f])])),
+      chars: resultChars, results: resultCount,
+      residualFileAccess: Math.round(FILE_ACCESS.reduce((s, f) => s + residual[f], 0)),
+      contributedFileAccess: Math.round(FILE_ACCESS.reduce((s, f) => s + contributed[f], 0)),
+      charsFileAccess: FILE_ACCESS.reduce((s, f) => s + resultChars[f], 0),
+    },
+  };
 }
 
-console.log(`\n=== ${file.split('/').pop()} ===`);
-console.log(`codegraph tools exposed: ${initTools ? initTools.length : '?'}`);
-console.log(`\nTool calls (${toolCalls.length}):`);
-const counts = {};
-for (const tc of toolCalls) { const n = tc.split(' ')[0]; counts[n] = (counts[n]||0)+1; }
-console.log('  by type:', JSON.stringify(counts));
-toolCalls.forEach((tc, i) => console.log(`  ${i+1}. ${tc}`));
-
-if (result) {
-  const u = result.usage || {};
-  const totalIn = (u.input_tokens||0) + (u.cache_read_input_tokens||0) + (u.cache_creation_input_tokens||0);
-  console.log(`\nResult: ${result.subtype} | duration ${(result.duration_ms/1000).toFixed(0)}s | turns ${result.num_turns}`);
-  console.log(`  tokens: in=${totalIn} out=${u.output_tokens||0} | cost $${(result.total_cost_usd||0).toFixed(3)}`);
+/** The occupancy block, as printed under a run and reused by the aggregator. */
+export function formatOccupancy(s, indent = '  ') {
+  const o = s.occupancy;
+  const n = (x) => x.toLocaleString('en-US');
+  const pctCtx = (t) => (o.ctxFinal > 0 ? ((t / o.ctxFinal) * 100).toFixed(1) : '0.0');
+  const pctWin = (t) => ((t / o.windowTokens) * 100).toFixed(1);
+  const rows = [];
+  const row = (label, tok, chars, results) => rows.push(
+    `${indent}  ${label.padEnd(18)}${(n(tok) + ' tok').padStart(12)}  ${(pctCtx(tok) + '%').padStart(6)} of ctx  ` +
+    `${(pctWin(tok) + '%').padStart(6)} of ${Math.round(o.windowTokens / 1000)}k win` +
+    (chars !== undefined ? `   (${n(chars)} chars, ${results} result${results === 1 ? '' : 's'})` : '')
+  );
+  const out = [`${indent}Residual context occupancy at end of run:`];
+  out.push(`${indent}  ${'final context'.padEnd(18)}${(n(o.ctxFinal) + ' tok').padStart(12)}  ${(pctWin(o.ctxFinal) + '%').padStart(6)} of ${Math.round(o.windowTokens / 1000)}k window`);
+  row('codegraph', o.residual.codegraph, o.chars.codegraph, o.results.codegraph);
+  row('Read', o.residual.read, o.chars.read, o.results.read);
+  row('Grep/Glob', o.residual.search, o.chars.search, o.results.search);
+  row('Bash', o.residual.bash, o.chars.bash, o.results.bash);
+  row('→ file-access', o.residualFileAccess, o.charsFileAccess,
+    o.results.read + o.results.search + o.results.bash);
+  row('other tools', o.residual.other, o.chars.other, o.results.other);
+  const toolTotal = Object.values(o.residual).reduce((a, b) => a + b, 0);
+  row('base (prompt+prose)', Math.max(0, o.ctxFinal - toolTotal));
+  out.push(...rows);
+  const dropped = o.contributed.codegraph + o.contributedFileAccess + o.contributed.other
+    - (o.residual.codegraph + o.residualFileAccess + o.residual.other);
+  out.push(
+    `${indent}  measure: ${o.charsPerToken.toFixed(2)} chars/tok ${o.calibrated ? 'measured' : '(FALLBACK — no clean gap to calibrate on)'}` +
+    ` · turns ${s.turns} · compactions ${o.compactions}` +
+    (o.evicted > 0 || dropped > 1 ? ` · evicted ${n(o.evicted)} tok` : '')
+  );
+  return out.join('\n');
+}
+
+// ---------------------------------------------------------------------------
+const isMain = process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href;
+if (isMain) {
+  const files = process.argv.slice(2).filter((a) => !a.startsWith('--'));
+  if (!files.length) { console.error('usage: parse-run.mjs <run.jsonl> [run.t2.jsonl ...]'); process.exit(1); }
+  const s = parseSession(files);
+
+  console.log(`\n=== ${files.map((f) => f.split('/').pop()).join(' + ')} ===`);
+  console.log(`codegraph tools exposed: ${s.initTools ? s.initTools.length : '?'}${s.raced ? '  [MCP COLD-START RACE — tool call hit "No such tool available"]' : ''}`);
+  console.log(`\nTool calls (${s.toolCalls.length}):`);
+  console.log('  by type:', JSON.stringify(s.counts));
+  s.toolCalls.forEach((tc, i) => console.log(`  ${i + 1}. ${tc}`));
+
+  if (s.result) {
+    const seg = s.results.length > 1 ? ` | ${s.results.length} segments (${s.results.map((r) => r.subtype).join(',')})` : '';
+    console.log(`\nResult: ${s.result.subtype} | duration ${s.dur.toFixed(0)}s | turns ${s.turns}${seg}`);
+    console.log(`  tokens processed: ${s.processed.toLocaleString('en-US')} | cost $${s.cost.toFixed(3)}`);
+  }
+  console.log('');
+  console.log(formatOccupancy(s));
 }

scripts/agent-eval/run-all.sh

@@ -5,6 +5,15 @@
 # without = empty MCP. Built-in Read/Grep/Bash stay available in both arms.
 #
 # Usage: run-all.sh <repo-path> "<question>" [headless|tmux|all]
+#
+# MULTI-TURN: separate questions with "||" to run them as ONE session —
+#   run-all.sh <repo> "How does X work?||Where is Y handled in that path?"
+# Turn 1 runs normally; every later turn `--resume`s the same session, so the
+# earlier turns' tool output is still in the window (that is the whole point:
+# residual context occupancy, the cost a single-question run cannot see).
+# Segments land in run-<label>.jsonl, run-<label>.t2.jsonl, … and parse-run.mjs
+# stitches them back into one session.
+#
 # Env:   CG_BIN          codegraph binary (default: command -v codegraph)
 #        AGENT_EVAL_OUT  output dir (default: /tmp/agent-eval)
 #        MODEL / EFFORT  claude model/effort (default: sonnet / high — the
@@ -14,6 +23,15 @@ set -uo pipefail
 REPO="${1:?usage: run-all.sh <repo-path> \"<question>\" [headless|tmux|all]}"
 Q="${2:?question required}"
 MODE="${3:-headless}"
+
+# Split "Q1||Q2||Q3" into turns (kept bash-3.2-safe: macOS ships 3.2).
+TURNS=()
+rest="$Q"
+while [ "$rest" != "${rest#*||}" ]; do
+  TURNS+=("${rest%%||*}")
+  rest="${rest#*||}"
+done
+TURNS+=("$rest")
 CG_BIN="${CG_BIN:-$(command -v codegraph)}"
 OUT="${AGENT_EVAL_OUT:-/tmp/agent-eval}"
 HARNESS="$(cd "$(dirname "$0")" && pwd)"
@@ -37,23 +55,52 @@ echo '{"mcpServers":{}}' > "$OUT/mcp-empty.json"
 
 echo "###### codegraph: $CG_BIN"
 echo "###### repo:      $REPO"
-echo "###### question:  $Q"
+echo "###### turns:     ${#TURNS[@]}"
+for t in "${TURNS[@]}"; do echo "######   - $t"; done
 echo
 
-# Headless arm: claude -p with stream-json -> exact tool sequence + tokens/cost.
+# Pull the session id out of a segment's result event so the next turn can
+# --resume it (rather than minting a --session-id, which needs a valid uuid).
+session_id_of() {
+  node -e '
+    const fs=require("fs");
+    for (const l of fs.readFileSync(process.argv[1],"utf8").split("\n").reverse()) {
+      if (!l) continue; let e; try { e=JSON.parse(l) } catch { continue }
+      if (e.session_id) { console.log(e.session_id); break }
+    }' "$1" 2>/dev/null
+}
+
+# Headless arm: claude -p with stream-json -> exact tool sequence + tokens/cost
+# + residual context occupancy. One session, one segment file per turn.
 headless() {
   local label="$1" cfg="$2"
   echo "############################## HEADLESS [$label] ##############################"
-  ( cd "$REPO" && claude -p "$Q" \
-      --output-format stream-json --verbose \
-      --permission-mode bypassPermissions \
-      --model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" \
-      --max-budget-usd 4 \
-      --strict-mcp-config --mcp-config "$cfg" \
-      > "$OUT/run-$label.jsonl" 2>"$OUT/run-$label.err" )
-  echo "exit $? -> $OUT/run-$label.jsonl ($(wc -l < "$OUT/run-$label.jsonl" | tr -d ' ') lines)"
+  local sid="" seg=0 out="" files=()
+  : > "$OUT/run-$label.err"
+  for q in "${TURNS[@]}"; do
+    seg=$((seg + 1))
+    out="$OUT/run-$label.jsonl"
+    [ "$seg" -gt 1 ] && out="$OUT/run-$label.t$seg.jsonl"
+    local resume=()
+    [ -n "$sid" ] && resume=(--resume "$sid")
+    ( cd "$REPO" && claude -p "$q" \
+        --output-format stream-json --verbose \
+        --permission-mode bypassPermissions \
+        --model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" \
+        --max-budget-usd 4 \
+        --strict-mcp-config --mcp-config "$cfg" \
+        ${resume[@]+"${resume[@]}"} \
+        </dev/null > "$out" 2>>"$OUT/run-$label.err" )
+    echo "exit $? -> $out ($(wc -l < "$out" | tr -d ' ') lines) [turn $seg/${#TURNS[@]}]"
+    files+=("$out")
+    sid="$(session_id_of "$out")"
+    if [ -z "$sid" ] && [ "$seg" -lt "${#TURNS[@]}" ]; then
+      echo "  WARN: no session_id in $out — later turns would start a FRESH context; stopping this arm"
+      break
+    fi
+  done
   tail -2 "$OUT/run-$label.err" 2>/dev/null
-  node "$HARNESS/parse-run.mjs" "$OUT/run-$label.jsonl" 2>&1 || true
+  node "$HARNESS/parse-run.mjs" "${files[@]}" 2>&1 || true
   echo
 }
 
@@ -65,11 +112,11 @@ fi
 if [ "$MODE" = tmux ] || [ "$MODE" = all ]; then
   echo "############################## INTERACTIVE [with] ##############################"
   CLAUDE_EXTRA_ARGS="--model ${MODEL:-sonnet} --effort ${EFFORT:-high} --strict-mcp-config --mcp-config $OUT/mcp-codegraph.json" \
-    bash "$HARNESS/itrun.sh" "$REPO" "int-with" "$Q" 2>&1 || echo "[itrun WITH failed]"
+    bash "$HARNESS/itrun.sh" "$REPO" "int-with" "${TURNS[0]}" 2>&1 || echo "[itrun WITH failed]"
   echo
   echo "############################## INTERACTIVE [without] ##############################"
   CLAUDE_EXTRA_ARGS="--model ${MODEL:-sonnet} --effort ${EFFORT:-high} --strict-mcp-config --mcp-config $OUT/mcp-empty.json" \
-    bash "$HARNESS/itrun.sh" "$REPO" "int-without" "$Q" 2>&1 || echo "[itrun WITHOUT failed]"
+    bash "$HARNESS/itrun.sh" "$REPO" "int-without" "${TURNS[0]}" 2>&1 || echo "[itrun WITHOUT failed]"
   echo
 fi
 echo "############################## RUN-ALL COMPLETE ##############################"