test(agent-eval): bucket every explore by what the agent did next (CG-8)

The agent's next action after a codegraph_explore is free ground truth about
whether the response was enough, and the harness was discarding it. Every run
now bucketed: explored again (insufficient), Read a file we returned
(allocation -- right file, wrong bytes), Read a file we did not return (recall),
Grep/Glob (recall, weak), or moved on (sufficient). The buckets are chosen so
each one names a distinct fix.

The classifier lives in parse-run.mjs next to the occupancy math and takes raw
events, so parse-session.mjs reuses it for interactive runs -- no new
scripts/agent-eval/*.mjs, which would score into the self-query eval fixture's
corpus.

Four rules, three of them found by validating against real transcripts rather
than reasoned up front:

  * A call in the SAME assistant message as the explore predates its response,
    so it is not a verdict on it. Stepped over, counted as `concurrent`.
  * ToolSearch/TodoWrite carry no signal; the call behind them is the verdict.
  * SUBAGENTS ARE A SEPARATE THREAD. Claude Code interleaves a subagent's calls
    into the same stream under parent_tool_use_id -- verified on a live
    excalidraw run where a delegated search's greps landed between the parent's
    own calls. Matching reactions across threads scored the subagent's grep as
    the parent's verdict on an explore it never saw. A delegation is judged by
    what the subagent did FIRST: before that, the same run reported 33%
    sufficient while the subagent was off grepping for the file, which is the
    one direction of error a tuning metric must not have. In interactive
    sessions the subagent is a separate FILE instead, so parse-session.mjs
    stitches the threads back with the toolUseId in agent-*.meta.json.
  * A re-read of a file an EARLIER explore shipped is still allocation, not
    recall -- filing it as recall aims the fix at the wrong end of the pipeline.

Shell file access counts too (`sed -n 100,200p f` reads, `grep`/`find` search),
since both arms have Bash and counting only the Read tool would score those
explores as sufficient. A heredoc or redirect is writing, not reading.

Validated by hand on cg22/ab-express/run-baseline-1 (explore, explore, Read of
lib/response.js which explore #2 returned -- the #1500 allocation bug as a
bucket instead of a hunch; the new-build arm is 100% sufficient) and on
cg15/ab-express/run-new-2 (four explores, the last returning lib/utils.js which
the agent then read at offset 195). Swept over all 76 A/B logs on this machine:
0 crashes, 176 calls bucketed. --selftest covers every bucket, both thread
rules, delegation, shell reads/searches and errored calls: 46/46.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

scripts/agent-eval/parse-run.mjs

@@ -9,6 +9,9 @@
 //   `--resume` does not replay prior messages, so the segments concatenate
 //   cleanly and token accounting carries across the boundary.
 //
+//   Every run also reports EXPLORE SUFFICIENCY — each codegraph_explore call
+//   bucketed by what the agent did next (see classifySufficiency).
+//
 //   `--envelope` additionally reports how the codegraph_explore responses were
 //   DIVIDED across files — the per-file share of the source envelope (#1500).
 //   `--answer <glob>` (repeatable, implies --envelope) marks the files that
@@ -300,6 +303,8 @@ export function parseSession(files) {
   return {
     files, toolCalls, counts, initTools, result, results, raced, cliCalls, cliContaminated,
     exploreTexts,
+    // What the agent did after each explore — the free sufficiency signal (CG-8).
+    sufficiency: classifySufficiency(events),
     ok: results.length > 0 && results.every((r) => r.subtype === 'success'),
     turns: reqIdx.length,
     tools: toolCalls.filter((t) => !t.startsWith('ToolSearch')).length,
@@ -420,6 +425,236 @@ export function formatEnvelope(exploreTexts, answerGlobs = [], indent = '  ') {
   return out.join('\n');
 }
 
+// ---------------------------------------------------------------------------
+// Explore sufficiency (CG-8)
+// ---------------------------------------------------------------------------
+// The agent's NEXT action after a codegraph_explore is free ground truth about
+// whether that response was enough. The buckets are chosen so each one maps to
+// a distinct fix:
+//
+//   another codegraph call      insufficient — the response did not answer
+//   Read of a file we RETURNED  allocation bug — right file, wrong bytes
+//   Read of a file we did NOT   recall bug — the file never surfaced
+//   Grep/Glob                   recall bug (weaker: the agent is still hunting)
+//   anything else / no tool     sufficient — the agent moved on
+//
+// Three rules keep this honest:
+//   * Only a call issued in a LATER assistant message counts as a reaction. A
+//     Read fired in the same message as the explore was issued before its
+//     response existed, so it cannot be a verdict on it (those are counted
+//     separately as `concurrent`).
+//   * ToolSearch/TodoWrite are stepped over: loading a deferred tool schema or
+//     ticking a checklist says nothing about the response.
+//   * SUBAGENT CALLS ARE A SEPARATE THREAD. Claude Code interleaves a subagent's
+//     tool calls into the same stream, tagged `parent_tool_use_id` — verified on
+//     a real run where a delegated search's greps landed between the parent's
+//     own calls. Reactions are matched within one thread, or the subagent's
+//     first grep would be scored as the parent's verdict on an explore it never
+//     saw.
+//
+// A delegation (`Agent`/`Task`) is judged by what the SUBAGENT did first, since
+// that thread is right there in the transcript. Scoring the delegation itself as
+// "moved on" would have called this run sufficient while the subagent was off
+// grepping for the file — the one direction of error a tuning metric must not
+// have. A delegation that never runs a tool stays "moved on".
+
+/** Tools that carry no signal about whether the previous response was enough. */
+const TRANSPARENT_TOOLS = new Set(['ToolSearch', 'TodoWrite']);
+/** Tools that hand the work to a subagent whose thread we then judge instead. */
+const DELEGATION_TOOLS = new Set(['Agent', 'Task']);
+
+/** Buckets, worst → best. Labels double as the summary rows. */
+const SUFFICIENCY = [
+  ['explore_again', 'explore again', 'insufficient: did not answer'],
+  ['read_returned', 'Read a file we returned', 'allocation: right file, wrong bytes'],
+  ['read_missed', 'Read a file we did not return', 'recall: file never surfaced'],
+  ['search', 'Grep/Glob', 'recall (weak): still hunting for the file'],
+  ['sufficient', 'moved on / answered', 'sufficient'],
+];
+const SUFFICIENCY_KEYS = SUFFICIENCY.map(([k]) => k);
+
+// Shell equivalents of Read and of Grep. Both arms have Bash, and on small
+// repos an agent reaches for `sed -n 100,200p file` as readily as for Read —
+// counting only the Read tool would score those explores as sufficient.
+const BASH_READ_RE = /(?:^|[;&|]|\$\(|`)\s*(?:sudo\s+)?(?:cat|bat|head|tail|less|more|nl|sed|awk)\s+([^\n|;&]*)/;
+const BASH_SEARCH_RE = /(?:^|[;&|]|\$\(|`)\s*(?:sudo\s+)?(?:grep|egrep|fgrep|rg|ag|ack|find|fd|ls|tree)\b/;
+
+/** What a Bash command is really doing, as far as retrieval is concerned. */
+function bashIntent(cmd) {
+  const c = String(cmd || '');
+  // A heredoc or a redirect is WRITING a file — `cat <<EOF > x` must not read
+  // as a Read.
+  if (!/<</.test(c) && !/>\s*\S/.test(c)) {
+    const m = BASH_READ_RE.exec(c);
+    if (m) {
+      // Drop flags and numeric arguments (`sed -n '100,200p' lib/x.js`), then
+      // take the last path-shaped token.
+      const args = m[1].split(/\s+/).filter((a) => a && !a.startsWith('-') && !/^['"]?\d/.test(a));
+      const path = args.reverse().find((a) => /[/.]/.test(a));
+      if (path) return { kind: 'read', path: path.replace(/^['"]|['"]$/g, '') };
+    }
+  }
+  if (BASH_SEARCH_RE.test(c)) return { kind: 'search' };
+  return null;
+}
+
+const normPath = (p) => String(p ?? '').replace(/\\/g, '/').replace(/^\.\//, '');
+/** Same file, with either side repo-relative and the other absolute. */
+function samePath(a, b) {
+  const x = normPath(a), y = normPath(b);
+  if (!x || !y) return false;
+  return x === y || x.endsWith('/' + y) || y.endsWith('/' + x);
+}
+
+/**
+ * The files whose SOURCE an explore response returned — its per-file sections,
+ * which start with the unique ``**` `` marker (FILE_SECTION_PREFIX in
+ * src/mcp/tools.ts). formatEnvelope keys off the same marker; it needs the byte
+ * offsets too, which is why it re-scans rather than calling this.
+ */
+export function exploreReturnedFiles(text) {
+  return [...String(text ?? '').matchAll(/^\*\*`([^`]+)`\*\*/gm)].map((m) => m[1]);
+}
+
+// Every path-shaped token anywhere in a response — flow steps, blast radius,
+// symbol lists. A file in here but NOT in the returned set was POINTED AT and
+// not delivered, which is a different (and more damning) miss than one the
+// response never mentioned at all.
+const PATH_TOKEN_RE = /(?:[\w@.+-]+\/)+[\w@.+-]+\.[A-Za-z]\w*/g;
+
+/**
+ * The reaction one action represents, given what the explore had returned.
+ * `earlier` is what PREVIOUS explores in the same thread returned: a re-read of
+ * a file we already shipped is an allocation miss wherever it was shipped, and
+ * filing it as recall would point the fix at the wrong end of the pipeline.
+ */
+function reactionOf(action, returned, mentioned, earlier = []) {
+  const { name, input } = action;
+  const readOf = (path, prefix) => {
+    const base = normPath(path).split('/').pop() || String(path ?? '');
+    if (returned.some((r) => samePath(path, r))) return { bucket: 'read_returned', next: `${prefix}Read ${base}` };
+    if (earlier.some((r) => samePath(path, r))) return { bucket: 'read_returned', next: `${prefix}Read ${base} (returned by an earlier explore)` };
+    const named = mentioned.some((m) => samePath(path, m));
+    return { bucket: 'read_missed', next: `${prefix}Read ${base}${named ? ' (named, not returned)' : ''}`, named };
+  };
+  if (/codegraph/.test(name)) return { bucket: 'explore_again', next: name.replace(/^mcp__[^_]*__/, '') };
+  if (name === 'Read' || name === 'NotebookRead') return readOf(input.file_path ?? input.notebook_path, '');
+  if (name === 'Grep' || name === 'Glob') return { bucket: 'search', next: name };
+  if (name === 'Bash') {
+    const intent = bashIntent(input.command);
+    if (intent?.kind === 'read') return readOf(intent.path, 'Bash ');
+    if (intent?.kind === 'search') return { bucket: 'search', next: 'Bash search' };
+  }
+  return { bucket: 'sufficient', next: name };
+}
+
+/** Is this action one of the ways an agent gets file bytes into its head? */
+const isFileAccess = (a) =>
+  a.name === 'Read' || a.name === 'NotebookRead' || a.name === 'Grep' || a.name === 'Glob'
+  || (a.name === 'Bash' && bashIntent(a.input?.command) !== null);
+
+/**
+ * Bucket every answered codegraph_explore call in a transcript by what the
+ * agent did next. Takes the raw JSONL events so it serves both transcript
+ * shapes: stream-json runs (parse-run.mjs) and interactive session logs
+ * (parse-session.mjs) — both emit one assistant event per content block with
+ * `message.id`, and tool results as `tool_result` blocks in user messages.
+ */
+export function classifySufficiency(events) {
+  // One action list PER THREAD: 'main', plus one per subagent (keyed by the
+  // delegating tool_use id, which is what `parent_tool_use_id` carries).
+  const threads = new Map();
+  const nameById = new Map();
+  const textById = new Map();    // explore tool_use_id -> response text
+  for (const ev of events) {
+    const content = ev?.message?.content;
+    if (!Array.isArray(content)) continue;
+    const thread = ev.parent_tool_use_id ?? 'main';
+    if (ev.type === 'assistant') {
+      if (!threads.has(thread)) threads.set(thread, []);
+      const list = threads.get(thread);
+      for (const b of content) {
+        if (b.type !== 'tool_use') continue;
+        nameById.set(b.id, b.name);
+        // No message.id (never seen on a real log) degrades to "every call is
+        // its own message", i.e. same-message calls read as reactions.
+        list.push({ msgId: ev.message.id || `#${thread}-${list.length}`, id: b.id, name: b.name, input: b.input || {} });
+      }
+    } else if (ev.type === 'user') {
+      for (const b of content) {
+        if (b.type !== 'tool_result') continue;
+        const name = nameById.get(b.tool_use_id) || '';
+        if (/codegraph_explore/.test(name) && !b.is_error) textById.set(b.tool_use_id, textOf(b.content));
+      }
+    }
+  }
+
+  const calls = [];
+  let errors = 0, concurrent = 0;
+  // What a delegation really did: the subagent's first substantive call. A
+  // nested delegation is skipped rather than followed, so a subagent that only
+  // spawns another subagent leaves the call as "moved on".
+  const throughDelegation = (action, returned, mentioned, earlier) => {
+    const first = (threads.get(action.id) || []).find((x) => !TRANSPARENT_TOOLS.has(x.name) && !DELEGATION_TOOLS.has(x.name));
+    if (!first) return { bucket: 'sufficient', next: action.name };
+    const r = reactionOf(first, returned, mentioned, earlier);
+    return { ...r, next: `${action.name} → ${r.next}` };
+  };
+  for (const [thread, actions] of threads) {
+    const earlier = [];  // files previous explores in THIS thread already shipped
+    for (let i = 0; i < actions.length; i++) {
+      const a = actions[i];
+      if (!/codegraph_explore/.test(a.name)) continue;
+      const text = textById.get(a.id);
+      // No response text = the call errored, or the run ended before it
+      // returned. Nothing to judge the sufficiency of; count it and move on.
+      if (text === undefined) { errors++; continue; }
+      const returned = exploreReturnedFiles(text);
+      const mentioned = text.match(PATH_TOKEN_RE) || [];
+      let reaction = { bucket: 'sufficient', next: '(final answer)' };
+      for (let j = i + 1; j < actions.length; j++) {
+        const b = actions[j];
+        if (b.msgId === a.msgId) { if (isFileAccess(b)) concurrent++; continue; }
+        if (TRANSPARENT_TOOLS.has(b.name)) continue;
+        reaction = DELEGATION_TOOLS.has(b.name)
+          ? throughDelegation(b, returned, mentioned, earlier)
+          : reactionOf(b, returned, mentioned, earlier);
+        break;
+      }
+      earlier.push(...returned);
+      calls.push({ thread, query: String(a.input.query ?? ''), files: returned.length, chars: text.length, ...reaction });
+    }
+  }
+
+  const counts = Object.fromEntries(SUFFICIENCY_KEYS.map((k) => [k, 0]));
+  for (const c of calls) counts[c.bucket]++;
+  return { calls, counts, errors, concurrent, answered: calls.length };
+}
+
+/** The sufficiency block, as printed under a run and reused by aggregators. */
+export function formatSufficiency(s, indent = '  ') {
+  const f = s.sufficiency;
+  if (!f.answered) {
+    return `${indent}Explore sufficiency: no answered codegraph_explore calls`
+      + (f.errors ? ` (${f.errors} errored or never returned)` : '');
+  }
+  const pct = (n) => ((n / f.answered) * 100).toFixed(0) + '%';
+  const out = [`${indent}Explore sufficiency — what the agent did NEXT (${f.answered} answered call${f.answered === 1 ? '' : 's'}):`];
+  for (const [key, label, meaning] of SUFFICIENCY) {
+    out.push(`${indent}  ${String(f.counts[key]).padStart(3)} ${pct(f.counts[key]).padStart(4)}  ${label.padEnd(31)}${meaning}`);
+  }
+  f.calls.forEach((c, i) => {
+    const q = c.query.length > 46 ? c.query.slice(0, 45) + '…' : c.query;
+    const where = c.thread && c.thread !== 'main' ? ' [subagent]' : '';
+    out.push(`${indent}  ${i + 1}.${where} "${q}" [${c.files} file${c.files === 1 ? '' : 's'}] → ${c.next}`);
+  });
+  const notes = [];
+  if (f.errors) notes.push(`${f.errors} errored/unanswered call${f.errors === 1 ? '' : 's'} (not bucketed)`);
+  if (f.concurrent) notes.push(`${f.concurrent} file-access call${f.concurrent === 1 ? '' : 's'} in the SAME message as an explore (not a reaction)`);
+  if (notes.length) out.push(`${indent}  note: ${notes.join(' · ')}`);
+  return out.join('\n');
+}
+
 // ---------------------------------------------------------------------------
 // `--selftest`: the occupancy math over synthetic transcripts with known
 // answers. It lives here rather than in a test file on purpose — a new
@@ -441,7 +676,7 @@ function selftest() {
     type: 'assistant',
     message: { id, content: [b], usage: { input_tokens: ctx, cache_read_input_tokens: 0, cache_creation_input_tokens: 0, output_tokens: 2 } },
   }));
-  const use = (id, name) => ({ type: 'tool_use', id, name, input: {} });
+  const use = (id, name, input = {}) => ({ type: 'tool_use', id, name, input });
   const res = (id, chars) => JSON.stringify({
     type: 'user', message: { content: [{ type: 'tool_result', tool_use_id: id, content: [{ type: 'text', text: 'x'.repeat(chars) }] }] },
   });
@@ -528,6 +763,186 @@ function selftest() {
   check('stitched final context', s.occupancy.ctxFinal, 14600, 0);
   check('stitched cost sums segments', s.cost * 100, 20, 0.1);
 
+  // ---- 6. Explore sufficiency: the bucket each explore call earns. --------
+  const checkIs = (name, got, want) => {
+    n++;
+    const ok = got === want;
+    if (!ok) failures++;
+    console.log(`${ok ? '  ok  ' : '  FAIL'} ${name}: got ${JSON.stringify(got)}, want ${JSON.stringify(want)}`);
+  };
+  const EXPLORE = 'mcp__codegraph__codegraph_explore';
+  // An explore response's shape that matters here: one `**`path`**` section per
+  // file whose source it returned, plus whatever else it named.
+  const exploreRes = (id, paths, extra = '', isError = false) => JSON.stringify({
+    type: 'user',
+    message: {
+      content: [{
+        type: 'tool_result', tool_use_id: id, ...(isError ? { is_error: true } : {}),
+        content: [{ type: 'text', text: paths.map((p) => `**\`${p}\`** — fn(function)\n\n1\tcode here\n`).join('\n') + extra }],
+      }],
+    },
+  });
+  const suff = (lines) => classifySufficiency(lines.map((l) => JSON.parse(l)));
+
+  // explore → explore is insufficient; the second explore → a Read of a file it
+  // RETURNED is the allocation bucket (this is the CG-22 express baseline).
+  let sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'res.send Content-Type ETag generation' })]),
+    exploreRes('e1', ['lib/response.js', 'lib/utils.js']),
+    ...req(12000, 'm2', [use('e2', EXPLORE, { query: 'response.js res.send function body' })]),
+    exploreRes('e2', ['lib/response.js']),
+    ...req(14000, 'm3', [use('r1', 'Read', { file_path: '/private/tmp/t-base/lib/response.js' })]),
+    res('r1', 3722),
+    ...req(15000, 'm4', [{ type: 'text', text: 'done' }]),
+    done(),
+  ]);
+  check('two answered explore calls', sf.answered, 2, 0);
+  checkIs('explore → explore = insufficient', sf.calls[0].bucket, 'explore_again');
+  checkIs('explore → Read of a returned file (abs path)', sf.calls[1].bucket, 'read_returned');
+
+  // A file the response NAMED but did not return is still a recall miss — and
+  // is flagged as named, since pointing without delivering is its own failure.
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' })]),
+    exploreRes('e1', ['lib/response.js'], '\n**Flow**\n1. lib/router/index.js:42 handle\n'),
+    ...req(12000, 'm2', [use('r1', 'Read', { file_path: '/t/lib/router/index.js' })]),
+    res('r1', 100),
+    done(),
+  ]);
+  checkIs('explore → Read of a named-but-unreturned file', sf.calls[0].bucket, 'read_missed');
+  checkIs('  …flagged as named', sf.calls[0].named, true);
+
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' })]),
+    exploreRes('e1', ['lib/response.js']),
+    ...req(12000, 'm2', [use('r1', 'Read', { file_path: '/t/lib/never/mentioned.js' })]),
+    res('r1', 100),
+    done(),
+  ]);
+  checkIs('explore → Read of a file never surfaced', sf.calls[0].bucket, 'read_missed');
+  checkIs('  …not flagged as named', sf.calls[0].named, false);
+
+  // Grep, and the shell equivalents of Read and Grep.
+  const oneShot = (next) => suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' })]),
+    exploreRes('e1', ['lib/response.js']),
+    ...req(12000, 'm2', [next]),
+    res(next.id, 100),
+    done(),
+  ]).calls[0];
+  checkIs('explore → Grep', oneShot(use('g1', 'Grep', { pattern: 'send' })).bucket, 'search');
+  checkIs('explore → Glob', oneShot(use('g1', 'Glob', { pattern: '**/*.js' })).bucket, 'search');
+  checkIs('explore → Bash sed of a returned file',
+    oneShot(use('b1', 'Bash', { command: "sed -n '100,200p' lib/response.js" })).bucket, 'read_returned');
+  checkIs('explore → Bash grep', oneShot(use('b1', 'Bash', { command: 'grep -rn send lib/' })).bucket, 'search');
+  checkIs('explore → Bash that writes a file is not a read',
+    oneShot(use('b1', 'Bash', { command: "cat > /tmp/note.md <<'EOF'\nx\nEOF" })).bucket, 'sufficient');
+  checkIs('explore → Bash npm test = moved on',
+    oneShot(use('b1', 'Bash', { command: 'npm test' })).bucket, 'sufficient');
+  checkIs('explore → Edit = sufficient',
+    oneShot(use('x1', 'Edit', { file_path: '/t/lib/response.js' })).bucket, 'sufficient');
+
+  // No further tool call at all: the agent answered from the response.
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' })]),
+    exploreRes('e1', ['lib/response.js']),
+    ...req(12000, 'm2', [{ type: 'text', text: 'here is how it works' }]),
+    done(),
+  ]);
+  checkIs('explore → final answer', sf.calls[0].bucket, 'sufficient');
+  checkIs('  …labelled as the final answer', sf.calls[0].next, '(final answer)');
+
+  // A Read issued in the SAME message as the explore predates its response, so
+  // it is not a verdict on it — step past it and count it separately.
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' }), use('r1', 'Read', { file_path: '/t/lib/response.js' })]),
+    exploreRes('e1', ['lib/response.js']),
+    res('r1', 100),
+    ...req(12000, 'm2', [use('x1', 'Edit', { file_path: '/t/lib/response.js' })]),
+    res('x1', 20),
+    done(),
+  ]);
+  checkIs('same-message Read is not a reaction', sf.calls[0].bucket, 'sufficient');
+  check('  …counted as concurrent instead', sf.concurrent, 1, 0);
+
+  // ToolSearch/TodoWrite carry no signal — the Read behind them is the verdict.
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' })]),
+    exploreRes('e1', ['lib/response.js']),
+    ...req(12000, 'm2', [use('t1', 'TodoWrite', {})]),
+    res('t1', 20),
+    ...req(13000, 'm3', [use('r1', 'Read', { file_path: '/t/lib/response.js' })]),
+    res('r1', 100),
+    done(),
+  ]);
+  checkIs('bookkeeping tools are stepped over', sf.calls[0].bucket, 'read_returned');
+
+  // A re-read of a file an EARLIER explore shipped is still an allocation miss:
+  // we returned it and clipped it wrong, just not on this call.
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'first' })]),
+    exploreRes('e1', ['lib/response.js', 'lib/utils.js']),
+    ...req(11000, 'm2', [use('e2', EXPLORE, { query: 'second' })]),
+    exploreRes('e2', ['lib/response.js']),
+    ...req(12000, 'm3', [use('r1', 'Read', { file_path: '/t/lib/utils.js' })]),
+    res('r1', 400),
+    done(),
+  ]);
+  checkIs('re-read of an earlier explore’s file is allocation, not recall',
+    sf.calls[1].bucket, 'read_returned');
+  checkIs('  …and says which explore returned it',
+    sf.calls[1].next, 'Read utils.js (returned by an earlier explore)');
+
+  // A subagent's calls are interleaved into the same stream under
+  // `parent_tool_use_id` (verified on a real excalidraw run). They belong to
+  // their own thread: the parent's verdict is the delegation, judged by what
+  // the subagent actually did first — here, grepping for a file we never
+  // returned.
+  const sub = (parent, obj) => JSON.stringify({ ...JSON.parse(obj), parent_tool_use_id: parent });
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' })]),
+    exploreRes('e1', ['lib/response.js']),
+    ...req(12000, 'm2', [use('a1', 'Agent', { subagent_type: 'Explore' })]),
+    ...req(0, 'sm1', [use('sb1', 'Bash', { command: 'grep -rn nonce lib/' })]).map((l) => sub('a1', l)),
+    sub('a1', res('sb1', 400)),
+    ...req(14000, 'm3', [{ type: 'text', text: 'done' }]),
+    res('a1', 900),
+    done(),
+  ]);
+  checkIs('delegation is judged by what the subagent did', sf.calls[0].bucket, 'search');
+  checkIs('  …and says so', sf.calls[0].next, 'Agent → Bash search');
+
+  // A subagent's Read must NOT be read as the parent's reaction to an explore
+  // the subagent never saw: the parent moved on, the subagent's own explore is
+  // judged inside its own thread.
+  sf = suff([
+    ...req(10000, 'm1', [use('a1', 'Agent', { subagent_type: 'Explore' })]),
+    ...req(0, 'sm1', [use('e1', EXPLORE, { query: 'sub q' })]).map((l) => sub('a1', l)),
+    sub('a1', exploreRes('e1', ['lib/response.js'])),
+    ...req(11000, 'm2', [use('e2', EXPLORE, { query: 'parent q' })]),
+    exploreRes('e2', ['lib/other.js']),
+    ...req(0, 'sm2', [use('sr1', 'Read', { file_path: '/t/lib/response.js' })]).map((l) => sub('a1', l)),
+    sub('a1', res('sr1', 400)),
+    ...req(13000, 'm3', [{ type: 'text', text: 'done' }]),
+    done(),
+  ]);
+  check('both threads bucketed', sf.answered, 2, 0);
+  checkIs('parent explore is not blamed for a subagent Read',
+    sf.calls.find((c) => c.query === 'parent q').bucket, 'sufficient');
+  checkIs('subagent explore is judged in its own thread',
+    sf.calls.find((c) => c.query === 'sub q').bucket, 'read_returned');
+
+  // An errored explore has no response to judge; it is counted, not bucketed.
+  sf = suff([
+    ...req(10000, 'm1', [use('e1', EXPLORE, { query: 'q' })]),
+    exploreRes('e1', [], 'not indexed', true),
+    ...req(12000, 'm2', [use('r1', 'Read', { file_path: '/t/lib/response.js' })]),
+    res('r1', 100),
+    done(),
+  ]);
+  check('errored explore is not bucketed', sf.answered, 0, 0);
+  check('  …but is counted', sf.errors, 1, 0);
+
   console.log(`\n${n - failures}/${n} checks passed`);
   return failures;
 }
@@ -567,6 +982,8 @@ if (isMain) {
   }
   console.log('');
   console.log(formatOccupancy(s));
+  console.log('');
+  console.log(formatSufficiency(s));
   if (wantEnvelope) {
     console.log('');
     console.log(formatEnvelope(s.exploreTexts, answerGlobs));

scripts/agent-eval/parse-session.mjs

@@ -6,6 +6,7 @@
 import { readFileSync, readdirSync, statSync, existsSync, realpathSync } from 'fs';
 import { join } from 'path';
 import { homedir } from 'os';
+import { classifySufficiency, formatSufficiency } from './parse-run.mjs';
 
 const projectArg = process.argv[2];
 if (!projectArg) { console.error('usage: parse-session.mjs <project-dir>'); process.exit(1); }
@@ -91,3 +92,29 @@ if (existsSync(subDir)) {
 }
 const k = (n) => (n / 1000).toFixed(1) + 'k';
 console.log(`TOKENS: gen ${k(tok.gen)} | fresh-in ${k(tok.fresh)} | cached-in ${k(tok.cached)} | billable≈ ${k(tok.gen + tok.fresh)}`);
+
+// What the agent did after each codegraph_explore (CG-8) — the same classifier
+// the headless A/B uses, over the interactive transcript.
+//
+// A subagent's calls live in their OWN file here (headless stream-json
+// interleaves them into one stream instead), so they are stitched back in:
+// each `agent-*.meta.json` carries the `toolUseId` of the Task that spawned it,
+// which is exactly the `parent_tool_use_id` the classifier keys threads on.
+// Without that, a delegated search would score as "the agent moved on".
+const parseLines = (file, parentToolUseId) => readFileSync(file, 'utf8').split('\n')
+  .filter(Boolean)
+  .map((l) => { try { return JSON.parse(l); } catch { return null; } })
+  .filter(Boolean)
+  .map((ev) => (parentToolUseId ? { ...ev, parent_tool_use_id: parentToolUseId } : ev));
+
+const events = parseLines(join(projDir, sessionId + '.jsonl'));
+if (existsSync(subDir)) {
+  for (const f of readdirSync(subDir).filter((f) => f.endsWith('.jsonl'))) {
+    let parent = null;
+    const meta = join(subDir, f.replace(/\.jsonl$/, '.meta.json'));
+    if (existsSync(meta)) { try { parent = JSON.parse(readFileSync(meta, 'utf8')).toolUseId ?? null; } catch { /* unreadable */ } }
+    events.push(...parseLines(join(subDir, f), parent ?? `subagent:${f}`));
+  }
+}
+console.log('');
+console.log(formatSufficiency({ sufficiency: classifySufficiency(events) }, ''));