Explorar o código

test(agent-eval): report residual direction by sign, not by hope (CG-13)

The occupancy summary hardcoded "% lower with codegraph". pct(w, wo) is the
reduction going with->without, so a negative value means the with-arm's
residual is LARGER -- and the line printed "-82% lower with codegraph" for the
case where codegraph in fact occupies 82% MORE. A double negative that reads as
a win and inverts the headline of the whole metric.

Direction now follows the sign in words, and the negative case says what the
shape actually is: codegraph front-loads one large verbatim payload that stays
resident, where Read/Grep churn many small results that evict. Fewer total
tokens processed and a larger persistent footprint are both true at once --
that pair is the axis issue #1500 reported.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Colby McHenry hai 1 mes
pai
achega
520ed9d933
Modificáronse 1 ficheiros con 19 adicións e 1 borrados
  1. 19 1
      scripts/agent-eval/parse-bench-readme.mjs

+ 19 - 1
scripts/agent-eval/parse-bench-readme.mjs

@@ -146,7 +146,25 @@ for (const { repo, W, WO } of rows) {
     `${wSw.toFixed(1)}%→${woSw.toFixed(1)}%`
   );
 }
-console.log(`\nAVERAGE: retrieval residual ${avg(occ.resid)}% lower with codegraph  ·  share-of-context ${avg(occ.shareCtx)}% lower`);
+// Direction must follow the SIGN, not the hope. `pct(w, wo)` is the reduction
+// going with→without, so a NEGATIVE value means the with-arm's residual is
+// LARGER. Hardcoding "lower" printed "-82% lower with codegraph" for the case
+// where codegraph in fact occupies 82% MORE — a double negative that reads as a
+// win and inverts the headline. Say which way it went, in words.
+const dir = (v) => (v < 0 ? 'HIGHER' : 'lower');
+const magn = (v) => Math.abs(v);
+console.log(
+  `\nAVERAGE: retrieval residual ${magn(avg(occ.resid))}% ${dir(avg(occ.resid))} with codegraph` +
+  `  ·  share-of-context ${magn(avg(occ.shareCtx))}% ${dir(avg(occ.shareCtx))}`
+);
+if (avg(occ.resid) < 0) {
+  console.log(
+    `  ^ codegraph front-loads one large verbatim payload that STAYS resident, where Read/Grep\n` +
+    `    churn many small results that evict. Read alongside the cost/token table above: fewer\n` +
+    `    total tokens processed can coexist with a larger persistent footprint. This is the axis\n` +
+    `    issue #1500 reported.`
+  );
+}
 
 console.log(
   `FIXED overhead: codegraph's tool schema + MCP instructions cost ${avg(occ.fixed) >= 0 ? '+' : ''}${avg(occ.fixed)} tok\n` +