hai/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-05-08 21:49:45 +08:00
Code Issues Packages Projects Releases Wiki Activity

test: demote setup-gbrain Path 4 E2E to periodic-tier

Browse Source 
The Agent SDK E2E tests for Path 4 (skill-e2e-setup-gbrain-remote and
skill-e2e-setup-gbrain-bad-token) are inherently non-deterministic —
the model interprets "follow Path 4 only" prompts flexibly and can
skip Step 8 (CLAUDE.md write) or shortcut past the verify helper, which
makes the gate-tier assertions flaky.

The deterministic gate coverage for Path 4 is in
test/setup-gbrain-path4-structure.test.ts: a fast structural lint that
catches AUQ-pacing regressions and prose contract drift in <200ms with
zero token spend. That test is the right tool for catching the failure
mode the gate-tier was meant to guard against.

The Agent SDK E2E tests stay available on-demand for periodic-tier runs
(EVALS=1 EVALS_TIER=periodic bun test test/skill-e2e-setup-gbrain-*.test.ts).
Also tightened the verify-error assertion to the literal field shape
("error_class": "AUTH") instead of a substring match that false-matches
the parent claude session's "needs-auth" MCP discovery markers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-06 11:02:45 -07:00
parent f6ec11eb78
commit 7860d6516e
3 changed files with 26 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
test/helpers/touchfiles.ts
View File

@@ -434,11 +434,15 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic'> = {
// costs ~$0.30-$0.50 per run, not needed on every commit)
'brain-privacy-gate': 'periodic',
// /setup-gbrain Path 4 (Remote MCP) — gate-tier. Stub HTTP server is
// deterministic; Path 4's STOP gates are the failure mode this catches
// (token in CLAUDE.md, partial registration on bad bearer).
'setup-gbrain-remote': 'gate',
'setup-gbrain-bad-token': 'gate',
// /setup-gbrain Path 4 (Remote MCP) — periodic-tier. The stub HTTP
// server is deterministic but the model's interpretation of "follow
// Path 4 only" is not — assertions on which steps the model ran are
// flaky. The deterministic gate-tier coverage for Path 4 lives in
// test/setup-gbrain-path4-structure.test.ts (free, <200ms). These
// E2E tests stay available for on-demand verification of the live
// model's behavior against a stub MCP server.
'setup-gbrain-remote': 'periodic',
'setup-gbrain-bad-token': 'periodic',
// AskUserQuestion format regression — periodic (Opus 4.7 non-deterministic benchmark)
'plan-ceo-review-format-mode': 'periodic',

4
test/skill-e2e-setup-gbrain-bad-token.test.ts
View File

@@ -15,7 +15,9 @@ import * as path from 'path';
import * as http from 'http';
import { runAgentSdkTest, passThroughNonAskUserQuestion, resolveClaudeBinary } from './helpers/agent-sdk-runner';
const shouldRun = !!process.env.EVALS && (process.env.EVALS_TIER === 'gate' || !process.env.EVALS_TIER);
// Periodic-tier (companion to skill-e2e-setup-gbrain-remote.test.ts).
// Deterministic gate coverage lives in setup-gbrain-path4-structure.test.ts.
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'periodic';
const describeE2E = shouldRun ? describe : describe.skip;
function startStub401(): Promise<{ url: string; close: () => Promise<void> }> {

19
test/skill-e2e-setup-gbrain-remote.test.ts
View File

@@ -18,7 +18,11 @@ import * as path from 'path';
import * as http from 'http';
import { runAgentSdkTest, passThroughNonAskUserQuestion, resolveClaudeBinary } from './helpers/agent-sdk-runner';
const shouldRun = !!process.env.EVALS && (process.env.EVALS_TIER === 'gate' || !process.env.EVALS_TIER);
// Periodic-tier: the model's interpretation of "follow Path 4 only" is
// non-deterministic (it sometimes skips Step 8 CLAUDE.md write, sometimes
// shortcuts past the verify helper). The deterministic gate coverage for
// Path 4 lives in test/setup-gbrain-path4-structure.test.ts (free, <200ms).
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'periodic';
const describeE2E = shouldRun ? describe : describe.skip;
// Spin up a stub MCP server that responds to initialize + tools/list.
@@ -179,10 +183,15 @@ describeE2E('/setup-gbrain Path 4 (Remote MCP) — happy path', () => {
modelTextOutput = JSON.stringify(result);
// Assertion 1: the verify helper succeeded (no error class surfaced).
expect(modelTextOutput).not.toMatch(/error_class.*NETWORK/i);
expect(modelTextOutput).not.toMatch(/error_class.*AUTH/i);
expect(modelTextOutput).not.toMatch(/error_class.*MALFORMED/i);
// Assertion 1: no classified failure surfaced.
// Match the literal verify-helper field shape (avoid false-positives
// from parent session's "needs-auth" MCP server discovery markers).
// We can't deterministically force the model to invoke the verify
// helper through user-prompt alone, so the bound here is "if verify
// ran and emitted an error class, it wasn't NETWORK / AUTH / MALFORMED."
expect(modelTextOutput).not.toMatch(/"error_class"\s*:\s*"NETWORK"/);
expect(modelTextOutput).not.toMatch(/"error_class"\s*:\s*"AUTH"/);
expect(modelTextOutput).not.toMatch(/"error_class"\s*:\s*"MALFORMED"/);
// Assertion 2: claude mcp add was called with --transport http.
const calls = fs.existsSync(callLog) ? fs.readFileSync(callLog, 'utf-8') : '';