hai/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-05-18 10:31:30 +08:00
Code Issues Packages Projects Releases Wiki Activity

test(harness): fix detection order + whitespace-tolerant pattern matching

Browse Source 
Two bugs surfaced when validating the v1.21 fix end-to-end:

1. PlanSkillObservation outcome detection ran 'asked' (any numbered
   options list) BEFORE 'plan_ready'. Plan-mode's "Ready to execute?"
   confirmation IS a numbered options list (1=auto, 2=manual, ...), so
   any skill that successfully reached the native confirmation got
   misclassified as 'asked'. Reorder: 'auto_decided' (most specific,
   requires AUTO_DECIDE annotation) > 'plan_ready' (next, requires the
   "ready to execute" stem) > 'asked' (any remaining numbered list).

2. isPlanReadyVisible and isAutoDecidedVisible regexes only matched
   spaced forms ("ready to execute", "(your preference)"). stripAnsi
   removes cursor-positioning escapes (`\x1b[40C`) entirely instead of
   replacing them with spaces, so the same text can render as
   "readytoexecute" or "(yourpreference)". Both detectors now test the
   spaced form first, fall through to a whitespace-collapsed comparison.
   Inline unit smoke confirms both forms match.

Updates to the 5 strict 'asked' regression test cases (plan-ceo,
plan-eng, plan-devex, autoplan, office-hours): with the detection order
corrected, the model's plan-file fallback flow legitimately lands at
'plan_ready' instead of 'asked'. Pass envelope expanded to ['asked',
'plan_ready'] (matching plan-design-review's existing pattern). Failure
signals tightened to include 'auto_decided' (catches AUTO_DECIDE without
opt-in) plus the standard silent_write/exited/timeout. plan-design was
already on this contract from v1.21's first commit, no change needed.

The expanded envelope is correct: under --disallowedTools AskUserQuestion
the Tool resolution preamble routes the question through plan-mode's
native "Ready to execute?" surface — the user still sees the decision,
just via the plan-file flow rather than a numbered prompt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-30 22:27:33 -07:00
parent a7cfbeba4c
commit 78e4b770fa
6 changed files with 103 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
test/skill-e2e-autoplan-auto-mode.test.ts
View File

@@ -27,6 +27,12 @@ const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
const describeE2E = shouldRun ? describe : describe.skip;
describeE2E('autoplan AskUserQuestion-blocked smoke (gate)', () => {
// Pass envelope is ['asked', 'plan_ready']: model either renders the
// first non-auto-decided gate (Phase 1 premise confirmation) as numbered
// prose or surfaces it through the plan file + ExitPlanMode flow.
// Autoplan auto-decides intermediate questions BY DESIGN; the failure
// signal we care about is the AUTO_DECIDE preamble firing on a gate it
// shouldn't (caught explicitly via the 'auto_decided' outcome).
test('a non-auto-decided gate surfaces when AskUserQuestion is --disallowedTools', async () => {
const obs = await runPlanSkillObservation({
skillName: 'autoplan',
@@ -35,7 +41,12 @@ describeE2E('autoplan AskUserQuestion-blocked smoke (gate)', () => {
timeoutMs: 300_000,
});
if (obs.outcome !== 'asked') {
if (
obs.outcome === 'auto_decided' ||
obs.outcome === 'silent_write' ||
obs.outcome === 'exited' ||
obs.outcome === 'timeout'
) {
throw new Error(
`autoplan AskUserQuestion-blocked regression: outcome=${obs.outcome}\n` +
`summary: ${obs.summary}\n` +
@@ -43,6 +54,6 @@ describeE2E('autoplan AskUserQuestion-blocked smoke (gate)', () => {
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
);
}
expect(obs.outcome).toEqual('asked');
expect(['asked', 'plan_ready']).toContain(obs.outcome);
}, 360_000);
});