feat: SKILL.md template system, 3-tier testing, DX tools (v0.3.3) (#41)

* refactor: extract command registry to commands.ts, add SNAPSHOT_FLAGS metadata

- NEW: browse/src/commands.ts — command sets + COMMAND_DESCRIPTIONS + load-time validation (zero side effects)
- server.ts imports from commands.ts instead of declaring sets inline
- snapshot.ts: SNAPSHOT_FLAGS array drives parseSnapshotArgs (metadata-driven, no duplication)
- All 186 existing tests pass

* feat: SKILL.md template system with auto-generated command references

- SKILL.md.tmpl + browse/SKILL.md.tmpl with {{COMMAND_REFERENCE}} and {{SNAPSHOT_FLAGS}} placeholders
- scripts/gen-skill-docs.ts generates SKILL.md from templates (supports --dry-run)
- Build pipeline runs gen:skill-docs before binary compilation
- Generated files have AUTO-GENERATED header, committed to git

* test: Tier 1 static validation — 34 tests for SKILL.md command correctness

- test/helpers/skill-parser.ts: extracts $B commands from code blocks, validates against registry
- test/skill-parser.test.ts: 13 parser/validator unit tests
- test/skill-validation.test.ts: 13 tests validating all SKILL.md files + registry consistency
- test/gen-skill-docs.test.ts: 8 generator tests (categories, sorting, freshness)

* feat: DX tools (skill:check, dev:skill) + Tier 2 E2E test scaffolding

- scripts/skill-check.ts: health summary for all SKILL.md files (commands, templates, freshness)
- scripts/dev-skill.ts: watch mode for template development
- test/helpers/session-runner.ts: Agent SDK wrapper for E2E skill tests
- test/skill-e2e.test.ts: 2 E2E tests + 3 stubs (auto-skip inside Claude Code sessions)
- E2E tests must run from plain terminal: SKILL_E2E=1 bun test test/skill-e2e.test.ts

* ci: SKILL.md freshness check on push/PR + TODO updates

- .github/workflows/skill-docs.yml: fails if generated SKILL.md files are stale
- TODO.md: add E2E cost tracking and model pinning to future ideas

* fix: restore rich descriptions lost in auto-generation

- Snapshot flags: add back value hints (-d <N>, -s <sel>, -o <path>)
- Snapshot flags: restore parenthetical context (@e refs, @c refs, etc.)
- Commands: is → includes valid states enum
- Commands: console → notes --errors filter behavior
- Commands: press → lists common keys (Enter, Tab, Escape)
- Commands: cookie-import-browser → describes picker UI
- Commands: dialog-accept → specifies alert/confirm/prompt
- Tips: restore → arrow (was downgraded to ->)

* test: quality evals for generated SKILL.md descriptions

Catches the exact regressions we shipped and caught in review:
- Snapshot flags must include value hints (-d <N>, -s <sel>, -o <path>)
- is command must list all valid states (visible/hidden/enabled/...)
- press command must list example keys (Enter, Tab, Escape)
- console command must describe --errors behavior
- Snapshot -i must mention @e refs, -C must mention @c refs
- All descriptions must be >= 8 chars (no empty stubs)
- Tips section must use → not ->

* feat: LLM-as-judge evals for SKILL.md documentation quality

4 eval tests using Anthropic API (claude-haiku, ~$0.01-0.03/run):
- Command reference table: clarity/completeness/actionability >= 4/5
- Snapshot flags section: same thresholds
- browse/SKILL.md overall quality
- Regression: generated version must score >= hand-maintained baseline

Requires ANTHROPIC_API_KEY. Auto-skips without it.
Run: bun run test:eval (or ANTHROPIC_API_KEY=sk-... bun test test/skill-llm-eval.test.ts)

* chore: bump version to 0.3.3, update changelog

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add ARCHITECTURE.md, update CLAUDE.md and CONTRIBUTING.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: conductor.json lifecycle hooks + .env propagation across worktrees

bin/dev-setup now copies .env from main worktree so API keys carry
over to Conductor workspaces automatically. conductor.json wires up
setup and archive hooks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: complete CHANGELOG for v0.3.3 (architecture, conductor, .env)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-03-13 21:08:12 -07:00
committed by GitHub
parent ea0c0dad5e
commit 5205070299
29 changed files with 2479 additions and 135 deletions

View File

@@ -12,6 +12,8 @@ allowed-tools:
- Read
---
<!-- AUTO-GENERATED from SKILL.md.tmpl — do not edit directly -->
<!-- Regenerate: bun run gen:skill-docs -->
# browse: QA Testing & Dogfooding
@@ -99,30 +101,115 @@ $B diff https://staging.app.com https://prod.app.com
## Snapshot Flags
The snapshot is your primary tool for understanding and interacting with pages.
```
-i Interactive elements only (buttons, links, inputs)
-i Interactive elements only (buttons, links, inputs) with @e refs
-c Compact (no empty structural nodes)
-d <N> Limit depth
-s <sel> Scope to CSS selector
-D Diff against previous snapshot
-D Diff against previous snapshot (what changed?)
-a Annotated screenshot with ref labels
-o <path> Output path for screenshot
-C Cursor-interactive elements (@c refs)
-C Cursor-interactive elements (@c refs — divs with pointer, onclick)
```
Combine: `$B snapshot -i -a -C -o /tmp/annotated.png`
Combine flags: `$B snapshot -i -a -C -o /tmp/annotated.png`
Use @refs after snapshot: `$B click @e3`, `$B fill @e4 "value"`, `$B click @c1`
After snapshot, use @refs everywhere:
```bash
$B click @e3 $B fill @e4 "value" $B hover @e1
$B html @e2 $B css @e5 "color" $B attrs @e6
$B click @c1 # cursor-interactive ref (from -C)
```
Refs are invalidated on navigation — run `snapshot` again after `goto`.
## Full Command List
**Navigate:** goto, back, forward, reload, url
**Read:** text, html, links, forms, accessibility
**Snapshot:** snapshot (with flags above)
**Interact:** click, fill, select, hover, type, press, scroll, wait, wait --networkidle, wait --load, viewport, upload, cookie-import, dialog-accept, dialog-dismiss
**Inspect:** js, eval, css, attrs, is, console, console --errors, network, dialog, cookies, storage, perf
**Visual:** screenshot, pdf, responsive
**Compare:** diff
**Multi-step:** chain (pipe JSON array)
**Tabs:** tabs, tab, newtab, closetab
**Server:** status, stop, restart
### Navigation
| Command | Description |
|---------|-------------|
| `back` | History back |
| `forward` | History forward |
| `goto <url>` | Navigate to URL |
| `reload` | Reload page |
| `url` | Print current URL |
### Reading
| Command | Description |
|---------|-------------|
| `accessibility` | Full ARIA tree |
| `forms` | Form fields as JSON |
| `html [selector]` | innerHTML |
| `links` | All links as "text → href" |
| `text` | Cleaned page text |
### Interaction
| Command | Description |
|---------|-------------|
| `click <sel>` | Click element |
| `cookie` | Set cookie |
| `cookie-import <json>` | Import cookies from JSON file |
| `cookie-import-browser [browser] [--domain d]` | Import cookies from real browser (opens picker UI, or direct with --domain) |
| `dialog-accept [text]` | Auto-accept next alert/confirm/prompt |
| `dialog-dismiss` | Auto-dismiss next dialog |
| `fill <sel> <val>` | Fill input |
| `header <name> <value>` | Set custom request header |
| `hover <sel>` | Hover element |
| `press <key>` | Press key (Enter, Tab, Escape, etc.) |
| `scroll [sel]` | Scroll element into view |
| `select <sel> <val>` | Select dropdown option |
| `type <text>` | Type into focused element |
| `upload <sel> <file...>` | Upload file(s) |
| `useragent <string>` | Set user agent |
| `viewport <WxH>` | Set viewport size |
| `wait <sel|--networkidle|--load>` | Wait for element/condition |
### Inspection
| Command | Description |
|---------|-------------|
| `attrs <sel|@ref>` | Element attributes as JSON |
| `console [--clear|--errors]` | Console messages (--errors filters to error/warning) |
| `cookies` | All cookies as JSON |
| `css <sel> <prop>` | Computed CSS value |
| `dialog [--clear]` | Dialog messages |
| `eval <file>` | Run JS file |
| `is <prop> <sel>` | State check (visible/hidden/enabled/disabled/checked/editable/focused) |
| `js <expr>` | Run JavaScript |
| `network [--clear]` | Network requests |
| `perf` | Page load timings |
| `storage [set k v]` | localStorage + sessionStorage |
### Visual
| Command | Description |
|---------|-------------|
| `diff <url1> <url2>` | Text diff between pages |
| `pdf [path]` | Save as PDF |
| `responsive [prefix]` | Mobile/tablet/desktop screenshots |
| `screenshot [path]` | Save screenshot |
### Snapshot
| Command | Description |
|---------|-------------|
| `snapshot [flags]` | Accessibility tree with @refs |
### Meta
| Command | Description |
|---------|-------------|
| `chain` | Multi-command from JSON stdin |
### Tabs
| Command | Description |
|---------|-------------|
| `closetab [id]` | Close tab |
| `newtab [url]` | Open new tab |
| `tab <id>` | Switch to tab |
| `tabs` | List open tabs |
### Server
| Command | Description |
|---------|-------------|
| `restart` | Restart server |
| `status` | Health check |
| `stop` | Shutdown server |

106
browse/SKILL.md.tmpl Normal file
View File

@@ -0,0 +1,106 @@
---
name: browse
version: 1.1.0
description: |
Fast headless browser for QA testing and site dogfooding. Navigate any URL, interact with
elements, verify page state, diff before/after actions, take annotated screenshots, check
responsive layouts, test forms and uploads, handle dialogs, and assert element states.
~100ms per command. Use when you need to test a feature, verify a deployment, dogfood a
user flow, or file a bug with evidence.
allowed-tools:
- Bash
- Read
---
# browse: QA Testing & Dogfooding
Persistent headless Chromium. First call auto-starts (~3s), then ~100ms per command.
State persists between calls (cookies, tabs, login sessions).
## Core QA Patterns
### 1. Verify a page loads correctly
```bash
$B goto https://yourapp.com
$B text # content loads?
$B console # JS errors?
$B network # failed requests?
$B is visible ".main-content" # key elements present?
```
### 2. Test a user flow
```bash
$B goto https://app.com/login
$B snapshot -i # see all interactive elements
$B fill @e3 "user@test.com"
$B fill @e4 "password"
$B click @e5 # submit
$B snapshot -D # diff: what changed after submit?
$B is visible ".dashboard" # success state present?
```
### 3. Verify an action worked
```bash
$B snapshot # baseline
$B click @e3 # do something
$B snapshot -D # unified diff shows exactly what changed
```
### 4. Visual evidence for bug reports
```bash
$B snapshot -i -a -o /tmp/annotated.png # labeled screenshot
$B screenshot /tmp/bug.png # plain screenshot
$B console # error log
```
### 5. Find all clickable elements (including non-ARIA)
```bash
$B snapshot -C # finds divs with cursor:pointer, onclick, tabindex
$B click @c1 # interact with them
```
### 6. Assert element states
```bash
$B is visible ".modal"
$B is enabled "#submit-btn"
$B is disabled "#submit-btn"
$B is checked "#agree-checkbox"
$B is editable "#name-field"
$B is focused "#search-input"
$B js "document.body.textContent.includes('Success')"
```
### 7. Test responsive layouts
```bash
$B responsive /tmp/layout # mobile + tablet + desktop screenshots
$B viewport 375x812 # or set specific viewport
$B screenshot /tmp/mobile.png
```
### 8. Test file uploads
```bash
$B upload "#file-input" /path/to/file.pdf
$B is visible ".upload-success"
```
### 9. Test dialogs
```bash
$B dialog-accept "yes" # set up handler
$B click "#delete-button" # trigger dialog
$B dialog # see what appeared
$B snapshot -D # verify deletion happened
```
### 10. Compare environments
```bash
$B diff https://staging.app.com https://prod.app.com
```
## Snapshot Flags
{{SNAPSHOT_FLAGS}}
## Full Command List
{{COMMAND_REFERENCE}}

107
browse/src/commands.ts Normal file
View File

@@ -0,0 +1,107 @@
/**
* Command registry — single source of truth for all browse commands.
*
* Dependency graph:
* commands.ts ──▶ server.ts (runtime dispatch)
* ──▶ gen-skill-docs.ts (doc generation)
* ──▶ skill-parser.ts (validation)
* ──▶ skill-check.ts (health reporting)
*
* Zero side effects. Safe to import from build scripts and tests.
*/
export const READ_COMMANDS = new Set([
'text', 'html', 'links', 'forms', 'accessibility',
'js', 'eval', 'css', 'attrs',
'console', 'network', 'cookies', 'storage', 'perf',
'dialog', 'is',
]);
export const WRITE_COMMANDS = new Set([
'goto', 'back', 'forward', 'reload',
'click', 'fill', 'select', 'hover', 'type', 'press', 'scroll', 'wait',
'viewport', 'cookie', 'cookie-import', 'cookie-import-browser', 'header', 'useragent',
'upload', 'dialog-accept', 'dialog-dismiss',
]);
export const META_COMMANDS = new Set([
'tabs', 'tab', 'newtab', 'closetab',
'status', 'stop', 'restart',
'screenshot', 'pdf', 'responsive',
'chain', 'diff',
'url', 'snapshot',
]);
export const ALL_COMMANDS = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
export const COMMAND_DESCRIPTIONS: Record<string, { category: string; description: string; usage?: string }> = {
// Navigation
'goto': { category: 'Navigation', description: 'Navigate to URL', usage: 'goto <url>' },
'back': { category: 'Navigation', description: 'History back' },
'forward': { category: 'Navigation', description: 'History forward' },
'reload': { category: 'Navigation', description: 'Reload page' },
'url': { category: 'Navigation', description: 'Print current URL' },
// Reading
'text': { category: 'Reading', description: 'Cleaned page text' },
'html': { category: 'Reading', description: 'innerHTML', usage: 'html [selector]' },
'links': { category: 'Reading', description: 'All links as "text → href"' },
'forms': { category: 'Reading', description: 'Form fields as JSON' },
'accessibility': { category: 'Reading', description: 'Full ARIA tree' },
// Inspection
'js': { category: 'Inspection', description: 'Run JavaScript', usage: 'js <expr>' },
'eval': { category: 'Inspection', description: 'Run JS file', usage: 'eval <file>' },
'css': { category: 'Inspection', description: 'Computed CSS value', usage: 'css <sel> <prop>' },
'attrs': { category: 'Inspection', description: 'Element attributes as JSON', usage: 'attrs <sel|@ref>' },
'is': { category: 'Inspection', description: 'State check (visible/hidden/enabled/disabled/checked/editable/focused)', usage: 'is <prop> <sel>' },
'console': { category: 'Inspection', description: 'Console messages (--errors filters to error/warning)', usage: 'console [--clear|--errors]' },
'network': { category: 'Inspection', description: 'Network requests', usage: 'network [--clear]' },
'dialog': { category: 'Inspection', description: 'Dialog messages', usage: 'dialog [--clear]' },
'cookies': { category: 'Inspection', description: 'All cookies as JSON' },
'storage': { category: 'Inspection', description: 'localStorage + sessionStorage', usage: 'storage [set k v]' },
'perf': { category: 'Inspection', description: 'Page load timings' },
// Interaction
'click': { category: 'Interaction', description: 'Click element', usage: 'click <sel>' },
'fill': { category: 'Interaction', description: 'Fill input', usage: 'fill <sel> <val>' },
'select': { category: 'Interaction', description: 'Select dropdown option', usage: 'select <sel> <val>' },
'hover': { category: 'Interaction', description: 'Hover element', usage: 'hover <sel>' },
'type': { category: 'Interaction', description: 'Type into focused element', usage: 'type <text>' },
'press': { category: 'Interaction', description: 'Press key (Enter, Tab, Escape, etc.)', usage: 'press <key>' },
'scroll': { category: 'Interaction', description: 'Scroll element into view', usage: 'scroll [sel]' },
'wait': { category: 'Interaction', description: 'Wait for element/condition', usage: 'wait <sel|--networkidle|--load>' },
'upload': { category: 'Interaction', description: 'Upload file(s)', usage: 'upload <sel> <file...>' },
'viewport':{ category: 'Interaction', description: 'Set viewport size', usage: 'viewport <WxH>' },
'cookie': { category: 'Interaction', description: 'Set cookie' },
'cookie-import': { category: 'Interaction', description: 'Import cookies from JSON file', usage: 'cookie-import <json>' },
'cookie-import-browser': { category: 'Interaction', description: 'Import cookies from real browser (opens picker UI, or direct with --domain)', usage: 'cookie-import-browser [browser] [--domain d]' },
'header': { category: 'Interaction', description: 'Set custom request header', usage: 'header <name> <value>' },
'useragent': { category: 'Interaction', description: 'Set user agent', usage: 'useragent <string>' },
'dialog-accept': { category: 'Interaction', description: 'Auto-accept next alert/confirm/prompt', usage: 'dialog-accept [text]' },
'dialog-dismiss': { category: 'Interaction', description: 'Auto-dismiss next dialog' },
// Visual
'screenshot': { category: 'Visual', description: 'Save screenshot', usage: 'screenshot [path]' },
'pdf': { category: 'Visual', description: 'Save as PDF', usage: 'pdf [path]' },
'responsive': { category: 'Visual', description: 'Mobile/tablet/desktop screenshots', usage: 'responsive [prefix]' },
'diff': { category: 'Visual', description: 'Text diff between pages', usage: 'diff <url1> <url2>' },
// Tabs
'tabs': { category: 'Tabs', description: 'List open tabs' },
'tab': { category: 'Tabs', description: 'Switch to tab', usage: 'tab <id>' },
'newtab': { category: 'Tabs', description: 'Open new tab', usage: 'newtab [url]' },
'closetab':{ category: 'Tabs', description: 'Close tab', usage: 'closetab [id]' },
// Server
'status': { category: 'Server', description: 'Health check' },
'stop': { category: 'Server', description: 'Shutdown server' },
'restart': { category: 'Server', description: 'Restart server' },
// Meta
'snapshot':{ category: 'Snapshot', description: 'Accessibility tree with @refs', usage: 'snapshot [flags]' },
'chain': { category: 'Meta', description: 'Multi-command from JSON stdin' },
};
// Load-time validation: descriptions must cover exactly the command sets
const allCmds = new Set([...READ_COMMANDS, ...WRITE_COMMANDS, ...META_COMMANDS]);
const descKeys = new Set(Object.keys(COMMAND_DESCRIPTIONS));
for (const cmd of allCmds) {
if (!descKeys.has(cmd)) throw new Error(`COMMAND_DESCRIPTIONS missing entry for: ${cmd}`);
}
for (const key of descKeys) {
if (!allCmds.has(key)) throw new Error(`COMMAND_DESCRIPTIONS has unknown command: ${key}`);
}

View File

@@ -110,28 +110,9 @@ const idleCheckInterval = setInterval(() => {
}
}, 60_000);
// ─── Command Sets (exported for chain command) ──────────────────
export const READ_COMMANDS = new Set([
'text', 'html', 'links', 'forms', 'accessibility',
'js', 'eval', 'css', 'attrs',
'console', 'network', 'cookies', 'storage', 'perf',
'dialog', 'is',
]);
export const WRITE_COMMANDS = new Set([
'goto', 'back', 'forward', 'reload',
'click', 'fill', 'select', 'hover', 'type', 'press', 'scroll', 'wait',
'viewport', 'cookie', 'cookie-import', 'cookie-import-browser', 'header', 'useragent',
'upload', 'dialog-accept', 'dialog-dismiss',
]);
export const META_COMMANDS = new Set([
'tabs', 'tab', 'newtab', 'closetab',
'status', 'stop', 'restart',
'screenshot', 'pdf', 'responsive',
'chain', 'diff',
'url', 'snapshot',
]);
// ─── Command Sets (from commands.ts — single source of truth) ───
import { READ_COMMANDS, WRITE_COMMANDS, META_COMMANDS } from './commands';
export { READ_COMMANDS, WRITE_COMMANDS, META_COMMANDS };
// ─── Server ────────────────────────────────────────────────────
const browserManager = new BrowserManager();

View File

@@ -40,6 +40,31 @@ interface SnapshotOptions {
cursorInteractive?: boolean; // -C / --cursor-interactive: scan cursor:pointer etc.
}
/**
* Snapshot flag metadata — single source of truth for CLI parsing and doc generation.
*
* Imported by:
* - gen-skill-docs.ts (generates {{SNAPSHOT_FLAGS}} tables)
* - skill-parser.ts (validates flags in SKILL.md examples)
*/
export const SNAPSHOT_FLAGS: Array<{
short: string;
long: string;
description: string;
takesValue?: boolean;
valueHint?: string;
optionKey: keyof SnapshotOptions;
}> = [
{ short: '-i', long: '--interactive', description: 'Interactive elements only (buttons, links, inputs) with @e refs', optionKey: 'interactive' },
{ short: '-c', long: '--compact', description: 'Compact (no empty structural nodes)', optionKey: 'compact' },
{ short: '-d', long: '--depth', description: 'Limit depth', takesValue: true, valueHint: '<N>', optionKey: 'depth' },
{ short: '-s', long: '--selector', description: 'Scope to CSS selector', takesValue: true, valueHint: '<sel>', optionKey: 'selector' },
{ short: '-D', long: '--diff', description: 'Diff against previous snapshot (what changed?)', optionKey: 'diff' },
{ short: '-a', long: '--annotate', description: 'Annotated screenshot with ref labels', optionKey: 'annotate' },
{ short: '-o', long: '--output', description: 'Output path for screenshot', takesValue: true, valueHint: '<path>', optionKey: 'outputPath' },
{ short: '-C', long: '--cursor-interactive', description: 'Cursor-interactive elements (@c refs — divs with pointer, onclick)', optionKey: 'cursorInteractive' },
];
interface ParsedNode {
indent: number;
role: string;
@@ -50,49 +75,24 @@ interface ParsedNode {
}
/**
* Parse CLI args into SnapshotOptions
* Parse CLI args into SnapshotOptions — driven by SNAPSHOT_FLAGS metadata.
*/
export function parseSnapshotArgs(args: string[]): SnapshotOptions {
const opts: SnapshotOptions = {};
for (let i = 0; i < args.length; i++) {
switch (args[i]) {
case '-i':
case '--interactive':
opts.interactive = true;
break;
case '-c':
case '--compact':
opts.compact = true;
break;
case '-d':
case '--depth':
opts.depth = parseInt(args[++i], 10);
const flag = SNAPSHOT_FLAGS.find(f => f.short === args[i] || f.long === args[i]);
if (!flag) throw new Error(`Unknown snapshot flag: ${args[i]}`);
if (flag.takesValue) {
const value = args[++i];
if (!value) throw new Error(`Usage: snapshot ${flag.short} <value>`);
if (flag.optionKey === 'depth') {
(opts as any)[flag.optionKey] = parseInt(value, 10);
if (isNaN(opts.depth!)) throw new Error('Usage: snapshot -d <number>');
break;
case '-s':
case '--selector':
opts.selector = args[++i];
if (!opts.selector) throw new Error('Usage: snapshot -s <selector>');
break;
case '-D':
case '--diff':
opts.diff = true;
break;
case '-a':
case '--annotate':
opts.annotate = true;
break;
case '-o':
case '--output':
opts.outputPath = args[++i];
if (!opts.outputPath) throw new Error('Usage: snapshot -o <path>');
break;
case '-C':
case '--cursor-interactive':
opts.cursorInteractive = true;
break;
default:
throw new Error(`Unknown snapshot flag: ${args[i]}`);
} else {
(opts as any)[flag.optionKey] = value;
}
} else {
(opts as any)[flag.optionKey] = true;
}
}
return opts;