superpowers/evals/docs/plan.md

Drill Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a tmux-based harness that drives AI coding agents through worktree scenarios and evaluates whether they follow superpowers skills.

Architecture: CLI (click) orchestrates an engine that sets up a test repo, launches an agent in tmux, drives it via an LLM actor (Anthropic SDK, structured tool_use), collects session logs + filesystem state, then evaluates compliance via an LLM verifier. Backend configs (YAML) define how to launch each agent CLI. Scenarios (YAML) define what to test.

Tech Stack: Python 3.11+, click, pyyaml, anthropic SDK, jinja2, pydantic, tmux

---

File Structure

drill/
├── drill/
│   ├── __init__.py          # Package init, version
│   ├── cli.py               # click CLI: run, compare, list
│   ├── engine.py            # Orchestrates full run lifecycle (7 steps)
│   ├── session.py           # tmux session management (create, send-keys, capture, kill)
│   ├── actor.py             # Actor LLM: rolling context, structured tool_use output
│   ├── verifier.py          # Verifier LLM: per-criterion evaluation, pydantic schema
│   ├── setup.py             # Template repo cloning, helper dispatch, assertion runner
│   ├── backend.py           # Loads backend YAML, builds CLI commands, idle detection
│   └── normalizer.py        # Normalizes backend-specific session logs to common schema
├── backends/
│   ├── claude.yaml          # Claude Code backend config
│   └── codex.yaml           # Codex backend config
├── prompts/
│   ├── actor.md             # Actor system prompt (jinja2 template)
│   └── verifier.md          # Verifier system prompt (jinja2 template)
├── scenarios/
│   ├── worktree-creation-from-main.yaml
│   ├── worktree-already-inside.yaml
│   ├── worktree-codex-detached-head.yaml
│   └── worktree-consent-flow.yaml
├── fixtures/
│   └── template-repo/       # Minimal git repo cloned per run
│       ├── package.json
│       ├── src/
│       │   ├── index.js
│       │   └── utils.js
│       └── README.md
├── setup_helpers/
│   ├── __init__.py          # Exports helper registry
│   ├── base.py              # create_base_repo
│   └── worktree.py          # add_worktree, detach_head, symlink_superpowers
├── tests/
│   ├── test_backend.py
│   ├── test_setup.py
│   ├── test_session.py
│   ├── test_actor.py
│   ├── test_verifier.py
│   ├── test_normalizer.py
│   ├── test_engine.py
│   └── test_cli.py
├── pyproject.toml
├── .gitignore
└── README.md

---

Task 1: Project Scaffold

Files:

• Create: pyproject.toml
• Create: drill/__init__.py
• Create: .gitignore

• Create: README.md

• [ ] Step 1: Create pyproject.toml

  [build-system]
  requires = ["setuptools>=68.0"]
  build-backend = "setuptools.backends._legacy:_Backend"
  
  [project]
  name = "drill"
  version = "0.1.0"
  description = "Superpowers skill compliance benchmark"
  requires-python = ">=3.11"
  dependencies = [
  "click>=8.1",
  "pyyaml>=6.0",
  "anthropic>=0.42",
  "jinja2>=3.1",
  "pydantic>=2.0",
  ]
  
  [project.optional-dependencies]
  dev = ["pytest>=8.0"]
  
  [project.scripts]
  drill = "drill.cli:main"
  
  [tool.setuptools.packages.find]
  include = ["drill*", "setup_helpers*"]
  

• [ ] Step 2: Create drill/init.py

  """Drill: Superpowers skill compliance benchmark."""
  
  __version__ = "0.1.0"
  

• [ ] Step 3: Create .gitignore

  results/
  __pycache__/
  *.pyc
  *.egg-info/
  dist/
  build/
  .venv/
  

• [ ] Step 4: Create README.md

  # Drill
  
  Superpowers skill compliance benchmark. Drives AI coding agents through
  tmux sessions and evaluates whether they follow superpowers workflows.
  
  See [docs/design.md](docs/design.md) for the full design spec.
  
  ## Setup
  
  

  bash

pip install -e ".[dev]"

## Usage

bash export SUPERPOWERS_ROOT=/path/to/superpowers export ANTHROPIC_API_KEY=sk-...

drill run worktree-creation-from-main --backend claude drill compare worktree-creation-from-main drill list

• Step 5: Install in dev mode and verify

Run: cd /Users/drewritter/prime-rad/drill && pip install -e ".[dev]" Expected: Installs successfully, drill --help shows usage

• [ ] Step 6: Commit

  git add pyproject.toml drill/__init__.py .gitignore README.md
  git commit -m "chore: project scaffold with pyproject.toml and drill entry point"
  

---

Task 2: Backend Config Loader

Files:

• Create: drill/backend.py
• Create: backends/claude.yaml
• Create: backends/codex.yaml

• Create: tests/test_backend.py

• [ ] Step 1: Write the failing test

  # tests/test_backend.py
  import os
  import pytest
  from pathlib import Path
  
  from drill.backend import Backend, load_backend
  
  
  @pytest.fixture
  def backends_dir():
  return Path(__file__).parent.parent / "backends"
  
  
  class TestLoadBackend:
  def test_loads_claude_backend(self, backends_dir):
      backend = load_backend("claude", backends_dir)
      assert backend.name == "claude"
      assert backend.cli == "claude"
      assert "--dangerously-skip-permissions" in backend.args
  
  def test_loads_codex_backend(self, backends_dir):
      backend = load_backend("codex", backends_dir)
      assert backend.name == "codex"
      assert backend.cli == "codex"
  
  def test_unknown_backend_raises(self, backends_dir):
      with pytest.raises(FileNotFoundError):
          load_backend("nonexistent", backends_dir)
  
  
  class TestBackendBuildCommand:
  def test_claude_build_command(self, backends_dir, monkeypatch):
      monkeypatch.setenv("SUPERPOWERS_ROOT", "/tmp/superpowers")
      backend = load_backend("claude", backends_dir)
      cmd = backend.build_command("/tmp/workdir")
      assert cmd[0] == "claude"
      assert "--plugin-dir" in cmd
      assert "/tmp/superpowers" in cmd
  
  def test_codex_build_command(self, backends_dir, monkeypatch):
      monkeypatch.setenv("SUPERPOWERS_ROOT", "/tmp/superpowers")
      backend = load_backend("codex", backends_dir)
      cmd = backend.build_command("/tmp/workdir")
      assert cmd[0] == "codex"
  
  
  class TestBackendEnvValidation:
  def test_missing_env_raises(self, backends_dir, monkeypatch):
      monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
      monkeypatch.delenv("SUPERPOWERS_ROOT", raising=False)
      backend = load_backend("claude", backends_dir)
      with pytest.raises(EnvironmentError, match="ANTHROPIC_API_KEY"):
          backend.validate_env()
  
  
  class TestBackendIdleDetection:
  def test_ready_pattern_matches(self, backends_dir):
      backend = load_backend("claude", backends_dir)
      assert backend.is_ready_line("❯ ")
      assert backend.is_ready_line("Human: ")
      assert not backend.is_ready_line("Running tool...")
  

• [ ] Step 2: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_backend.py -v Expected: FAIL — ModuleNotFoundError: No module named 'drill.backend'

• Step 3: Create backend YAML files

Create backends/claude.yaml:

name: claude
cli: claude
args:
  - "--dangerously-skip-permissions"
  - "--plugin-dir"
  - "${SUPERPOWERS_ROOT}"
required_env:
  - ANTHROPIC_API_KEY
  - SUPERPOWERS_ROOT
hooks:
  pre_run: []
  post_run: []
shutdown: "/exit"
idle:
  quiescence_seconds: 3
  ready_pattern: "^❯|^\\$|Human:"
startup_timeout: 30
terminal:
  cols: 200
  rows: 50
session_logs:
  pattern: "~/.claude/projects/**/session-*.jsonl"

Create backends/codex.yaml:

name: codex
cli: codex
args:
  - "--dangerously-bypass-approvals-and-sandbox"
required_env:
  - OPENAI_API_KEY
  - SUPERPOWERS_ROOT
hooks:
  pre_run:
    - symlink_superpowers
  post_run: []
shutdown: "<<KEY:ctrl-d>>"
idle:
  quiescence_seconds: 5
  ready_pattern: "codex>|^>"
startup_timeout: 30
terminal:
  cols: 200
  rows: 50
session_logs:
  pattern: "~/.codex/sessions/rollout-*.jsonl"

• [ ] Step 4: Write the implementation

  # drill/backend.py
  """Backend config loader and command builder."""
  
  from __future__ import annotations
  
  import os
  import re
  from dataclasses import dataclass, field
  from pathlib import Path
  
  import yaml
  
  
  @dataclass
  class Backend:
  name: str
  cli: str
  args: list[str]
  required_env: list[str]
  hooks: dict[str, list[str]]
  shutdown: str
  idle: dict[str, any]
  startup_timeout: int
  terminal: dict[str, int]
  session_logs: dict[str, str]
  
  def build_command(self, workdir: str) -> list[str]:
      """Build the full CLI invocation with env var interpolation."""
      resolved = []
      for arg in self.args:
          resolved.append(_interpolate_env(arg))
      return [self.cli, *resolved]
  
  def validate_env(self) -> None:
      """Raise EnvironmentError if any required env vars are missing."""
      missing = [v for v in self.required_env if not os.environ.get(v)]
      if missing:
          raise EnvironmentError(
              f"Missing required environment variables for {self.name} backend: "
              + ", ".join(missing)
          )
  
  def is_ready_line(self, line: str) -> bool:
      """Check if a terminal line matches the idle ready pattern."""
      pattern = self.idle.get("ready_pattern", "")
      return bool(re.search(pattern, line))
  
  @property
  def quiescence_seconds(self) -> float:
      return self.idle.get("quiescence_seconds", 5)
  
  @property
  def cols(self) -> int:
      return self.terminal.get("cols", 200)
  
  @property
  def rows(self) -> int:
      return self.terminal.get("rows", 50)
  
  
  def load_backend(name: str, backends_dir: Path) -> Backend:
  """Load a backend config from YAML."""
  path = backends_dir / f"{name}.yaml"
  if not path.exists():
      raise FileNotFoundError(f"Backend config not found: {path}")
  with open(path) as f:
      data = yaml.safe_load(f)
  return Backend(
      name=data["name"],
      cli=data["cli"],
      args=data.get("args", []),
      required_env=data.get("required_env", []),
      hooks=data.get("hooks", {"pre_run": [], "post_run": []}),
      shutdown=data.get("shutdown", "/exit"),
      idle=data.get("idle", {}),
      startup_timeout=data.get("startup_timeout", 30),
      terminal=data.get("terminal", {"cols": 200, "rows": 50}),
      session_logs=data.get("session_logs", {}),
  )
  
  
  def _interpolate_env(value: str) -> str:
  """Replace ${VAR} with environment variable values."""
  def replacer(match):
      var = match.group(1)
      val = os.environ.get(var)
      if val is None:
          raise EnvironmentError(f"Environment variable {var} not set")
      return val
  return re.sub(r"\$\{(\w+)\}", replacer, value)
  

• [ ] Step 5: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_backend.py -v Expected: All tests PASS

• [ ] Step 6: Commit

  git add drill/backend.py backends/ tests/test_backend.py
  git commit -m "feat: backend config loader with YAML parsing and env validation"
  

---

Task 3: tmux Session Manager

Files:

• Create: drill/session.py

• Create: tests/test_session.py

• [ ] Step 1: Write the failing test

  # tests/test_session.py
  import subprocess
  import time
  import pytest
  
  from drill.session import TmuxSession
  
  
  class TestTmuxSession:
  def test_create_and_kill(self):
      session = TmuxSession(name="drill-test-create", cols=80, rows=24)
      session.create()
      # Verify session exists
      result = subprocess.run(
          ["tmux", "has-session", "-t", "drill-test-create"],
          capture_output=True,
      )
      assert result.returncode == 0
      session.kill()
      # Verify session is gone
      result = subprocess.run(
          ["tmux", "has-session", "-t", "drill-test-create"],
          capture_output=True,
      )
      assert result.returncode != 0
  
  def test_send_keys_and_capture(self):
      session = TmuxSession(name="drill-test-keys", cols=80, rows=24)
      session.create()
      try:
          session.send_keys("echo hello-drill-test")
          time.sleep(0.5)
          output = session.capture()
          assert "hello-drill-test" in output
      finally:
          session.kill()
  
  def test_launch_command(self, tmp_path):
      session = TmuxSession(name="drill-test-launch", cols=80, rows=24)
      session.create()
      try:
          session.launch(["python3", "-c", "import time; time.sleep(30)"], cwd=str(tmp_path))
          time.sleep(0.5)
          output = session.capture()
          # Process should be running, not showing shell prompt
          assert session.is_process_alive()
      finally:
          session.kill()
  
  def test_send_special_key(self):
      session = TmuxSession(name="drill-test-special", cols=80, rows=24)
      session.create()
      try:
          session.send_keys("cat")  # start cat, which reads stdin
          time.sleep(0.3)
          session.send_special_key("ctrl-c")
          time.sleep(0.3)
          # After ctrl-c, cat should have exited
          output = session.capture()
          assert "^C" in output or output.endswith("$")
      finally:
          session.kill()
  

• [ ] Step 2: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_session.py -v Expected: FAIL — ModuleNotFoundError: No module named 'drill.session'

• [ ] Step 3: Write the implementation

  # drill/session.py
  """tmux session management for driving agent CLI sessions."""
  
  from __future__ import annotations
  
  import subprocess
  import time
  
  
  class TmuxSession:
  """Manages a tmux session for driving an agent CLI."""
  
  def __init__(self, name: str, cols: int = 200, rows: int = 50):
      self.name = name
      self.cols = cols
      self.rows = rows
  
  def create(self) -> None:
      """Create a new detached tmux session."""
      subprocess.run(
          [
              "tmux", "new-session",
              "-d",
              "-s", self.name,
              "-x", str(self.cols),
              "-y", str(self.rows),
          ],
          check=True,
      )
  
  def launch(self, command: list[str], cwd: str) -> None:
      """Launch a command inside the tmux session."""
      cmd_str = " ".join(command)
      self.send_keys(f"cd {cwd} && {cmd_str}")
  
  def send_keys(self, text: str) -> None:
      """Send keystrokes to the tmux session, followed by Enter."""
      subprocess.run(
          ["tmux", "send-keys", "-t", self.name, text, "Enter"],
          check=True,
      )
  
  def send_special_key(self, key: str) -> None:
      """Send a special key like ctrl-c, ctrl-d."""
      key_map = {
          "ctrl-c": "C-c",
          "ctrl-d": "C-d",
          "ctrl-z": "C-z",
          "enter": "Enter",
          "escape": "Escape",
      }
      tmux_key = key_map.get(key, key)
      subprocess.run(
          ["tmux", "send-keys", "-t", self.name, tmux_key],
          check=True,
      )
  
  def capture(self) -> str:
      """Capture the current terminal pane content."""
      result = subprocess.run(
          ["tmux", "capture-pane", "-t", self.name, "-p"],
          capture_output=True,
          text=True,
          check=True,
      )
      return result.stdout
  
  def is_process_alive(self) -> bool:
      """Check if the process in the pane is still running."""
      result = subprocess.run(
          [
              "tmux", "list-panes", "-t", self.name,
              "-F", "#{pane_dead}",
          ],
          capture_output=True,
          text=True,
      )
      return result.stdout.strip() == "0"
  
  def kill(self) -> None:
      """Kill the tmux session."""
      subprocess.run(
          ["tmux", "kill-session", "-t", self.name],
          capture_output=True,  # don't fail if already dead
      )
  

• [ ] Step 4: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_session.py -v Expected: All tests PASS

• [ ] Step 5: Commit

  git add drill/session.py tests/test_session.py
  git commit -m "feat: tmux session manager with send-keys, capture, and special key support"
  

---

Task 4: Setup Helpers and Template Repo

Files:

• Create: setup_helpers/__init__.py
• Create: setup_helpers/base.py
• Create: setup_helpers/worktree.py
• Create: fixtures/template-repo/ (with contents)
• Create: drill/setup.py

• Create: tests/test_setup.py

• [ ] Step 1: Create the template repo fixture

  cd /Users/drewritter/prime-rad/drill
  mkdir -p fixtures/template-repo/src
  

Create fixtures/template-repo/package.json:

{
  "name": "drill-test-project",
  "version": "1.0.0",
  "description": "Test project for Drill scenarios",
  "main": "src/index.js"
}

Create fixtures/template-repo/src/index.js:

const { greet } = require('./utils');

function main() {
  console.log(greet('world'));
}

main();

Create fixtures/template-repo/src/utils.js:

function greet(name) {
  return `Hello, ${name}!`;
}

module.exports = { greet };

Create fixtures/template-repo/README.md:

# Test Project

A minimal project for Drill test scenarios.

Initialize git history:

cd fixtures/template-repo
git init
git add package.json README.md
git commit -m "initial commit"
git add src/utils.js
git commit -m "add utils module"
git add src/index.js
git commit -m "add entry point"
cd ../..

• [ ] Step 2: Write the failing test

  # tests/test_setup.py
  import os
  import subprocess
  import pytest
  from pathlib import Path
  
  from drill.setup import clone_template, run_assertions
  from setup_helpers.base import create_base_repo
  from setup_helpers.worktree import add_worktree, detach_head, symlink_superpowers
  
  
  @pytest.fixture
  def fixtures_dir():
  return Path(__file__).parent.parent / "fixtures"
  
  
  @pytest.fixture
  def work_dir(tmp_path):
  return tmp_path / "test-repo"
  
  
  class TestCloneTemplate:
  def test_clones_template_repo(self, fixtures_dir, work_dir):
      clone_template(fixtures_dir / "template-repo", work_dir)
      assert (work_dir / "package.json").exists()
      assert (work_dir / "src" / "index.js").exists()
      # Should have git history
      result = subprocess.run(
          ["git", "log", "--oneline"],
          cwd=work_dir,
          capture_output=True,
          text=True,
      )
      assert "initial commit" in result.stdout
  
  
  class TestCreateBaseRepo:
  def test_creates_base_repo(self, fixtures_dir, work_dir):
      create_base_repo(work_dir, fixtures_dir / "template-repo")
      assert (work_dir / "package.json").exists()
      result = subprocess.run(
          ["git", "branch", "--show-current"],
          cwd=work_dir,
          capture_output=True,
          text=True,
      )
      assert result.stdout.strip() == "main"
  
  
  class TestWorktreeHelpers:
  def test_add_worktree(self, fixtures_dir, work_dir):
      create_base_repo(work_dir, fixtures_dir / "template-repo")
      wt_path = work_dir.parent / "feature-wt"
      add_worktree(work_dir, "feature-branch", str(wt_path))
      assert wt_path.exists()
      result = subprocess.run(
          ["git", "worktree", "list"],
          cwd=work_dir,
          capture_output=True,
          text=True,
      )
      assert "feature-branch" in result.stdout
  
  def test_detach_head(self, fixtures_dir, work_dir):
      create_base_repo(work_dir, fixtures_dir / "template-repo")
      wt_path = work_dir.parent / "detached-wt"
      add_worktree(work_dir, "tmp-branch", str(wt_path))
      detach_head(str(wt_path))
      result = subprocess.run(
          ["git", "branch", "--show-current"],
          cwd=wt_path,
          capture_output=True,
          text=True,
      )
      assert result.stdout.strip() == ""  # detached = no branch
  
  def test_symlink_superpowers(self, fixtures_dir, work_dir, tmp_path):
      create_base_repo(work_dir, fixtures_dir / "template-repo")
      fake_superpowers = tmp_path / "superpowers" / "skills"
      fake_superpowers.mkdir(parents=True)
      symlink_superpowers(work_dir, str(tmp_path / "superpowers"))
      link = work_dir / ".agents" / "skills" / "superpowers"
      assert link.is_symlink()
  
  
  class TestRunAssertions:
  def test_passing_assertions(self, fixtures_dir, work_dir):
      create_base_repo(work_dir, fixtures_dir / "template-repo")
      assertions = [
          "git rev-parse --is-inside-work-tree",
          "git branch --show-current | grep main",
      ]
      # Should not raise
      run_assertions(assertions, work_dir)
  
  def test_failing_assertion_raises(self, fixtures_dir, work_dir):
      create_base_repo(work_dir, fixtures_dir / "template-repo")
      assertions = ["git branch --show-current | grep nonexistent"]
      with pytest.raises(AssertionError, match="Setup assertion failed"):
          run_assertions(assertions, work_dir)
  

• [ ] Step 3: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_setup.py -v Expected: FAIL — ModuleNotFoundError

• Step 4: Write setup_helpers

Create setup_helpers/__init__.py:

"""Setup helpers for Drill scenarios."""

from setup_helpers.base import create_base_repo
from setup_helpers.worktree import add_worktree, detach_head, symlink_superpowers

HELPER_REGISTRY = {
    "create_base_repo": create_base_repo,
    "add_worktree": add_worktree,
    "detach_head": detach_head,
    "symlink_superpowers": symlink_superpowers,
}

Create setup_helpers/base.py:

"""Base setup helpers."""

from __future__ import annotations

import subprocess
from pathlib import Path


def create_base_repo(workdir: Path, template_dir: Path) -> None:
    """Clone the template repo to workdir."""
    subprocess.run(
        ["git", "clone", str(template_dir), str(workdir)],
        check=True,
        capture_output=True,
    )

Create setup_helpers/worktree.py:

"""Worktree-specific setup helpers."""

from __future__ import annotations

import os
import subprocess
from pathlib import Path


def add_worktree(repo_dir: Path, branch: str, worktree_path: str) -> None:
    """Create a git worktree at the given path."""
    subprocess.run(
        ["git", "worktree", "add", "-b", branch, worktree_path],
        cwd=repo_dir,
        check=True,
        capture_output=True,
    )


def detach_head(worktree_path: str) -> None:
    """Detach HEAD in a worktree (simulates Codex App state)."""
    # Get current commit hash
    result = subprocess.run(
        ["git", "rev-parse", "HEAD"],
        cwd=worktree_path,
        capture_output=True,
        text=True,
        check=True,
    )
    commit = result.stdout.strip()
    # Get the branch name so we can delete it after detaching
    result = subprocess.run(
        ["git", "branch", "--show-current"],
        cwd=worktree_path,
        capture_output=True,
        text=True,
        check=True,
    )
    branch = result.stdout.strip()
    # Detach HEAD
    subprocess.run(
        ["git", "checkout", "--detach", commit],
        cwd=worktree_path,
        check=True,
        capture_output=True,
    )
    # Delete the temporary branch
    if branch:
        subprocess.run(
            ["git", "branch", "-D", branch],
            cwd=worktree_path,
            capture_output=True,
        )


def symlink_superpowers(workdir: Path, superpowers_root: str) -> None:
    """Create .agents/skills/superpowers symlink for Codex discovery."""
    skills_dir = Path(workdir) / ".agents" / "skills"
    skills_dir.mkdir(parents=True, exist_ok=True)
    target = Path(superpowers_root) / "skills"
    link = skills_dir / "superpowers"
    link.symlink_to(target)

• [ ] Step 5: Write drill/setup.py

  # drill/setup.py
  """Test repo setup: template cloning, helper dispatch, assertion runner."""
  
  from __future__ import annotations
  
  import subprocess
  from pathlib import Path
  
  from setup_helpers import HELPER_REGISTRY
  
  
  def clone_template(template_dir: Path, workdir: Path) -> None:
  """Clone the template repo to a working directory."""
  subprocess.run(
      ["git", "clone", str(template_dir), str(workdir)],
      check=True,
      capture_output=True,
  )
  
  
  def run_helpers(
  helper_names: list[str],
  workdir: Path,
  fixtures_dir: Path,
  ) -> None:
  """Run named setup helpers against the working directory."""
  for name in helper_names:
      helper = HELPER_REGISTRY.get(name)
      if helper is None:
          raise ValueError(f"Unknown setup helper: {name}")
      if name == "create_base_repo":
          helper(workdir, fixtures_dir / "template-repo")
      elif name == "symlink_superpowers":
          import os
          helper(workdir, os.environ["SUPERPOWERS_ROOT"])
      else:
          # All other helpers take workdir as single arg
          helper(workdir)
  
  
  def run_assertions(assertions: list[str], workdir: Path) -> None:
  """Run shell assertion commands. Raise if any fail."""
  for assertion in assertions:
      result = subprocess.run(
          assertion,
          shell=True,
          cwd=workdir,
          capture_output=True,
          text=True,
      )
      if result.returncode != 0:
          raise AssertionError(
              f"Setup assertion failed: {assertion}\n"
              f"stdout: {result.stdout}\n"
              f"stderr: {result.stderr}"
          )
  

• [ ] Step 6: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_setup.py -v Expected: All tests PASS

• [ ] Step 7: Commit

  git add fixtures/ setup_helpers/ drill/setup.py tests/test_setup.py
  git commit -m "feat: template repo, setup helpers, and assertion runner"
  

---

Task 5: Actor LLM

Files:

• Create: drill/actor.py
• Create: prompts/actor.md

• Create: tests/test_actor.py

• [ ] Step 1: Write the failing test

  # tests/test_actor.py
  import json
  import pytest
  from unittest.mock import MagicMock, patch
  
  from drill.actor import Actor, ActorAction
  
  
  class TestActorAction:
  def test_parse_type_action(self):
      action = ActorAction.from_tool_result({"action": "type", "text": "create a worktree"})
      assert action.action == "type"
      assert action.text == "create a worktree"
  
  def test_parse_done_action(self):
      action = ActorAction.from_tool_result({"action": "done"})
      assert action.action == "done"
  
  def test_parse_stuck_action(self):
      action = ActorAction.from_tool_result({"action": "stuck"})
      assert action.action == "stuck"
  
  def test_parse_key_action(self):
      action = ActorAction.from_tool_result({"action": "key", "key": "ctrl-c"})
      assert action.action == "key"
      assert action.key == "ctrl-c"
  
  
  class TestActorPrompt:
  def test_builds_system_prompt_naive(self):
      actor = Actor(model="claude-sonnet-4-6", temperature=0.7)
      prompt = actor.build_system_prompt(
          posture="naive",
          intents=["Ask the agent to create a worktree"],
      )
      assert "naive" in prompt.lower() or "plain language" in prompt.lower()
      assert "create a worktree" in prompt
  
  def test_builds_system_prompt_spec_aware(self):
      actor = Actor(model="claude-sonnet-4-6", temperature=0.7)
      prompt = actor.build_system_prompt(
          posture="spec-aware",
          intents=["Use the worktree skill to create an isolated workspace"],
      )
      assert "spec" in prompt.lower() or "skill" in prompt.lower()
  
  
  class TestActorContext:
  def test_appends_terminal_captures(self):
      actor = Actor(model="claude-sonnet-4-6", temperature=0.7)
      actor.append_capture("Screen 1: Welcome to Claude")
      actor.append_capture("Screen 2: ❯ ")
      messages = actor.build_messages()
      assert len(messages) == 2
      assert "Screen 1" in messages[0]["content"]
      assert "Screen 2" in messages[1]["content"]
  

• [ ] Step 2: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_actor.py -v Expected: FAIL — ModuleNotFoundError

• Step 3: Create the actor prompt template

Create prompts/actor.md:

You are simulating a user interacting with an AI coding agent in a terminal.

{% if posture == "naive" %}
You are a developer who wants to accomplish a task. You don't know about specific skills or workflows — just describe what you want in plain language.
{% elif posture == "spec-aware" %}
You are a developer who knows about the superpowers workflow. You may reference specific skills or conventions by name (e.g., "use the worktree skill", "follow the using-git-worktrees pattern").
{% endif %}

Goals (in rough priority order):
{% for intent in intents %}
- {{ intent }}
{% endfor %}

Rules:
- Decide what to do based on what's currently on screen.
- Goals are not a script — some are conditional. Act on them when relevant.
- Type natural, concise messages like a real developer would.
- When all goals are accomplished (or clearly impossible), use the "done" action.
- If you're stuck and cannot make progress, use the "stuck" action.

• [ ] Step 4: Write the implementation

  # drill/actor.py
  """Actor LLM: simulates a user driving an agent session."""
  
  from __future__ import annotations
  
  from dataclasses import dataclass, field
  from pathlib import Path
  
  import anthropic
  from jinja2 import Template
  
  
  ACTOR_TOOL = {
  "name": "terminal_action",
  "description": "Send an action to the terminal session.",
  "input_schema": {
      "type": "object",
      "properties": {
          "action": {
              "type": "string",
              "enum": ["type", "done", "stuck", "key"],
              "description": "The action to take.",
          },
          "text": {
              "type": "string",
              "description": "Text to type (only for 'type' action).",
          },
          "key": {
              "type": "string",
              "description": "Special key to send (only for 'key' action, e.g., 'ctrl-c').",
          },
      },
      "required": ["action"],
  },
  }
  
  
  @dataclass
  class ActorAction:
  action: str  # "type", "done", "stuck", "key"
  text: str | None = None
  key: str | None = None
  
  @classmethod
  def from_tool_result(cls, data: dict) -> ActorAction:
      return cls(
          action=data["action"],
          text=data.get("text"),
          key=data.get("key"),
      )
  
  
  class Actor:
  """Drives agent sessions by deciding what a simulated user would type."""
  
  def __init__(self, model: str = "claude-sonnet-4-6", temperature: float = 0.7):
      self.model = model
      self.temperature = temperature
      self.captures: list[str] = []
      self._system_prompt: str | None = None
      self._client = anthropic.Anthropic()
  
  def build_system_prompt(self, posture: str, intents: list[str]) -> str:
      """Render the actor system prompt from template."""
      template_path = Path(__file__).parent.parent / "prompts" / "actor.md"
      template = Template(template_path.read_text())
      self._system_prompt = template.render(posture=posture, intents=intents)
      return self._system_prompt
  
  def append_capture(self, terminal_output: str) -> None:
      """Append a terminal capture to the rolling context."""
      self.captures.append(terminal_output)
  
  def build_messages(self) -> list[dict]:
      """Build the message list from terminal captures."""
      messages = []
      for capture in self.captures:
          messages.append({"role": "user", "content": capture})
      return messages
  
  def decide(self) -> ActorAction:
      """Call the LLM to decide the next action."""
      response = self._client.messages.create(
          model=self.model,
          max_tokens=1024,
          temperature=self.temperature,
          system=self._system_prompt,
          tools=[ACTOR_TOOL],
          tool_choice={"type": "tool", "name": "terminal_action"},
          messages=self.build_messages(),
      )
      # Extract the tool use block
      for block in response.content:
          if block.type == "tool_use":
              return ActorAction.from_tool_result(block.input)
      raise RuntimeError("Actor did not return a tool_use block")
  

• [ ] Step 5: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_actor.py -v Expected: All tests PASS (no live API calls — only testing parsing and prompt building)

• [ ] Step 6: Commit

  git add drill/actor.py prompts/actor.md tests/test_actor.py
  git commit -m "feat: actor LLM with structured tool_use output and prompt template"
  

---

Task 6: Verifier LLM

Files:

• Create: drill/verifier.py
• Create: prompts/verifier.md

• Create: tests/test_verifier.py

• [ ] Step 1: Write the failing test

  # tests/test_verifier.py
  import json
  import pytest
  from unittest.mock import MagicMock, patch
  
  from drill.verifier import Verifier, Verdict, CriterionResult
  
  
  class TestVerdict:
  def test_parse_valid_verdict(self):
      data = {
          "criteria": [
              {
                  "criterion": "Agent detected on main",
                  "verdict": "pass",
                  "evidence": "Terminal showed 'main branch detected'",
                  "rationale": "Agent correctly identified the branch",
              }
          ],
          "observations": ["Agent was very fast"],
          "summary": "Passed all checks",
      }
      verdict = Verdict.model_validate(data)
      assert len(verdict.criteria) == 1
      assert verdict.criteria[0].verdict == "pass"
      assert verdict.score == "1/1"
  
  def test_score_calculation(self):
      data = {
          "criteria": [
              {"criterion": "A", "verdict": "pass", "evidence": "e", "rationale": "r"},
              {"criterion": "B", "verdict": "fail", "evidence": "e", "rationale": "r"},
              {"criterion": "C", "verdict": "pass", "evidence": "e", "rationale": "r"},
          ],
          "observations": [],
          "summary": "Mixed results",
      }
      verdict = Verdict.model_validate(data)
      assert verdict.score == "2/3"
      assert verdict.passed is False
  
  def test_all_pass(self):
      data = {
          "criteria": [
              {"criterion": "A", "verdict": "pass", "evidence": "e", "rationale": "r"},
          ],
          "observations": [],
          "summary": "Good",
      }
      verdict = Verdict.model_validate(data)
      assert verdict.passed is True
  
  
  class TestVerifierPrompt:
  def test_builds_system_prompt(self):
      verifier = Verifier(model="claude-sonnet-4-6", temperature=0.0)
      prompt = verifier.build_system_prompt()
      assert "criterion" in prompt.lower()
      assert "evidence" in prompt.lower()
      assert "JSON" in prompt
  

• [ ] Step 2: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_verifier.py -v Expected: FAIL — ModuleNotFoundError

• Step 3: Create the verifier prompt template

Create prompts/verifier.md:

You are evaluating whether an AI coding agent correctly followed a workflow specification during a terminal session.

You will receive:
1. Terminal session log (what was displayed on screen)
2. Filesystem state after the session (file tree, git state, worktree list)
3. Tool call log (structured record of every tool the agent invoked)

Evaluate each criterion independently. For each, respond with:
- verdict: pass or fail
- evidence: specific quotes from the logs or filesystem state
- rationale: why this constitutes a pass or fail

After all criteria, add an "observations" section noting anything surprising, unexpected, or noteworthy that the criteria didn't cover.

Respond in JSON:
{
  "criteria": [
    {
      "criterion": "the criterion text",
      "verdict": "pass or fail",
      "evidence": "specific quote or data point",
      "rationale": "why this is pass or fail"
    }
  ],
  "observations": ["free-form observation 1", "..."],
  "summary": "one-line overall assessment"
}

• [ ] Step 4: Write the implementation

  # drill/verifier.py
  """Verifier LLM: evaluates agent session against criteria."""
  
  from __future__ import annotations
  
  import json
  from pathlib import Path
  
  import anthropic
  from jinja2 import Template
  from pydantic import BaseModel
  
  
  class CriterionResult(BaseModel):
  criterion: str
  verdict: str  # "pass" or "fail"
  evidence: str
  rationale: str
  
  
  class Verdict(BaseModel):
  criteria: list[CriterionResult]
  observations: list[str]
  summary: str
  
  @property
  def score(self) -> str:
      passed = sum(1 for c in self.criteria if c.verdict == "pass")
      return f"{passed}/{len(self.criteria)}"
  
  @property
  def passed(self) -> bool:
      return all(c.verdict == "pass" for c in self.criteria)
  
  
  class Verifier:
  """Evaluates agent sessions against verification criteria."""
  
  MAX_RETRIES = 3
  
  def __init__(self, model: str = "claude-sonnet-4-6", temperature: float = 0.0):
      self.model = model
      self.temperature = temperature
      self._client = anthropic.Anthropic()
  
  def build_system_prompt(self) -> str:
      """Render the verifier system prompt from template."""
      template_path = Path(__file__).parent.parent / "prompts" / "verifier.md"
      return template_path.read_text()
  
  def verify(
      self,
      session_log: str,
      filesystem_json: str,
      tool_calls_jsonl: str,
      criteria: list[str],
  ) -> Verdict:
      """Run the verifier against a completed session."""
      system = self.build_system_prompt()
      user_content = (
          "## Terminal Session Log\n\n"
          f"```\n{session_log}\n```\n\n"
          "## Filesystem State\n\n"
          f"```json\n{filesystem_json}\n```\n\n"
          "## Tool Call Log\n\n"
          f"```jsonl\n{tool_calls_jsonl}\n```\n\n"
          "## Criteria to Evaluate\n\n"
          + "\n".join(f"- {c}" for c in criteria)
      )
  
      for attempt in range(self.MAX_RETRIES):
          response = self._client.messages.create(
              model=self.model,
              max_tokens=4096,
              temperature=self.temperature,
              system=system,
              messages=[{"role": "user", "content": user_content}],
          )
          text = response.content[0].text
          # Extract JSON from response (may be wrapped in markdown fences)
          json_str = _extract_json(text)
          try:
              return Verdict.model_validate_json(json_str)
          except Exception:
              if attempt == self.MAX_RETRIES - 1:
                  raise
              continue
  
      raise RuntimeError("Verifier failed to return valid JSON")
  
  
  def _extract_json(text: str) -> str:
  """Extract JSON from text that may be wrapped in markdown code fences."""
  # Try to find JSON in code fences
  if "```json" in text:
      start = text.index("```json") + 7
      end = text.index("```", start)
      return text[start:end].strip()
  if "```" in text:
      start = text.index("```") + 3
      end = text.index("```", start)
      return text[start:end].strip()
  # Try raw JSON
  start = text.index("{")
  end = text.rindex("}") + 1
  return text[start:end]
  

• [ ] Step 5: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_verifier.py -v Expected: All tests PASS

• [ ] Step 6: Commit

  git add drill/verifier.py prompts/verifier.md tests/test_verifier.py
  git commit -m "feat: verifier LLM with pydantic verdict schema and retry logic"
  

---

Task 7: Log Normalizer

Files:

• Create: drill/normalizer.py

• Create: tests/test_normalizer.py

• [ ] Step 1: Write the failing test

  # tests/test_normalizer.py
  import json
  import pytest
  from pathlib import Path
  
  from drill.normalizer import normalize_claude_logs, normalize_codex_logs, snapshot_log_dir, collect_new_logs
  
  
  class TestSnapshotAndCollect:
  def test_snapshot_and_collect_new_files(self, tmp_path):
      log_dir = tmp_path / "logs"
      log_dir.mkdir()
      # Pre-existing file
      (log_dir / "old.jsonl").write_text('{"old": true}\n')
      snapshot = snapshot_log_dir(log_dir)
      # Simulate new file created during session
      (log_dir / "new.jsonl").write_text('{"new": true}\n')
      new_files = collect_new_logs(log_dir, snapshot)
      assert len(new_files) == 1
      assert new_files[0].name == "new.jsonl"
  
  def test_empty_dir_returns_empty(self, tmp_path):
      log_dir = tmp_path / "logs"
      log_dir.mkdir()
      snapshot = snapshot_log_dir(log_dir)
      new_files = collect_new_logs(log_dir, snapshot)
      assert new_files == []
  
  
  class TestNormalizeClaudeLogs:
  def test_normalizes_tool_use(self):
      lines = [
          json.dumps({
              "type": "tool_use",
              "name": "EnterWorktree",
              "input": {"branch": "add-login"},
          }),
          json.dumps({
              "type": "tool_use",
              "name": "Bash",
              "input": {"command": "git status"},
          }),
          json.dumps({
              "type": "text",
              "text": "I'll create a worktree",
          }),
      ]
      raw = "\n".join(lines)
      normalized = normalize_claude_logs(raw)
      assert len(normalized) == 2
      assert normalized[0]["tool"] == "EnterWorktree"
      assert normalized[0]["source"] == "native"
      assert normalized[1]["tool"] == "Bash"
      assert normalized[1]["source"] == "shell"
  
  
  class TestNormalizeCodexLogs:
  def test_normalizes_local_shell_call(self):
      lines = [
          json.dumps({
              "type": "response_item",
              "item": {
                  "type": "local_shell_call",
                  "action": {"command": ["git", "worktree", "add", "feature"]},
                  "status": "completed",
              }
          }),
          json.dumps({
              "type": "response_item",
              "item": {
                  "type": "message",
                  "content": [{"text": "Creating worktree"}],
              }
          }),
      ]
      raw = "\n".join(lines)
      normalized = normalize_codex_logs(raw)
      assert len(normalized) == 1
      assert normalized[0]["tool"] == "Bash"
      assert "git worktree add" in normalized[0]["args"]["command"]
      assert normalized[0]["source"] == "shell"
  

• [ ] Step 2: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_normalizer.py -v Expected: FAIL — ModuleNotFoundError

• [ ] Step 3: Write the implementation

  # drill/normalizer.py
  """Normalizes backend-specific session logs to a common tool call schema."""
  
  from __future__ import annotations
  
  import json
  from pathlib import Path
  
  # Tools that are native (not shell commands)
  NATIVE_TOOLS = {
  "EnterWorktree", "ExitWorktree", "EnterPlanMode", "ExitPlanMode",
  "TaskCreate", "TaskUpdate", "TaskList", "TaskGet",
  "Skill", "Agent", "Read", "Write", "Edit", "Glob", "Grep",
  }
  
  
  def snapshot_log_dir(log_dir: Path) -> set[str]:
  """Snapshot the current files in a log directory."""
  if not log_dir.exists():
      return set()
  return {f.name for f in log_dir.iterdir() if f.is_file()}
  
  
  def collect_new_logs(log_dir: Path, snapshot: set[str]) -> list[Path]:
  """Find files created after the snapshot was taken."""
  if not log_dir.exists():
      return []
  current = {f.name for f in log_dir.iterdir() if f.is_file()}
  new_names = current - snapshot
  return [log_dir / name for name in sorted(new_names)]
  
  
  def normalize_claude_logs(raw_content: str) -> list[dict]:
  """Normalize Claude Code session log to common schema."""
  results = []
  for line in raw_content.strip().split("\n"):
      if not line.strip():
          continue
      try:
          entry = json.loads(line)
      except json.JSONDecodeError:
          continue
      if entry.get("type") == "tool_use":
          tool_name = entry.get("name", "")
          source = "native" if tool_name in NATIVE_TOOLS else "shell"
          results.append({
              "tool": tool_name,
              "args": entry.get("input", {}),
              "source": source,
          })
  return results
  
  
  def normalize_codex_logs(raw_content: str) -> list[dict]:
  """Normalize Codex rollout log to common schema."""
  results = []
  for line in raw_content.strip().split("\n"):
      if not line.strip():
          continue
      try:
          entry = json.loads(line)
      except json.JSONDecodeError:
          continue
      if entry.get("type") != "response_item":
          continue
      item = entry.get("item", {})
      item_type = item.get("type", "")
      if item_type == "local_shell_call":
          action = item.get("action", {})
          cmd = action.get("command", [])
          cmd_str = " ".join(cmd) if isinstance(cmd, list) else str(cmd)
          results.append({
              "tool": "Bash",
              "args": {"command": cmd_str},
              "source": "shell",
          })
      elif item_type == "function_call":
          name = item.get("name", "")
          source = "native" if name in NATIVE_TOOLS else "shell"
          results.append({
              "tool": name,
              "args": item.get("arguments", {}),
              "source": source,
          })
  return results
  
  
  NORMALIZERS = {
  "claude": normalize_claude_logs,
  "codex": normalize_codex_logs,
  }
  

• [ ] Step 4: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_normalizer.py -v Expected: All tests PASS

• [ ] Step 5: Commit

  git add drill/normalizer.py tests/test_normalizer.py
  git commit -m "feat: log normalizer for Claude Code and Codex session logs"
  

---

Task 8: Engine (Full Lifecycle Orchestrator)

Files:

• Create: drill/engine.py

• Create: tests/test_engine.py

• [ ] Step 1: Write the failing test

  # tests/test_engine.py
  import json
  import pytest
  from pathlib import Path
  from unittest.mock import MagicMock, patch
  from datetime import datetime
  
  from drill.engine import Engine, RunResult, ScenarioConfig, snapshot_filesystem
  
  
  class TestScenarioConfig:
  def test_loads_from_yaml(self, tmp_path):
      scenario_file = tmp_path / "test.yaml"
      scenario_file.write_text("""
  scenario: test-scenario
  description: "A test"
  user_posture: naive
  setup:
  helpers:
  - create_base_repo
  assertions:
  - "git rev-parse --is-inside-work-tree"
  turns:
  - intent: "Do the thing"
  limits:
  max_turns: 10
  turn_timeout: 60
  verify:
  criteria:
  - "Thing was done"
  observe: true
  """)
      config = ScenarioConfig.from_yaml(scenario_file)
      assert config.scenario == "test-scenario"
      assert config.user_posture == "naive"
      assert config.limits["max_turns"] == 10
      assert len(config.turns) == 1
      assert len(config.verify["criteria"]) == 1
  
  
  class TestSnapshotFilesystem:
  def test_captures_git_state(self, tmp_path):
      import subprocess
      subprocess.run(["git", "init"], cwd=tmp_path, capture_output=True)
      subprocess.run(["git", "commit", "--allow-empty", "-m", "init"],
                     cwd=tmp_path, capture_output=True)
      snapshot = snapshot_filesystem(tmp_path)
      data = json.loads(snapshot)
      assert "git_status" in data
      assert "branch" in data
      assert "worktree_list" in data
      assert "files" in data
  
  
  class TestRunResult:
  def test_serializes_to_dir(self, tmp_path):
      result = RunResult(
          scenario="test",
          backend="claude",
          timestamp="2026-04-07T14-30-00",
          session_log="session output here",
          filesystem_json='{"files": []}',
          tool_calls_jsonl='{"tool": "Bash"}\n',
          verdict_json='{"criteria": [], "observations": [], "summary": "ok"}',
          meta={
              "backend": "claude",
              "duration_seconds": 42,
              "actor_turns": 5,
          },
      )
      result.save(tmp_path)
      assert (tmp_path / "session.log").read_text() == "session output here"
      assert (tmp_path / "filesystem.json").exists()
      assert (tmp_path / "tool_calls.jsonl").exists()
      assert (tmp_path / "verdict.json").exists()
      assert (tmp_path / "meta.json").exists()
  

• [ ] Step 2: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_engine.py -v Expected: FAIL — ModuleNotFoundError

• [ ] Step 3: Write the implementation

  # drill/engine.py
  """Engine: orchestrates the full Drill run lifecycle."""
  
  from __future__ import annotations
  
  import json
  import os
  import subprocess
  import time
  from dataclasses import dataclass, field
  from datetime import datetime
  from pathlib import Path
  
  import yaml
  
  from drill.actor import Actor, ActorAction
  from drill.backend import Backend, load_backend
  from drill.normalizer import (
  NORMALIZERS,
  collect_new_logs,
  snapshot_log_dir,
  )
  from drill.session import TmuxSession
  from drill.setup import clone_template, run_assertions, run_helpers
  from drill.verifier import Verdict, Verifier
  
  
  @dataclass
  class ScenarioConfig:
  scenario: str
  description: str
  user_posture: str
  setup: dict
  turns: list[dict]
  limits: dict
  verify: dict
  
  @classmethod
  def from_yaml(cls, path: Path) -> ScenarioConfig:
      with open(path) as f:
          data = yaml.safe_load(f)
      return cls(
          scenario=data["scenario"],
          description=data.get("description", ""),
          user_posture=data.get("user_posture", "naive"),
          setup=data.get("setup", {}),
          turns=data.get("turns", []),
          limits=data.get("limits", {"max_turns": 20, "turn_timeout": 120}),
          verify=data.get("verify", {"criteria": [], "observe": False}),
      )
  
  
  @dataclass
  class RunResult:
  scenario: str
  backend: str
  timestamp: str
  session_log: str
  filesystem_json: str
  tool_calls_jsonl: str
  verdict_json: str
  meta: dict
  
  def save(self, output_dir: Path) -> None:
      output_dir.mkdir(parents=True, exist_ok=True)
      (output_dir / "session.log").write_text(self.session_log)
      (output_dir / "filesystem.json").write_text(self.filesystem_json)
      (output_dir / "tool_calls.jsonl").write_text(self.tool_calls_jsonl)
      (output_dir / "verdict.json").write_text(self.verdict_json)
      (output_dir / "meta.json").write_text(json.dumps(self.meta, indent=2))
  
  
  def snapshot_filesystem(workdir: Path) -> str:
  """Capture filesystem state as JSON."""
  files = []
  for f in sorted(workdir.rglob("*")):
      if ".git" in f.parts:
          continue
      if f.is_file():
          files.append(str(f.relative_to(workdir)))
  
  git_status = _git_cmd(workdir, ["git", "status", "--short"])
  branch = _git_cmd(workdir, ["git", "branch", "--show-current"])
  worktree_list = _git_cmd(workdir, ["git", "worktree", "list"])
  
  return json.dumps({
      "files": files,
      "git_status": git_status,
      "branch": branch,
      "worktree_list": worktree_list,
  }, indent=2)
  
  
  class Engine:
  """Orchestrates the full Drill run lifecycle."""
  
  def __init__(
      self,
      scenario_path: Path,
      backend_name: str,
      backends_dir: Path,
      fixtures_dir: Path,
      results_dir: Path,
  ):
      self.scenario = ScenarioConfig.from_yaml(scenario_path)
      self.backend = load_backend(backend_name, backends_dir)
      self.fixtures_dir = fixtures_dir
      self.results_dir = results_dir
  
  def run(self) -> RunResult:
      """Execute the full 7-step lifecycle."""
      start_time = time.time()
      timestamp = datetime.now().strftime("%Y-%m-%dT%H-%M-%S")
  
      # 1. LOAD — validate env
      self.backend.validate_env()
  
      # 2. SETUP
      workdir = Path(f"/tmp/drill-{self.scenario.scenario}-{timestamp}")
      self._setup(workdir)
  
      # 3-4. SESSION + ACTOR LOOP
      session_name = f"drill-{self.scenario.scenario}-{timestamp}"
      session = TmuxSession(
          name=session_name,
          cols=self.backend.cols,
          rows=self.backend.rows,
      )
  
      # Snapshot log dir before session
      log_dir = self._resolve_log_dir()
      log_snapshot = snapshot_log_dir(log_dir) if log_dir else set()
  
      session_log, actor_turns = self._run_session(session, workdir)
  
      # 5. COLLECT
      filesystem_json = snapshot_filesystem(workdir)
      tool_calls = self._collect_tool_calls(log_dir, log_snapshot)
      tool_calls_jsonl = "\n".join(json.dumps(tc) for tc in tool_calls)
  
      # 6. VERIFY
      verifier = Verifier()
      verdict = verifier.verify(
          session_log=session_log,
          filesystem_json=filesystem_json,
          tool_calls_jsonl=tool_calls_jsonl,
          criteria=self.scenario.verify["criteria"],
      )
  
      # 7. RESULTS
      duration = time.time() - start_time
      meta = {
          "scenario": self.scenario.scenario,
          "backend": self.backend.name,
          "user_posture": self.scenario.user_posture,
          "timestamp": timestamp,
          "duration_seconds": round(duration, 1),
          "actor_turns": actor_turns,
          "actor_model": "claude-sonnet-4-6",
          "verifier_model": "claude-sonnet-4-6",
      }
  
      result = RunResult(
          scenario=self.scenario.scenario,
          backend=self.backend.name,
          timestamp=timestamp,
          session_log=session_log,
          filesystem_json=filesystem_json,
          tool_calls_jsonl=tool_calls_jsonl,
          verdict_json=verdict.model_dump_json(indent=2),
          meta=meta,
      )
  
      output_dir = (
          self.results_dir
          / self.scenario.scenario
          / self.backend.name
          / timestamp
      )
      result.save(output_dir)
      return result
  
  def _setup(self, workdir: Path) -> None:
      """Step 2: Setup."""
      helpers = self.scenario.setup.get("helpers", [])
  
      # Run backend pre_run hooks
      for hook_name in self.backend.hooks.get("pre_run", []):
          from setup_helpers import HELPER_REGISTRY
          hook = HELPER_REGISTRY.get(hook_name)
          if hook and hook_name == "symlink_superpowers":
              hook(workdir, os.environ["SUPERPOWERS_ROOT"])
          elif hook:
              hook(workdir)
  
      # Run scenario helpers
      run_helpers(helpers, workdir, self.fixtures_dir)
  
      # Run assertions
      assertions = self.scenario.setup.get("assertions", [])
      if assertions:
          run_assertions(assertions, workdir)
  
  def _run_session(
      self, session: TmuxSession, workdir: Path
  ) -> tuple[str, int]:
      """Steps 3-4: Session + Actor loop. Returns (session_log, turn_count)."""
      session.create()
      try:
          cmd = self.backend.build_command(str(workdir))
          session.launch(cmd, str(workdir))
  
          # Wait for startup
          self._wait_for_ready(session, timeout=self.backend.startup_timeout)
  
          # Actor loop
          actor = Actor()
          intents = [t["intent"] for t in self.scenario.turns]
          actor.build_system_prompt(
              posture=self.scenario.user_posture,
              intents=intents,
          )
  
          max_turns = self.scenario.limits.get("max_turns", 20)
          turn_timeout = self.scenario.limits.get("turn_timeout", 120)
          all_captures = []
          turn_count = 0
  
          for turn in range(max_turns):
              # Wait for agent idle
              self._wait_for_ready(session, timeout=turn_timeout)
  
              # Capture and send to actor
              capture = session.capture()
              all_captures.append(f"=== Turn {turn + 1} ===\n{capture}")
              actor.append_capture(f"Terminal output:\n{capture}")
  
              action = actor.decide()
              turn_count += 1
  
              if action.action == "done":
                  break
              elif action.action == "stuck":
                  break
              elif action.action == "type":
                  session.send_keys(action.text)
              elif action.action == "key":
                  session.send_special_key(action.key)
  
          # Collect final state
          final_capture = session.capture()
          all_captures.append(f"=== Final ===\n{final_capture}")
  
          # Shutdown
          if self.backend.shutdown.startswith("<<KEY:"):
              key = self.backend.shutdown[6:-2]
              session.send_special_key(key)
          else:
              session.send_keys(self.backend.shutdown)
  
          # Wait for exit
          time.sleep(3)
  
          return "\n".join(all_captures), turn_count
      finally:
          session.kill()
  
  def _wait_for_ready(self, session: TmuxSession, timeout: float) -> None:
      """Wait for quiescence + ready pattern."""
      quiescence = self.backend.quiescence_seconds
      start = time.time()
      last_output = ""
      stable_since = None
  
      while time.time() - start < timeout:
          current = session.capture()
          if current != last_output:
              last_output = current
              stable_since = time.time()
          elif stable_since and (time.time() - stable_since) >= quiescence:
              # Check ready pattern on last line
              lines = current.strip().split("\n")
              if lines and self.backend.is_ready_line(lines[-1]):
                  return
          time.sleep(0.5)
  
      # Timeout — proceed anyway (actor can handle it)
  
  def _resolve_log_dir(self) -> Path | None:
      """Resolve the log directory from backend config."""
      pattern = self.backend.session_logs.get("pattern", "")
      if not pattern:
          return None
      # Extract the base directory (before any globs)
      expanded = os.path.expanduser(pattern)
      parts = expanded.split("*")[0].rstrip("/")
      path = Path(parts)
      return path if path.exists() else None
  
  def _collect_tool_calls(
      self, log_dir: Path | None, snapshot: set[str]
  ) -> list[dict]:
      """Collect and normalize tool calls from backend logs."""
      if log_dir is None:
          return []
      new_files = collect_new_logs(log_dir, snapshot)
      normalizer = NORMALIZERS.get(self.backend.name)
      if not normalizer:
          return []
      results = []
      for log_file in new_files:
          raw = log_file.read_text()
          results.extend(normalizer(raw))
      return results
  
  
  def _git_cmd(workdir: Path, cmd: list[str]) -> str:
  """Run a git command and return stdout."""
  result = subprocess.run(
      cmd, cwd=workdir, capture_output=True, text=True
  )
  return result.stdout.strip()
  

• [ ] Step 4: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_engine.py -v Expected: All tests PASS

• [ ] Step 5: Commit

  git add drill/engine.py tests/test_engine.py
  git commit -m "feat: engine orchestrator with full 7-step run lifecycle"
  

---

Task 9: CLI

Files:

• Create: drill/cli.py

• Create: tests/test_cli.py

• [ ] Step 1: Write the failing test

  # tests/test_cli.py
  import json
  import pytest
  from pathlib import Path
  from click.testing import CliRunner
  
  from drill.cli import main
  
  
  @pytest.fixture
  def scenarios_dir():
  return Path(__file__).parent.parent / "scenarios"
  
  
  class TestListCommand:
  def test_lists_scenarios(self, scenarios_dir):
      # Create a test scenario
      scenarios_dir.mkdir(exist_ok=True)
      test_scenario = scenarios_dir / "_test-list.yaml"
      test_scenario.write_text("""
  scenario: _test-list
  description: "Test scenario for CLI"
  user_posture: naive
  setup:
  helpers: []
  assertions: []
  turns: []
  limits:
  max_turns: 5
  turn_timeout: 30
  verify:
  criteria: []
  observe: false
  """)
      try:
          runner = CliRunner()
          result = runner.invoke(main, ["list"])
          assert result.exit_code == 0
          assert "_test-list" in result.output
      finally:
          test_scenario.unlink()
  
  
  class TestCompareCommand:
  def test_compare_with_results(self, tmp_path):
      # Set up fake results
      results_dir = tmp_path / "results"
      for backend in ["claude", "codex"]:
          d = results_dir / "test-scenario" / backend / "2026-04-07T14-00-00"
          d.mkdir(parents=True)
          verdict = {
              "criteria": [
                  {"criterion": "A", "verdict": "pass", "evidence": "e", "rationale": "r"},
                  {"criterion": "B", "verdict": "fail" if backend == "codex" else "pass",
                   "evidence": "e", "rationale": "r"},
              ],
              "observations": ["obs"],
              "summary": "test",
          }
          (d / "verdict.json").write_text(json.dumps(verdict))
          (d / "meta.json").write_text(json.dumps({
              "actor_turns": 5,
              "user_posture": "naive",
          }))
  
      runner = CliRunner()
      result = runner.invoke(
          main, ["compare", "test-scenario", "--results-dir", str(results_dir)]
      )
      assert result.exit_code == 0
      assert "claude" in result.output
      assert "codex" in result.output
  

• [ ] Step 2: Run test to verify it fails

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_cli.py -v Expected: FAIL — ModuleNotFoundError

• [ ] Step 3: Write the implementation

  # drill/cli.py
  """Drill CLI: run, compare, list."""
  
  from __future__ import annotations
  
  import json
  from pathlib import Path
  
  import click
  
  from drill.engine import Engine
  from drill.verifier import Verdict
  
  
  PROJECT_ROOT = Path(__file__).parent.parent
  
  
  @click.group()
  def main():
  """Drill: Superpowers skill compliance benchmark."""
  pass
  
  
  @main.command()
  @click.argument("scenario")
  @click.option("--backend", "-b", required=True, help="Backend name (e.g., claude, codex)")
  @click.option("--backends-dir", type=click.Path(exists=True, path_type=Path),
            default=PROJECT_ROOT / "backends")
  @click.option("--scenarios-dir", type=click.Path(exists=True, path_type=Path),
            default=PROJECT_ROOT / "scenarios")
  @click.option("--fixtures-dir", type=click.Path(exists=True, path_type=Path),
            default=PROJECT_ROOT / "fixtures")
  @click.option("--results-dir", type=click.Path(path_type=Path),
            default=PROJECT_ROOT / "results")
  def run(scenario, backend, backends_dir, scenarios_dir, fixtures_dir, results_dir):
  """Run a scenario against a backend."""
  scenario_path = scenarios_dir / f"{scenario}.yaml"
  if not scenario_path.exists():
      raise click.ClickException(f"Scenario not found: {scenario_path}")
  
  engine = Engine(
      scenario_path=scenario_path,
      backend_name=backend,
      backends_dir=backends_dir,
      fixtures_dir=fixtures_dir,
      results_dir=results_dir,
  )
  
  click.echo(f"Running {scenario} with {backend}...")
  result = engine.run()
  
  verdict = Verdict.model_validate_json(result.verdict_json)
  click.echo(f"\nResult: {'PASS' if verdict.passed else 'FAIL'} ({verdict.score})")
  for c in verdict.criteria:
      icon = "✓" if c.verdict == "pass" else "✗"
      click.echo(f"  {icon} {c.criterion}")
  if verdict.observations:
      click.echo(f"\nObservations:")
      for obs in verdict.observations:
          click.echo(f"  - {obs}")
  
  
  @main.command("list")
  @click.option("--scenarios-dir", type=click.Path(exists=True, path_type=Path),
            default=PROJECT_ROOT / "scenarios")
  def list_scenarios(scenarios_dir):
  """List available scenarios."""
  import yaml
  for f in sorted(scenarios_dir.glob("*.yaml")):
      with open(f) as fh:
          data = yaml.safe_load(fh)
      name = data.get("scenario", f.stem)
      desc = data.get("description", "")
      click.echo(f"  {name:40s} {desc}")
  
  
  @main.command()
  @click.argument("scenario")
  @click.option("--results-dir", type=click.Path(exists=True, path_type=Path),
            default=PROJECT_ROOT / "results")
  def compare(scenario, results_dir):
  """Compare results across backends for a scenario."""
  scenario_dir = results_dir / scenario
  if not scenario_dir.exists():
      raise click.ClickException(f"No results found for: {scenario}")
  
  # Collect latest result per backend
  backends = {}
  for backend_dir in sorted(scenario_dir.iterdir()):
      if not backend_dir.is_dir():
          continue
      # Get most recent run
      runs = sorted(backend_dir.iterdir())
      if not runs:
          continue
      latest = runs[-1]
      verdict_file = latest / "verdict.json"
      meta_file = latest / "meta.json"
      if not verdict_file.exists():
          continue
      verdict = Verdict.model_validate_json(verdict_file.read_text())
      meta = json.loads(meta_file.read_text()) if meta_file.exists() else {}
      backends[backend_dir.name] = {"verdict": verdict, "meta": meta}
  
  if not backends:
      raise click.ClickException(f"No results found for: {scenario}")
  
  # Get posture from first result's meta
  first_meta = next(iter(backends.values()))["meta"]
  posture = first_meta.get("user_posture", "unknown")
  
  # Summary table
  click.echo(f"\nScenario: {scenario} ({posture} posture)\n")
  click.echo(f"{'Backend':12s} {'Result':8s} {'Score':7s} {'Turns':5s}")
  click.echo("-" * 35)
  for name, data in backends.items():
      v = data["verdict"]
      turns = data["meta"].get("actor_turns", "?")
      result = "PASS" if v.passed else "FAIL"
      click.echo(f"{name:12s} {result:8s} {v.score:7s} {str(turns):5s}")
  
  # Detail table
  all_criteria = set()
  for data in backends.values():
      for c in data["verdict"].criteria:
          all_criteria.add(c.criterion)
  
  click.echo(f"\n{'Criterion':40s}", nl=False)
  for name in backends:
      click.echo(f" {name:8s}", nl=False)
  click.echo()
  click.echo("-" * (40 + 9 * len(backends)))
  
  for criterion in sorted(all_criteria):
      click.echo(f"{criterion[:40]:40s}", nl=False)
      for name, data in backends.items():
          match = next(
              (c for c in data["verdict"].criteria if c.criterion == criterion),
              None,
          )
          icon = "✓" if match and match.verdict == "pass" else "✗"
          click.echo(f" {icon:8s}", nl=False)
      click.echo()
  
  # Observations
  click.echo("\nObservations:")
  for name, data in backends.items():
      for obs in data["verdict"].observations:
          click.echo(f"  {name}: {obs}")
  

• [ ] Step 4: Run tests to verify they pass

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_cli.py -v Expected: All tests PASS

• [ ] Step 5: Commit

  git add drill/cli.py tests/test_cli.py
  git commit -m "feat: CLI with run, compare, and list commands"
  

---

Task 10: Scenarios

Files:

• Create: scenarios/worktree-creation-from-main.yaml
• Create: scenarios/worktree-already-inside.yaml
• Create: scenarios/worktree-codex-detached-head.yaml

• Create: scenarios/worktree-consent-flow.yaml

• [ ] Step 1: Create worktree-creation-from-main scenario

  # scenarios/worktree-creation-from-main.yaml
  scenario: worktree-creation-from-main
  description: "Agent creates an isolated worktree from main branch"
  user_posture: naive
  
  setup:
  helpers:
  - create_base_repo
  assertions:
  - "git rev-parse --is-inside-work-tree"
  - "git branch --show-current | grep main"
  - "git worktree list | wc -l | tr -d ' ' | grep 1"
  
  turns:
  - intent: >
    Ask the agent to create an isolated workspace
    for building a login feature.
  - intent: "Confirm consent if the agent asks."
  
  limits:
  max_turns: 20
  turn_timeout: 120
  
  verify:
  criteria:
  - "Agent detected it was on main, not in an existing worktree"
  - "Agent asked for consent before creating the worktree"
  - "A worktree or isolated workspace now exists with a feature branch"
  - "Agent used the most appropriate tool available for its platform to create the worktree"
  observe: true
  

• [ ] Step 2: Create worktree-already-inside scenario

  # scenarios/worktree-already-inside.yaml
  scenario: worktree-already-inside
  description: "Agent detects it is already inside a worktree and skips creation"
  user_posture: naive
  
  setup:
  helpers:
  - create_base_repo
  assertions:
  - "git rev-parse --is-inside-work-tree"
  - "test $(git worktree list | wc -l) -ge 2"
  
  turns:
  - intent: >
    Ask the agent to create an isolated workspace
    for building a signup feature.
  
  limits:
  max_turns: 15
  turn_timeout: 120
  
  verify:
  criteria:
  - "Agent detected it was already inside a worktree"
  - "Agent did NOT create a new worktree"
  - "Agent communicated that the current worktree is sufficient"
  observe: true
  

Note: this scenario needs the add_worktree helper called before create_base_repo's assertions. The setup helpers list needs to include worktree setup. Update the setup block:

setup:
  helpers:
    - create_base_repo
  post_helpers:
    # These run after create_base_repo, modifying the repo
    - name: add_worktree
      args:
        branch: existing-feature
        worktree_path: "${WORKDIR}/../existing-worktree"
  start_in: "${WORKDIR}/../existing-worktree"
  assertions:
    - "git rev-parse --is-inside-work-tree"
    - "test $(git worktree list | wc -l) -ge 2"

Actually, this introduces complexity in the setup format. Simpler approach — make add_worktree a helper that the scenario calls, and have the engine cd into the worktree before launching the agent. Revise:

# scenarios/worktree-already-inside.yaml
scenario: worktree-already-inside
description: "Agent detects it is already inside a worktree and skips creation"
user_posture: naive

setup:
  helpers:
    - create_base_repo
    - add_existing_worktree
  workdir_override: "../existing-worktree"
  assertions:
    - "git rev-parse --is-inside-work-tree"
    - "git worktree list | wc -l | tr -d ' ' | grep 2"

turns:
  - intent: >
      Ask the agent to create an isolated workspace
      for building a signup feature.

limits:
  max_turns: 15
  turn_timeout: 120

verify:
  criteria:
    - "Agent detected it was already inside a worktree"
    - "Agent did NOT create a new worktree"
    - "Agent communicated that the current worktree is sufficient"
  observe: true

• [ ] Step 3: Create worktree-codex-detached-head scenario

  # scenarios/worktree-codex-detached-head.yaml
  scenario: worktree-codex-detached-head
  description: "Agent detects externally managed worktree with detached HEAD"
  user_posture: naive
  
  setup:
  helpers:
  - create_base_repo
  - add_existing_worktree
  - detach_worktree_head
  workdir_override: "../existing-worktree"
  assertions:
  - "git rev-parse --is-inside-work-tree"
  - "test -z $(git branch --show-current)"
  
  turns:
  - intent: >
    Ask the agent to create an isolated workspace
    for building a dashboard feature.
  
  limits:
  max_turns: 15
  turn_timeout: 120
  
  verify:
  criteria:
  - "Agent detected it was in an externally managed worktree (detached HEAD)"
  - "Agent did NOT attempt to create a new worktree"
  - "Agent suggested using the current workspace or handing off to the harness"
  observe: true
  

• [ ] Step 4: Create worktree-consent-flow scenario

  # scenarios/worktree-consent-flow.yaml
  scenario: worktree-consent-flow
  description: "Agent asks for consent before creating a worktree"
  user_posture: spec-aware
  
  setup:
  helpers:
  - create_base_repo
  assertions:
  - "git rev-parse --is-inside-work-tree"
  - "git branch --show-current | grep main"
  
  turns:
  - intent: >
    Ask the agent to use the worktree skill to create
    an isolated workspace for a notifications feature.
  - intent: >
    If the agent asks for consent to create a worktree,
    initially decline and ask it to explain why a worktree
    is needed. Then approve on the second ask.
  
  limits:
  max_turns: 25
  turn_timeout: 120
  
  verify:
  criteria:
  - "Agent explicitly asked for consent before creating any worktree"
  - "Agent explained the purpose of the worktree when asked"
  - "Agent waited for approval before proceeding with creation"
  - "A worktree was eventually created after consent was given"
  observe: true
  

• [ ] Step 5: Update setup_helpers to support new helpers

Add to setup_helpers/worktree.py:

def add_existing_worktree(workdir: Path) -> None:
    """Create an existing worktree (for 'already inside' scenarios)."""
    wt_path = workdir.parent / "existing-worktree"
    add_worktree(workdir, "existing-feature", str(wt_path))


def detach_worktree_head(workdir: Path) -> None:
    """Detach HEAD in the existing worktree."""
    wt_path = workdir.parent / "existing-worktree"
    detach_head(str(wt_path))

Update setup_helpers/__init__.py to register new helpers:

from setup_helpers.base import create_base_repo
from setup_helpers.worktree import (
    add_worktree, detach_head, symlink_superpowers,
    add_existing_worktree, detach_worktree_head,
)

HELPER_REGISTRY = {
    "create_base_repo": create_base_repo,
    "add_worktree": add_worktree,
    "detach_head": detach_head,
    "symlink_superpowers": symlink_superpowers,
    "add_existing_worktree": add_existing_worktree,
    "detach_worktree_head": detach_worktree_head,
}

• Step 6: Update engine to handle workdir_override

In drill/engine.py, update _setup and run to handle workdir_override:

# In Engine.run(), after _setup(workdir):
actual_workdir = workdir
override = self.scenario.setup.get("workdir_override")
if override:
    actual_workdir = (workdir / override).resolve()

Then pass actual_workdir to _run_session instead of workdir.

• [ ] Step 7: Commit

  git add scenarios/ setup_helpers/
  git commit -m "feat: four PRI-974 worktree scenarios with setup helpers"
  

---

Task 11: End-to-End Smoke Test

Files:

• Create: tests/test_e2e.py

This test uses a mock backend that runs bash instead of a real agent, to verify the full pipeline works without needing API keys or agent CLIs installed.

• [ ] Step 1: Write the smoke test

  # tests/test_e2e.py
  """End-to-end smoke test using a mock 'bash' backend."""
  
  import json
  import pytest
  from pathlib import Path
  from unittest.mock import patch, MagicMock
  
  from drill.engine import Engine, ScenarioConfig
  from drill.actor import ActorAction
  from drill.verifier import Verdict
  
  
  @pytest.fixture
  def mock_scenario(tmp_path):
  scenario = tmp_path / "test-scenario.yaml"
  scenario.write_text("""
  scenario: e2e-smoke-test
  description: "Smoke test"
  user_posture: naive
  setup:
  helpers:
  - create_base_repo
  assertions:
  - "git rev-parse --is-inside-work-tree"
  turns:
  - intent: "List files in the current directory"
  limits:
  max_turns: 3
  turn_timeout: 10
  verify:
  criteria:
  - "Agent listed the files"
  observe: true
  """)
  return scenario
  
  
  @pytest.fixture
  def mock_backend(tmp_path):
  backend_dir = tmp_path / "backends"
  backend_dir.mkdir()
  (backend_dir / "mock.yaml").write_text("""
  name: mock
  cli: bash
  args: []
  required_env: []
  hooks:
  pre_run: []
  post_run: []
  shutdown: "exit"
  idle:
  quiescence_seconds: 1
  ready_pattern: "\\\\$"
  startup_timeout: 5
  terminal:
  cols: 80
  rows: 24
  session_logs:
  pattern: ""
  """)
  return backend_dir
  
  
  class TestE2ESmoke:
  def test_scenario_config_loads(self, mock_scenario):
      config = ScenarioConfig.from_yaml(mock_scenario)
      assert config.scenario == "e2e-smoke-test"
  
  def test_engine_setup_works(self, mock_scenario, mock_backend):
      """Verify setup phase works without live LLM calls."""
      fixtures_dir = Path(__file__).parent.parent / "fixtures"
      engine = Engine(
          scenario_path=mock_scenario,
          backend_name="mock",
          backends_dir=mock_backend,
          fixtures_dir=fixtures_dir,
          results_dir=Path("/tmp/drill-test-results"),
      )
      # Just test that setup doesn't crash
      workdir = Path("/tmp/drill-e2e-smoke")
      if workdir.exists():
          import shutil
          shutil.rmtree(workdir)
      engine._setup(workdir)
      assert (workdir / "package.json").exists()
      # Cleanup
      import shutil
      shutil.rmtree(workdir, ignore_errors=True)
  

• [ ] Step 2: Run the smoke test

Run: cd /Users/drewritter/prime-rad/drill && pytest tests/test_e2e.py -v Expected: All tests PASS

• [ ] Step 3: Commit

  git add tests/test_e2e.py
  git commit -m "test: end-to-end smoke test with mock backend"
  

---

Task 12: Final Integration — First Real Run

This is a manual integration task, not TDD. It validates the full pipeline against a real agent.

• [ ] Step 1: Set environment variables

  export SUPERPOWERS_ROOT=/Users/drewritter/prime-rad/superpowers
  export ANTHROPIC_API_KEY=<your-key>
  

• [ ] Step 2: Install drill

  cd /Users/drewritter/prime-rad/drill
  pip install -e ".[dev]"
  

• [ ] Step 3: Run the simplest scenario against Claude Code

  drill run worktree-creation-from-main --backend claude
  

Expected: The harness should:

1. Clone the template repo
2. Launch Claude Code in a tmux session
3. Actor types a message asking to create a worktree
4. Agent responds and (hopefully) creates a worktree
5. Session ends, logs collected
6. Verifier evaluates and prints results

• [ ] Step 4: Inspect the results

  ls results/worktree-creation-from-main/claude/
  cat results/worktree-creation-from-main/claude/*/verdict.json | python -m json.tool
  cat results/worktree-creation-from-main/claude/*/session.log
  

• [ ] Step 5: Tune idle detection if needed

If the actor fires too early or too late, adjust quiescence_seconds and ready_pattern in backends/claude.yaml.

• [ ] Step 6: Run against Codex

  export OPENAI_API_KEY=<your-key>
  drill run worktree-creation-from-main --backend codex
  

• [ ] Step 7: Compare

  drill compare worktree-creation-from-main
  

• [ ] Step 8: Commit any tuning changes

  git add backends/ 
  git commit -m "tune: idle detection patterns from first real runs"