Преглед изворни кода

Add Claude Code skills test framework

Created automated test suite for testing superpowers skills using
Claude Code CLI in headless mode.

New files:
- tests/claude-code/run-skill-tests.sh - Main test runner
- tests/claude-code/test-helpers.sh - Helper functions for testing
- tests/claude-code/test-subagent-driven-development.sh - First test
- tests/claude-code/README.md - Documentation

Test framework features:
- Run Claude Code with prompts and capture output
- Assertion helpers (contains, not_contains, count, order)
- Test project creation helpers
- Timeout support (default 5 minutes)
- Verbose mode for debugging
- Specific test selection

First test verifies subagent-driven-development skill:
- Skill loading
- Workflow ordering (spec compliance before code quality)
- Self-review requirements
- Plan reading efficiency (read once)
- Spec compliance reviewer skepticism
- Review loops
- Task context provision

Run with: cd tests/claude-code && ./run-skill-tests.sh
Jesse Vincent пре 9 месеци
родитељ
комит
2d011053d5

+ 128 - 0
tests/claude-code/README.md

@@ -0,0 +1,128 @@
+# Claude Code Skills Tests
+
+Automated tests for superpowers skills using Claude Code CLI.
+
+## Overview
+
+This test suite verifies that skills are loaded correctly and Claude follows them as expected. Tests invoke Claude Code in headless mode (`claude -p`) and verify the behavior.
+
+## Requirements
+
+- Claude Code CLI installed and in PATH (`claude --version` should work)
+- Local superpowers plugin installed (see main README for installation)
+
+## Running Tests
+
+### Run all tests:
+```bash
+./run-skill-tests.sh
+```
+
+### Run specific test:
+```bash
+./run-skill-tests.sh --test test-subagent-driven-development.sh
+```
+
+### Run with verbose output:
+```bash
+./run-skill-tests.sh --verbose
+```
+
+### Set custom timeout (default 300s):
+```bash
+./run-skill-tests.sh --timeout 600
+```
+
+## Test Structure
+
+### test-helpers.sh
+Common functions for skills testing:
+- `run_claude "prompt" [timeout]` - Run Claude with prompt
+- `assert_contains output pattern name` - Verify pattern exists
+- `assert_not_contains output pattern name` - Verify pattern absent
+- `assert_count output pattern count name` - Verify exact count
+- `assert_order output pattern_a pattern_b name` - Verify order
+- `create_test_project` - Create temp test directory
+- `create_test_plan project_dir` - Create sample plan file
+
+### Test Files
+
+Each test file:
+1. Sources `test-helpers.sh`
+2. Runs Claude Code with specific prompts
+3. Verifies expected behavior using assertions
+4. Returns 0 on success, non-zero on failure
+
+## Example Test
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/test-helpers.sh"
+
+echo "=== Test: My Skill ==="
+
+# Ask Claude about the skill
+output=$(run_claude "What does the my-skill skill do?" 30)
+
+# Verify response
+assert_contains "$output" "expected behavior" "Skill describes behavior"
+
+echo "=== All tests passed ==="
+```
+
+## Current Tests
+
+### test-subagent-driven-development.sh
+Tests the updated subagent-driven-development workflow:
+- Skill loading
+- Workflow ordering (spec compliance before code quality)
+- Self-review requirements
+- Plan reading efficiency (read once)
+- Spec compliance reviewer skepticism
+- Review loops
+- Task context provision (full text, not file reading)
+
+## Adding New Tests
+
+1. Create new test file: `test-<skill-name>.sh`
+2. Source test-helpers.sh
+3. Write tests using `run_claude` and assertions
+4. Add to test list in `run-skill-tests.sh`
+5. Make executable: `chmod +x test-<skill-name>.sh`
+
+## Timeout Considerations
+
+- Default timeout: 5 minutes per test
+- Claude Code may take time to respond
+- Adjust with `--timeout` if needed
+- Tests should be focused to avoid long runs
+
+## Debugging Failed Tests
+
+With `--verbose`, you'll see full Claude output:
+```bash
+./run-skill-tests.sh --verbose --test test-subagent-driven-development.sh
+```
+
+Without verbose, only failures show output.
+
+## CI/CD Integration
+
+To run in CI:
+```bash
+# Run with explicit timeout for CI environments
+./run-skill-tests.sh --timeout 900
+
+# Exit code 0 = success, non-zero = failure
+```
+
+## Notes
+
+- Tests verify skill *instructions*, not full execution
+- Full workflow tests would be very slow
+- Focus on verifying key skill requirements
+- Tests should be deterministic
+- Avoid testing implementation details

+ 162 - 0
tests/claude-code/run-skill-tests.sh

@@ -0,0 +1,162 @@
+#!/usr/bin/env bash
+# Test runner for Claude Code skills
+# Tests skills by invoking Claude Code CLI and verifying behavior
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+cd "$SCRIPT_DIR"
+
+echo "========================================"
+echo " Claude Code Skills Test Suite"
+echo "========================================"
+echo ""
+echo "Repository: $(cd ../.. && pwd)"
+echo "Test time: $(date)"
+echo "Claude version: $(claude --version 2>/dev/null || echo 'not found')"
+echo ""
+
+# Check if Claude Code is available
+if ! command -v claude &> /dev/null; then
+    echo "ERROR: Claude Code CLI not found"
+    echo "Install Claude Code first: https://code.claude.com"
+    exit 1
+fi
+
+# Parse command line arguments
+VERBOSE=false
+SPECIFIC_TEST=""
+TIMEOUT=300  # Default 5 minute timeout per test
+
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --verbose|-v)
+            VERBOSE=true
+            shift
+            ;;
+        --test|-t)
+            SPECIFIC_TEST="$2"
+            shift 2
+            ;;
+        --timeout)
+            TIMEOUT="$2"
+            shift 2
+            ;;
+        --help|-h)
+            echo "Usage: $0 [options]"
+            echo ""
+            echo "Options:"
+            echo "  --verbose, -v      Show verbose output"
+            echo "  --test, -t NAME    Run only the specified test"
+            echo "  --timeout SECONDS  Set timeout per test (default: 300)"
+            echo "  --help, -h         Show this help"
+            echo ""
+            echo "Tests:"
+            echo "  test-subagent-driven-development.sh  Test subagent-driven-development workflow"
+            exit 0
+            ;;
+        *)
+            echo "Unknown option: $1"
+            echo "Use --help for usage information"
+            exit 1
+            ;;
+    esac
+done
+
+# List of skill tests to run
+tests=(
+    "test-subagent-driven-development.sh"
+)
+
+# Filter to specific test if requested
+if [ -n "$SPECIFIC_TEST" ]; then
+    tests=("$SPECIFIC_TEST")
+fi
+
+# Track results
+passed=0
+failed=0
+skipped=0
+
+# Run each test
+for test in "${tests[@]}"; do
+    echo "----------------------------------------"
+    echo "Running: $test"
+    echo "----------------------------------------"
+
+    test_path="$SCRIPT_DIR/$test"
+
+    if [ ! -f "$test_path" ]; then
+        echo "  [SKIP] Test file not found: $test"
+        skipped=$((skipped + 1))
+        continue
+    fi
+
+    if [ ! -x "$test_path" ]; then
+        echo "  Making $test executable..."
+        chmod +x "$test_path"
+    fi
+
+    start_time=$(date +%s)
+
+    if [ "$VERBOSE" = true ]; then
+        if timeout "$TIMEOUT" bash "$test_path"; then
+            end_time=$(date +%s)
+            duration=$((end_time - start_time))
+            echo ""
+            echo "  [PASS] $test (${duration}s)"
+            passed=$((passed + 1))
+        else
+            exit_code=$?
+            end_time=$(date +%s)
+            duration=$((end_time - start_time))
+            echo ""
+            if [ $exit_code -eq 124 ]; then
+                echo "  [FAIL] $test (timeout after ${TIMEOUT}s)"
+            else
+                echo "  [FAIL] $test (${duration}s)"
+            fi
+            failed=$((failed + 1))
+        fi
+    else
+        # Capture output for non-verbose mode
+        if output=$(timeout "$TIMEOUT" bash "$test_path" 2>&1); then
+            end_time=$(date +%s)
+            duration=$((end_time - start_time))
+            echo "  [PASS] (${duration}s)"
+            passed=$((passed + 1))
+        else
+            exit_code=$?
+            end_time=$(date +%s)
+            duration=$((end_time - start_time))
+            if [ $exit_code -eq 124 ]; then
+                echo "  [FAIL] (timeout after ${TIMEOUT}s)"
+            else
+                echo "  [FAIL] (${duration}s)"
+            fi
+            echo ""
+            echo "  Output:"
+            echo "$output" | sed 's/^/    /'
+            failed=$((failed + 1))
+        fi
+    fi
+
+    echo ""
+done
+
+# Print summary
+echo "========================================"
+echo " Test Results Summary"
+echo "========================================"
+echo ""
+echo "  Passed:  $passed"
+echo "  Failed:  $failed"
+echo "  Skipped: $skipped"
+echo ""
+
+if [ $failed -gt 0 ]; then
+    echo "STATUS: FAILED"
+    exit 1
+else
+    echo "STATUS: PASSED"
+    exit 0
+fi

+ 195 - 0
tests/claude-code/test-helpers.sh

@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# Helper functions for Claude Code skill tests
+
+# Run Claude Code with a prompt and capture output
+# Usage: run_claude "prompt text" [timeout_seconds]
+run_claude() {
+    local prompt="$1"
+    local timeout="${2:-60}"
+    local output_file=$(mktemp)
+
+    # Run Claude in headless mode with timeout
+    if timeout "$timeout" claude -p "$prompt" > "$output_file" 2>&1; then
+        cat "$output_file"
+        rm -f "$output_file"
+        return 0
+    else
+        local exit_code=$?
+        cat "$output_file" >&2
+        rm -f "$output_file"
+        return $exit_code
+    fi
+}
+
+# Check if output contains a pattern
+# Usage: assert_contains "output" "pattern" "test name"
+assert_contains() {
+    local output="$1"
+    local pattern="$2"
+    local test_name="${3:-test}"
+
+    if echo "$output" | grep -q "$pattern"; then
+        echo "  [PASS] $test_name"
+        return 0
+    else
+        echo "  [FAIL] $test_name"
+        echo "  Expected to find: $pattern"
+        echo "  In output:"
+        echo "$output" | sed 's/^/    /'
+        return 1
+    fi
+}
+
+# Check if output does NOT contain a pattern
+# Usage: assert_not_contains "output" "pattern" "test name"
+assert_not_contains() {
+    local output="$1"
+    local pattern="$2"
+    local test_name="${3:-test}"
+
+    if echo "$output" | grep -q "$pattern"; then
+        echo "  [FAIL] $test_name"
+        echo "  Did not expect to find: $pattern"
+        echo "  In output:"
+        echo "$output" | sed 's/^/    /'
+        return 1
+    else
+        echo "  [PASS] $test_name"
+        return 0
+    fi
+}
+
+# Check if output matches a count
+# Usage: assert_count "output" "pattern" expected_count "test name"
+assert_count() {
+    local output="$1"
+    local pattern="$2"
+    local expected="$3"
+    local test_name="${4:-test}"
+
+    local actual=$(echo "$output" | grep -c "$pattern" || echo "0")
+
+    if [ "$actual" -eq "$expected" ]; then
+        echo "  [PASS] $test_name (found $actual instances)"
+        return 0
+    else
+        echo "  [FAIL] $test_name"
+        echo "  Expected $expected instances of: $pattern"
+        echo "  Found $actual instances"
+        echo "  In output:"
+        echo "$output" | sed 's/^/    /'
+        return 1
+    fi
+}
+
+# Check if pattern A appears before pattern B
+# Usage: assert_order "output" "pattern_a" "pattern_b" "test name"
+assert_order() {
+    local output="$1"
+    local pattern_a="$2"
+    local pattern_b="$3"
+    local test_name="${4:-test}"
+
+    # Get line numbers where patterns appear
+    local line_a=$(echo "$output" | grep -n "$pattern_a" | head -1 | cut -d: -f1)
+    local line_b=$(echo "$output" | grep -n "$pattern_b" | head -1 | cut -d: -f1)
+
+    if [ -z "$line_a" ]; then
+        echo "  [FAIL] $test_name: pattern A not found: $pattern_a"
+        return 1
+    fi
+
+    if [ -z "$line_b" ]; then
+        echo "  [FAIL] $test_name: pattern B not found: $pattern_b"
+        return 1
+    fi
+
+    if [ "$line_a" -lt "$line_b" ]; then
+        echo "  [PASS] $test_name (A at line $line_a, B at line $line_b)"
+        return 0
+    else
+        echo "  [FAIL] $test_name"
+        echo "  Expected '$pattern_a' before '$pattern_b'"
+        echo "  But found A at line $line_a, B at line $line_b"
+        return 1
+    fi
+}
+
+# Create a temporary test project directory
+# Usage: test_project=$(create_test_project)
+create_test_project() {
+    local test_dir=$(mktemp -d)
+    echo "$test_dir"
+}
+
+# Cleanup test project
+# Usage: cleanup_test_project "$test_dir"
+cleanup_test_project() {
+    local test_dir="$1"
+    if [ -d "$test_dir" ]; then
+        rm -rf "$test_dir"
+    fi
+}
+
+# Create a simple plan file for testing
+# Usage: create_test_plan "$project_dir" "$plan_name"
+create_test_plan() {
+    local project_dir="$1"
+    local plan_name="${2:-test-plan}"
+    local plan_file="$project_dir/docs/plans/$plan_name.md"
+
+    mkdir -p "$(dirname "$plan_file")"
+
+    cat > "$plan_file" <<'EOF'
+# Test Implementation Plan
+
+## Task 1: Create Hello Function
+
+Create a simple hello function that returns "Hello, World!".
+
+**File:** `src/hello.js`
+
+**Implementation:**
+```javascript
+export function hello() {
+  return "Hello, World!";
+}
+```
+
+**Tests:** Write a test that verifies the function returns the expected string.
+
+**Verification:** `npm test`
+
+## Task 2: Create Goodbye Function
+
+Create a goodbye function that takes a name and returns a goodbye message.
+
+**File:** `src/goodbye.js`
+
+**Implementation:**
+```javascript
+export function goodbye(name) {
+  return `Goodbye, ${name}!`;
+}
+```
+
+**Tests:** Write tests for:
+- Default name
+- Custom name
+- Edge cases (empty string, null)
+
+**Verification:** `npm test`
+EOF
+
+    echo "$plan_file"
+}
+
+# Export functions for use in tests
+export -f run_claude
+export -f assert_contains
+export -f assert_not_contains
+export -f assert_count
+export -f assert_order
+export -f create_test_project
+export -f cleanup_test_project
+export -f create_test_plan

+ 139 - 0
tests/claude-code/test-subagent-driven-development.sh

@@ -0,0 +1,139 @@
+#!/usr/bin/env bash
+# Test: subagent-driven-development skill
+# Verifies that the skill is loaded and follows correct workflow
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/test-helpers.sh"
+
+echo "=== Test: subagent-driven-development skill ==="
+echo ""
+
+# Test 1: Verify skill can be loaded
+echo "Test 1: Skill loading..."
+
+output=$(run_claude "What is the subagent-driven-development skill? Describe its key steps briefly." 30)
+
+if assert_contains "$output" "subagent-driven-development" "Skill is recognized"; then
+    : # pass
+else
+    exit 1
+fi
+
+if assert_contains "$output" "Load Plan\|read.*plan\|extract.*tasks" "Mentions loading plan"; then
+    : # pass
+else
+    exit 1
+fi
+
+echo ""
+
+# Test 2: Verify skill describes correct workflow order
+echo "Test 2: Workflow ordering..."
+
+output=$(run_claude "In the subagent-driven-development skill, what comes first: spec compliance review or code quality review? Be specific about the order." 30)
+
+if assert_order "$output" "spec.*compliance" "code.*quality" "Spec compliance before code quality"; then
+    : # pass
+else
+    exit 1
+fi
+
+echo ""
+
+# Test 3: Verify self-review is mentioned
+echo "Test 3: Self-review requirement..."
+
+output=$(run_claude "Does the subagent-driven-development skill require implementers to do self-review? What should they check?" 30)
+
+if assert_contains "$output" "self-review\|self review" "Mentions self-review"; then
+    : # pass
+else
+    exit 1
+fi
+
+if assert_contains "$output" "completeness\|Completeness" "Checks completeness"; then
+    : # pass
+else
+    exit 1
+fi
+
+echo ""
+
+# Test 4: Verify plan is read once
+echo "Test 4: Plan reading efficiency..."
+
+output=$(run_claude "In subagent-driven-development, how many times should the controller read the plan file? When does this happen?" 30)
+
+if assert_contains "$output" "once\|one time\|single" "Read plan once"; then
+    : # pass
+else
+    exit 1
+fi
+
+if assert_contains "$output" "Step 1\|beginning\|start\|Load Plan" "Read at beginning"; then
+    : # pass
+else
+    exit 1
+fi
+
+echo ""
+
+# Test 5: Verify spec compliance reviewer is skeptical
+echo "Test 5: Spec compliance reviewer mindset..."
+
+output=$(run_claude "What is the spec compliance reviewer's attitude toward the implementer's report in subagent-driven-development?" 30)
+
+if assert_contains "$output" "not trust\|don't trust\|skeptical\|verify.*independently\|suspiciously" "Reviewer is skeptical"; then
+    : # pass
+else
+    exit 1
+fi
+
+if assert_contains "$output" "read.*code\|inspect.*code\|verify.*code" "Reviewer reads code"; then
+    : # pass
+else
+    exit 1
+fi
+
+echo ""
+
+# Test 6: Verify review loops
+echo "Test 6: Review loop requirements..."
+
+output=$(run_claude "In subagent-driven-development, what happens if a reviewer finds issues? Is it a one-time review or a loop?" 30)
+
+if assert_contains "$output" "loop\|again\|repeat\|until.*approved\|until.*compliant" "Review loops mentioned"; then
+    : # pass
+else
+    exit 1
+fi
+
+if assert_contains "$output" "implementer.*fix\|fix.*issues" "Implementer fixes issues"; then
+    : # pass
+else
+    exit 1
+fi
+
+echo ""
+
+# Test 7: Verify full task text is provided
+echo "Test 7: Task context provision..."
+
+output=$(run_claude "In subagent-driven-development, how does the controller provide task information to the implementer subagent? Does it make them read a file or provide it directly?" 30)
+
+if assert_contains "$output" "provide.*directly\|full.*text\|paste\|include.*prompt" "Provides text directly"; then
+    : # pass
+else
+    exit 1
+fi
+
+if assert_not_contains "$output" "read.*file\|open.*file" "Doesn't make subagent read file"; then
+    : # pass
+else
+    exit 1
+fi
+
+echo ""
+
+echo "=== All subagent-driven-development skill tests passed ==="