소스 검색

Add comprehensive testing documentation

Documents:
- How to run integration tests
- subagent-driven-development test details
- Token analysis tool usage
- Troubleshooting common issues
- Writing new integration tests
- Session transcript format
Jesse Vincent 9 달 전
부모
커밋
2ea4f36b0f
1개의 변경된 파일303개의 추가작업 그리고 0개의 파일을 삭제
  1. 303 0
      docs/testing.md

+ 303 - 0
docs/testing.md

@@ -0,0 +1,303 @@
+# Testing Superpowers Skills
+
+This document describes how to test Superpowers skills, particularly the integration tests for complex skills like `subagent-driven-development`.
+
+## Overview
+
+Testing skills that involve subagents, workflows, and complex interactions requires running actual Claude Code sessions in headless mode and verifying their behavior through session transcripts.
+
+## Test Structure
+
+```
+tests/
+├── claude-code/
+│   ├── test-helpers.sh                    # Shared test utilities
+│   ├── test-subagent-driven-development-integration.sh
+│   ├── analyze-token-usage.py             # Token analysis tool
+│   └── run-skill-tests.sh                 # Test runner (if exists)
+```
+
+## Running Tests
+
+### Integration Tests
+
+Integration tests execute real Claude Code sessions with actual skills:
+
+```bash
+# Run the subagent-driven-development integration test
+cd tests/claude-code
+./test-subagent-driven-development-integration.sh
+```
+
+**Note:** Integration tests can take 10-30 minutes as they execute real implementation plans with multiple subagents.
+
+### Requirements
+
+- Must run from the **superpowers plugin directory** (not from temp directories)
+- Claude Code must be installed and available as `claude` command
+- Local dev marketplace must be enabled: `"superpowers@superpowers-dev": true` in `~/.claude/settings.json`
+
+## Integration Test: subagent-driven-development
+
+### What It Tests
+
+The integration test verifies the `subagent-driven-development` skill correctly:
+
+1. **Plan Loading**: Reads the plan once at the beginning
+2. **Full Task Text**: Provides complete task descriptions to subagents (doesn't make them read files)
+3. **Self-Review**: Ensures subagents perform self-review before reporting
+4. **Review Order**: Runs spec compliance review before code quality review
+5. **Review Loops**: Uses review loops when issues are found
+6. **Independent Verification**: Spec reviewer reads code independently, doesn't trust implementer reports
+
+### How It Works
+
+1. **Setup**: Creates a temporary Node.js project with a minimal implementation plan
+2. **Execution**: Runs Claude Code in headless mode with the skill
+3. **Verification**: Parses the session transcript (`.jsonl` file) to verify:
+   - Skill tool was invoked
+   - Subagents were dispatched (Task tool)
+   - TodoWrite was used for tracking
+   - Implementation files were created
+   - Tests pass
+   - Git commits show proper workflow
+4. **Token Analysis**: Shows token usage breakdown by subagent
+
+### Test Output
+
+```
+========================================
+ Integration Test: subagent-driven-development
+========================================
+
+Test project: /tmp/tmp.xyz123
+
+=== Verification Tests ===
+
+Test 1: Skill tool invoked...
+  [PASS] subagent-driven-development skill was invoked
+
+Test 2: Subagents dispatched...
+  [PASS] 7 subagents dispatched
+
+Test 3: Task tracking...
+  [PASS] TodoWrite used 5 time(s)
+
+Test 6: Implementation verification...
+  [PASS] src/math.js created
+  [PASS] add function exists
+  [PASS] multiply function exists
+  [PASS] test/math.test.js created
+  [PASS] Tests pass
+
+Test 7: Git commit history...
+  [PASS] Multiple commits created (3 total)
+
+Test 8: No extra features added...
+  [PASS] No extra features added
+
+=========================================
+ Token Usage Analysis
+=========================================
+
+Usage Breakdown:
+----------------------------------------------------------------------------------------------------
+Agent           Description                          Msgs      Input     Output      Cache     Cost
+----------------------------------------------------------------------------------------------------
+main            Main session (coordinator)             34         27      3,996  1,213,703 $   4.09
+3380c209        implementing Task 1: Create Add Function     1          2        787     24,989 $   0.09
+34b00fde        implementing Task 2: Create Multiply Function     1          4        644     25,114 $   0.09
+3801a732        reviewing whether an implementation matches...   1          5        703     25,742 $   0.09
+4c142934        doing a final code review...                    1          6        854     25,319 $   0.09
+5f017a42        a code reviewer. Review Task 2...               1          6        504     22,949 $   0.08
+a6b7fbe4        a code reviewer. Review Task 1...               1          6        515     22,534 $   0.08
+f15837c0        reviewing whether an implementation matches...   1          6        416     22,485 $   0.07
+----------------------------------------------------------------------------------------------------
+
+TOTALS:
+  Total messages:         41
+  Input tokens:           62
+  Output tokens:          8,419
+  Cache creation tokens:  132,742
+  Cache read tokens:      1,382,835
+
+  Total input (incl cache): 1,515,639
+  Total tokens:             1,524,058
+
+  Estimated cost: $4.67
+  (at $3/$15 per M tokens for input/output)
+
+========================================
+ Test Summary
+========================================
+
+STATUS: PASSED
+```
+
+## Token Analysis Tool
+
+### Usage
+
+Analyze token usage from any Claude Code session:
+
+```bash
+python3 tests/claude-code/analyze-token-usage.py ~/.claude/projects/<project-dir>/<session-id>.jsonl
+```
+
+### Finding Session Files
+
+Session transcripts are stored in `~/.claude/projects/` with the working directory path encoded:
+
+```bash
+# Example for /Users/jesse/Documents/GitHub/superpowers/superpowers
+SESSION_DIR="$HOME/.claude/projects/-Users-jesse-Documents-GitHub-superpowers-superpowers"
+
+# Find recent sessions
+ls -lt "$SESSION_DIR"/*.jsonl | head -5
+```
+
+### What It Shows
+
+- **Main session usage**: Token usage by the coordinator (you or main Claude instance)
+- **Per-subagent breakdown**: Each Task invocation with:
+  - Agent ID
+  - Description (extracted from prompt)
+  - Message count
+  - Input/output tokens
+  - Cache usage
+  - Estimated cost
+- **Totals**: Overall token usage and cost estimate
+
+### Understanding the Output
+
+- **High cache reads**: Good - means prompt caching is working
+- **High input tokens on main**: Expected - coordinator has full context
+- **Similar costs per subagent**: Expected - each gets similar task complexity
+- **Cost per task**: Typical range is $0.05-$0.15 per subagent depending on task
+
+## Troubleshooting
+
+### Skills Not Loading
+
+**Problem**: Skill not found when running headless tests
+
+**Solutions**:
+1. Ensure you're running FROM the superpowers directory: `cd /path/to/superpowers && tests/...`
+2. Check `~/.claude/settings.json` has `"superpowers@superpowers-dev": true` in `enabledPlugins`
+3. Verify skill exists in `skills/` directory
+
+### Permission Errors
+
+**Problem**: Claude blocked from writing files or accessing directories
+
+**Solutions**:
+1. Use `--permission-mode bypassPermissions` flag
+2. Use `--add-dir /path/to/temp/dir` to grant access to test directories
+3. Check file permissions on test directories
+
+### Test Timeouts
+
+**Problem**: Test takes too long and times out
+
+**Solutions**:
+1. Increase timeout: `timeout 1800 claude ...` (30 minutes)
+2. Check for infinite loops in skill logic
+3. Review subagent task complexity
+
+### Session File Not Found
+
+**Problem**: Can't find session transcript after test run
+
+**Solutions**:
+1. Check the correct project directory in `~/.claude/projects/`
+2. Use `find ~/.claude/projects -name "*.jsonl" -mmin -60` to find recent sessions
+3. Verify test actually ran (check for errors in test output)
+
+## Writing New Integration Tests
+
+### Template
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/test-helpers.sh"
+
+# Create test project
+TEST_PROJECT=$(create_test_project)
+trap "cleanup_test_project $TEST_PROJECT" EXIT
+
+# Set up test files...
+cd "$TEST_PROJECT"
+
+# Run Claude with skill
+PROMPT="Your test prompt here"
+cd "$SCRIPT_DIR/../.." && timeout 1800 claude -p "$PROMPT" \
+  --allowed-tools=all \
+  --add-dir "$TEST_PROJECT" \
+  --permission-mode bypassPermissions \
+  2>&1 | tee output.txt
+
+# Find and analyze session
+WORKING_DIR_ESCAPED=$(echo "$SCRIPT_DIR/../.." | sed 's/\\//-/g' | sed 's/^-//')
+SESSION_DIR="$HOME/.claude/projects/$WORKING_DIR_ESCAPED"
+SESSION_FILE=$(find "$SESSION_DIR" -name "*.jsonl" -type f -mmin -60 | sort -r | head -1)
+
+# Verify behavior by parsing session transcript
+if grep -q '"name":"Skill".*"skill":"your-skill-name"' "$SESSION_FILE"; then
+    echo "[PASS] Skill was invoked"
+fi
+
+# Show token analysis
+python3 "$SCRIPT_DIR/analyze-token-usage.py" "$SESSION_FILE"
+```
+
+### Best Practices
+
+1. **Always cleanup**: Use trap to cleanup temp directories
+2. **Parse transcripts**: Don't grep user-facing output - parse the `.jsonl` session file
+3. **Grant permissions**: Use `--permission-mode bypassPermissions` and `--add-dir`
+4. **Run from plugin dir**: Skills only load when running from the superpowers directory
+5. **Show token usage**: Always include token analysis for cost visibility
+6. **Test real behavior**: Verify actual files created, tests passing, commits made
+
+## Session Transcript Format
+
+Session transcripts are JSONL (JSON Lines) files where each line is a JSON object representing a message or tool result.
+
+### Key Fields
+
+```json
+{
+  "type": "assistant",
+  "message": {
+    "content": [...],
+    "usage": {
+      "input_tokens": 27,
+      "output_tokens": 3996,
+      "cache_read_input_tokens": 1213703
+    }
+  }
+}
+```
+
+### Tool Results
+
+```json
+{
+  "type": "user",
+  "toolUseResult": {
+    "agentId": "3380c209",
+    "usage": {
+      "input_tokens": 2,
+      "output_tokens": 787,
+      "cache_read_input_tokens": 24989
+    },
+    "prompt": "You are implementing Task 1...",
+    "content": [{"type": "text", "text": "..."}]
+  }
+}
+```
+
+The `agentId` field links to subagent sessions, and the `usage` field contains token usage for that specific subagent invocation.