Explorar o código

feat(skills): add e2e browser and CLI/TUI driving recipes

Jesse Vincent hai 2 meses
pai
achega
cb7b11ab2a

+ 93 - 0
skills/agentic-end-to-end-testing/driving-cli-tui.md

@@ -0,0 +1,93 @@
+# Driving a CLI / TUI (tmux)
+
+Each scenario gets its own named tmux session (cleanup needs a deterministic
+name). Fix the size for deterministic capture; prefer the app's plain-text/inline
+mode if it has one.
+
+## The four-command recipe
+
+```bash
+tmux new-session -d -s <name> -x 200 -y 50 "<cmd> 2>/tmp/<name>-stderr.log"
+tmux send-keys -t <name> -l "literal text"   # -l = no key-name parsing (paths, slashes)
+tmux send-keys -t <name> Enter
+tmux capture-pane -t <name> -p                # -p = plain text; add -e only for styling
+```
+
+- `-x 200 -y 50` fixes the pane size so `capture-pane` output is deterministic
+  run to run — a resized pane reflows text differently.
+- Always `-l` for user-typed strings; without it a literal path like
+  `/foo/bar` gets parsed as arrow-key escapes instead of typed characters.
+- Redirect stderr to a file — panics, log lines, and debug probes land there,
+  not in the pane, so they won't show up in a `capture-pane` snapshot at all.
+
+Kill any leftover session with the same name before starting a new one, so
+reruns don't attach to a stale process:
+
+```bash
+tmux kill-session -t <name> 2>/dev/null   # idempotent: fine if nothing to kill
+```
+
+## Form fill: send-keys patterns
+
+`send-keys` parses keystrokes by name (`Enter`, `BTab`, `C-u`) unless you pass
+`-l` for literal text. A typical field-by-field fill mixes both:
+
+```bash
+tmux send-keys -t <name> BTab                 # shift-tab to a prior field
+tmux send-keys -t <name> C-u                  # clear the current line
+tmux send-keys -t <name> -l "some/literal/path"   # literal — no key parsing
+tmux send-keys -t <name> Tab                  # forward to the next field
+tmux send-keys -t <name> Enter
+```
+
+`sleep 0.3` between keys is usually enough; bump to 0.5–1.0s for field
+transitions where the UI re-renders.
+
+## Polling capture-pane for state
+
+Poll `capture-pane -p` for a state string and grep the **glyph or word**, not
+the color — `-p` drops ANSI styling by default (add `-e` only if you need
+styling), and colors are also just harder to grep reliably than a fixed
+glyph:
+
+```bash
+for i in $(seq 1 30); do
+  pane=$(tmux capture-pane -t <name> -p)
+  echo "$pane" | grep -q "state: processing" && break
+  sleep 1
+done
+```
+
+TUIs commonly use a distinct glyph per state, e.g. a Braille spinner (`⠋`)
+while pending and an X mark (`✗`) on failure, with the glyph simply removed
+once reconciled. Grep for the glyph itself, not for a color code.
+
+## Two captures for optimistic UI
+
+Mirror the web sync/async pattern: capture the pane immediately after the
+triggering keypress, then again after a reconcile window. Without the
+immediate capture you can't tell "rendered then reconciled" from "never
+rendered":
+
+```bash
+tmux send-keys -t <name> -l "trigger the optimistic action"
+tmux send-keys -t <name> Enter
+echo "=== synchronous ===" ; tmux capture-pane -t <name> -p | grep -E "pending-glyph"
+sleep 6
+echo "=== reconciled  ===" ; tmux capture-pane -t <name> -p | grep -E "pending-glyph" || echo "[no pending — reconciled]"
+```
+
+## Plain-text mode over the alt-screen buffer
+
+If the TUI has a flag that disables its alternate-screen buffer (a debug or
+plain-output mode), use it when launching under tmux. `capture-pane` then sees
+plain scrollback text instead of raw escape sequences from a full-screen
+redraw, which is much easier to grep.
+
+## Non-interactive CLIs don't need tmux
+
+If the surface under test is a one-shot command rather than an interactive
+session, skip tmux entirely — run the command and capture its stdout/stderr
+directly. The tmux machinery exists for interaction, not for driving a binary
+in general. Still run it against a real, freshly built instance, not a stale
+one left over from an earlier session.

+ 95 - 0
skills/agentic-end-to-end-testing/driving-web-browser.md

@@ -0,0 +1,95 @@
+# Driving a Web UI (Browser)
+
+Use a Chrome/CDP browser tool. After authenticated navigation, drive the page
+through `eval` against the app's own JS entry points rather than synthesizing
+clicks where possible — it's more robust to layout change than clicking
+coordinates or brittle selectors.
+
+## Authenticated navigation
+
+If the app's login flow is a token-bearing redirect (e.g. a URL like
+`/auth?token=<TOKEN>&next=<path>`), navigate straight to that URL and then wait
+for an element you expect to exist once the session is live:
+
+```text
+navigate http://<host>/auth?token=<TOKEN>&next=<path>
+await_element [data-some-marker]
+```
+
+Use the literal token value, not the path to the file that contains it. Passing
+the path instead of the token itself typically renders as an "invalid token"
+page rather than an obvious stack trace — if you see that error, check which
+one you passed.
+
+## Optimistic-vs-settled assertions
+
+For any "did the optimistic UI update happen before the request resolved?"
+scenario, fire the action but *don't await it*, take a synchronous DOM
+snapshot (the pending placeholder is there *now*), then await and snapshot
+again:
+
+```javascript
+(async () => {
+  const before = {
+    pendingCount: document.querySelectorAll(".optimistic-pending").length,
+  };
+  // Fire — capture the promise but don't await yet.
+  const promise = window.App.doAction(id, payload).catch(e => e);
+  // Synchronous: the pending placeholder is in the DOM RIGHT NOW.
+  const sync = {
+    pendingCount: document.querySelectorAll(".optimistic-pending").length,
+    pendingText: document.querySelector(".optimistic-pending")?.textContent,
+  };
+  await promise;
+  await new Promise(r => setTimeout(r, 200));  // let the DOM settle
+  const after = {
+    pendingCount: document.querySelectorAll(".optimistic-pending").length,
+    failedCount: document.querySelectorAll(".optimistic-failed").length,
+    reason: document.querySelector(".optimistic-failed-reason")?.textContent,
+  };
+  return JSON.stringify({ before, sync, after }, null, 2);
+})()
+```
+
+Without the no-await capture you can't tell "rendered then reconciled" from
+"never rendered" — both look identical in the post-await snapshot alone.
+
+## Return a plain string from eval
+
+Join your findings into a string (e.g. `JSON.stringify(..., null, 2)` or
+`\n`-joined lines) before returning from `eval`. Some bridges stringify a
+returned object as `[object Object]`, silently discarding everything you
+wanted to inspect.
+
+## Probing internal state when the DOM is ambiguous
+
+Inspect the app's singleton via `window.<App>?.state` (or whatever it exposes)
+when the DOM alone can't tell you what happened:
+
+```javascript
+JSON.stringify({
+  state: window.App?.state,          // idle | processing | …
+  hydrated: window.App?.hydrated,
+  pendingType: typeof window.App?.pending,
+  windowKeys: Object.keys(window).filter(k => k.toLowerCase().includes("app")),
+})
+```
+
+The `windowKeys` scan is useful when you don't already know the singleton's
+name — grep the result for something plausible. If a hydration/connection
+flag is `false` when you expect `true`, or a registry that should be an object
+comes back `"undefined"`, that's usually the real bug, not a DOM timing issue.
+
+## Prefer labels over selectors
+
+When a step needs a concrete locator, prefer a label the user actually sees
+(button text, aria-label, visible heading) over a brittle structural selector
+like `#nav > li:nth-child(3)`. A layout shuffle breaks the selector; it rarely
+changes the label.
+
+## When console capture is unreliable
+
+If the browser tool's console-log capture is flaky or stubbed, route debug
+output through `eval` instead: push entries to a `window.__DEBUG_LOG` array
+from the page, then read it back with a follow-up `eval` call. This sidesteps
+the capture path entirely and gives you an ordinary string to inspect.