Просмотр исходного кода

feat(skills): add e2e desktop computer-use driving recipe

Jesse Vincent 2 месяцев назад
Родитель
Сommit
f48daf84f7
1 измененных файлов с 76 добавлено и 0 удалено
  1. 76 0
      skills/agentic-end-to-end-testing/driving-computer-use.md

+ 76 - 0
skills/agentic-end-to-end-testing/driving-computer-use.md

@@ -0,0 +1,76 @@
+# Driving a Desktop App (Computer Use)
+
+Drive the live app through its accessibility tree, not screen-pixel guesses,
+whenever an accessibility-driven tool is available. The worked example
+throughout is macOS accessibility automation (an app-state dump plus
+element-indexed click/type actions); the same dump-act-re-dump discipline
+applies to any platform's accessibility layer.
+
+## Dump, act, re-dump
+
+Before touching anything, pull a full app-state dump — the accessibility
+tree, not a screenshot. Read every element index and role off *that* dump;
+never guess or reuse an index from a previous dump, since insertions and
+removals renumber the tree.
+
+```text
+get_app_state {app}
+click {app, element_index}       # index/role read from the dump above
+type_text {app, text}
+get_app_state {app}              # re-dump — did the field you predicted change?
+```
+
+Re-dump after every action, not just at the end. An action without a
+following dump is a click you can't prove happened — you only have proof once
+you've read the state back and it shows the change.
+
+## Quote the observed state into the record
+
+The evidence is the before → after value read from the dump, quoted directly
+into the report or commit — not a description of the click. A counter that
+should now read a higher page, a selection whose label changed after a
+"next" action: put the literal *old value* and *new value* side by side so a
+reader can re-run the same action and check for the same transition. "I
+clicked the button" proves nothing; "field X read `A`, then `B`" is
+falsifiable.
+
+## Isolate before you drive
+
+Copy the built app to a throwaway location under a distinct bundle
+identifier and reset its permission grants before scripting it, so a driving
+session can't corrupt the real app's session state or permissions. Build any
+harness the driving needs outside the project's own repo — end-to-end
+driving should never mutate the project under test.
+
+## The escalation ladder
+
+Accessibility automation on a real desktop is not always available cleanly.
+Climb a ladder of approaches, and when a rung is blocked, record *why* before
+trying the next one:
+
+1. **Scripting API** (e.g. an OS-level accessibility scripting bridge) — the
+   cheap default. Blocked signature: a permission error before any command
+   runs (no Accessibility grant).
+2. **UI-test harness** (a first-class automated-testing framework for the
+   platform) — the "proper" way to drive the real app end to end. Blocked
+   signature: the harness process itself never establishes its automation
+   session (e.g. an unsigned test runner killed before it attaches) — that's
+   the harness failing to bootstrap, not a bug in the app under test.
+3. **Raw input injection** (a coordinate-based click/keystroke tool plus a
+   screenshot after each action) — the fallback of last resort when both of
+   the above are blocked. Coarser than element-indexed driving, so screenshot
+   after every action and confirm the click landed on the intended window
+   before trusting the result.
+
+Every rung you tried belongs in the report, including the ones that failed —
+not only the one that worked. Diagnose each blocked rung enough to state the
+failure cleanly (permission denied, session never attached, wrong window
+frontmost) before moving on; a rung abandoned without a stated reason is
+indistinguishable from one you never tried.
+
+## A blocked ladder is a report, not an excuse
+
+If every rung is blocked, that is the result: write down what you tried, what
+each rung's failure looked like, and stop there. Never fall back to
+describing what the UI "should" do, and never fabricate a dump or a
+before/after value you didn't actually read back from the running app.