|
|
@@ -10,7 +10,7 @@ The web GUI ships as a real assembled chain — chromium page → client plugin
|
|
|
|
|
|
## Decision
|
|
|
|
|
|
-`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay` against the real in-process web composition, asserting a normalized conversation aria golden plus in-process world state. No new package; the product deltas are additive `dsh-llm-replay` surfaces (`paceMs`, `ReplayHandle`, and the `{ patches }` override form: indexed augmentation over the derived script so a sidecar expresses "call N throws / hangs, everything else replays as recorded" without copying recorded chunks), one `dsh-llm` fix the retry scenario exposed (a carried `failure` snapshot is honored on any Error — the `instanceof` gate dropped provider codes across dual package copies, source-plane replay over a lib-plane boot), and the `llm-retry` row the web composition was missing.
|
|
|
+`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replay through `@deepseek-ai/dsh-llm-replay` against the real in-process web composition, with normalized aria goldens for user-visible states and in-process assertions for durable world state. The supporting product contracts are `dsh-llm-replay` pacing, consumption checks, and validated indexed override patches; cross-package `dsh-llm` failures retain validated provider facts through own data properties; and the shipped web composition mounts `llm-retry` for transient model failures.
|
|
|
|
|
|
### Scaffold: `apps/web/tests/scaffold.ts`
|
|
|
|
|
|
@@ -32,25 +32,17 @@ Every scenario fails on any pageerror and on the client's connection-loss/gap-re
|
|
|
|
|
|
### Expected outputs
|
|
|
|
|
|
-At least one committed golden per scenario, and one per DISTINCT end-state for the interactive scenarios (cancel/error/retry, waiting/answered, mid-steer/settled, panel-open, post-reload): a normalized `ariaSnapshot()` of the scenario's owning region — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride root-context session events inline (which tool call produced which durable result, whether `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping.
|
|
|
+Scenarios with a stable owning region commit a normalized `ariaSnapshot()` for each distinct user-visible state; cross-region workspace-management states instead use semantic DOM assertions plus authoritative host-state checks. UUID, cwd, workspace basename, and duration volatility collapse to stable tokens; captures poll until consecutive normalized reads agree. Role and text anchors remain semantic guards around the reviewable goldens and own cross-region states directly. World-state assertions use root-context session events rather than a second committed log golden because the ACP, headless, and TUI suites already pin the persisted-log surface through the same loop and persistence. `refresh` is the sole golden writer; a missing replay golden fails with the regeneration command.
|
|
|
|
|
|
-The typecheck plane split is structural: the three files that boot the host spine (`scaffold`, `replay-round-trip.e2e`, and `seeded-history.e2e`) are excluded from the client-registered `apps/web` project. Those files and their shared `support.ts` are included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges.
|
|
|
+The typecheck plane split is structural: the host scaffold, its support module, and every web spec that boots or inspects the host composition are excluded from the client-registered `apps/web` project and included file-by-file in `tsconfig.host.json`. One program cannot hold both sides of the Cordis `Context` merges.
|
|
|
|
|
|
### Modes and fixtures
|
|
|
|
|
|
-`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` tokenization; a follow-up keyless refresh regenerates the aria goldens. Every prompting scenario's fixture was recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point with no run-local `rpcId`). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred.
|
|
|
+`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless). Prompting specs separate drive steps shared by all modes from replay/refresh assertions; record mode drives the live composer, harvests the in-memory session header and events, scrubs request headers, and tokenizes run-local session, cwd, and RPC identities. A follow-up keyless refresh regenerates aria goldens. Each prompt is checked against its fixture's recorded `user/message`, and each scenario directory has a closed inventory whose JSONL files are scrub fixed points. Web fixtures scrub headers everywhere and pin no header class; see Deferred.
|
|
|
|
|
|
-### Scenarios
|
|
|
+### Coverage contract
|
|
|
|
|
|
-1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (the bash call's durable result is exactly `WEB_E2E_OK\n`, completed `turn/end`, >10 chunk events).
|
|
|
-2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed.
|
|
|
-3. **`live-interactions`** — one tool-free recorded turn serves three replay-only scenarios through override sidecars whose CONTENT is authored in the spec and minted as a per-run file in a spec-owned temp dir (the derived success entry for the retry append is re-derived from the fixture via `deriveReplayScript`, never copied into a committed sidecar). Cancel: a `{ patches }` `hang` with a `readyFile` marker — the marker's existence proves the stream is parked mid-turn before the test clicks Stop, making mid-stream cancellation deterministic by construction (`turn/end` reason `aborted`, composer re-enabled). AUTH error: a pre-chunk `throw` outside llm-retry's retryable set (`turn/end` reason `error`, zero `llm/retry` events, composer recovers). SERVER retry: `throw` at call 0 + the fixture's own success appended at 1, proving llm-retry end-to-end in the browser via the durable `llm/retry` record (`request/header` logs only on change, so attempt count is invisible there). Each scenario pins its terminal surface as a golden: `cancel.expected.md` (frozen `partial`, 已停止 marker), `error-auth.expected.md` (the prompt bubble alone — the committed artifact of the web-error-surface gap, the diff that flips when error rendering lands), `retry.expected.md` (indistinguishable from a clean completion — retries are deliberately invisible in the transcript).
|
|
|
-4. **`question-composer`** — the shipped composition's resident `ask_user_question` takeover: a recorded turn blocks mid-step on the real userInteraction seam, the composer (`[data-question-key]`) renders in the browser, the test answers through it (the ONE sanctioned place a drive step reacts to model content: the turn cannot complete without the answer, in record and replay alike), and the tool result carries the chosen label. Goldens: the composer's stable waiting state (`ui.expected.md`) and the answered transcript (`answered.expected.md` — the question resolved into its tool round trip plus the final reply, takeover gone).
|
|
|
-5. **`steering`** — mid-turn steer while the question composer blocks the step (the deterministic mid-turn window; no timing dependence). The composer locks while running, so the steer POSTs `session.prompt` `mode:'steer'` from the page over the same same-origin `/api` wire the client uses (`TODO(web-steer-composer)`: drive a composer gesture once one exists); everything downstream is product — gateway → `Agent.steer` → step-boundary drain → durable `steering/message` → SSE → badged interjection bubble. Record-mode fixture honesty: the recording is rejected unless the live model's final reply obeys an instruction only the steering message carries. Goldens pin the timing semantics visually: `mid-steer.expected.md` captures the accepted-but-invisible state (the loop drains steering only at the step boundary, so no interjection bubble exists while the question still blocks — if the client ever renders pending steers eagerly, this golden flips first) and `settled.expected.md` the badged bubble plus obeying reply.
|
|
|
-6. **`navigation-panes`** — one rich two-turn seed (turn 1: bash + two parallel reads in one assistant message; turn 2: a markdown-heavy reply) rendered cold through the seeded-history pattern (zero model calls), serving four surfaces: sidebar search (client-side title filter — asserted only after the durable title lands with the attach baseline, because a cold `SessionSummary` carries no title and search matches the `displayTitle` the user sees; negative query empties the tree, positive narrows, clear restores), the Trajectory tab (turn sections + the step group's tool mix plus the view-area aria golden), the Waterfall tab (span stats + one lane per span — the P-I fold counts a turn-0 prologue span because only assistant/steering nodes carry a turn number, pinned as-is), and the details column (the bash toolview row routes click to openDetails; open/closed is asserted on the frame's `data-details-collapsed` attribute because close collapses the grid column to width 0 without unmounting the subtree). Goldens: `trajectory.expected.md` and `waterfall.expected.md` (each tab's view area) plus `details-open.expected.md` (the open panel: tool-name header, Input args, Output result).
|
|
|
-7. **`lifecycle-chrome`** — one tiny recorded text turn drives three whole-page concerns. Workspace flow over the real wire: the empty-state hero's first send materializes a real Workspace + Session (the jsdom `workspace-flow.snapshot.ts` suite pins this state machine over the fixture client; this scenario pins it through HTTP RPC + SSE + the gateway), proven durably by the session header's cwd being the create-by-name target `<workspaceRoot>/workspace`, plus the hero waiting-state aria golden. Reload recovery: collapse the sidebar (persisted `dsh.layout.panels`), `page.reload`, and the surface comes back whole from persistence alone — layout collapsed, selection restored (`dsh.sessions.current`), the recorded turn re-rendered from `session.history` with zero model calls (the drained replay cursor makes any stray request fail loud at close), and `reloaded.expected.md` pins the rebuilt conversation region — rendering the same settled transcript from persistence alone IS the recovery claim. Dark mode: the scenario drives the ThemeService's DOM contract seam directly — the `body[data-ds-dark-theme]` attribute — and pins the shipped cascade (alias token flips, a painted surface repaints, removal restores the light sample exactly), independent of the settings surface whose real user gesture `settings-chrome` owns; per the scope ruling there is no theme/layout golden (aria is color-blind).
|
|
|
-8. **`settings-chrome`** — the settings surface (#644), zero model calls on a blank frame. The modal shell: sidebar-foot trigger (`aria-haspopup`/`aria-expanded`) opens `role=dialog` 设置, General active by default with the skeleton rows plus the functional Language and Appearance rows (dialog aria golden), section switch moves `aria-current` to the deliberately empty Models, closes via Escape and the header close button. The Appearance row is the REAL theme gesture (retiring the lifecycle scenario's `TODO(web-theme-gesture)`): clicking 深色 runs the whole chain — `aria-pressed`, persisted `dsh.theme`, `body[data-ds-dark-theme]`, alias-token flip — and survives reload; `system` follows the emulated OS scheme both ways (`page.emulateMedia`), and the spec restores the light default for inter-spec hygiene. The Language row switches the settings-scoped copy to English (`dsh.locale` persisted, dialog re-registers as Settings/General/Appearance), survives reload, and restores zh — only the settings namespaces are localized today, so the scenario asserts exactly that surface. Intentional reloads tear the SSE stream, so the spec drains exactly the reconnect warnings its own reloads caused; the tripwire still fails on any unexpected connection loss.
|
|
|
-9. **`workspace-management`** — the workspace browser operations (#643), zero model calls (workspace.create/rename are host RPCs; the one session row comes from re-seeding seeded-history's committed seed, so no new fixture is recorded). Create-by-name twice through the region-header ＋ dialog (`workspace.create` mkdirs and prepends to the durable registry — asserted host-side via `ctx.workspace.list()`). Rename end to end: the hover-revealed row-actions menu (the button is `display:none` until its row hovers) → Rename dialog → the duplicate-name pre-check raises the inline `role=alert` and disables the primary button before any wire call → a fresh name goes through the `workspace.rename` RPC, updates the row, persists on the host, and survives reload. The flat "In one list" view: the Group by menu flips the section label to Sessions, drops group headers (seeded session becomes a top-level row), persists in `dsh.workspace.view` across reload, and the spec restores grouped mode. The session hover card renders after the dwell (display-only, no aria role — text anchors) and closes when the pointer leaves. Deliberately NOT driven: the visual-only menu rows this iteration ships inert (session Rename/Fork/Delete, workspace Delete) and drag reorder — see Deferred.
|
|
|
+The lane covers three behavior families. Live-turn scenarios pin ordinary tool execution, cancellation, non-retryable failure, transient retry, resident questions, and mid-turn steering; synchronization uses durable events, `whenIdle()`, or an explicit replay marker rather than delays. Cold-history scenarios seed through the real persistence API and cover history rendering, sidebar search, trajectory and waterfall views, and tool details without model calls. Browser-lifecycle scenarios cover first-send workspace materialization, reload recovery, layout persistence, theme and locale preferences, and workspace create/rename/view operations. Each family asserts the browser surface and the authoritative host state; a stray model call or under-consumed fixture fails teardown.
|
|
|
|
|
|
### CI stance
|
|
|
|
|
|
@@ -70,7 +62,7 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
|
|
|
|
|
**Placeholder `DEEPSEEK_API_KEY` + replay interception instead of disabling the adapter row.** Rejected despite zero composition change and two in-tree precedents: it satisfies `llm-deepseek`'s fail-loud key check with a lie and leaves a dead adapter mounted-but-intercepted; the disabled row (the ACP overlay's move) is honest keylessness and fails loud at the earliest resolvable point.
|
|
|
|
|
|
-**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free.
|
|
|
+**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and the scenario-specific interactions have not produced a stable browser-free contract beyond the helpers already exported from gated packages and the local scaffold. Reconsider when a second web-shaped consumer or demonstrably repeated lifecycle code establishes that contract.
|
|
|
|
|
|
**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on root-context events keep the world-verification duty.
|
|
|
|
|
|
@@ -80,11 +72,11 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
|
|
|
|
|
**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement.
|
|
|
|
|
|
-**A client `data-dsh-busy` settled signal.** Deferred: the multi-condition settled polls proved sufficient at two scenarios and the host-side `whenIdle` barrier does the heavy lifting. Re-entry trigger: the first settled-poll flake, or a scenario needing a state the DOM does not expose.
|
|
|
+**A client `data-dsh-busy` settled signal.** Deferred: the host-side `whenIdle` barrier plus stable DOM polls cover the current scenarios. Reconsider after the first settled-poll flake or when a required state is not observable in the DOM.
|
|
|
|
|
|
## Testing
|
|
|
|
|
|
-The lane itself: `pnpm run test:web` runs every scenario keylessly alongside the existing smoke pair; `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` re-records a scenario's fixture against the live model; `DSH_SNAPSHOT=refresh` rewrites the aria goldens keylessly. `paceMs` validation, pacing floor, abort-during-pace, both `assertConsumed` failure shapes, and the `{ patches }` acceptance/rejection paths (index swap keeps siblings, `at == length` appends, out-of-range/non-integer loud) are pinned in `packages/support/llm-replay/tests/llm-replay.spec.ts`.
|
|
|
+`pnpm run test:web` runs the lane keylessly. `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` records a prompting scenario against the live model, and `DSH_SNAPSHOT=refresh` rewrites aria goldens keylessly. `dsh-llm-replay` unit coverage pins pacing, cancellation, consumption diagnostics, sidecar validation, indexed replacement, and the single append position.
|
|
|
|
|
|
## Deferred
|
|
|
|
|
|
@@ -93,7 +85,7 @@ The lane itself: `pnpm run test:web` runs every scenario keylessly alongside the
|
|
|
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
|
|
|
- **Web error surface**: the client consumes no `agent/error` frames and a pre-chunk failure freezes no partial, so a non-retryable provider failure renders no error copy — the user sees the send simply stop. The AUTH scenario pins the current contract (no crash, composer recovers, turn logged `error`) and `FIXME(web-error-surface)` marks where visible error text gets asserted once the UI grows an error rendering.
|
|
|
- **Composer steering gesture**: the input locks while running (stop-or-wait), so the steering scenario steers over the wire from the page; `TODO(web-steer-composer)` upgrades the drive step to a real composer gesture when the product grows one.
|
|
|
-- **Drag session reorder**: `workspace.insertSessionBefore` (manual ordering, #643) has no browser scenario yet — it needs two sessions materialized in ONE workspace (a two-script recorded fixture) plus synthesized HTML5 drag events; add it when that surface changes or regresses. The inert menu rows (session Rename/Fork/Delete, workspace Delete) get scenarios when they gain behavior.
|
|
|
+- **Drag session reorder**: `workspace.insertSessionBefore` has no browser scenario; it needs two sessions materialized in one workspace plus synthesized HTML5 drag events. Add it when that surface changes or regresses. The inert session Rename/Fork/Delete and workspace Delete menu rows get scenarios when they gain behavior.
|
|
|
|
|
|
## Consequences
|
|
|
|
Tianyi Cui