Browse Source

docs(playbook): first P7 agent A/B numbers (proshop, Sonnet, 2 runs/arm) and the plan's P7 status

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01REFyW9hmNrxhwN5wxRoAkC
Colby McHenry 1 week ago
parent
commit
7f461269c4

+ 1 - 1
docs/design/dynamic-dispatch-coverage-playbook.md

@@ -278,7 +278,7 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
 (Verify the exact supported set against `src/extraction/languages/` and
 `src/resolution/frameworks/` before starting — this table is a starting point.)
 | TypeScript/JS | Next.js (App Router + Pages Router) | page → `<Link>` / `router.push` / `redirect` → page; page load → data; client component → `'use server'` action → DB → `redirect`; `route.ts` handler → DB → response | R + S | ✅ 2026-08-28 (`frameworks/nextjs.ts`, `next-router-synthesizer.ts`, Steps `load` trigger + server-action crossing): fixture `__tests__/nextjs.test.ts` end to end (Screens `routed`, the push attributed back under its condition, the action's redirect); `leerob/next-saas-starter` indexes its pages and links. 🔬 agent A/B (`--model sonnet`, ≥2 runs/arm) not run yet |
-| TypeScript/JS | Express + React (MERN monorepo) | client `axios` / `fetch` literal path → own route → handler → Mongoose → response rows | S + R | ✅ 2026-08-28 (`tier-synthesizer.ts` `http-client` + Express mounts / chained `router.route()` / wrapped `const h = asyncHandler(…)` handlers / nested `package.json` detection): `bradtraversy/proshop_mern` 49 routes (19 pages + 30 endpoints), 23 client→route edges, every one spot-checked correct; `login → ⇢ POST /api/users/login → User.findOne → 401 rows → jwt.sign`. Node count stable across re-index. 🔬 A/B not run |
+| TypeScript/JS | Express + React (MERN monorepo) | client `axios` / `fetch` literal path → own route → handler → Mongoose → response rows | S + R | ✅ 2026-08-28 (`tier-synthesizer.ts` `http-client` + Express mounts / chained `router.route()` / wrapped `const h = asyncHandler(…)` handlers / nested `package.json` detection): `bradtraversy/proshop_mern` 49 routes (19 pages + 30 endpoints), 23 client→route edges, every one spot-checked correct; `login → ⇢ POST /api/users/login → User.findOne → 401 rows → jwt.sign`. Node count stable across re-index. **Agent A/B (2026-08-28, `run-all.sh`, Sonnet/high, 2 runs/arm, warm daemon, CLI shim on — 0 CLI leaks):** *"How does submitting the login form reach the database, and what does the API respond with when the password is wrong?"* — with: **14s / 14s, 2 tool calls each, 0 Read, 0 Grep, 1 and 2 explores** (explore sufficiency: answered after 1 call in run 1, after 2 in run 2), answer names `submitHandler → login → POST /api/users/login → authUser → User.findOne → matchPassword → 401` end to end; without: **18s / 37s, 14 and 8 tool calls, 7 and 1 Read** (+7 Bash `cat`s in run 2, a subagent in run 1). Tokens processed 153k/165k vs 237k/369k. Pass bar met on the small repo (0 Read/Grep, faster, within 1–2 explores against a budget of 1). 🔬 medium/large (Ghost) and ≥3 prompts still to run |
 | TypeScript/JS | NestJS queues / events / sockets | `queue.add('job')` → `@Process('job')`; `emit('x')` → `@OnEvent('x')`; `socket.emit` → `@SubscribeMessage` and `server.emit` → `socket.on` | S | ✅ 2026-08-28 (`tier-synthesizer.ts` `queue-job`, `event-bus`): `nestjs/nest` `sample/26-queues` (`transcode` → `handleTranscode`) and `sample/30-event-emitter` (`order.created` → its listener) exact, 0 wrong edges after the `e2e/` and generic-event guards; `immich-app/immich` 0 edges — a generated SDK client and a wrapped queue API carry no literal, so silence (correct). 🔬 A/B not run |
 | Python | FastAPI (prefixed routers) | request → `Depends` → handler → session → `HTTPException` rows | R | ✅ 2026-08-28 `python.ts` `postExtract` composes `APIRouter(prefix=)` + literal `include_router(prefix=)`: `fastapi/full-stack-fastapi-template` 23 routes named `GET /items/{id}`, `POST /login/access-token` (were `GET /`); a computed mount prefix is skipped. 🔬 A/B not run |
 | C# | ASP.NET Minimal API endpoint groups | request → group handler → `ISender.Send` → `TypedResults` rows | R | ✅ 2026-08-28 `csharp.ts` handler-first `MapPost(Handler[, path])` under the class + `$"/api/{groupName}"` head (`jasontaylordev/CleanArchitecture` shape); fixture end to end in `ui-steps-api-servers.test.ts` (`PUT /api/TodoItems/{id}` → `204 · 400` rows); `jasontaylordev/CleanArchitecture` 10 routes, all the app's endpoints (were 0). 🔬 A/B not run |

+ 16 - 1
docs/plans/2026-08-28-steps-and-screens-for-apis-and-web.md

@@ -2,7 +2,8 @@
 
 **Status:** plan, written 2026-08-28 at the end of the session that built the Steps view and the
 readings it rests on (Expo + React Native app, `amniservices-mobile-app`). **Updated the same day, later
-sessions: P0–P6 are built** (see the per-item notes marked *Built*); P7's agent A/B numbers are open. Every claim about what a
+sessions: P0–P6 are built** (see the per-item notes marked *Built*); P7 has its first agent A/B (proshop, small) and
+the medium / large rows are open. Every claim about what a
 resolver emits *today* was verified against the source on this date — re-verify before building on it,
 the resolvers move. What was learned building it, beyond the plan: the index keeps only the LAST
 segment of a deep member call (`create` for `prisma.user.create`) and name-matches it — often to the
@@ -354,6 +355,8 @@ its `redirect` on the FILE node (Screens now re-attributes a file-scope navigati
 `useActionState(signIn, …)` leaves no function-as-value edge (a plain call argument — Screens now falls back to the
 functions that MENTION the value in the files importing it, read from the source; the principled fix is an extractor
 fnRef rule for `useActionState` / `useFormState` / `startTransition` arguments, with the Rust kernel twin).
+With both: 12 screens, 17 links, 8 origins — `/sign-in → /dashboard via Login > signIn WHEN userWithTeam.length !== 0 &&
+isPasswordValid && redirectTo !== 'checkout'`, `/dashboard/security → /sign-in via deleteAccount WHEN isPasswordValid`.
 
 *Where:* `resolution/frameworks/react.ts` (split a `nextjs.ts` out of it — the pages/app routing is
 already there), a `next-router-synthesizer.ts` modelled on `expo-router-synthesizer.ts`.
@@ -421,6 +424,18 @@ names and the frameworks detected, so the viewer does not guess).
 
 ### P7 — Validation set and the numbers
 
+*First numbers* (2026-08-28, later session) — the deterministic half is in the playbook §6 rows (Next.js, MERN, Nest
+channels, FastAPI prefixed routers, ASP.NET endpoint groups: node counts, edge precision spot-checks, pictures). One agent
+A/B so far, `bradtraversy/proshop_mern` (small, Express + React), `scripts/agent-eval/run-all.sh`, Sonnet/high, 2 runs per
+arm, a daemon pre-warmed before each with-run, the CLI shim on (0 leaks): *"How does submitting the login form reach the
+database, and what does the API respond with when the password is wrong?"* — with codegraph 14s / 14s, 2 tool calls, 0 Read,
+0 Grep, 1–2 explores, the full path named (`submitHandler → login → POST /api/users/login → authUser → User.findOne →
+matchPassword → 401`); without 18s / 37s, 14 / 8 tool calls, 7 / 1 Read plus Bash `cat`s and a subagent. Tokens 153k/165k vs
+237k/369k. The pass bar (§4 of the playbook) holds on the small repo. **Still open:** the medium / large rows (Ghost,
+immich, cal.com / twenty), ≥3 prompts per framework, and a control repo. The driver lives in the session scratchpad
+(`ab-proshop.sh`: pre-warm `serve --mcp` with `CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` high and `CODEGRAPH_WASM_RELAUNCHED=1`,
+then `run-all.sh` per run with its own `AGENT_EVAL_OUT`); re-create it from the memory note.
+
 Small fixtures live in the tests. For the real bar, index these and record the results in
 `docs/design/dynamic-dispatch-coverage-playbook.md` (new rows) exactly as the playbook asks
 (**≥3 flow prompts × small/medium/large, node count stable, synthesized-edge precision spot-check,