1
0
Эх сурвалжийг харах

docs(playbook): first P7 agent A/B numbers (proshop, Sonnet, 2 runs/arm) and the plan's P7 status

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01REFyW9hmNrxhwN5wxRoAkC
Colby McHenry 1 долоо хоног өмнө
parent
commit
7f461269c4

+ 1 - 1
docs/design/dynamic-dispatch-coverage-playbook.md

@@ -278,7 +278,7 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
 (Verify the exact supported set against `src/extraction/languages/` and
 (Verify the exact supported set against `src/extraction/languages/` and
 `src/resolution/frameworks/` before starting — this table is a starting point.)
 `src/resolution/frameworks/` before starting — this table is a starting point.)
 | TypeScript/JS | Next.js (App Router + Pages Router) | page → `<Link>` / `router.push` / `redirect` → page; page load → data; client component → `'use server'` action → DB → `redirect`; `route.ts` handler → DB → response | R + S | ✅ 2026-08-28 (`frameworks/nextjs.ts`, `next-router-synthesizer.ts`, Steps `load` trigger + server-action crossing): fixture `__tests__/nextjs.test.ts` end to end (Screens `routed`, the push attributed back under its condition, the action's redirect); `leerob/next-saas-starter` indexes its pages and links. 🔬 agent A/B (`--model sonnet`, ≥2 runs/arm) not run yet |
 | TypeScript/JS | Next.js (App Router + Pages Router) | page → `<Link>` / `router.push` / `redirect` → page; page load → data; client component → `'use server'` action → DB → `redirect`; `route.ts` handler → DB → response | R + S | ✅ 2026-08-28 (`frameworks/nextjs.ts`, `next-router-synthesizer.ts`, Steps `load` trigger + server-action crossing): fixture `__tests__/nextjs.test.ts` end to end (Screens `routed`, the push attributed back under its condition, the action's redirect); `leerob/next-saas-starter` indexes its pages and links. 🔬 agent A/B (`--model sonnet`, ≥2 runs/arm) not run yet |
-| TypeScript/JS | Express + React (MERN monorepo) | client `axios` / `fetch` literal path → own route → handler → Mongoose → response rows | S + R | ✅ 2026-08-28 (`tier-synthesizer.ts` `http-client` + Express mounts / chained `router.route()` / wrapped `const h = asyncHandler(…)` handlers / nested `package.json` detection): `bradtraversy/proshop_mern` 49 routes (19 pages + 30 endpoints), 23 client→route edges, every one spot-checked correct; `login → ⇢ POST /api/users/login → User.findOne → 401 rows → jwt.sign`. Node count stable across re-index. 🔬 A/B not run |
+| TypeScript/JS | Express + React (MERN monorepo) | client `axios` / `fetch` literal path → own route → handler → Mongoose → response rows | S + R | ✅ 2026-08-28 (`tier-synthesizer.ts` `http-client` + Express mounts / chained `router.route()` / wrapped `const h = asyncHandler(…)` handlers / nested `package.json` detection): `bradtraversy/proshop_mern` 49 routes (19 pages + 30 endpoints), 23 client→route edges, every one spot-checked correct; `login → ⇢ POST /api/users/login → User.findOne → 401 rows → jwt.sign`. Node count stable across re-index. **Agent A/B (2026-08-28, `run-all.sh`, Sonnet/high, 2 runs/arm, warm daemon, CLI shim on — 0 CLI leaks):** *"How does submitting the login form reach the database, and what does the API respond with when the password is wrong?"* — with: **14s / 14s, 2 tool calls each, 0 Read, 0 Grep, 1 and 2 explores** (explore sufficiency: answered after 1 call in run 1, after 2 in run 2), answer names `submitHandler → login → POST /api/users/login → authUser → User.findOne → matchPassword → 401` end to end; without: **18s / 37s, 14 and 8 tool calls, 7 and 1 Read** (+7 Bash `cat`s in run 2, a subagent in run 1). Tokens processed 153k/165k vs 237k/369k. Pass bar met on the small repo (0 Read/Grep, faster, within 1–2 explores against a budget of 1). 🔬 medium/large (Ghost) and ≥3 prompts still to run |
 | TypeScript/JS | NestJS queues / events / sockets | `queue.add('job')` → `@Process('job')`; `emit('x')` → `@OnEvent('x')`; `socket.emit` → `@SubscribeMessage` and `server.emit` → `socket.on` | S | ✅ 2026-08-28 (`tier-synthesizer.ts` `queue-job`, `event-bus`): `nestjs/nest` `sample/26-queues` (`transcode` → `handleTranscode`) and `sample/30-event-emitter` (`order.created` → its listener) exact, 0 wrong edges after the `e2e/` and generic-event guards; `immich-app/immich` 0 edges — a generated SDK client and a wrapped queue API carry no literal, so silence (correct). 🔬 A/B not run |
 | TypeScript/JS | NestJS queues / events / sockets | `queue.add('job')` → `@Process('job')`; `emit('x')` → `@OnEvent('x')`; `socket.emit` → `@SubscribeMessage` and `server.emit` → `socket.on` | S | ✅ 2026-08-28 (`tier-synthesizer.ts` `queue-job`, `event-bus`): `nestjs/nest` `sample/26-queues` (`transcode` → `handleTranscode`) and `sample/30-event-emitter` (`order.created` → its listener) exact, 0 wrong edges after the `e2e/` and generic-event guards; `immich-app/immich` 0 edges — a generated SDK client and a wrapped queue API carry no literal, so silence (correct). 🔬 A/B not run |
 | Python | FastAPI (prefixed routers) | request → `Depends` → handler → session → `HTTPException` rows | R | ✅ 2026-08-28 `python.ts` `postExtract` composes `APIRouter(prefix=)` + literal `include_router(prefix=)`: `fastapi/full-stack-fastapi-template` 23 routes named `GET /items/{id}`, `POST /login/access-token` (were `GET /`); a computed mount prefix is skipped. 🔬 A/B not run |
 | Python | FastAPI (prefixed routers) | request → `Depends` → handler → session → `HTTPException` rows | R | ✅ 2026-08-28 `python.ts` `postExtract` composes `APIRouter(prefix=)` + literal `include_router(prefix=)`: `fastapi/full-stack-fastapi-template` 23 routes named `GET /items/{id}`, `POST /login/access-token` (were `GET /`); a computed mount prefix is skipped. 🔬 A/B not run |
 | C# | ASP.NET Minimal API endpoint groups | request → group handler → `ISender.Send` → `TypedResults` rows | R | ✅ 2026-08-28 `csharp.ts` handler-first `MapPost(Handler[, path])` under the class + `$"/api/{groupName}"` head (`jasontaylordev/CleanArchitecture` shape); fixture end to end in `ui-steps-api-servers.test.ts` (`PUT /api/TodoItems/{id}` → `204 · 400` rows); `jasontaylordev/CleanArchitecture` 10 routes, all the app's endpoints (were 0). 🔬 A/B not run |
 | C# | ASP.NET Minimal API endpoint groups | request → group handler → `ISender.Send` → `TypedResults` rows | R | ✅ 2026-08-28 `csharp.ts` handler-first `MapPost(Handler[, path])` under the class + `$"/api/{groupName}"` head (`jasontaylordev/CleanArchitecture` shape); fixture end to end in `ui-steps-api-servers.test.ts` (`PUT /api/TodoItems/{id}` → `204 · 400` rows); `jasontaylordev/CleanArchitecture` 10 routes, all the app's endpoints (were 0). 🔬 A/B not run |

+ 16 - 1
docs/plans/2026-08-28-steps-and-screens-for-apis-and-web.md

@@ -2,7 +2,8 @@
 
 
 **Status:** plan, written 2026-08-28 at the end of the session that built the Steps view and the
 **Status:** plan, written 2026-08-28 at the end of the session that built the Steps view and the
 readings it rests on (Expo + React Native app, `amniservices-mobile-app`). **Updated the same day, later
 readings it rests on (Expo + React Native app, `amniservices-mobile-app`). **Updated the same day, later
-sessions: P0–P6 are built** (see the per-item notes marked *Built*); P7's agent A/B numbers are open. Every claim about what a
+sessions: P0–P6 are built** (see the per-item notes marked *Built*); P7 has its first agent A/B (proshop, small) and
+the medium / large rows are open. Every claim about what a
 resolver emits *today* was verified against the source on this date — re-verify before building on it,
 resolver emits *today* was verified against the source on this date — re-verify before building on it,
 the resolvers move. What was learned building it, beyond the plan: the index keeps only the LAST
 the resolvers move. What was learned building it, beyond the plan: the index keeps only the LAST
 segment of a deep member call (`create` for `prisma.user.create`) and name-matches it — often to the
 segment of a deep member call (`create` for `prisma.user.create`) and name-matches it — often to the
@@ -354,6 +355,8 @@ its `redirect` on the FILE node (Screens now re-attributes a file-scope navigati
 `useActionState(signIn, …)` leaves no function-as-value edge (a plain call argument — Screens now falls back to the
 `useActionState(signIn, …)` leaves no function-as-value edge (a plain call argument — Screens now falls back to the
 functions that MENTION the value in the files importing it, read from the source; the principled fix is an extractor
 functions that MENTION the value in the files importing it, read from the source; the principled fix is an extractor
 fnRef rule for `useActionState` / `useFormState` / `startTransition` arguments, with the Rust kernel twin).
 fnRef rule for `useActionState` / `useFormState` / `startTransition` arguments, with the Rust kernel twin).
+With both: 12 screens, 17 links, 8 origins — `/sign-in → /dashboard via Login > signIn WHEN userWithTeam.length !== 0 &&
+isPasswordValid && redirectTo !== 'checkout'`, `/dashboard/security → /sign-in via deleteAccount WHEN isPasswordValid`.
 
 
 *Where:* `resolution/frameworks/react.ts` (split a `nextjs.ts` out of it — the pages/app routing is
 *Where:* `resolution/frameworks/react.ts` (split a `nextjs.ts` out of it — the pages/app routing is
 already there), a `next-router-synthesizer.ts` modelled on `expo-router-synthesizer.ts`.
 already there), a `next-router-synthesizer.ts` modelled on `expo-router-synthesizer.ts`.
@@ -421,6 +424,18 @@ names and the frameworks detected, so the viewer does not guess).
 
 
 ### P7 — Validation set and the numbers
 ### P7 — Validation set and the numbers
 
 
+*First numbers* (2026-08-28, later session) — the deterministic half is in the playbook §6 rows (Next.js, MERN, Nest
+channels, FastAPI prefixed routers, ASP.NET endpoint groups: node counts, edge precision spot-checks, pictures). One agent
+A/B so far, `bradtraversy/proshop_mern` (small, Express + React), `scripts/agent-eval/run-all.sh`, Sonnet/high, 2 runs per
+arm, a daemon pre-warmed before each with-run, the CLI shim on (0 leaks): *"How does submitting the login form reach the
+database, and what does the API respond with when the password is wrong?"* — with codegraph 14s / 14s, 2 tool calls, 0 Read,
+0 Grep, 1–2 explores, the full path named (`submitHandler → login → POST /api/users/login → authUser → User.findOne →
+matchPassword → 401`); without 18s / 37s, 14 / 8 tool calls, 7 / 1 Read plus Bash `cat`s and a subagent. Tokens 153k/165k vs
+237k/369k. The pass bar (§4 of the playbook) holds on the small repo. **Still open:** the medium / large rows (Ghost,
+immich, cal.com / twenty), ≥3 prompts per framework, and a control repo. The driver lives in the session scratchpad
+(`ab-proshop.sh`: pre-warm `serve --mcp` with `CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` high and `CODEGRAPH_WASM_RELAUNCHED=1`,
+then `run-all.sh` per run with its own `AGENT_EVAL_OUT`); re-create it from the memory note.
+
 Small fixtures live in the tests. For the real bar, index these and record the results in
 Small fixtures live in the tests. For the real bar, index these and record the results in
 `docs/design/dynamic-dispatch-coverage-playbook.md` (new rows) exactly as the playbook asks
 `docs/design/dynamic-dispatch-coverage-playbook.md` (new rows) exactly as the playbook asks
 (**≥3 flow prompts × small/medium/large, node count stable, synthesized-edge precision spot-check,
 (**≥3 flow prompts × small/medium/large, node count stable, synthesized-edge precision spot-check,