Jelajahi Sumber

test(ralph): snapshot fresh-agent rounds in headless app

Tianyi Cui 2 bulan lalu
induk
melakukan
9f6f87cf69

+ 2 - 2
.agents/notes/implemented/feature/2026-07-19-fresh-agent-ralph-workflow-tool.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write
-2026-07-19-fresh-agent-ralph-workflow-tool.md: 159ad7e39602d8b84ffafdb396ad1083f9c73009
-2026-07-19-fresh-agent-ralph-workflow-tool.zh.md: bb6025952ada35526459101d53b3ea1fea622163
+2026-07-19-fresh-agent-ralph-workflow-tool.md: 46816b8cc06f0acf5615ffb44035c79ea13b6794
+2026-07-19-fresh-agent-ralph-workflow-tool.zh.md: 188a0bf50a7a6a683cd6aa1004c4d78673d3631d

+ 1 - 1
.agents/notes/implemented/feature/2026-07-19-fresh-agent-ralph-workflow-tool.md

@@ -46,7 +46,7 @@ ACP and terminal presentation use a generic `ralph` card whose raw input is the
 
 Unit tests cover config and call-cap resolution, provider capability rejection, fixed start-request routing and child ceiling, all successful terminal outcomes, ordinary child-failure envelopes, malformed and oversized boundary values, exact successful-result truncation, abort timing, disposal, render intent, prompt lifecycle, and namespace-plugin shape at per-file 100% coverage. Worker-engine tests prove synchronous provider-route validation, per-run child ceilings below the deployment ceiling, and that a provider override selects every child without changing the configured default, including the built `lib/worker.cjs` under plain Node.
 
-A keyless real-stack integration drives the fixed script through the actual worker-thread engine, spawn provider, structured-output runtime, and agent loop. It proves distinct child identities, absent `seedLength`, inherited cwd, no parent-history markers in either child request, exact previous-report handoff only in the following round, one phase event, terminal completion, and disposal of both children. The same real stack covers blocker and round-limit outcomes, unnormalized and semantically invalid reports, oversized handoffs, ordinary child failure with the last good handoff, and cancellation to child quiescence. Tool tests pin generic call/result presentation, while ACP replay header snapshots pin the shipped schema and prompt-guidance transcript surface.
+A keyless real-stack integration drives the fixed script through the actual worker-thread engine, spawn provider, structured-output runtime, and agent loop. It proves distinct child identities, absent `seedLength`, inherited cwd, no parent-history markers in either child request, exact previous-report handoff only in the following round, one phase event, terminal completion, and disposal of both children. The same real stack covers blocker and round-limit outcomes, unnormalized and semantically invalid reports, oversized handoffs, ordinary child failure with the last good handoff, and cancellation to child quiescence. A shipped keyless headless snapshot additionally boots the real `examples/headless-agent` composition, invokes `ralph`, pins the parent stream transcript, and inspects persisted logs for two distinct unseeded child sessions and the round-one handoff appearing only in round two. Tool tests pin generic call/result presentation, while ACP replay header snapshots pin the shipped schema and prompt-guidance transcript surface.
 
 ## Alternatives considered
 

+ 1 - 1
.agents/notes/implemented/feature/2026-07-19-fresh-agent-ralph-workflow-tool.zh.md

@@ -46,7 +46,7 @@ ACP 和终端展示使用通用 `ralph` 卡片,并把目标作为原始输入
 
 单元测试覆盖配置与调用上限解析、provider 能力拒绝、固定启动请求路由与子 agent 上限、全部成功终止结果、普通子 agent 失败外层值、畸形及过大边界值、成功结果精确截断、中止时序、处置、渲染意图、提示生命周期和命名空间插件形状,并达到逐文件 100% 覆盖率。工作流引擎测试证明 provider 路由会同步验证、每次运行的子 agent 上限可低于部署上限,并且 provider 覆盖会选择每个子 agent 且不改变配置默认值,其中包括普通 Node 下构建后的 `lib/worker.cjs`。
 
-一项无密钥真实栈集成测试通过实际工作线程引擎、spawn provider、结构化输出运行时和 agent loop 驱动固定脚本。它证明子 agent 标识不同、没有 `seedLength`、继承 cwd、两个子请求都不含父历史标记、上一份报告只精确出现在下一轮交接中、只产生一个阶段事件、终止完成以及两个子 agent 都被处置。同一真实栈还覆盖阻塞与轮次上限结果、未规范化及语义无效报告、过大交接、保留上一份有效交接的普通子 agent 失败,以及取消后达到子 agent 静止状态。工具测试固定通用调用/结果展示,而 ACP 重放请求头快照固定发布的 schema 与提示指导转录表面。
+一项无密钥真实栈集成测试通过实际工作线程引擎、spawn provider、结构化输出运行时和 agent loop 驱动固定脚本。它证明子 agent 标识不同、没有 `seedLength`、继承 cwd、两个子请求都不含父历史标记、上一份报告只精确出现在下一轮交接中、只产生一个阶段事件、终止完成以及两个子 agent 都被处置。同一真实栈还覆盖阻塞与轮次上限结果、未规范化及语义无效报告、过大交接、保留上一份有效交接的普通子 agent 失败,以及取消后达到子 agent 静止状态。一项已发布的无密钥无头快照还会启动真实的 `examples/headless-agent` 组合、调用 `ralph`、固定父级流式转录,并检查持久化日志中存在两个不同且无种子的子会话,且第一轮交接只出现在第二轮。工具测试固定通用调用/结果展示,而 ACP 重放请求头快照固定发布的 schema 与提示指导转录表面。
 
 ## 考虑过的替代方案
 

+ 12 - 0
examples/headless-agent/ralph.cordis.snapshot.yml

@@ -0,0 +1,12 @@
+# Replay counterpart to cordis.yml for the shipped Ralph-loop snapshot.
+- id: base
+  name: '@cordisjs/plugin-include'
+  config:
+    path: ./cordis.yml
+    patches:
+      - id: llm-deepseek
+        name: '@deepseek-ai/dsh-llm-deepseek'
+        disabled: true
+      - insert:
+          - id: llm-replay
+            name: '@deepseek-ai/dsh-llm-replay'

+ 75 - 0
examples/headless-agent/tests/headless.snapshot.ts

@@ -17,6 +17,8 @@ const advancedStreamExpected = join(advancedScenarioDir, 'stream-json.expected.j
 const advancedConfigPath = fileURLToPath(new URL('../advanced.cordis.snapshot.yml', import.meta.url))
 const goalScenarioDir = join(snapshotsDir, 'goal-tools')
 const goalConfigPath = fileURLToPath(new URL('../goal.cordis.snapshot.yml', import.meta.url))
+const ralphScenarioDir = join(snapshotsDir, 'ralph-loop')
+const ralphConfigPath = fileURLToPath(new URL('../ralph.cordis.snapshot.yml', import.meta.url))
 const binScript = fileURLToPath(new URL('../../../packages/examples/cli-demo/src/bin.ts', import.meta.url))
 const tsconfigPath = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
 const refreshing = process.env.DSH_SNAPSHOT === 'refresh'
@@ -216,4 +218,77 @@ describe('headless stream-json snapshots', () => {
     if (refreshing) await writeFile(streamExpected, normalized)
     expect(normalized).toBe(await readFile(streamExpected, 'utf8'))
   }, LOADER_SMOKE_TEST_TIMEOUT_MS)
+
+  it('replays two fresh Ralph rounds through the one-shot app', async () => {
+    const prompt = await scenarioPrompt(ralphScenarioDir, 'ralph-loop')
+    const streamExpected = join(ralphScenarioDir, 'stream-json.expected.jsonl')
+    let runCwd = ''
+    const result = await runLoaderSmoke({
+      label: 'Ralph loop headless stream-json snapshot',
+      tempDirPrefix: 'headless-snapshot-ralph-loop-',
+      binScript,
+      configPath: ralphConfigPath,
+      binArgs: ['--config', ralphConfigPath, '--output-format', 'stream-json', prompt],
+      tsconfigPath,
+      env: {
+        DSH_SNAPSHOT: 'replay',
+        DSH_SNAPSHOT_FILE: join(ralphScenarioDir, 'session.jsonl'),
+        DSH_SNAPSHOT_OVERRIDE: join(ralphScenarioDir, 'replay.override.json'),
+        DSH_SNAPSHOT_CHILD_FILES: [
+          join(ralphScenarioDir, 'session.1.jsonl'),
+          join(ralphScenarioDir, 'session.2.jsonl'),
+        ].join(delimiter),
+        NODE_OPTIONS: [process.env.NODE_OPTIONS, '--disable-warning=ExperimentalWarning'].filter(Boolean).join(' '),
+      },
+      prepare: (cwd) => { runCwd = cwd },
+      inspect: async (cwd) => {
+        const logs = await persistedLogs(cwd)
+        expect(logs).toHaveLength(3)
+        const parent = logs.find(log => typeof log.header.parentSession !== 'string')
+        if (parent === undefined) throw new Error('Ralph snapshot did not persist its parent session')
+        const parentId = parent.header.id
+        expect(typeof parentId).toBe('string')
+        const children = logs.filter(log => typeof log.header.parentSession === 'string')
+          .sort((left, right) => Number(left.header.createdAt) - Number(right.header.createdAt))
+        expect(children).toHaveLength(2)
+        expect(children.map(child => child.header.parentSession)).toEqual([parentId, parentId])
+        expect(children.map(child => child.header.cwd)).toEqual([parent.header.cwd, parent.header.cwd])
+        expect(children.map(child => child.header.seedLength)).toEqual([undefined, undefined])
+        expect(new Set(children.map(child => child.header.id)).size).toBe(2)
+
+        const parentRecords = parseJsonl(parent.content)
+        const parentCalls = parentRecords.filter(record => record.type === 'tool/call')
+        expect(parentCalls.map(record => (record.data as JsonObject | undefined)?.name)).toEqual(['ralph'])
+        const parentResult = parentRecords.find(record => record.type === 'tool/result')
+        const parentResultData = parentResult?.data as JsonObject | undefined
+        expect(parentResultData?.isError).toBe(false)
+        expect(JSON.stringify(parentResultData?.content)).toContain('reported completion after 2 rounds')
+
+        const childRecords = children.map(child => parseJsonl(child.content))
+        const childPrompts = childRecords.map((records) => {
+          const message = records.find(record => record.type === 'user/message')
+          return JSON.stringify((message?.data as JsonObject | undefined)?.content)
+        })
+        expect(childPrompts[0]).toContain('Ralph round: 1 of 2.')
+        expect(childPrompts[0]).toContain('(none — this is the first round)')
+        expect(childPrompts[0]).not.toContain('ROUND_ONE_HANDOFF')
+        expect(childPrompts[1]).toContain('Ralph round: 2 of 2.')
+        expect(childPrompts[1]).toContain('ROUND_ONE_HANDOFF')
+        for (const childPrompt of childPrompts) {
+          expect(childPrompt).toContain('Prove two fresh Ralph rounds through the shipped headless app.')
+          expect(childPrompt).not.toContain('Run a two-round fresh-agent Ralph loop')
+        }
+        for (const records of childRecords) {
+          const calls = records.filter(record => record.type === 'tool/call')
+          expect(calls.map(record => (record.data as JsonObject | undefined)?.name))
+            .toEqual(['structured_output'])
+        }
+      },
+    })
+
+    expect(result.stderr).toBe('')
+    const normalized = normalizeHeadlessStream(result.stdout, runCwd)
+    if (refreshing) await writeFile(streamExpected, normalized)
+    expect(normalized).toBe(await readFile(streamExpected, 'utf8'))
+  }, LOADER_SMOKE_TEST_TIMEOUT_MS)
 })

+ 8 - 0
examples/headless-agent/tests/snapshots/ralph-loop/input.json

@@ -0,0 +1,8 @@
+{
+  "steps": [
+    {
+      "op": "prompt",
+      "text": "Run a two-round fresh-agent Ralph loop to prove the shipped headless integration."
+    }
+  ]
+}

+ 22 - 0
examples/headless-agent/tests/snapshots/ralph-loop/replay.override.json

@@ -0,0 +1,22 @@
+[
+  {
+    "kind": "chunks",
+    "chunks": [
+      { "type": "block-start", "index": 0, "blockType": "tool-call" },
+      { "type": "tool-call-delta", "index": 0, "id": "call_ralph", "name": "ralph", "argumentsDelta": "{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}" },
+      { "type": "block-end", "index": 0, "block": { "type": "tool-call", "id": "call_ralph", "name": "ralph", "arguments": "{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}" } },
+      { "type": "usage", "usage": { "inputTokens": 20, "outputTokens": 8 } },
+      { "type": "finish", "reason": { "kind": "tool-calls" } }
+    ]
+  },
+  {
+    "kind": "chunks",
+    "chunks": [
+      { "type": "block-start", "index": 0, "blockType": "text" },
+      { "type": "text-delta", "index": 0, "text": "RALPH SNAPSHOT COMPLETE" },
+      { "type": "block-end", "index": 0, "block": { "type": "text", "text": "RALPH SNAPSHOT COMPLETE" } },
+      { "type": "usage", "usage": { "inputTokens": 30, "outputTokens": 4 } },
+      { "type": "finish", "reason": { "kind": "stop" } }
+    ]
+  }
+]

+ 6 - 0
examples/headless-agent/tests/snapshots/ralph-loop/session.1.jsonl

@@ -0,0 +1,6 @@
+{"type":"session","version":0,"id":"42222222-2222-4222-8222-222222222222","createdAt":1783951001000,"cwd":"/tmp/ralph-headless","parentSession":"41111111-1111-4111-8111-111111111111"}
+{"type":"assistant/chunk","seq":0,"time":1783951001001,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}}}
+{"type":"assistant/chunk","seq":1,"time":1783951001002,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":0,"id":"round-one-report","name":"structured_output","argumentsDelta":"{\"status\":\"continue\",\"summary\":\"ROUND_ONE_HANDOFF\",\"evidence\":[\"Round one inspected the workspace.\"],\"nextSteps\":[\"Finish the snapshot objective.\"],\"blocker\":\"\"}"}}}
+{"type":"assistant/chunk","seq":2,"time":1783951001003,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"round-one-report","name":"structured_output","arguments":"{\"status\":\"continue\",\"summary\":\"ROUND_ONE_HANDOFF\",\"evidence\":[\"Round one inspected the workspace.\"],\"nextSteps\":[\"Finish the snapshot objective.\"],\"blocker\":\"\"}"}}}}
+{"type":"assistant/chunk","seq":3,"time":1783951001004,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":30,"outputTokens":12}}}}
+{"type":"assistant/chunk","seq":4,"time":1783951001005,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}

+ 6 - 0
examples/headless-agent/tests/snapshots/ralph-loop/session.2.jsonl

@@ -0,0 +1,6 @@
+{"type":"session","version":0,"id":"43333333-3333-4333-8333-333333333333","createdAt":1783951002000,"cwd":"/tmp/ralph-headless","parentSession":"41111111-1111-4111-8111-111111111111"}
+{"type":"assistant/chunk","seq":0,"time":1783951002001,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}}}
+{"type":"assistant/chunk","seq":1,"time":1783951002002,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":0,"id":"round-two-report","name":"structured_output","argumentsDelta":"{\"status\":\"complete\",\"summary\":\"The Ralph snapshot objective is complete.\",\"evidence\":[\"Two fresh rounds completed through the shipped app.\"],\"nextSteps\":[],\"blocker\":\"\"}"}}}
+{"type":"assistant/chunk","seq":2,"time":1783951002003,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"round-two-report","name":"structured_output","arguments":"{\"status\":\"complete\",\"summary\":\"The Ralph snapshot objective is complete.\",\"evidence\":[\"Two fresh rounds completed through the shipped app.\"],\"nextSteps\":[],\"blocker\":\"\"}"}}}}
+{"type":"assistant/chunk","seq":3,"time":1783951002004,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":40,"outputTokens":12}}}}
+{"type":"assistant/chunk","seq":4,"time":1783951002005,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}

+ 1 - 0
examples/headless-agent/tests/snapshots/ralph-loop/session.jsonl

@@ -0,0 +1 @@
+{"type":"session","version":0,"id":"41111111-1111-4111-8111-111111111111","createdAt":1783951000000,"cwd":"/tmp/ralph-headless"}

+ 23 - 0
examples/headless-agent/tests/snapshots/ralph-loop/stream-json.expected.jsonl

@@ -0,0 +1,23 @@
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"turn/start","seq":0,"time":0,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user"}}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"user/message","seq":1,"time":0,"data":{"content":[{"type":"text","text":"Run a two-round fresh-agent Ralph loop to prove the shipped headless integration."}],"source":{"kind":"user"}},"surfaceOp":"append"}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/start","seq":2,"time":0,"data":{"turn":1,"step":1}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"request/header","seq":3,"time":0,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":4,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":5,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":0,"id":"call_ralph","name":"ralph","argumentsDelta":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":6,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"call_ralph","name":"ralph","arguments":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":7,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":20,"outputTokens":8}}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":8,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/message","seq":9,"time":0,"data":{"turn":1,"step":1,"content":[{"type":"tool-call","id":"call_ralph","name":"ralph","arguments":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":20,"outputTokens":8}},"sourceEventSeqs":[4,5,6,7,8],"surfaceOp":"append"}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"tool/call","seq":10,"time":0,"data":{"turn":1,"step":1,"callId":"call_ralph","name":"ralph","arguments":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"tool/result","seq":11,"time":0,"data":{"turn":1,"step":1,"callId":"call_ralph","content":[{"type":"text","text":"Ralph worker reported completion after 2 rounds.\nFinal report:\n{\n  \"status\": \"complete\",\n  \"summary\": \"The Ralph snapshot objective is complete.\",\n  \"evidence\": [\n    \"Two fresh rounds completed through the shipped app.\"\n  ],\n  \"nextSteps\": [],\n  \"blocker\": \"\"\n}"}],"isError":false},"sourceEventSeqs":[10],"surfaceOp":"append"}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/end","seq":12,"time":0,"data":{"turn":1,"step":1}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/start","seq":13,"time":0,"data":{"turn":1,"step":2}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":14,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":0,"blockType":"text"}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":15,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":0,"text":"RALPH SNAPSHOT COMPLETE"}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":16,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"RALPH SNAPSHOT COMPLETE"}}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":17,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"usage","usage":{"inputTokens":30,"outputTokens":4}}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":18,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"finish","reason":{"kind":"stop"}}}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/message","seq":19,"time":0,"data":{"turn":1,"step":2,"content":[{"type":"text","text":"RALPH SNAPSHOT COMPLETE"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":30,"outputTokens":4}},"sourceEventSeqs":[14,15,16,17,18],"surfaceOp":"append"}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/end","seq":20,"time":0,"data":{"turn":1,"step":2}}}
+{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"turn/end","seq":21,"time":0,"data":{"turn":1,"reason":{"kind":"completed"}}}}
+{"type":"result","success":true,"sessionId":"{{sessionId}}","turn":1,"result":"RALPH SNAPSHOT COMPLETE","reason":{"kind":"completed"},"usage":{"inputTokens":50,"outputTokens":12}}