Browse Source

test: respect startup and teardown budgets in Windows fixtures

creatixchu 4 weeks ago
parent
commit
a43fa81f82

+ 2 - 2
.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md
-2026-09-08-ci-readiness-and-completion.md: 2249df5467189975aca2d73ec56c6cf82ec7b62f
-2026-09-08-ci-readiness-and-completion.zh.md: ccf8e7c16903b66e39bc48e99460e5c5179ab51b
+2026-09-08-ci-readiness-and-completion.md: c72048ef90b987e7d27992b1754736b2ce895093
+2026-09-08-ci-readiness-and-completion.zh.md: 1188d66249b3092416435da2e8496a657ad00677

+ 4 - 0
.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md

@@ -14,6 +14,8 @@ The [ACP coverage run](https://github.com/deepseek-harness/deepseek-harness/acti
 
 A [worker-runtime coverage failure](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34248221544/job/102135631932) exhausts the slow-binding fixture's one-second compute allowance. Concurrent native Windows reproductions exceed that allowance before calling the binding. Worker initialization contributes measured active time; the delayed binding contributes idle time.
 
+The [Windows coverage run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34324325375/job/102377982193) reports an SDK subprocess exit beyond a fixture's 200 ms confirmation window and an Inspector Worker startup beyond its ten-second default. Neither the protocol-error routing case nor the Cordis tree projection case measures those latency guarantees.
+
 ## Decision
 
 The [webhook browser test](../../../../apps/web/tests/github-ready-review.e2e.ts) observes the model request caused by delivery before checking Session registration. The [feedback test](../../../../apps/web/tests/feedback-command.e2e.ts) waits for the empty composer and enabled attachment control before comparing ARIA output. Matching consecutive snapshots cannot prove that the command RPC has settled: its event stream can publish the acknowledgement first.
@@ -28,6 +30,8 @@ The [subagent teardown decision](2026-09-07-subagent-teardown-test-budgets.md) o
 
 The [worker-runtime binding test](../../../../packages/code-runtime/code-runtime-worker-thread/tests/runtime.spec.ts) allows five seconds of compute for source-worker initialization and delays the binding for 6.5 seconds. Charging that idle delay would still exceed the entire compute allowance. The case retains its 15-second test limit and 30-second wall ceiling, registers Context and reply-timer cleanup, and leaves the hot-loop, decoy-dispatch, wall-ceiling, and abort controls at their existing limits. Production budgets remain unchanged.
 
+The [SDK subagent protocol-error test](../../../../packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts) uses the provider's normal shutdown and exit grace periods and registers disposal before its assertions. The [Inspector tree tests](../../../../packages/experimental/inspector/tests/cordis-tree.host.spec.ts) pass the active test budget to Worker startup and register cleanup while startup is still pending. A cancelled test cannot receive a late-ready handle; cleanup awaits initialization and closes a successfully started Worker. Failed initialization already terminates the Worker before rejecting. A controlled late-start test verifies cancellation and closure through the real Worker's HTTP endpoint. Production defaults remain unchanged.
+
 ## Alternatives considered
 
 **Larger independent waits.** Rejected where a completion promise already exists. A separate polling deadline continues to compete with the execution lane's budget.

+ 4 - 0
.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.zh.md

@@ -14,6 +14,8 @@ Status: implemented
 
 一次 [worker runtime coverage 失败](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34248221544/job/102135631932)耗尽了慢 binding 夹具的一秒计算额度。原生 Windows 并发复现在调用 binding 前已超过该额度。Worker 初始化会累计所测的活跃时间;延迟的 binding 累计空闲时间。
 
+[Windows 覆盖率运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34324325375/job/102377982193)报告了 SDK 子进程退出超过测试设置的 200 毫秒确认期限,以及 Inspector Worker 启动超过十秒默认期限。协议错误转发用例和 Cordis 树投影用例都不衡量这些延迟保证。
+
 ## 决策
 
 [Webhook 浏览器测试](../../../../apps/web/tests/github-ready-review.e2e.ts)观察投递触发的模型请求后再检查 Session 注册。[反馈测试](../../../../apps/web/tests/feedback-command.e2e.ts)在比较 ARIA 输出前等待输入框清空且附件按钮启用。连续两次快照相同不能证明命令 RPC 已完成:事件流可能先发布确认消息。
@@ -28,6 +30,8 @@ Status: implemented
 
 [Worker runtime binding 测试](../../../../packages/code-runtime/code-runtime-worker-thread/tests/runtime.spec.ts)为源码 worker 初始化保留五秒计算额度,并将 binding 延迟设为 6.5 秒。若将该空闲延迟计费,仍会超过整个计算额度。用例保留 15 秒测试期限与 30 秒墙钟上限,登记 Context 和回复定时器的清理,并保持热循环、诱饵 dispatch、墙钟上限及取消控制用例的原有限制。生产预算不变。
 
+[SDK 子 Agent 协议错误测试](../../../../packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts)使用提供方正常的关闭和退出等待时间,并在断言前登记清理。[Inspector 树测试](../../../../packages/experimental/inspector/tests/cordis-tree.host.spec.ts)将当前测试预算传给 Worker 启动,并在启动尚未完成时登记清理。取消后的测试不会收到随后才就绪的实例;清理等待初始化完成,并关闭成功启动的 Worker。初始化失败时,启动操作会在拒绝前终止 Worker。受控的延迟启动测试通过真实 Worker 的 HTTP 端点验证取消和关闭。生产默认值不变。
+
 ## 考虑过的替代方案
 
 **增大独立等待时限。** 已有完成 Promise 时不采用。独立轮询期限仍会与执行通道的预算竞争。

+ 67 - 11
packages/experimental/inspector/tests/cordis-tree.host.spec.ts

@@ -2,10 +2,11 @@
 
 import { Context } from '@deepseek-ai/cordis'
 import WebSocket, { type RawData } from 'ws'
-import { afterEach, describe, expect, it, vi } from 'vitest'
+import { afterEach, describe, expect, it, vi, type TestContext } from 'vitest'
 import { CordisTreeCollector } from '../src/shared/cordis/collector.ts'
 import { observeCordisTree } from '../src/shared/cordis/observer.ts'
-import { startInspector, type InspectorHandle } from '../src/host/bridge/controller.ts'
+import * as inspectorBridge from '../src/host/bridge/controller.ts'
+import type { InspectorHandle, InspectorOptions } from '../src/host/bridge/controller.ts'
 import { publishCordisTree as publishHostCordisTree } from '../src/host/inspection/cordis.ts'
 import { parseCordisTreeSnapshot, type CordisTreeNode } from '../src/shared/cordis/snapshot.ts'
 import { inspectorId } from '../src/shared/bridge/ids.ts'
@@ -76,6 +77,19 @@ class CdpClient {
   }
 }
 
+/** Own pending startup through cancellation; tree assertions do not measure Worker cold-start latency. */
+async function startTestInspector(options: InspectorOptions, test: TestContext): Promise<InspectorHandle> {
+  const pending = inspectorBridge.startInspector({ ...options, startupTimeoutMs: test.task.timeout })
+  test.onTestFinished(async () => {
+    // Failed starts terminate their Worker before rejecting.
+    const [started] = await Promise.allSettled([pending])
+    if (started.status === 'fulfilled') await started.value.close()
+  })
+  const handle = await pending
+  test.signal.throwIfAborted()
+  return handle
+}
+
 describe('Cordis tree inspection', () => {
   let inspector: InspectorHandle | undefined
   let cdp: CdpClient | undefined
@@ -84,6 +98,48 @@ describe('Cordis tree inspection', () => {
   const observers: Array<() => void> = []
   const fibers: Array<{ dispose(): Promise<void> }> = []
 
+  it('closes a late-starting Worker after test cancellation without returning its handle', async (test) => {
+    const release = Promise.withResolvers<undefined>()
+    const controller = new AbortController()
+    const cleanups: Array<Parameters<TestContext['onTestFinished']>[0]> = []
+    let starting: Promise<InspectorHandle> | undefined
+    const start = inspectorBridge.startInspector
+    const spy = vi.spyOn(inspectorBridge, 'startInspector').mockImplementation(async (options) => {
+      starting = start(options)
+      const handle = await starting
+      await release.promise
+      return handle
+    })
+    test.onTestFinished(async () => {
+      release.resolve(undefined)
+      spy.mockRestore()
+      if (starting === undefined) return
+      const [started] = await Promise.allSettled([starting])
+      if (started.status === 'fulfilled') await started.value.close()
+    })
+    const pending = startTestInspector({ port: 0, captureFetch: false }, {
+      ...test,
+      signal: controller.signal,
+      onTestFinished: (cleanup) => { cleanups.push(cleanup) },
+    })
+    const result = pending.then(
+      handle => ({ status: 'fulfilled' as const, handle }),
+      (reason: unknown) => ({ status: 'rejected' as const, reason }),
+    )
+    if (starting === undefined) throw new Error('Inspector startup was not called')
+    const handle = await starting
+    expect(spy).toHaveBeenCalledWith(expect.objectContaining({ startupTimeoutMs: test.task.timeout }))
+    expect(cleanups).toHaveLength(1)
+    const cancelled = new Error('test cancelled during startup')
+    controller.abort(cancelled)
+    release.resolve(undefined)
+    const outcome = await result
+    expect(outcome.status).toBe('rejected')
+    expect('reason' in outcome ? outcome.reason : undefined).toBe(cancelled)
+    await cleanups[0]!(test)
+    await expect(fetch(handle.endpoint.httpUrl)).rejects.toThrow()
+  })
+
   afterEach(async () => {
     for (const dispose of observers.splice(0).reverse()) dispose()
     for (const fiber of fibers.splice(0).reverse()) await fiber.dispose()
@@ -362,8 +418,8 @@ describe('Cordis tree inspection', () => {
     backend.close()
   })
 
-  it('projects Host and Client trees and resolves both node kinds to RemoteObjects', async () => {
-    inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 })
+  it('projects Host and Client trees and resolves both node kinds to RemoteObjects', async (test) => {
+    inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test)
     const host = new Context()
     const hostFiber = host.plugin({ name: 'host-child', apply() {} })
     fibers.push(hostFiber)
@@ -540,8 +596,8 @@ describe('Cordis tree inspection', () => {
     expect(disconnectedTree.clients[0]?.connection.state).toBe('disconnected')
   })
 
-  it('emits only node-level DOM changes for Client snapshots', async () => {
-    inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 })
+  it('emits only node-level DOM changes for Client snapshots', async (test) => {
+    inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test)
     cdp = await CdpClient.connect(inspector.endpoint.webSocketDebuggerUrl)
     const initialDocument = (await cdp.call('DOM.getDocument')).result?.root as CdpNode
     const clientsNode = initialDocument.children?.find(node => node.localName === 'clients')
@@ -598,8 +654,8 @@ describe('Cordis tree inspection', () => {
     })
   })
 
-  it('serves three document levels by default and withheld levels on demand', async () => {
-    inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 })
+  it('serves three document levels by default and withheld levels on demand', async (test) => {
+    inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test)
     const host = new Context()
     let innerFiber: { uid: number | null } | undefined
     const outer = host.plugin({
@@ -680,14 +736,14 @@ describe('Cordis tree inspection', () => {
     expect(secondCdp.events.slice(offset).some(event => event.method === 'DOM.setChildNodes')).toBe(false)
   })
 
-  it('restores a disconnected Client tree from a new transport generation', async () => {
-    inspector = await startInspector({
+  it('restores a disconnected Client tree from a new transport generation', async (test) => {
+    inspector = await startTestInspector({
       port: 0,
       captureFetch: false,
       maxCordisNodes: 100,
       clientReconnectBaseMs: 10,
       clientReconnectMaxMs: 20,
-    })
+    }, test)
     clientSource = await InspectorClientFixture.start(inspector.endpoint.client, { label: 'Reconnect Client' })
     cdp = await CdpClient.connect(inspector.endpoint.webSocketDebuggerUrl)
     await cdp.call('Runtime.enable')

+ 5 - 7
packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts

@@ -6,7 +6,7 @@
  * quiescent disposal are all exercised end to end. No model, no key.
  */
 
-import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from 'vitest'
 import { Context } from '@deepseek-ai/cordis'
 import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
 import { tmpdir } from 'node:os'
@@ -751,19 +751,17 @@ describe('dsh-subagent-dsh-sdk provider', () => {
       cwd: process.cwd(),
       provider: 'p',
       model: 'm',
-      // The fake dies as soon as the prompt arrives: FAKE_HANG_PROMPT plus a
-      // short-lived process is simulated by killing via dispose below instead;
-      // here use FAKE_MALFORMED to make the prompt reply violate the protocol.
       env: { FAKE_MALFORMED_PROMPT: '1' },
-      shutdownTimeoutMs: 100,
-      disposeEofGraceMs: 200,
-      disposeGraceMs: 200,
+      shutdownTimeoutMs: DEFAULT_SHUTDOWN_TIMEOUT_MS,
+      disposeEofGraceMs: DEFAULT_DISPOSE_EOF_GRACE_MS,
+      disposeGraceMs: DEFAULT_DISPOSE_GRACE_MS,
       onError: (error) => {
         seen.push(error.message)
         throw new Error('sink failure must be contained')
       },
     }
     const run = await startSdkRun(request(), spec)
+    onTestFinished(() => run.dispose())
     const result = await run.result
     expect(result.stopReason).toBe('error')
     expect(result.diagnostic).toBe(