Browse Source

perf(session): bundle migration verifier dependencies

fz 1 month ago
parent
commit
0f6e7442d7

+ 2 - 2
.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md
-2026-08-31-released-session-format-migrations.md: eb5eb14f28a6459bd388caa2ea106ec01d1ebbe6
-2026-08-31-released-session-format-migrations.zh.md: b06bfac059541e9c397ff46fe7d169690c5b1213
+2026-08-31-released-session-format-migrations.md: 286737730198ad895de2f03a48f9c74848839582
+2026-08-31-released-session-format-migrations.zh.md: 228b09e03916c1fa447398edb45508a7a6ab61e5

+ 2 - 0
.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md

@@ -94,6 +94,8 @@ The JSONL provider scans frame boundaries once, reuses one Zstandard decoder, pa
 
 Current encoding is record based. The provider serializes about 1 MiB of plaintext per main-thread slice, streams it through one Zstandard context with source-error propagation, writes compressed output in 4 MiB batches to an exclusively created same-directory temporary file, and syncs it before publication. A process-wide scheduler admits at most two full verification Workers and hands a released permit directly to the oldest waiter.
 
+The shipped `lib/worker.cjs` bundles its JavaScript workspace dependencies so each fresh verifier avoids resolving and compiling their runtime module graph. This is safe because the Worker communicates through plain request/result messages and shares no service or class identity with its host. The Host build applies the existing TypeScript and Typert transforms; the Client pass skips this Node-only package instead of replacing its worker with untransformed source. Native add-ons remain external. Verification, scheduler admission, termination, and durable publication still complete before writable open returns. The built-worker smoke copies the package manifest and worker into an isolated temporary package with ambient module paths removed, accepts a valid generation, and rejects an incorrect event count.
+
 Preparation forwards cancellation through source reads and observes it at the existing approximately 500 ms Decode yield boundary. Once `publish()` starts, encode, Worker verification, and publication do not receive caller cancellation and run to settlement; write open checks its caller signal again afterward. A published generation is never rolled back.
 
 The Stage pipeline ends at one prepared current artifact. [Historical Session read preparation](2026-09-05-read-only-session-migration-preparation.md) defines how read open consumes that artifact immediately while write open performs encode, verification, and publication before returning append access.

+ 2 - 0
.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.zh.md

@@ -94,6 +94,8 @@ JSONL provider 只扫描一次 frame boundary,复用一个 Zstandard decoder
 
 Current encode 以单条 record 为单位。Provider 在主线程每个 slice 序列化约 1 MiB plaintext,通过一个会传播 source error 的 Zstandard context 流式压缩,以 4 MiB batch 写入同目录排他创建的临时文件,并在 publication 前 sync。进程级 scheduler 最多允许两个完整 verification Worker 并行,并把释放的 permit 直接交给最早的 waiter。
 
+发布的 `lib/worker.cjs` 将 JavaScript workspace 依赖一起打包,使每个新 verifier 无需解析并编译它们的运行时模块图。Worker 只通过普通 request/result 消息通信,与 host 不共享 service 或 class identity,因此可以这样处理。Host build 应用现有 TypeScript 与 Typert 转换;Client pass 跳过这个 Node-only package,不会用未经转换的源代码覆盖 worker。Native add-on 保持 external。Verification、scheduler admission、termination 与 durable publication 仍在 writable open 返回前完成。Built-worker 冒烟测试把 package manifest 与 worker 复制到隔离的临时 package,移除环境中的模块搜索路径,接受有效 generation,并拒绝错误的 event count。
+
 Preparation 会把 cancellation 传给 source read,并在现有的约 500 ms Decode yield 边界观察它。`publish()` 一旦开始,encode、Worker verification 与 publication 不接收 caller cancellation,并运行到终态;write open 会在之后再次检查 caller signal。已经发布的 generation 绝不会回滚。
 
 Stage pipeline 终止于一份 prepared current artifact。[历史 Session 只读迁移准备](2026-09-05-read-only-session-migration-preparation.zh.md)定义 read open 如何立即消费该 artifact,以及 write open 如何在返回 append 权限前完成 encode、verification 与 publication。

+ 2 - 2
.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.md
-2026-09-04-session-open-performance-gate.md: d5447f93666a5cb27af2073a4a66f99d0c39f6ee
-2026-09-04-session-open-performance-gate.zh.md: 657f77fb76cf84afc36301d22d9a7798c559075b
+2026-09-04-session-open-performance-gate.md: d69e7424c5b34bc9f50d4a1d355279b65b7dfd8a
+2026-09-04-session-open-performance-gate.zh.md: cce4ae3921d70d43dde823d64dca5a0976344100

+ 2 - 0
.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.md

@@ -56,6 +56,8 @@ The pre-stack implementation keeps V0 as its current format, so first open does
 
 The [standard two-CPU run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384/job/101461539961) at `ca3ffe95dac2c55eefeb16ed9b61067bbd19ee90` uses Node 24.20.0 x64 and Ubuntu image `20260831.293.1`. Its five current-generation `open` samples are 49.2, 47.4, 49.1, 48.6, and 48.1 ms: median 48.6 ms, maximum 49.2 ms. The rounded 50 ms CI expectation gives a 63 ms limit without reapplying the 2× machine scale. The log identifies two available CPUs but not their model; it does not isolate hardware from the Node-version change. This is endpoint-specific runner calibration, not evidence of an application optimization or a new reference-machine measurement. Every other benchmark passes its existing budget. Deterministic controls reject the observed median at the historical 30 ms limit, accept it at 63 ms, reject a synthetic 75 ms reopen median, and reject a synthetic 4,000 ms first-open duration at its unchanged 550 ms limit. These controls verify budget enforcement, not a measured new regression.
 
+A cold-verifier packaging change removes runtime workspace-module loading without changing these budgets or the measured endpoint. On macOS arm64, Node 24.18.0, the same 127,400-event fixture at `ac48359b195558806ee5a2286697074fd1a52815` takes 164.2, 162.4, 159.7, 149.3, and 167.7 ms for first writable resume (median 162.4 ms). Bundling the verifier through the workspace build gives 119.8, 120.9, 121.9, 122.3, and 121.8 ms (median 121.8 ms, 25% lower). Retained heap stays at 5.4 MB; median peak RSS changes from 144.9 to 143.7 MB. Reopen medians are 27.5 and 27.1 ms, and all 16 Session cases, including the 128 MB completion checks, pass. A CPU profile attributes part of the old verifier cost to module resolution and compilation. The isolated-package built-worker test fails on the original worker because its workspace imports cannot resolve, and passes with the bundled worker, including rejection of an incorrect event count. These local results do not establish Linux runner timing; the existing 450 ms CI gate remains the acceptance check.
+
 The calibrated source budgets are:
 
 | Measurement | Reference expectation | CI budget |

+ 2 - 0
.agents/notes/implemented/testing/2026-09-04-session-open-performance-gate.zh.md

@@ -56,6 +56,8 @@ Session benchmark 使用固定参数合成 released-v0 输入:200 轮,每轮
 
 `ca3ffe95dac2c55eefeb16ed9b61067bbd19ee90` 上的[标准双 CPU 运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384/job/101461539961)使用 Node 24.20.0 x64 和 Ubuntu 镜像 `20260831.293.1`。当前 generation `open` 的五次样本为 49.2、47.4、49.1、48.6 和 48.1 ms:中位数 48.6 ms,最大值 49.2 ms。取整后的 50 ms CI 预期值给出 63 ms 上限,不重复乘以 2 倍机器系数。日志标明两个可用 CPU,但未记录型号;它无法区分硬件变化与 Node 版本变化的影响。这是端点专属的运行器校准,不是应用优化或参考机器新测量的证据。其他每项 benchmark 均通过既有预算。确定性正反例在历史 30 ms 上限下拒绝实测中位数,在 63 ms 下接受它,拒绝合成的 75 ms reopen 中位数,并以未改变的 550 ms 上限拒绝合成的 4,000 ms 首次打开耗时。这些正反例验证预算执行,不代表测得新的退化。
 
+一次冷 verifier 打包调整移除了运行时 workspace 模块加载,未改变这些预算或测量终点。在 macOS arm64、Node 24.18.0 上,`ac48359b195558806ee5a2286697074fd1a52815` 对同一份 127,400-event fixture 的首次 writable resume 耗时为 164.2、162.4、159.7、149.3、167.7 ms(中位数 162.4 ms)。通过 workspace build 打包 verifier 后为 119.8、120.9、121.9、122.3、121.8 ms(中位数 121.8 ms,降低 25%)。Retained heap 保持 5.4 MB;peak RSS 中位数从 144.9 变为 143.7 MB。Reopen 中位数为 27.5 和 27.1 ms,包含 128 MB completion check 的全部 16 项 Session 用例通过。CPU profile 将旧 verifier 的部分成本归因于模块解析和编译。隔离 package 的 built-worker 测试在旧 worker 上因无法解析 workspace import 而失败,在打包后的 worker 上通过,同时验证错误的 event count 会被拒绝。这些本地结果不能证明 Linux runner 耗时;现有 450 ms CI gate 仍是验收检查。
+
 校准后的源码预算如下:
 
 | 测量项 | 参考机预期 | CI 预算 |

+ 35 - 6
packages/session/session-persistence-jsonl/tests/built-migration-worker.e2e.ts

@@ -9,11 +9,12 @@ const built = ['lib/index.js', 'lib/worker.cjs']
   .every(path => existsSync(join(packageRoot, path)))
 
 describe.skipIf(!built)('built migration verifier (plain node)', () => {
-  it('publishes a historical generation through the bundled worker', async () => {
+  it('publishes history and verifies it without runtime workspace imports', async () => {
     const script = `
-      import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
+      import { copyFile, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
       import { tmpdir } from 'node:os'
       import { join } from 'node:path'
+      import { Worker } from 'node:worker_threads'
       import { Context } from '@deepseek-ai/cordis'
       import JsonlSessionPersistence from '@deepseek-ai/dsh-session-persistence-jsonl'
 
@@ -30,8 +31,30 @@ describe.skipIf(!built)('built migration verifier (plain node)', () => {
         const handle = await ctx.sessionPersistence.open(id, 'write')
         await handle.close()
         await ctx.sessionPersistence.flush()
-        const header = JSON.parse((await readFile(join(directory, 'session.v3.jsonl'), 'utf8')).trim())
-        console.log(JSON.stringify({ id: header.id, version: header.version }))
+        const currentPath = join(directory, 'session.v3.jsonl')
+        const header = JSON.parse((await readFile(currentPath, 'utf8')).trim())
+        await mkdir(join(root, 'lib'))
+        await copyFile('package.json', join(root, 'package.json'))
+        const workerPath = join(root, 'lib', 'worker.cjs')
+        await copyFile('lib/worker.cjs', workerPath)
+        async function verify(expectedEventCount) {
+          const worker = new Worker(workerPath, { workerData: {
+            path: currentPath, compression: 'none', expectedId: id, expectedEventCount,
+          } })
+          try {
+            return await new Promise((resolve, reject) => {
+              worker.once('message', resolve)
+              worker.once('error', reject)
+              worker.once('exit', code => reject(new Error('verifier exited before a result: ' + code)))
+            })
+          } finally {
+            await worker.terminate()
+          }
+        }
+        const verified = await verify(0)
+        const refused = await verify(1)
+        console.log(JSON.stringify({ id: header.id, version: header.version,
+          verified: verified.ok, refused: refused.ok, refusal: refused.message }))
       } finally {
         await ctx.fiber.dispose()
         await rm(root, { recursive: true, force: true })
@@ -40,10 +63,16 @@ describe.skipIf(!built)('built migration verifier (plain node)', () => {
     const { exitCode, stdout, stderr } = await execa(
       process.execPath,
       ['--input-type=module', '-e', script],
-      { cwd: packageRoot, stdin: 'ignore', timeout: 30_000, killSignal: 'SIGKILL', reject: false },
+      {
+        cwd: packageRoot, env: { NODE_PATH: undefined, NODE_OPTIONS: undefined },
+        stdin: 'ignore', timeout: 30_000, killSignal: 'SIGKILL', reject: false,
+      },
     )
 
     expect(exitCode, `stderr:\n${stderr}`).toBe(0)
-    expect(JSON.parse(stdout.trim())).toEqual({ id: 'built-migration-worker', version: 3 })
+    expect(JSON.parse(stdout.trim())).toEqual({
+      id: 'built-migration-worker', version: 3, verified: true, refused: false,
+      refusal: 'current session generation contains 0 events, expected 1',
+    })
   })
 })

+ 4 - 1
packages/session/session-persistence-jsonl/tsdown.config.ts

@@ -1,7 +1,7 @@
 import { defineConfig } from 'tsdown'
 
 /** Build the backend and its path-loaded verifier as separate bundles. */
-export default defineConfig([
+export default defineConfig(({ env }) => env?.DSH_BUILD_FACE === 'client' ? [] : [
   {
     entry: ['lib/types/index.js'],
     outDir: 'lib',
@@ -14,6 +14,9 @@ export default defineConfig([
   },
   {
     entry: ['lib/types/worker.js'],
+    // Fresh verifiers need no host singleton identity. Inline their JavaScript
+    // closure to avoid resolving and compiling workspace packages on every open.
+    deps: { alwaysBundle: [/^@deepseek-ai\/(?!node-addon-system)/] },
     outDir: 'lib',
     format: ['cjs'],
     platform: 'node',