Selaa lähdekoodia

ci: size PR previews on measured standard hosted runners (#3628)

* ci: measure hosted preview runner sizes [preview-sizing-benchmark]

* ci: type benchmark job keys explicitly [preview-sizing-benchmark]

* ci: size PR previews on measured standard hosted runners
Tianyi Cui 2 viikkoa sitten
vanhempi
sitoutus
ceb3136bef

+ 6 - 0
.agents/notes/implemented/process/2026-09-06-preview-hosted-runner-sizing.i18n.yaml

@@ -0,0 +1,6 @@
+# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
+# side as of the last confirmed-consistent state. Both languages carry equal authority;
+# after editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-09-06-preview-hosted-runner-sizing.md
+2026-09-06-preview-hosted-runner-sizing.md: 87298e94f11aa7e483afde31e0963a56f523febc
+2026-09-06-preview-hosted-runner-sizing.zh.md: 285b21d60d755e76db582e4e55c5cde913f7fc2e

+ 46 - 0
.agents/notes/implemented/process/2026-09-06-preview-hosted-runner-sizing.md

@@ -0,0 +1,46 @@
+# Agent Note: Measured GitHub-hosted PR preview sizing
+
+Status: implemented
+
+English | [中文](2026-09-06-preview-hosted-runner-sizing.zh.md)
+
+## Problem
+
+PR previews build the full workspace and browser-worker VFS image. A lower per-minute runner price does not guarantee lower job cost because GitHub rounds each job upward to whole minutes. Moving previews to persistent self-hosted machines also changes isolation and is outside this decision.
+
+## Decision
+
+The [preview workflow](../../../../.github/workflows/build-preview-cloudflare.yml) uses standard GitHub-hosted `ubuntu-24.04`. Build, cache, deployment, protected-image verification, and comment semantics remain unchanged. The [sizing reference](../../../../.github/preview-sizing/README.md) owns comparison requirements. The separate CI [failover runbook](2026-07-26-ci-failover-runbook.md) retains its independent runner-switch decision; previews do not use those switches.
+
+### Measurements
+
+[Experiment 34012729982](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34012729982) succeeds for all eight size/cache combinations plus one cache seed. Every measured job checks out SHA `9149d7e7ef945b5601711badd3cf63d58ab384f5`, uses Node 24.19.0 and pnpm 11.7.0, and executes immutable install, full workspace build, preview/VFS packing, and local upload shaping with gzip integrity verification. Warm jobs restore one exact run-private pnpm cache; cold jobs skip restoration but contain pnpm bootstrap files. No compiled outputs are restored.
+
+| Runner | Cold / warm job seconds | Rounded minutes each | USD each | Workspace seconds cold / warm | Preview seconds cold / warm |
+|---|---:|---:|---:|---:|---:|
+| standard, 2 vCPU | 202 / 203 | 4 | 0.024 | 138.92 / 147.21 | 12.65 / 12.94 |
+| larger, 4 vCPU | 177 / 162 | 3 | 0.036 | 124.21 / 114.86 | 10.77 / 9.88 |
+| larger, 8 vCPU | 154 / 154 | 3 | 0.066 | 110.51 / 110.77 | 9.20 / 9.21 |
+| larger, 16 vCPU | 124 / 125 | 3 | 0.126 | 90.57 / 84.99 | 7.62 / 7.33 |
+
+Using [published rates](https://docs.github.com/en/billing/reference/actions-runner-pricing), measured jobs total $0.504; the 60-second standard seed adds $0.006. The $0.510 gross compute estimate includes setup, restoration, measurement upload, and cleanup, but excludes storage and account discounts. Standard costs 80.95% less than 16-core and 33.33% less than 4-core in each sampled cache state. It adds 78 seconds against the corresponding 16-core job.
+
+Standard jobs expose two vCPUs and 7.75 GiB RAM. Workspace maximum process RSS is 2.86 / 2.76 GiB; preview maximum process RSS is 0.76 / 0.74 GiB. Both complete without an OOM or timeout. GNU time RSS is not simultaneous process-tree memory. These samples establish successful execution, not a permanent memory guarantee.
+
+The comparison fixes source, lockfile, commands, and runtime versions, not physical CPUs or image release: standard and 4-core use image 20260831.293.1; 8-core and 16-core use 20260823.283.1. CPUs vary among AMD EPYC 9V74/7763 and Intel Xeon 8370C/8573C. One sample per cache state measures the offered labels, not isolated CPU scaling or statistical repeatability.
+
+The experiment does not deploy or access Cloudflare credentials. Measurement upload takes zero to one second; warm-cache restore takes six to ten seconds. For context, [production job 101428009994](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34011495156/job/101428009994) spends 14 seconds uploading, one second verifying, and two seconds commenting on a different SHA. Adding that overhead to this experiment is a projection, not a measured standard-runner publication result. The actual PR preview workflow owns deployment confirmation.
+
+## Alternatives considered
+
+**Keep 16-core.** It provides the shortest measured job, but costs $0.102 more per sample for a 78-second improvement. Preview builds do not justify that premium for this cost-focused decision.
+
+**Select 4-core or 8-core.** Both succeed and shorten builds, but their rounded sample costs exceed standard Ubuntu. Four-core retains more RAM and disk headroom if future workloads exhaust standard capacity; such a change requires new measurements.
+
+**Move to self-hosted.** Rejected by scope: previews remain on GitHub CI. The existing Linux and Windows registrations can share persistent hosts; their dependency, store-volume, and cleanup assumptions do not apply to fresh hosted VMs. No failover or trust condition changes.
+
+## Consequences
+
+Previews trade approximately 78 seconds of sampled build-job latency for lower compute cost. Production Cloudflare latency, image rollout variance, future build growth, and broader success rates remain observable limitations. No hourly or monthly savings are extrapolated from this single experiment. The temporary benchmark workflow and its safety test are absent from the final tree; the experiment commits and linked run preserve the method and evidence.
+
+The executed [focused regression](../../../../scripts/preview-workflow.spec.ts) pins hosted routing, PR triggers and permissions, immutable full builds, restore-only caching, publication shaping, protected-image checks, and idempotent comments. A physical self-hosted routing mutation fails its routing assertion; restoration passes all three tests. No model-visible runtime behavior changes, so no Session snapshot changes are required.

+ 46 - 0
.agents/notes/implemented/process/2026-09-06-preview-hosted-runner-sizing.zh.md

@@ -0,0 +1,46 @@
+# Agent Note: 基于测量的 GitHub 托管 PR 预览规格
+
+Status: implemented
+
+[English](2026-09-06-preview-hosted-runner-sizing.md) | 中文
+
+## 问题
+
+PR(Pull Request)预览构建完整工作区及浏览器 worker VFS 镜像。较低的每分钟运行器价格不能保证较低的作业成本,因为 GitHub 将每个作业向上取整至整分钟。将预览移至持久化自托管机器还会改变隔离方式,不属于本决策范围。
+
+## 决策
+
+[预览工作流](../../../../.github/workflows/build-preview-cloudflare.yml) 使用标准 GitHub 托管 `ubuntu-24.04`。构建、缓存、部署、受保护镜像验证及评论语义保持不变。[规格参考](../../../../.github/preview-sizing/README.zh.md) 负责比较要求。独立的 CI [故障切换手册](2026-07-26-ci-failover-runbook.zh.md) 保留其运行器切换决策;预览不使用这些开关。
+
+### 测量
+
+[实验 34012729982](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34012729982) 的八种规格/缓存组合及一个缓存预热作业均成功。每个测量作业检出 SHA `9149d7e7ef945b5601711badd3cf63d58ab384f5`,使用 Node 24.19.0 与 pnpm 11.7.0,并执行不可变安装、完整工作区构建、预览/VFS 打包,以及含 gzip 完整性验证的本地上传内容整理。热作业恢复同一个运行私有精确 pnpm 缓存;冷作业跳过恢复,但包含 pnpm 引导安装文件。不恢复编译产物。
+
+| 运行器 | 冷 / 热作业秒数 | 各自取整分钟数 | 各自美元费用 | 冷 / 热工作区秒数 | 冷 / 热预览秒数 |
+|---|---:|---:|---:|---:|---:|
+| 标准,2 vCPU | 202 / 203 | 4 | 0.024 | 138.92 / 147.21 | 12.65 / 12.94 |
+| 大型,4 vCPU | 177 / 162 | 3 | 0.036 | 124.21 / 114.86 | 10.77 / 9.88 |
+| 大型,8 vCPU | 154 / 154 | 3 | 0.066 | 110.51 / 110.77 | 9.20 / 9.21 |
+| 大型,16 vCPU | 124 / 125 | 3 | 0.126 | 90.57 / 84.99 | 7.62 / 7.33 |
+
+按[公开费率](https://docs.github.com/en/billing/reference/actions-runner-pricing),测量作业合计 $0.504;60 秒标准预热作业增加 $0.006。$0.510 总计算费用估算包含设置、恢复、测量上传及清理,但不含存储和账户折扣。在每种采样缓存状态下,标准运行器比 16 核低 80.95%,比 4 核低 33.33%。相比对应的 16 核作业增加 78 秒。
+
+标准作业提供两个 vCPU 与 7.75 GiB 内存。工作区最大进程 RSS 为 2.86 / 2.76 GiB;预览最大进程 RSS 为 0.76 / 0.74 GiB。两者均未发生 OOM 或超时并完成。GNU time RSS 不是进程树同时占用的内存总量。这些样本证明成功执行,而非永久内存保证。
+
+比较固定源代码、锁文件、命令和运行时版本,但不固定物理 CPU 或镜像版本:标准与 4 核使用镜像 20260831.293.1;8 核与 16 核使用 20260823.283.1。CPU 包括 AMD EPYC 9V74/7763 与 Intel Xeon 8370C/8573C。每种缓存状态的单个样本测量所提供的标签,而非独立 CPU 扩展性或统计可重复性。
+
+实验不部署,也不访问 Cloudflare 凭据。测量上传耗时零至一秒;热缓存恢复耗时六至十秒。作为背景,[生产作业 101428009994](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34011495156/job/101428009994) 在不同 SHA 上上传耗时 14 秒、验证一秒、评论两秒。将该开销加至本实验属于推算,而非已测量的标准运行器发布结果。实际 PR 预览工作流负责部署确认。
+
+## 考虑过的替代方案
+
+**保留 16 核。** 它提供最短的测量作业,但为 78 秒改善使每个样本增加 $0.102。对于本次以成本为重点的决策,预览构建不值得这项溢价。
+
+**选择 4 核或 8 核。** 两者均成功并缩短构建,但取整后的样本费用高于标准 Ubuntu。若未来工作负载耗尽标准容量,4 核可保留更多内存与磁盘余量;这样的变更需要新测量。
+
+**移至自托管。** 因范围限制而拒绝:预览保留在 GitHub CI。现有 Linux 与 Windows 注册实例可能共享持久化主机;其依赖、store 卷及清理假设不适用于全新的托管 VM。不改变故障切换或信任条件。
+
+## 影响
+
+预览以约 78 秒采样构建作业延迟换取更低的计算费用。生产 Cloudflare 延迟、镜像发布差异、未来构建增长及更广泛的成功率仍是可观测限制。不从本次单一实验外推每小时或每月节省。最终文件树不包含临时基准工作流及其安全测试;实验提交与链接的运行保留方法和证据。
+
+已执行的[针对性回归](../../../../scripts/preview-workflow.spec.ts) 固定托管路由、PR 触发器与权限、不可变完整构建、只恢复缓存、发布内容整理、受保护镜像检查及幂等评论。实际修改为自托管路由会使路由断言失败;恢复后全部三个测试通过。不改变模型可见运行时行为,因此不需要修改 Session 快照。

+ 6 - 0
.github/preview-sizing/README.i18n.yaml

@@ -0,0 +1,6 @@
+# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
+# side as of the last confirmed-consistent state. Both languages carry equal authority;
+# after editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write .github/preview-sizing/README.md
+README.md: 142854e582523eb1ed48924a73fbe29ea6164bf6
+README.zh.md: a71325189dc7966df87e14bbb591af3c76a3d498

+ 35 - 0
.github/preview-sizing/README.md

@@ -0,0 +1,35 @@
+# PR preview runner sizing
+
+English | [中文](README.zh.md)
+
+## Summary
+
+The [preview workflow](../workflows/build-preview-cloudflare.yml) builds pull-request previews on standard GitHub-hosted `ubuntu-24.04`. Runner sizing compares complete job cost, not price per minute or core count alone.
+
+## Table of Contents
+
+- [Comparison requirements](#comparison-requirements)
+- [Publication semantics](#publication-semantics)
+- [Dev Note](#dev-note)
+
+<a id="comparison-requirements"></a>
+
+## Comparison requirements
+
+A sizing experiment holds checkout SHA, lockfile, Node and pnpm versions, workspace build, and preview/VFS packing commands constant. Each runner starts without build outputs. Cold installs do not restore dependency caches; pnpm bootstrap files may already exist. Warm installs restore the same exact cache without prefix fallback. Record the actual runner image, CPU, RAM, disk, cache outcome, phase duration, exit status, and peak memory. GNU time maximum RSS reports a process maximum, not simultaneous aggregate memory across the build process tree.
+
+Calculate estimated gross compute as the sum of each completed job’s elapsed minutes rounded upward, multiplied by that runner’s rate. Include setup, cache restoration, cleanup, failures, and measurement-upload overhead. Report seed jobs separately. Queue delay is a latency observation, not executed job time. These estimates are not invoice totals; standard-runner included minutes and storage are separate.
+
+A build-only benchmark does not deploy, access Cloudflare credentials, or post pull-request comments. Its cost does not establish complete preview publication cost. Confirm the selected runner through the actual preview workflow before treating deployment latency and protected-image delivery as verified.
+
+<a id="publication-semantics"></a>
+
+## Publication semantics
+
+Runner selection does not alter pull-request events, per-PR cancellation, immutable installation, restore-only dependency caching, full workspace build, preview packing, sourcemap removal, or the preview page copied to the deployment root. Cloudflare uploads only the built site to the PR branch alias. The protected-image check requires HTTP 200, no transport content encoding, and gzip magic bytes; the URL comment remains idempotent. Dependabot and other PR authors remain on GitHub-hosted machines.
+
+<a id="dev-note"></a>
+
+## Dev Note
+
+The [runner decision](../../.agents/notes/implemented/process/2026-09-06-preview-hosted-runner-sizing.md) records measurements, cost estimates, and image/CPU variation. The build-only experiment does not verify production deployment.

+ 35 - 0
.github/preview-sizing/README.zh.md

@@ -0,0 +1,35 @@
+# PR 预览运行器规格
+
+[English](README.md) | 中文
+
+## 摘要
+
+[预览工作流](../workflows/build-preview-cloudflare.yml) 在标准 GitHub 托管 `ubuntu-24.04` 上构建 PR(Pull Request)预览。运行器规格选择比较完整作业成本,而非仅比较每分钟价格或核心数。
+
+## 目录
+
+- [比较要求](#comparison-requirements)
+- [发布语义](#publication-semantics)
+- [开发备注](#dev-note)
+
+<a id="comparison-requirements"></a>
+
+## 比较要求
+
+规格实验保持检出 SHA、锁文件、Node 与 pnpm 版本、工作区构建以及预览/VFS 打包命令一致。每个运行器启动时均无构建产物。冷安装不恢复依赖缓存,但 pnpm 引导安装文件可能已存在;热安装恢复同一个精确缓存,不使用前缀回退。记录实际运行器镜像、CPU、内存、磁盘、缓存结果、各阶段耗时、退出状态与内存峰值。GNU time 最大 RSS 表示进程最大值,而非构建进程树同时占用的内存总量。
+
+估算总计算费用时,将每个已完成作业的运行分钟数向上取整,乘以对应运行器费率后求和。纳入设置、缓存恢复、清理、失败及测量数据上传的开销。单独报告缓存预热作业。排队延迟属于延迟观测,不属于作业执行时间。这些估算不是账单总额;标准运行器的套餐内分钟数及存储另行计算。
+
+仅构建的基准测试不部署、不访问 Cloudflare 凭据,也不发布 PR 评论。其成本不能证明完整预览发布成本。在将部署延迟与受保护镜像交付视为已验证之前,须通过实际预览工作流确认所选运行器。
+
+<a id="publication-semantics"></a>
+
+## 发布语义
+
+运行器选择不改变 PR 事件、按 PR 取消、不可变安装、只恢复的依赖缓存、完整工作区构建、预览打包、sourcemap 删除,以及复制到部署根目录的预览页面。Cloudflare 仅将构建站点上传至 PR 分支别名。受保护镜像检查要求 HTTP 200、无传输内容编码及 gzip 魔数字节;URL 评论保持幂等。Dependabot 与其他 PR 作者仍使用 GitHub 托管机器。
+
+<a id="dev-note"></a>
+
+## 开发备注
+
+[运行器决策](../../.agents/notes/implemented/process/2026-09-06-preview-hosted-runner-sizing.zh.md) 记录测量、成本估算及镜像/CPU 差异。仅构建实验不验证生产部署。

+ 1 - 1
.github/workflows/build-preview-cloudflare.yml

@@ -32,7 +32,7 @@ env:
 
 jobs:
   preview:
-    runs-on: dsh-ubuntu-24-04-16core
+    runs-on: ubuntu-24.04
     name: cloudflare pages preview
     steps:
       - uses: actions/checkout@v6

+ 63 - 0
scripts/preview-workflow.spec.ts

@@ -0,0 +1,63 @@
+import { readFileSync } from 'node:fs'
+import { resolve } from 'node:path'
+import * as yaml from 'js-yaml'
+import { describe, expect, it } from 'vitest'
+
+const workflow = yaml.load(readFileSync(resolve(import.meta.dirname, '../.github/workflows/build-preview-cloudflare.yml'), 'utf8')) as {
+  on: unknown
+  permissions: unknown
+  concurrency: unknown
+  env: Record<string, string>
+  jobs: Record<'preview', {
+    'runs-on': string
+    steps: Array<{ name?: string; uses?: string; run?: string; with?: Record<string, unknown>; env?: Record<string, string> }>
+  }>
+}
+const preview = workflow.jobs.preview
+
+describe('PR preview workflow', () => {
+  it('keeps every PR author on the selected GitHub-hosted runner', () => {
+    expect(Object.keys(workflow.jobs)).toEqual(['preview'])
+    expect(preview['runs-on']).toBe('ubuntu-24.04')
+    expect(workflow.on).toEqual({ pull_request: { types: ['opened', 'synchronize', 'reopened'] } })
+    expect(workflow.permissions).toEqual({ contents: 'read', 'pull-requests': 'write' })
+    expect(preview.steps.find(step => step.uses === 'actions/checkout@v6')?.with).toEqual({ 'persist-credentials': false })
+  })
+
+  it('keeps the immutable full build and restore-only dependency cache', () => {
+    expect(workflow.env.PRIMARY_NODE_VERSION).toBe('24')
+    expect(workflow.env.DSH_TELEMETRY_DISABLED).toBe('1')
+    const commands = preview.steps.map(step => step.run)
+    expect(commands).toContain('pnpm install --frozen-lockfile')
+    expect(commands).toContain('pnpm run build')
+    expect(commands).toContain('pnpm --filter @deepseek-ai/dsh-web-frontend run build:preview')
+    expect(commands.indexOf('pnpm run build')).toBeLessThan(commands.indexOf('pnpm --filter @deepseek-ai/dsh-web-frontend run build:preview'))
+    expect(preview.steps.filter(step => step.uses?.startsWith('actions/cache'))).toHaveLength(1)
+    expect(preview.steps.find(step => step.uses === 'actions/cache/restore@v4')?.with).toMatchObject({
+      key: "${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}",
+    })
+  })
+
+  it('retains per-PR deployment, protected image verification, and idempotent URL comments', () => {
+    expect(workflow.concurrency).toEqual({
+      group: 'build-preview-cloudflare-${{ github.event.pull_request.number }}',
+      'cancel-in-progress': true,
+    })
+    expect(workflow.env.CF_PROJECT).toBe('dsh-build-preview')
+    const shape = preview.steps.find(step => step.name === 'Shape the upload')!
+    expect(shape.run).toContain("find apps/web/dist -name '*.map' -delete")
+    expect(shape.run).toContain('cp apps/web/dist/preview.html apps/web/dist/index.html')
+    const deploy = preview.steps.find(step => step.name === 'Upload to Cloudflare Pages')!
+    expect(deploy.run).toContain('npx --yes wrangler@4 pages deploy apps/web/dist')
+    expect(deploy.run).toContain('--branch "pr-${{ github.event.pull_request.number }}"')
+    const verify = preview.steps.find(step => step.name === 'Verify the protected deployment serves the image')!
+    expect(verify.run).toContain('/preview/vfs-image.tar.gz')
+    expect(verify.run).toContain('"$code" != "200"')
+    expect(verify.run).toContain('content-encoding:')
+    expect(verify.run).toContain('"$magic" != "1f8b"')
+    expect(verify.env?.CF_ACCESS_CLIENT_SECRET).toBe('${{ secrets.CF_ACCESS_CLIENT_SECRET }}')
+    const comment = preview.steps.find(step => step.name === 'Comment the preview URL')!
+    expect(comment.run).toContain('<!-- dsh-preview-url -->')
+    expect(comment.run).toContain('gh pr comment "$PR" --body-file -')
+  })
+})