test(ui-deliverables): preserve file-reference prompt evaluations

.agents/notes/implemented/feature/2026-09-15-markdown-file-preview-links.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-09-15-markdown-file-preview-links.md
-2026-09-15-markdown-file-preview-links.md: 07177c15d134b805042d93f48f8caae6bc165620
-2026-09-15-markdown-file-preview-links.zh.md: bc6b343c0091339e9be25d062a006905c6e9477f
+2026-09-15-markdown-file-preview-links.md: 337385c2218d8b38b292094e90f12a46f0a7297c
+2026-09-15-markdown-file-preview-links.zh.md: a14974bbd078f895b2598fe18943c8a8d55cbb88

.agents/notes/implemented/feature/2026-09-15-markdown-file-preview-links.md

@@ -45,3 +45,5 @@ The development comparison ran four read-only tasks three times per variant. E c
 | Balanced 11-answer coverage | 95.2% | 93.0% | 79.0% |
 
 The colon revision is a separate development run. Its first three answers contained 28 mixed `:start-Lend` suffixes, retained as failures. Explicitly forbidding both `#` and `L` in the suffix yielded 51 structurally valid links in one repeated explanation, but no colon labels, three missing links, and 27 unnecessary directory qualifiers. A subsequent user-accepted demo produced 18 structurally valid links, including three colon labels and no mixed suffix. Neither run establishes reliable compliance or exhaustive factual accuracy. Authentication failures, the timeout, and all first answers remain in the local experiment archive; PR #4255 carries the extended 69-row metrics and task-rubric tables.
+
+Reusable [evaluation inputs and rubrics](../../../../packages/client/ui-deliverables/evals/file-references/README.md) live beside the prompt owner, with compact historical observations and failure records. Their manual runner uses the supported headless profile and keeps generated outputs outside version control. Package ownership keeps prompt changes and their regression criteria together; the repository performance benchmark tree has different input and timing requirements.

.agents/notes/implemented/feature/2026-09-15-markdown-file-preview-links.zh.md

@@ -45,3 +45,5 @@ Web 文件引用提示词要求链接命令、配置表达式和代码块以外
 | 平衡的 11 份回答覆盖率 | 95.2% | 93.0% | 79.0% |
 
 冒号修订是独立的开发验证。最初三份回答包含 28 个混合 `:start-Lend` 后缀，均保留为失败。明确禁止后缀包含 `#` 和 `L` 后，一次重复讲解生成 51 个结构有效的链接，但没有冒号标签，仍有三次漏链和 27 个不必要的目录限定。随后用户接受的 demo 生成 18 个结构有效的链接，其中三个使用冒号标签，没有混合后缀。这些运行都不能证明稳定遵循或完整事实正确性。认证失败、超时和所有首答仍保留在本地实验归档中；PR #4255 包含扩展的 69 行指标及任务 rubric 表。
+
+可复用的[评测输入和 rubric](../../../../packages/client/ui-deliverables/evals/file-references/README.zh.md) 与提示词所有者放在一起，并保留精简历史观察和失败记录。人工触发的运行器使用受支持的 headless profile，生成输出不进入版本控制。包内归属让提示词变化与回归标准一起维护；仓库性能 benchmark 目录有不同的输入和计时要求。

packages/client/ui-deliverables/README.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write packages/client/ui-deliverables/README.md
-README.md: 62ab7c1b3352e36a27fe374941be82e6927b2506
-README.zh.md: 82aa5e96b5233aeda9f80beb63cde6b5739cd227
+README.md: 2cc174a1be69cec37ef1eb07c017e70562e9a3a0
+README.zh.md: a25b046ef2fa76a8f787265a8992572830efa255

packages/client/ui-deliverables/README.md

@@ -63,6 +63,7 @@ Native opening uses an authenticated POST addressed by the viewed Session, event
 
 Read these pages when the deliverables surface is not enough. They move from the row to the turn-tail hole and the decisions behind the vocabulary.
 
+- [File-reference evaluation](evals/file-references/README.md) — frozen prompts, tasks, manual rubrics, and reproducible development-result summaries.
 - [ui-conversation](../ui-conversation/README.md) — declares the `conversation.chat.turnTail` hole and renders the closing prose.
 - [Workspace file links](../../../.agents/notes/implemented/feature/2026-07-31-web-workspace-file-links.md) — the decision behind the produced-files row; its Host open path is superseded by the [right Sidebar](../../../.agents/notes/implemented/feature/2026-09-04-right-sidebar-docking-infrastructure.md).
 - [Inline file mentions](../../../.agents/notes/archived/feature/2026-08-07-web-inline-file-mentions.md) — the decision behind clickable mentions in the closing prose.

packages/client/ui-deliverables/README.zh.md

@@ -63,6 +63,7 @@ Node 半部注册[模型体验](#model-experience)所述的静态 `ui:deliverabl
 
 当产出物面不够用时阅读以下页面。它们从该行进入 turn-tail 洞与词表背后的决策。
 
+- [文件引用评测](evals/file-references/README.zh.md) — 冻结提示词、任务、人工 rubric 和可复算的开发评测汇总。
 - [ui-conversation](../ui-conversation/README.zh.md)——声明 `conversation.chat.turnTail` 洞并渲染收尾正文。
 - [工作区文件链接](../../../.agents/notes/implemented/feature/2026-07-31-web-workspace-file-links.zh.md)——产出文件行背后的决策；其 Host 打开路径已被[右侧 Sidebar](../../../.agents/notes/implemented/feature/2026-09-04-right-sidebar-docking-infrastructure.zh.md)取代。
 - [行内文件提及](../../../.agents/notes/archived/feature/2026-08-07-web-inline-file-mentions.md)——收尾正文可点击提及背后的决策。

packages/client/ui-deliverables/evals/file-references/README.i18n.yaml

@@ -0,0 +1,6 @@
+# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
+# side as of the last confirmed-consistent state. Both languages carry equal authority;
+# after editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write packages/client/ui-deliverables/evals/file-references/README.md
+README.md: e6c5520cf92cacc80d917f600fdeb0352f6c5c7b
+README.zh.md: ea94964ec996ee6e206b6f141e6b71063e2e2a88

packages/client/ui-deliverables/evals/file-references/README.md

@@ -0,0 +1,58 @@
+---
+description: "File-reference prompt evaluation inputs, manual rubrics, and preserved development results for prompt maintainers."
+kind: "package-reference"
+---
+
+# File-reference prompt evaluation
+
+English | [中文](README.zh.md)
+
+## Summary
+
+This owner-local evaluation compares file-reference guidance using four unchanged development tasks. It preserves A/B/C, the final colon-label revision, eligibility rules, task rubrics, and historical failures. Real-model runs are manual; the existing package tests and Web Session snapshots own keyless product regressions. The runner and aggregator use Python's standard library; the treatment enters the logged system prompt through a regular plugin.
+
+## Table of Contents
+
+- [Run a cohort](#run-a-cohort)
+- [Review and summarize](#review-and-summarize)
+- [Preserved results](#preserved-results)
+- [Dev Note](#dev-note)
+
+<a id="run-a-cohort"></a>
+## Run a cohort
+
+Requires Python 3.9+, a POSIX host, Node and repository dependencies, and a built checkout (`pnpm run build`). Run from the repository root. Supply the normal DSH environment file with `--env-file`, or inherit its environment; the runner does not read credential values. `--model` is explicit, with the DeepSeek provider and high reasoning effort. Each attempt gets fresh DSH home, agent home, Session, and an archived HEAD workspace; the executable is this checkout's built `dsh --profile headless` launcher.
+
+For one real smoke, set `DSH_EVAL_ENV` to your normal environment-file path and run:
+
+```sh
+python3 packages/client/ui-deliverables/evals/file-references/run.py --variant a-colon --case plan --repetitions 1 --model deepseek-flash --env-file "$DSH_EVAL_ENV"
+```
+
+Omit `--variant`, `--case`, and `--repetitions` for the original three variants, four tasks, and three repetitions. `--prepare-only` records inputs without calling the API. The default 480-second attempt bound can be shortened with `--timeout`. Runs execute serially and stop at the first failed attempt; no retry or resume replaces an output. The printed directory lives under ignored `.artifacts/file-reference-evals/`, retaining input hashes, source archive, outputs, workspaces, Sessions, and failures. A supplied `--output` must be new and inside `.artifacts/`.
+
+The portable runner uses the normal read-only sandbox and never-approval policy. Evaluator files are removed from the task workspace, but host/runtime/network resources are shared: this is not hermetic isolation. Inspect recorded tool activity for out-of-workspace access. Its controls differ from historical R3's macOS-specific guards and parallel scheduling; compare only within the new cohort, not by pooling it with R3. The archive pins source HEAD while the launcher hash identifies the build; rebuild after runtime changes.
+
+<a id="review-and-summarize"></a>
+## Review and summarize
+
+Apply [rubric.md](rubric.md) to the full first answer and preserved workspace. Keep evaluators separate from execution; rubric contents are not passed to the model. Occurrence eligibility, unnecessary directories, ambiguity, and factual support require human review. Preserve the occurrence and task ledgers with each run, then write hash-bound `annotation.json` as documented in the rubric. Unreviewed answers remain unreviewed; an absent answer never counts as a perfect response.
+
+The summarizer accepts the printed run directory, or the committed historical observations:
+
+```sh
+python3 packages/client/ui-deliverables/evals/file-references/summarize.py packages/client/ui-deliverables/evals/file-references/results/2026-09-16/observations.json
+python3 -m unittest discover -s packages/client/ui-deliverables/evals/file-references -p 'test_*.py' -v
+```
+
+It validates nonnegative counts, E = L + M + I, duplicate attempts, and fresh-answer hashes. It reports planned/attempted/answered counts, missing audits, valid-link coverage, perfect audited answers, and timeouts. It does not automatically infer missing-file mentions or judge semantic correctness. These keyless Python checks validate the evaluation tools; they are manually invoked and do not add a real-API CI job.
+
+<a id="preserved-results"></a>
+## Preserved results
+
+[The R3 table](results/2026-09-16/metrics.md) retains all 69 original metrics and task rubrics. [Observations](results/2026-09-16/observations.json) reproduce the core counts and include the timeout. [Limitations and later runs](results/2026-09-16/notes.md) keep the colon revisions and guided recording separate. The raw first answers, full Sessions, authentication logs, and original audit ledgers remain in the private experiment archive, not in this package or its npm tarball. Historical aggregate counts support recomputation, not independent re-auditing without those raw records.
+
+<a id="dev-note"></a>
+## Dev Note
+
+The [file-preview decision](../../../../../.agents/notes/implemented/feature/2026-09-15-markdown-file-preview-links.md) owns prompt selection. This directory owns reusable evaluation inputs and methods; repository `benchmarks/` owns cross-package performance gates.

packages/client/ui-deliverables/evals/file-references/README.zh.md

@@ -0,0 +1,58 @@
+---
+description: "供提示词维护者使用的文件引用提示词评测输入、人工 rubric 和保留的开发评测结果。"
+kind: "package-reference"
+---
+
+# 文件引用提示词评测
+
+[English](README.md) | 中文
+
+## 摘要
+
+这份包内评测使用四个未修改的开发任务比较文件引用指导。它保留 A/B/C、最终冒号标签版、计数规则、任务 rubric 和历史失败。真实模型运行由人工触发；现有包测试和 Web Session 快照负责无密钥产品回归。运行器和汇总器使用 Python 标准库；评测提示词通过普通插件进入已记录的系统提示词。
+
+## 目录
+
+- [运行一组评测](#run-a-cohort)
+- [评审与汇总](#review-and-summarize)
+- [保留的结果](#preserved-results)
+- [开发者笔记](#dev-note)
+
+<a id="run-a-cohort"></a>
+## 运行一组评测
+
+需要 Python 3.9+、POSIX 主机、Node 和仓库依赖，以及已构建的检出目录（`pnpm run build`）。从仓库根目录执行。通过 `--env-file` 指定正常使用的 DSH 环境文件，或继承其环境；运行器不读取凭据值。`--model` 必须显式提供，使用 DeepSeek provider 和 high 推理强度。每次尝试使用新的 DSH home、agent home、Session 和 HEAD 归档工作区；可执行程序是本检出目录构建的 `dsh --profile headless` 启动器。
+
+执行一次真实冒烟验证时，将 `DSH_EVAL_ENV` 设为正常环境文件路径，然后运行：
+
+```sh
+python3 packages/client/ui-deliverables/evals/file-references/run.py --variant a-colon --case plan --repetitions 1 --model deepseek-flash --env-file "$DSH_EVAL_ENV"
+```
+
+省略 `--variant`、`--case` 和 `--repetitions` 即使用原三版、四个任务和三次重复。`--prepare-only` 只记录输入，不调用 API。默认每次尝试上限为 480 秒，可用 `--timeout` 缩短。运行串行执行，遇到首次失败即停止；重试或恢复不会替换输出。打印的目录位于被忽略的 `.artifacts/file-reference-evals/` 下，保留输入哈希、源码归档、输出、工作区、Session 和失败。指定的 `--output` 必须是 `.artifacts/` 内的新目录。
+
+可移植运行器使用正常的只读 sandbox 和 never approval 策略。评测文件会从任务工作区移除，但宿主、运行时和网络资源仍共享，因此不是完全隔离环境。应检查记录的工具活动中是否存在工作区外访问。它的控制措施与历史 R3 的 macOS 专用防护及并行调度不同；只在新组内比较，不与 R3 合并统计。归档固定 HEAD 源码，启动器哈希标识构建；运行时代码变化后应重新构建。
+
+<a id="review-and-summarize"></a>
+## 评审与汇总
+
+依据 [rubric.md](rubric.md) 检查完整首答和保留的工作区。评审与执行分开；rubric 内容不传给模型。引用是否应计数、目录是否多余、是否有歧义及事实依据都需要人工评审。保留每次运行的引用和任务台账，再按 rubric 写入绑定答案哈希的 `annotation.json`。未经评审的回答保持未评审；缺失回答不计为完美回答。
+
+汇总器接受打印的运行目录，或已提交的历史观察数据：
+
+```sh
+python3 packages/client/ui-deliverables/evals/file-references/summarize.py packages/client/ui-deliverables/evals/file-references/results/2026-09-16/observations.json
+python3 -m unittest discover -s packages/client/ui-deliverables/evals/file-references -p 'test_*.py' -v
+```
+
+它验证非负计数、E = L + M + I、重复尝试和新答案哈希，并报告计划数、尝试数、回答数、缺失审计、有效链接覆盖、已审计完美回答和超时。它不自动推断遗漏的文件提及，也不判断语义正确性。这些无密钥 Python 检查验证评测工具，由人工调用，不增加真实 API CI 作业。
+
+<a id="preserved-results"></a>
+## 保留的结果
+
+[R3 表格](results/2026-09-16/metrics.md) 保留全部 69 项原始指标和任务 rubric。[观察数据](results/2026-09-16/observations.json) 可复算核心计数，并包含超时。[限制及后续运行](results/2026-09-16/notes.md) 将冒号修订和指定格式的录屏单列。原始首答、完整 Session、认证日志和原始审计台账保留在私有实验归档中，不进入此包或其 npm tarball。历史聚合计数支持复算；没有原始记录时，不支持独立重新审计。
+
+<a id="dev-note"></a>
+## 开发者笔记
+
+[文件预览决策](../../../../../.agents/notes/implemented/feature/2026-09-15-markdown-file-preview-links.zh.md) 负责提示词选择。本目录负责可复用评测输入和方法；仓库 `benchmarks/` 负责跨包性能门禁。

packages/client/ui-deliverables/evals/file-references/cases/diagnose.txt

@@ -0,0 +1 @@
+使用 str_replace_editor 时，我先 view 了一个文件，外部程序随后改了它，再执行 str_replace 报 FS_STALE_VERSION。请基于 packages/fs 里的实现、测试和文档给出排查步骤，说明怎样正确重试、恢复旧会话时有什么不同，以及从哪些文件继续定位；可以给必要的调用示例。只读分析，不修改文件，不调用子代理。

packages/client/ui-deliverables/evals/file-references/cases/explain.txt

@@ -0,0 +1 @@
+讲讲 packages/bundle/headless 是怎么把命令行任务跑成一次 Agent 会话的：任务从哪里来，如何选模型和工作目录，什么时候打印结果，失败和恢复会话如何处理。只读分析，不修改文件，不调用子代理。

packages/client/ui-deliverables/evals/file-references/cases/plan.txt

@@ -0,0 +1 @@
+想让 packages/util/atomic-write 的 writeFileAtomic 支持 Buffer，同时保留现有字符串调用。请给一个最小修改方案，列出涉及的实现、测试、文档与调用方，说明兼容性和验证方式。只分析方案，不修改文件，不调用子代理。

packages/client/ui-deliverables/evals/file-references/cases/review.txt

@@ -0,0 +1 @@
+请 review packages/util/lazy-require 的实现、测试和 README，指出值得修复的问题；没有确认的问题也请直说，并说明你检查了哪些行为。只读分析，不修改文件，不调用子代理。

packages/client/ui-deliverables/evals/file-references/prompts/a-colon.txt

@@ -0,0 +1,2 @@
+Outside commands, configuration expressions, and code blocks, link every mention of an existing file, including repeats and tables, to its full path relative to the working directory or absolute; append #L24 or #L24-L30 to the target for known lines.
+Use the filename or a clear alias as the label, adding only enough parent directories to distinguish files; keep full paths out of labels. Default to the name alone; when precise locations matter, append :24 or :24–30, with no # or L in the line suffix.

packages/client/ui-deliverables/evals/file-references/prompts/baseline.txt

@@ -0,0 +1,2 @@
+Outside commands, configuration expressions, and code blocks, link every mention of an existing file, including repeats and tables, to its full path relative to the working directory or absolute; append #L24 or #L24-L30 to the target for known lines.
+Use the filename or a clear alias as the label, adding only enough parent directories to distinguish files; keep full paths out of labels.

packages/client/ui-deliverables/evals/file-references/prompts/display.txt

@@ -0,0 +1,2 @@
+Outside commands, configuration expressions, and code blocks, link every mention of an existing file, including repeats and tables, to its full path relative to the working directory or absolute; append #L24 or #L24-L30 to the target for known lines.
+Keep full paths in link targets, with only a filename or short, clear alias visible; use the surrounding text or a brief qualifier to distinguish same-named files.

packages/client/ui-deliverables/evals/file-references/prompts/filename.txt

@@ -0,0 +1,2 @@
+Outside commands, configuration expressions, and code blocks, link every mention of an existing file, including repeats and tables, to its full path relative to the working directory or absolute; append #L24 or #L24-L30 to the target for known lines.
+Use the filename alone as the label by default; if ambiguous in context, use a short, clear alias or brief qualifier. Keep directory paths in the target.

packages/client/ui-deliverables/evals/file-references/results/2026-09-16/metrics.md

@@ -0,0 +1,108 @@
+# R3 补充 metrics 与 rubric 表格
+
+A = baseline，B = filename，C = display。除标明“首次尝试”或“平衡子集”的行外，使用全部非空首答：A 11 份、B/C 各 12 份。A 的 review 第 2 轮 480.17 秒超时保留，空答不进入内容分母；没有重跑。这里补充汇总既有台账，未修改冻结 rubric、首答或失败记录。
+
+数字是描述量，不能相加为总分。任务要求被覆盖，不等于其中断言正确；事实问题和限制另列。耗时含工具工作、网络与主机调度，仅描述本轮，不能归因为 SP 性能差异。排除项与待定项统计台账条目，集体 README 条目可能代表多个对象，因此不能直接加回 E。
+
+## 执行与稳定性
+
+| 指标 / rubric | A | B | C |
+| --- | ---: | ---: | ---: |
+| 首次尝试 | 12 | 12 | 12 |
+| 完整首答 / 首次尝试 | 11/12 (91.7%) | 12/12 (100.0%) | 12/12 (100.0%) |
+| 无最终回答的超时 | 1 | 0 | 0 |
+| 首答有效引用全部链接（M=I=0） | 2/11 (18.2%) | 5/12 (41.7%) | 4/12 (33.3%) |
+| 首答完全没有文件链接 | 0/11 (0.0%) | 0/12 (0.0%) | 3/12 (25.0%) |
+| 完成首答耗时中位数（秒） | 77.19 | 94.28 | 86.09 |
+| 完成首答耗时范围（秒） | 58.97–244.59 | 64.84–306.79 | 66.34–368.05 |
+| 所有首次尝试总墙钟耗时之和（秒；并发任务相加） | 1639.39 | 1493.39 | 1615.84 |
+| 工作区未变更 / 首次尝试 | 12/12 (100.0%) | 12/12 (100.0%) | 12/12 (100.0%) |
+
+## 引用、标签与歧义
+
+| 指标 / rubric | A | B | C |
+| --- | ---: | ---: | ---: |
+| 有效链接覆盖 L/E | 453/476 (95.2%) | 539/585 (92.1%) | 467/592 (78.9%) |
+| 已生成链接中的结构有效率 L/(L+I) | 453/453 (100.0%) | 539/540 (99.8%) | 467/468 (99.8%) |
+| 漏链 / 无效链接 | 23 / 0 | 45 / 1 | 124 / 1 |
+| 越界行号引用 | 0 | 1 | 1 |
+| 不同文件数（跨首答按仓库相对路径去重） | 89 | 94 | 104 |
+| 每份首答不同文件数之和（跨答不去重） | 186 | 207 | 230 |
+| 待定引用台账条目（不进 E） | 12 | 16 | 17 |
+| 排除项台账条目（不是全体排除引用次数） | 67 | 64 | 104 |
+| 同名文件上下文歧义：引用次数 / 首答数 | 0 / 0 | 4 / 1 | 2 / 1 |
+| 不必要目录标签 / 文件名或别名标签 | 149/373 (39.9%) | 200/471 (42.5%) | 166/403 (41.2%) |
+| 完整路径标签 / 文件名或别名标签 | 44/373 (11.8%) | 89/471 (18.9%) | 118/403 (29.3%) |
+| 未链接引用中仍显示完整路径 | 1 | 2 | 59 |
+| 文件名或别名长度中位数 / P90（字符） | 19 / 39 | 16 / 43 | 20 / 42 |
+| 文件名或别名长度均值（字符） | 21.58 | 21.86 | 23.37 |
+| 实际数字标签 / 明确行号标签（不进名称长度） | 79 / 1 | 68 / 1 | 65 / 0 |
+
+## 平衡子集：每版相同的 11 个任务轮次
+
+| 指标 / rubric | A | B | C |
+| --- | ---: | ---: | ---: |
+| 有效链接覆盖 | 453/476 (95.2%) | 506/544 (93.0%) | 432/547 (79.0%) |
+| 全引用有效链接的首答 | 2/11 (18.2%) | 5/11 (45.5%) | 4/11 (36.4%) |
+| 不必要目录标签 / 文件名或别名标签 | 149/373 (39.9%) | 185/438 (42.2%) | 152/368 (41.3%) |
+| 同名歧义引用次数 | 0 | 4 | 2 |
+| 完成耗时中位数（秒） | 77.19 | 93.92 | 84.02 |
+
+## 原任务 rubric 分项
+
+| 指标 / rubric | A | B | C |
+| --- | ---: | ---: | ---: |
+| P1 Buffer 最小实现与 string 支持 | 3/3 已回应 | 3/3 已回应 | 3/3 已回应 |
+| P2 实现、测试、文档、调用方四个面 | 3/3 已回应 | 3/3 已回应 | 3/3 已回应 |
+| P3 兼容性 | 3/3 已回应 | 3/3 已回应 | 2/3 已回应；1 份有矛盾 |
+| P4 验证方案 | 3/3 已回应 | 3/3 已回应 | 3/3 已回应 |
+| P5 只读／无子代理 | 3/3 已回应 | 3/3 已回应 | 3/3 已回应 |
+| D1 陈旧版本时序与排查步骤 | 3/3 已回应 | 1/3 已回应；2 份有矛盾 | 2/3 已回应；1 份有矛盾 |
+| D2 正确重试流程 | 2/3 已回应；1 份有矛盾 | 3/3 已回应 | 3/3 已回应 |
+| D3 恢复旧会话的区别 | 3/3 已回应 | 2/3 已回应；1 份有矛盾 | 2/3 已回应；1 份有矛盾 |
+| D4 实现、测试、文档依据 | 3/3 已回应 | 3/3 已回应 | 3/3 已回应 |
+| D5 给出的调用示例符合实现 | 2/3 已回应；1 份有矛盾 | 2/3 已回应；1 份有矛盾 | 3/3 已回应 |
+| D6 只读／无子代理 | 3/3 已回应 | 3/3 已回应 | 3/3 已回应 |
+| R1 实现检查 | 2/2 已覆盖；另 1 次超时 | 3/3 已覆盖 | 3/3 已覆盖 |
+| R2 测试检查 | 2/2 已覆盖；另 1 次超时 | 3/3 已覆盖 | 3/3 已覆盖 |
+| R3 README 检查 | 2/2 已覆盖；另 1 次超时 | 3/3 已覆盖 | 3/3 已覆盖 |
+| R4 明确的 findings 结论 | 2/2 已覆盖；另 1 次超时 | 3/3 已覆盖 | 3/3 已覆盖 |
+| R5 说明检查过的行为 | 2/2 已覆盖；另 1 次超时 | 3/3 已覆盖 | 3/3 已覆盖 |
+| R6 核查 findings 的事实依据 | 2/2 有审计；均保留证据限制；另 1 次超时 | 3/3 有审计；均保留证据限制 | 3/3 有审计；均保留证据限制 |
+| R7 只读／无子代理 | 2/2 记录支持；另 1 次超时 | 3/3 记录支持 | 3/3 记录支持 |
+| X1 任务来源 | 3/3 已覆盖 | 3/3 已覆盖 | 3/3 已覆盖 |
+| X2 模型选择 | 3/3 已覆盖；1 份有矛盾 | 3/3 已覆盖；1 份有矛盾 | 3/3 已覆盖 |
+| X3 工作目录 | 3/3 已覆盖 | 3/3 已覆盖 | 3/3 已覆盖 |
+| X4 输出时机 | 3/3 已覆盖 | 3/3 已覆盖 | 3/3 已覆盖 |
+| X5 失败处理 | 3/3 已覆盖；1 份有矛盾 | 3/3 已覆盖 | 3/3 已覆盖 |
+| X6 恢复会话 | 3/3 已覆盖 | 3/3 已覆盖 | 3/3 已覆盖 |
+| X7 贯通会话流程 | 3/3 已覆盖 | 3/3 已覆盖 | 3/3 已覆盖 |
+| X8 事实及引用依据 | 3/3 记录问题或证据限制 | 3/3 记录问题或证据限制 | 3/3 记录问题或证据限制 |
+| X9 只读／无子代理 | 3/3 记录支持 | 3/3 记录支持 | 3/3 记录支持 |
+
+## 问题记录：各任务保留各自分类，不相加成总分
+
+| 指标 / rubric | A | B | C |
+| --- | ---: | ---: | ---: |
+| plan：源码支持的矛盾条目 | 1 | 2 | 2 |
+| diagnose：源码支持的矛盾条目 | 5 | 6 | 6 |
+| plan：仓库指令不符条目 | 2 | 1 | 2 |
+| plan：无依据断言 / 文档遗漏条目 | 1 / 1 | 1 / 0 | 0 / 1 |
+| explain：X8 问题或证据限制条目（含轻微措辞、引用问题） | 7 | 7 | 7 |
+| review：R6 仍未证明的事实／运行结论 | 2/2 保留限制 | 3/3 保留限制 | 3/3 保留限制 |
+
+## 实验与 GUI rubric 状态
+
+| 指标 / rubric | A | B | C |
+| --- | ---: | ---: | ---: |
+| SP 两条物理行，通用指导 | 2 | 2 | 2 |
+| 原任务逐字保留；三次首次尝试 | 已核对 | 已核对 | 已核对 |
+| GUI 原始首答导入（方案第 1、2 轮） | 2 份；字节一致 | 2 份；字节一致 | 2 份；字节一致 |
+| GUI 显示、换行、截断与表格布局 | 未验证：浏览器阻止访问 | 未验证：浏览器阻止访问 | 未验证：浏览器阻止访问 |
+| 实际点击并打开目标／定位行号 | 未验证 | 未验证 | 未验证 |
+| 主观美观／偏好 | 未评；无数值分数 | 未评；无数值分数 | 未评；无数值分数 |
+| 未见过的 holdout | 无；仅开发／回归任务 | 无；仅开发／回归任务 | 无；仅开发／回归任务 |
+
+同名歧义的人工记录集中于 B diagnose 第 2 轮的 4 次和 C diagnose 第 1 轮的 2 次；若把相邻角色的列举顺序视为足够对应，两者敏感性下界均为 0。0 表示没有观察到歧义，不代表全局唯一性得到证明。长度延续主汇总的顺序统计口径（偶数样本中位数取较低中位项，P90 取上整秩），只描述文件名/别名，排除数字与明确行号标签。
+
+结构有效率只检查 Markdown、解析、文件存在性和行号范围。语义引用支持未逐条穷尽；例如 A explain 第 2 轮、B explain 第 1 轮仍有范围存在但不能完全支持断言的引用。越界行号的 2 次分属 B/C explain 第 3 轮；不应误报为文件不存在。不同评审的事实问题分类与粒度不同，尤其 review 的 R6 是逐例文字审计，故没有将它强行变成与其他任务可加总的错误次数。

packages/client/ui-deliverables/evals/file-references/results/2026-09-16/notes.md

@@ -0,0 +1,14 @@
+# R3 and subsequent colon-label observations
+
+R3 contains 36 first attempts on four known development tasks, three repetitions per A/B/C treatment. One A review attempt timed out at 480.17 seconds without a final answer; it remains in execution counts and outside content denominators. Earlier authentication failures and earlier unsuccessful cohorts remain separately archived. No retry replaces an original first answer.
+
+The [original 69-row table](metrics.md) preserves its audit-time GUI status. The [compact observations](observations.json) preserve E/L/M/I for every attempted slot and exact SHA-256 hashes of the original prompts and tasks. These aggregates do not contain complete answer text or occurrence-level evidence; independent semantic re-audit requires the private archive. The original controls included macOS-specific host-read restrictions, maximum concurrency four, and a 480-second wall limit. Two runs had recorded host-access limitations; another read unfrozen external history. No cohort is claimed hermetic.
+
+| Later evidence | Valid links | Missing links | Label observations | Scope |
+| --- | ---: | ---: | --- | --- |
+| First A colon revision: plan, review, explain | 102 | 13 | 28 mixed `:start-Lend` suffixes | Retained failures; separate revision. |
+| Corrected A: explanation | 51 | 3 | No colon labels; 27 unnecessary directory qualifiers | Colon branch inactive. |
+| Corrected A: accepted GUI demo | 18 | Not fully audited | 3 colon labels; no mixed suffix | One task; user accepted the display. |
+| PR recording | 5 | Not fully audited | Explicit `filename:line` request; five colon labels | Guided UI demonstration; not autonomous adherence evidence. |
+
+A leads original occurrence coverage; B has more wholly linked answers and completes all attempts. C has three answers without Markdown file links. All variants have factual problems or unresolved claims. No weighted score, statistically established winner, or unseen holdout is available. The final colon revision has not received the full four-task, three-repeat comparison.

packages/client/ui-deliverables/evals/file-references/results/2026-09-16/observations.json

@@ -0,0 +1,415 @@
+{
+  "cohort": "historical-r3",
+  "runs": [
+    {
+      "name": "plan-baseline-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 36,
+        "L": 34,
+        "M": 2,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-filename-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 36,
+        "L": 35,
+        "M": 1,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-display-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 30,
+        "L": 0,
+        "M": 30,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-baseline-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 76,
+        "L": 74,
+        "M": 2,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-filename-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 65,
+        "L": 65,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-display-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 85,
+        "L": 85,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-baseline-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 37,
+        "L": 33,
+        "M": 4,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-filename-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 36,
+        "L": 25,
+        "M": 11,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-display-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 31,
+        "L": 19,
+        "M": 12,
+        "I": 0
+      }
+    },
+    {
+      "name": "explain-baseline-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 28,
+        "L": 27,
+        "M": 1,
+        "I": 0
+      }
+    },
+    {
+      "name": "explain-filename-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 53,
+        "L": 53,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "explain-display-1",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 53,
+        "L": 52,
+        "M": 1,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-filename-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 34,
+        "L": 30,
+        "M": 4,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-display-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 27,
+        "L": 20,
+        "M": 7,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-baseline-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 30,
+        "L": 28,
+        "M": 2,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-filename-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 66,
+        "L": 66,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-display-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 57,
+        "L": 57,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-baseline-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 65,
+        "L": 64,
+        "M": 1,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-filename-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 41,
+        "L": 33,
+        "M": 8,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-display-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 45,
+        "L": 35,
+        "M": 10,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-baseline-2",
+      "nonemptyFinal": false,
+      "timedOut": true,
+      "counts": null
+    },
+    {
+      "name": "explain-filename-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 41,
+        "L": 39,
+        "M": 2,
+        "I": 0
+      }
+    },
+    {
+      "name": "explain-display-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 61,
+        "L": 61,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "explain-baseline-2",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 38,
+        "L": 38,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-display-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 31,
+        "L": 0,
+        "M": 31,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-baseline-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 26,
+        "L": 25,
+        "M": 1,
+        "I": 0
+      }
+    },
+    {
+      "name": "plan-filename-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 46,
+        "L": 46,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-display-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 57,
+        "L": 57,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-baseline-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 58,
+        "L": 57,
+        "M": 1,
+        "I": 0
+      }
+    },
+    {
+      "name": "diagnose-filename-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 81,
+        "L": 81,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-display-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 32,
+        "L": 0,
+        "M": 32,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-baseline-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 29,
+        "L": 20,
+        "M": 9,
+        "I": 0
+      }
+    },
+    {
+      "name": "review-filename-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 33,
+        "L": 20,
+        "M": 13,
+        "I": 0
+      }
+    },
+    {
+      "name": "explain-display-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 83,
+        "L": 81,
+        "M": 1,
+        "I": 1
+      }
+    },
+    {
+      "name": "explain-baseline-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 53,
+        "L": 53,
+        "M": 0,
+        "I": 0
+      }
+    },
+    {
+      "name": "explain-filename-3",
+      "nonemptyFinal": true,
+      "timedOut": false,
+      "counts": {
+        "E": 53,
+        "L": 46,
+        "M": 6,
+        "I": 1
+      }
+    }
+  ],
+  "inputHashes": {
+    "prompts": {
+      "baseline": "a26cb88160f3078cda6ad3bb87714eee5faa7e2a898511d971412c40e2695f69",
+      "filename": "fcec33296928dcb369f9b13b96dc1433cc339665abf25fcce956957df41cb906",
+      "display": "48f2e4816785357713fed14717847d5964583862f9ea9de8fa02bd5fee4bd354"
+    },
+    "tasks": {
+      "plan": "4c010a99a3d0a3713e15fdc220bd35ecb5f7f35bd1089805159ca12a08a153ec",
+      "diagnose": "afe4c6646c8265e5716244566522cf0f9559585e0d3f1f1d5897f95d61075cee",
+      "review": "eb3d6274371135e4cfb95a987fdccc983b4fbc627e7d9d03642a6277d08f0b72",
+      "explain": "236e36fadefb1475ed0335532eacbd318efc7368eec7cb6b82d563f2f44a8bd4"
+    }
+  },
+  "limitations": [
+    "Development tasks; no holdout or weighted score.",
+    "A review repetition 2 timed out; no retry replaced it.",
+    "Two runs had host-access limitations; another read unfrozen external history.",
+    "Portable runner has different controls; do not pool cohorts."
+  ]
+}

packages/client/ui-deliverables/evals/file-references/rubric.md

@@ -0,0 +1,75 @@
+---
+description: "Frozen file-reference eligibility, label judgments, and original-task checks for prompt maintainers."
+kind: "package-reference"
+---
+
+# File-reference evaluation rubric
+
+## Summary
+
+These checks preserve the user-calibrated R3 rules and four original tasks. Task coverage, supported factual contradictions, unresolved claims, linking, and presentation are separate observations. No weighted score, character cutoff, significance claim, or universal-winner threshold is defined.
+
+## Table of Contents
+
+- [References and labels](#references-and-labels)
+- [Original tasks](#original-tasks)
+- [Review records](#review-records)
+- [GUI and experiment limits](#gui-and-experiment-limits)
+
+## References and labels
+
+Count every reader-facing mention of an identifiable existing file, including repeats, headings, lists, tables, aliases, and bare filenames in inline code. Commands, configuration expressions, code blocks, directories, globs, hypothetical files, and unverified candidates do not enter the verified denominator. Record exclusions and uncertainty with their reasons; false existence claims remain factual issues.
+
+E is eligible mentions, L valid links, M missing links, and I invalid links: E = L + M + I. Zero E is inactive, never 100%. A timed-out attempt remains a failure; an absent answer has no content denominator. Keep all first attempts and report missing reviews explicitly.
+
+One filename followed by plain line numbers is one occurrence. Repeating only a plain line number creates no additional linking obligation. An actual numeric-only Markdown file link counts once. Full workspace-relative and absolute targets are valid; known-line anchors use `#L24` or `#L24-L30`. A displayed colon suffix is not a DSH destination anchor. Check parser acceptance, intended file identity, existence, and range separately from whether the cited text supports the claim.
+
+Labels may use a filename, short suffix, clear alias, or contextual numeric reference. Prefer an identifiable basename over its full repository path. Judge unnecessary directories and same-basename ambiguity in visible nearby context; hidden URLs and tooltips do not establish visible clarity. A basename need not be globally unique. Keep composite README references, generic groups, and unresolved aliases outside decisive counts, with sensitivity bounds where useful.
+
+Report full-path frequency, unnecessary qualifiers, label lengths, distinct files, numeric labels, malformed targets, wrong-file targets, and broken ranges separately. Exclude numeric-only and explicit line-reference labels from filename/alias lengths. Count unlinked full-path mentions too, so missing links cannot masquerade as shorter labels. Use the lower middle observation for even-sample medians and the ceiling rank for P90, matching the historical report.
+
+## Original tasks
+
+Each criterion receives addressed, omitted, contradicted, or unverified, with output spans and source evidence. Covering a topic does not establish correctness. Suggested edits do not violate read-only scope. No requirement to execute tests or produce findings is added to a task that asks for inspection.
+
+| ID | Requirement |
+| --- | --- |
+| P1 | Minimal Buffer support grounded in implementation, retaining string support. |
+| P2 | Address implementation, tests, documentation, and callers; evidence may show a surface needs no edit. |
+| P3 | Explain compatibility without requiring an unsupported overload or API design. |
+| P4 | Give verification appropriate to the proposed change. |
+| P5 | Read-only analysis, no subagents, assessed from execution evidence. |
+| D1 | Explain the stale-version sequence and usable investigation steps. |
+| D2 | Give the supported retry procedure and current basis for retry. |
+| D3 | Explain restored-session differences from source evidence. |
+| D4 | Identify relevant implementation, tests, documentation, and useful investigation files. |
+| D5 | Invocation examples are optional; supplied examples must agree with implementation. |
+| D6 | Read-only analysis, no subagents, assessed from execution evidence. |
+| R1 | Inspect or discuss the implementation. |
+| R2 | Inspect or discuss the tests. |
+| R3 | Inspect README and documented expectations. |
+| R4 | Give supported findings or explicitly state no issue was confirmed. |
+| R5 | Explain the behaviors checked beyond a bare assertion of correctness. |
+| R6 | Verify material findings against preserved source; distinguish confirmed issues, limitations, and speculation. |
+| R7 | Read-only analysis, no subagents, assessed from execution evidence. |
+| X1 | Explain CLI task origin and entry into Session execution. |
+| X2 | Explain source-supported model selection inputs and defaults. |
+| X3 | Explain working-directory selection. |
+| X4 | Explain output timing relative to execution and completion. |
+| X5 | Explain failure handling without unsupported recovery promises. |
+| X6 | Explain Session resume and relevant task/model/directory interactions. |
+| X7 | Connect the topics into a coherent Session lifecycle account. |
+| X8 | Verify material factual claims and citations against preserved source. |
+| X9 | Read-only analysis, no subagents, assessed from execution evidence. |
+
+## Review records
+
+For each first answer, review the complete text and retain an occurrence ledger containing line, column, text, identified file, L/M/I classification, exclusions, and ambiguity. Record task-criterion observations and source-supported contradictions separately. Reviewers must not infer E by counting only Markdown links. A self-review is supplemental evidence, not authoritative scoring.
+
+Create `annotation.json` beside a reviewed run's `final.md`, with `finalSha256`, `counts` containing E/L/M/I, and the occurrence/task ledgers. The summarizer verifies the answer hash and arithmetic; it does not validate human identity judgments or semantic claims. Missing annotations remain unreviewed. Historical observations contain aggregate counts imported from the original audit; their full raw ledgers remain in the private experiment archive.
+
+## GUI and experiment limits
+
+Review scanability, informative labels, same-basename clarity, visible/clickable links, correct destinations, wrapping, clipping, tables, line-number placement, and retained task explanation. Match viewport, theme, and zoom across prompts; inspect raw rendered answers without rewriting them. Screenshots alone do not verify clicking. Overall attractiveness and acceptable density require human preference, with no numeric beauty score.
+
+Freeze exact tasks, treatments, permitted resources, and bounds before execution. Keep evaluator material outside the task workspace. Run three fresh repetitions to describe variance; preserve failures and never replace them with silent retries. Known development tasks are not holdouts. An inactive formatting branch remains unverified. Later colon-label revisions and guided GUI demos are separate evidence, not new scores for the original A/B/C cohort.

packages/client/ui-deliverables/evals/file-references/run.py

@@ -0,0 +1,174 @@
+"""Run preserved file-reference tasks through built dsh profiles on POSIX hosts."""
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+import shutil
+import signal
+import subprocess
+import tempfile
+import time
+
+OWNER = Path(__file__).resolve().parent
+REPO = OWNER.parents[4]
+
+
+def digest(path):
+    """Hash exact input or output bytes without parsing their contents."""
+    return hashlib.sha256(path.read_bytes()).hexdigest()
+
+
+def execute(command, workspace, task, slot, timeout):
+    """Preserve first output and reap the owned process group, including on interruption."""
+    started = time.monotonic()
+    timed_out = False
+    interrupted = False
+    with task.open('rb') as stdin, (slot / 'stdout.jsonl').open('xb') as stdout, (slot / 'stderr.txt').open('xb') as stderr:
+        child = subprocess.Popen(command, cwd=workspace, stdin=stdin, stdout=stdout,
+                                 stderr=stderr, start_new_session=True)
+        try:
+            child.wait(timeout=timeout)
+        except subprocess.TimeoutExpired:
+            timed_out = True
+        except KeyboardInterrupt:
+            interrupted = True
+        finally:
+            # Terminate descendants that inherit the launcher process group.
+            try:
+                os.killpg(child.pid, signal.SIGTERM)
+            except ProcessLookupError:
+                pass
+            try:
+                child.wait(timeout=5)
+            except subprocess.TimeoutExpired:
+                os.killpg(child.pid, signal.SIGKILL)
+                child.wait()
+    # A descendant may ignore TERM after the leader exits; never leave it running.
+    try:
+        os.killpg(child.pid, signal.SIGKILL)
+    except ProcessLookupError:
+        pass
+    deadline = time.monotonic() + 5
+    teardown_error = None
+    while True:
+        try:
+            os.killpg(child.pid, 0)
+        except ProcessLookupError:
+            break
+        if time.monotonic() >= deadline:
+            teardown_error = 'Owned process group did not finish teardown'
+            break
+        time.sleep(.01)
+    finals = []
+    output_error = None
+    for line in (slot / 'stdout.jsonl').read_text().splitlines():
+        try:
+            event = json.loads(line)
+        except json.JSONDecodeError:
+            output_error = 'stdout contains a non-JSON line; inspect preserved output'
+            break
+        if event.get('type') == 'final':
+            finals.append(event['text'])
+    final = finals[-1] if finals else ''
+    if final.strip():
+        (slot / 'final.md').write_text(final + '\n')
+    result = {'exitCode': child.returncode, 'timedOut': timed_out,
+              'interrupted': interrupted, 'outputError': output_error, 'teardownError': teardown_error, 'durationSeconds': time.monotonic() - started,
+              'nonemptyFinal': bool(final.strip()),
+              'finalSha256': digest(slot / 'final.md') if final.strip() else None}
+    (slot / 'execution.json').write_text(json.dumps(result, indent=2) + '\n')
+    return result
+
+
+def main():
+    """Freeze inputs before serial execution; stop at the first failed attempt."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--variant', nargs='+', choices=['baseline', 'filename', 'display', 'a-colon'], default=['baseline', 'filename', 'display'])
+    parser.add_argument('--case', nargs='+', choices=['plan', 'diagnose', 'review', 'explain'], default=['plan', 'diagnose', 'review', 'explain'])
+    parser.add_argument('--repetitions', type=int, default=3)
+    parser.add_argument('--timeout', type=float, default=480)
+    parser.add_argument('--model', required=True, help='Explicit DeepSeek model id, recorded with every run')
+    parser.add_argument('--env-file', type=Path, help='Normal DSH .env source; values are never read by this runner')
+    parser.add_argument('--output', type=Path, help='New directory under repository .artifacts/')
+    parser.add_argument('--prepare-only', action='store_true', help='Freeze the cohort without calling the API')
+    args = parser.parse_args()
+    if os.name != 'posix':
+        parser.error('Process-group cleanup requires a POSIX host')
+    if args.repetitions < 1 or not 0 < args.timeout <= 480:
+        parser.error('Use positive repetitions and a timeout no greater than 480 seconds')
+    if len(set(args.variant)) != len(args.variant) or len(set(args.case)) != len(args.case):
+        parser.error('Cases and variants must be unique')
+    launcher = REPO / 'apps/cli/lib/bin.js'
+    if not launcher.is_file():
+        parser.error('Build this checkout with pnpm run build first')
+    root = REPO / '.artifacts/file-reference-evals'
+    root.mkdir(parents=True, exist_ok=True)
+    output = args.output.resolve() if args.output else Path(tempfile.mkdtemp(prefix='run-', dir=root))
+    if args.output:
+        if not output.is_relative_to(REPO / '.artifacts'):
+            parser.error('--output must be inside repository .artifacts/')
+        output.mkdir(mode=0o700)
+    output.chmod(0o700)
+    env_args = ['--env-file=' + str(args.env_file.resolve())] if args.env_file else []
+    revision = subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=REPO, text=True).strip()
+    archive = output / 'source.tar'
+    with archive.open('xb') as target:
+        subprocess.run(['git', 'archive', 'HEAD'], cwd=REPO, stdout=target, check=True)
+    frozen = output / 'inputs'
+    frozen.mkdir()
+    for folder in ['cases', 'prompts']:
+        shutil.copytree(OWNER / folder, frozen / folder)
+    shutil.copyfile(OWNER / 'treatment.mjs', frozen / 'treatment.mjs')
+    shutil.copyfile(OWNER / 'rubric.md', frozen / 'rubric.md')
+    names = [f'{case}-{variant}-{repeat}' for repeat in range(1, args.repetitions + 1)
+             for case in args.case for variant in args.variant]
+    manifest = {'sourceRevision': revision, 'launcherSha256': digest(launcher),
+                'sourceArchiveSha256': digest(archive), 'runs': names,
+                'model': args.model, 'reasoningEffort': 'high', 'timeoutSeconds': args.timeout,
+                'preparedOnly': args.prepare_only, 'cohort': 'portable-owner-eval',
+                'inputs': {str(p.relative_to(frozen)): digest(p) for p in frozen.rglob('*') if p.is_file()}}
+    (output / 'manifest.json').write_text(json.dumps(manifest, indent=2) + '\n')
+    print(output, flush=True)
+    if args.prepare_only:
+        return
+    for name in names:
+        case, variant = name.rsplit('-', 1)[0].split('-', 1)
+        slot = output / name
+        slot.mkdir()
+        workspace = slot / 'workspace'
+        workspace.mkdir()
+        subprocess.run(['tar', '-xf', str(archive), '-C', str(workspace)], check=True)
+        # Evaluator instructions and historical scores are not task inputs.
+        evals = workspace / OWNER.relative_to(REPO)
+        if evals.exists():
+            shutil.rmtree(evals)
+        before = {str(p.relative_to(workspace)): digest(p) for p in workspace.rglob('*') if p.is_file()}
+        home = slot / 'home'
+        home.mkdir()
+        (home / 'settings.yaml').write_text(json.dumps({'agent-default-model': {
+            'provider': 'deepseek-official', 'model': args.model, 'reasoningEffort': 'high'}}))
+        patch = [
+            {'id': 'sandbox-policy', 'config': {'mode': 'read-only'}},
+            {'id': 'approval', 'config': {'policy': 'never'}},
+            {'id': 'agent-instructions', 'config': {'maxBytes': 65536, 'projectRootMarkers': ['pnpm-workspace.yaml'], 'dshHome': str(slot / 'agents')}},
+            {'insert': [{'id': 'file-reference-treatment', 'name': str(frozen / 'treatment.mjs'),
+                         'config': {'promptFile': str(frozen / 'prompts' / (variant + '.txt'))}}]},
+        ]
+        overlay = slot / 'overlay.yml'
+        overlay.write_text(json.dumps(patch))
+        command = ['env', 'DSH_HOME=' + str(home), 'DSH_AGENTS_HOME=' + str(slot / 'agents'),
+                   'node', *env_args, str(launcher), '--profile', 'headless', '--patch', str(overlay), '--json', '-']
+        task = frozen / 'cases' / (case + '.txt')
+        result = execute(command, workspace, task, slot, args.timeout)
+        after = {str(p.relative_to(workspace)): digest(p) for p in workspace.rglob('*') if p.is_file()}
+        changed = sorted(k for k in before.keys() | after.keys() if before.get(k) != after.get(k))
+        (slot / 'workspace-audit.json').write_text(json.dumps({'unchanged': not changed, 'changedFiles': changed}, indent=2) + '\n')
+        print(json.dumps({'run': name, **result, 'workspaceUnchanged': not changed}), flush=True)
+        if result['exitCode'] != 0 or result['timedOut'] or result['interrupted'] or result['outputError'] or result['teardownError'] or not result['nonemptyFinal'] or changed:
+            (output / 'stop.json').write_text(json.dumps({'failed': name, 'unstarted': names[names.index(name) + 1:]}, indent=2) + '\n')
+            raise SystemExit(1)
+
+
+if __name__ == '__main__':
+    main()

packages/client/ui-deliverables/evals/file-references/summarize.py

@@ -0,0 +1,81 @@
+"""Aggregate reviewed occurrence counts without treating unreviewed answers as passes."""
+import argparse
+import json
+from pathlib import Path
+from run import digest
+
+
+def observations(source):
+    """Read historical records or join fresh outputs with hash-bound human annotations."""
+    if source.is_file():
+        return json.loads(source.read_text())['runs']
+    manifest = json.loads((source / 'manifest.json').read_text())
+    rows = []
+    for name in manifest['runs']:
+        slot = source / name
+        if not (slot / 'execution.json').exists():
+            if (slot / 'stdout.jsonl').exists():
+                raise ValueError(f'{name}: incomplete execution record; inspect retained output')
+            rows.append({'name': name, 'unstarted': True})
+            continue
+        row = {'name': name, **json.loads((slot / 'execution.json').read_text()), 'counts': None}
+        annotation = slot / 'annotation.json'
+        if annotation.exists():
+            review = json.loads(annotation.read_text())
+            if not row['nonemptyFinal'] or not review['finalSha256'] == row['finalSha256'] == digest(slot / 'final.md'):
+                raise ValueError(f'{name}: annotation does not match the preserved final answer')
+            row['counts'] = review['counts']
+        rows.append(row)
+    return rows
+
+
+def aggregate(rows):
+    """Keep execution and content denominators separate; reject invalid ledgers."""
+    result = {}
+    seen = set()
+    for row in rows:
+        name = row['name']
+        if name in seen:
+            raise ValueError(f'Duplicate attempt: {name}')
+        seen.add(name)
+        variant = name.rsplit('-', 1)[0].split('-', 1)[1]
+        group = result.setdefault(variant, dict(planned=0, attempted=0, completed=0, timedOut=0,
+                                               audited=0, unreviewed=0, perfect=0, E=0, L=0, M=0, I=0))
+        group['planned'] += 1
+        if row.get('unstarted'):
+            continue
+        group['attempted'] += 1
+        group['completed'] += int(row['nonemptyFinal'])
+        group['timedOut'] += int(row['timedOut'])
+        counts = row.get('counts')
+        if counts is None:
+            group['unreviewed'] += int(row['nonemptyFinal'])
+            continue
+        if not row['nonemptyFinal']:
+            raise ValueError(f'{name}: an absent final answer cannot have content counts')
+        if any(type(counts.get(k)) is not int or counts[k] < 0 for k in ['E', 'L', 'M', 'I']):
+            raise ValueError(f'{name}: E/L/M/I must be nonnegative integers')
+        if counts['E'] != counts['L'] + counts['M'] + counts['I']:
+            raise ValueError(f'{name}: E must equal L + M + I')
+        group['audited'] += 1
+        group['perfect'] += int(counts['E'] > 0 and counts['M'] == counts['I'] == 0)
+        for key in ['E', 'L', 'M', 'I']:
+            group[key] += counts[key]
+    return result
+
+
+def main():
+    """Print the same compact result table for archived and newly reviewed cohorts."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('source', type=Path, help='Run directory or historical observations.json')
+    args = parser.parse_args()
+    groups = aggregate(observations(args.source))
+    print('| Prompt | Answers / attempts / planned | Audited | Unreviewed | L/E | Missing / invalid | Perfect | Timeouts |')
+    print('| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |')
+    for variant, row in groups.items():
+        rate = f"{row['L']}/{row['E']} ({row['L'] / row['E']:.1%})" if row['E'] else 'inactive'
+        print(f"| {variant} | {row['completed']}/{row['attempted']}/{row['planned']} | {row['audited']} | {row['unreviewed']} | {rate} | {row['M']} / {row['I']} | {row['perfect']} | {row['timedOut']} |")
+
+
+if __name__ == '__main__':
+    main()

packages/client/ui-deliverables/evals/file-references/test_eval.py

@@ -0,0 +1,115 @@
+"""Keyless checks for retained failures and honest evaluation denominators."""
+import json
+import os
+from pathlib import Path
+import sys
+import tempfile
+import unittest
+from run import execute, digest
+from summarize import aggregate, observations
+
+
+class EvaluationTests(unittest.TestCase):
+    def test_missing_audits_and_unstarted_slots_are_not_passes(self):
+        rows = [
+            {'name': 'plan-baseline-1', 'nonemptyFinal': True, 'timedOut': False},
+            {'name': 'plan-baseline-2', 'nonemptyFinal': False, 'timedOut': True},
+            {'name': 'plan-baseline-3', 'unstarted': True},
+        ]
+        self.assertEqual(aggregate(rows)['baseline'], dict(planned=3, attempted=2, completed=1,
+                         timedOut=1, audited=0, unreviewed=1, perfect=0, E=0, L=0, M=0, I=0))
+
+    def test_interrupted_record_is_not_an_unstarted_slot(self):
+        with tempfile.TemporaryDirectory() as root:
+            source = Path(root)
+            (source / 'manifest.json').write_text(json.dumps({'runs': ['plan-baseline-1']}))
+            slot = source / 'plan-baseline-1'
+            slot.mkdir()
+            (slot / 'stdout.jsonl').write_text('{}\n')
+            with self.assertRaisesRegex(ValueError, 'incomplete execution record'):
+                observations(source)
+
+    def test_bad_denominators_and_duplicate_attempts_fail(self):
+        row = {'name': 'plan-a-colon-1', 'nonemptyFinal': True, 'timedOut': False,
+               'counts': {'E': 2, 'L': 1, 'M': 0, 'I': 0}}
+        with self.assertRaisesRegex(ValueError, 'E must equal'):
+            aggregate([row])
+        row['counts']['M'] = 1
+        self.assertEqual(aggregate([row])['a-colon']['M'], 1)
+        with self.assertRaisesRegex(ValueError, 'Duplicate'):
+            aggregate([row, row])
+        row['counts']['E'] = True
+        with self.assertRaisesRegex(ValueError, 'nonnegative integers'):
+            aggregate([row])
+
+    def test_frozen_inputs_match_original_hashes(self):
+        owner = Path(__file__).parent
+        hashes = json.loads((owner / 'results/2026-09-16/observations.json').read_text())['inputHashes']
+        for kind, expected in hashes.items():
+            folder = 'cases' if kind == 'tasks' else 'prompts'
+            for name, value in expected.items():
+                self.assertEqual(digest(owner / folder / (name + '.txt')), value)
+
+    def test_descendant_ignoring_term_is_reaped(self):
+        with tempfile.TemporaryDirectory() as root:
+            slot = Path(root)
+            task = slot / 'task.txt'
+            task.write_text('task')
+            code = 'import json, subprocess, sys, time; child=subprocess.Popen([sys.executable,"-c","import signal,time; signal.signal(signal.SIGTERM,signal.SIG_IGN); time.sleep(60)"]); print(json.dumps({"pid":child.pid}),flush=True); time.sleep(60)'
+            result = execute([sys.executable, '-c', code], slot, task, slot, .3)
+            self.assertTrue(result['timedOut'])
+            pid = json.loads((slot / 'stdout.jsonl').read_text())['pid']
+            with self.assertRaises(ProcessLookupError):
+                os.kill(pid, 0)
+
+    def test_historical_counts_reproduce_report(self):
+        source = Path(__file__).parent / 'results/2026-09-16/observations.json'
+        groups = aggregate(observations(source))
+        for variant, expected in [('baseline', (476, 453, 23, 0)),
+                                  ('filename', (585, 539, 45, 1)), ('display', (592, 467, 124, 1))]:
+            self.assertEqual(tuple(groups[variant][k] for k in ['E', 'L', 'M', 'I']), expected)
+        self.assertEqual(groups['baseline']['timedOut'], 1)
+
+    def test_annotation_cannot_reclassify_changed_first_answer(self):
+        with tempfile.TemporaryDirectory() as root:
+            source = Path(root)
+            (source / 'manifest.json').write_text(json.dumps({'runs': ['plan-baseline-1']}))
+            slot = source / 'plan-baseline-1'
+            slot.mkdir()
+            final = slot / 'final.md'
+            final.write_text('first answer')
+            original = digest(final)
+            (slot / 'execution.json').write_text(json.dumps({'nonemptyFinal': True, 'timedOut': False, 'finalSha256': original}))
+            (slot / 'annotation.json').write_text(json.dumps({'finalSha256': original, 'counts': {'E': 1, 'L': 1, 'M': 0, 'I': 0}}))
+            self.assertEqual(aggregate(observations(source))['baseline']['L'], 1)
+            final.write_text('changed answer')
+            with self.assertRaisesRegex(ValueError, 'does not match'):
+                observations(source)
+
+    def test_malformed_output_retains_execution_failure(self):
+        with tempfile.TemporaryDirectory() as root:
+            slot = Path(root)
+            task = slot / 'task.txt'
+            task.write_text('task')
+            result = execute([sys.executable, '-c', 'print("not json")'], slot, task, slot, 10)
+            self.assertIsNotNone(result['outputError'])
+            self.assertTrue((slot / 'execution.json').exists())
+
+    def test_timeout_preserves_output_and_reaps_process(self):
+        with tempfile.TemporaryDirectory() as root:
+            slot = Path(root)
+            task = slot / 'task.txt'
+            task.write_text('task')
+            result = execute([sys.executable, '-c', 'import time; print("{}", flush=True); time.sleep(60)'],
+                             slot, task, slot, .2)
+            self.assertTrue(result['timedOut'])
+            self.assertFalse(result['nonemptyFinal'])
+            self.assertIsNotNone(result['exitCode'])
+            self.assertEqual((slot / 'stdout.jsonl').read_text(), '{}\n')
+            self.assertEqual(json.loads((slot / 'execution.json').read_text())['timedOut'], True)
+            with self.assertRaises(FileExistsError):
+                execute([sys.executable, '-c', 'pass'], slot, task, slot, 1)
+
+
+if __name__ == '__main__':
+    unittest.main()

packages/client/ui-deliverables/evals/file-references/treatment.mjs

@@ -0,0 +1,14 @@
+/** Frozen evaluation treatment registered through the ordinary system-prompt plugin. */
+import { readFileSync } from 'node:fs'
+
+export const name = 'file-reference-evaluation'
+export const inject = ['systemPrompt']
+
+export function apply(ctx, config) {
+  ctx.systemPrompt.section({
+    name: 'evaluation:file-references',
+    order: ctx.systemPrompt.getSectionOrder('DELIVERABLE_FILE_REFERENCES'),
+    text: readFileSync(config.promptFile, 'utf8'),
+    interpolate: false,
+  })
+}