fix: projection error

.agents/notes/implemented/bug-fix/2026-08-06-token-surface-unpriced-replace-compatibility.i18n.yaml

@@ -0,0 +1,6 @@
+# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
+# side as of the last confirmed-consistent state. Both languages carry equal authority;
+# after editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-08-06-token-surface-unpriced-replace-compatibility.md
+2026-08-06-token-surface-unpriced-replace-compatibility.md: 77b778fba695f560eb68af6da2f416e61ccff181
+2026-08-06-token-surface-unpriced-replace-compatibility.zh.md: dd4c5883c4c6d37809c2f8100f266655d8ba600c

.agents/notes/implemented/bug-fix/2026-08-06-token-surface-unpriced-replace-compatibility.md

@@ -0,0 +1,35 @@
+# Agent Note: unpriced surface replacements fold neutrally
+
+Status: implemented
+
+English | [中文](2026-08-06-token-surface-unpriced-replace-compatibility.zh.md)
+
+## Problem
+
+The `contextPressure` and `contextBreakdown` projections keep a running surface-token total plus at most one pending shadow-price claim, so their persisted checkpoints stay O(1) over a session's life. Current replace producers append a `compact/summary` or `compact/prune` metering event immediately before the replacement; its `shadowedTokenCount` prices the exact replaced range, and `foldSurfaceProjection` turns that into the signed delta.
+
+Sessions recorded before the shadow-price protocol log replacements with no adjacent metering event. The O(1) state cannot reconstruct the replaced range's price, and the fold treated every unpriced replacement as a contract violation and threw — so replaying such a session died at its first replacement (`token surface: replace at seq … has no adjacent shadow price`), leaving the session permanently unopenable.
+
+## Decision
+
+A replace that arrives with no armed claim folds price-neutrally: `foldSurfaceProjection` returns `deltaTokens: 0`, pricing the replaced range as if it had cost exactly what its replacement costs, and replay continues. A claim expired by an intervening event reaches the same neutral path, since the fold cannot distinguish it from a log that never metered.
+
+An armed claim naming a **different** range still throws. There the metering event was adjacent, so the producer wrote contradictory adjacent events — a live shadow-price contract violation, not historical data, and it must fail loud rather than let the total drift silently.
+
+Both projections share the one fold, so neither gains state fields nor bumps its `stateVersion`. `surface-fold.ts` and `ctx.tokenMeter.measure()` are unaffected: they hold the per-node priced surface and never needed the claim protocol.
+
+## Alternatives considered
+
+**Keep throwing.** Preserves the strict producer contract, but every pre-protocol session stays permanently unreplayable, and the projections exist to serve replay.
+
+**Persist the full priced surface in the projection state.** Could price any replaced range exactly, but grows the checkpoint by one node per model-visible message without bound — defeating the O(1) constraint the shadow-price protocol exists to preserve (see [the context-meter note](2026-08-05-context-meter-blind-to-compaction.md)).
+
+## Consequences
+
+An unpriced replacement holds the total still instead of shrinking it, so the compacted-away span stays counted: `contextBreakdown.messageTokens` retains the overcount, and `contextPressure.projectedTokens` overestimates occupancy only until the next usage sample re-anchors it, because that figure tracks movement since the sample rather than the absolute level. The error direction is safe — overestimating occupancy at worst invites an earlier compaction.
+
+The loud failure survives where it still means something: a range-mismatched adjacent claim is a current producer bug and still throws.
+
+## Testing
+
+`packages/llm/token-meter/tests/context-breakdown-projection.spec.ts` pins the neutral fold for the no-claim and expired-claim replacements, the throw for a mismatched claim, and the exact pricing for a matched one. `packages/llm/token-meter/tests/token-usage-projection.spec.ts` pins `contextPressure` holding still across an unpriced replacement.

.agents/notes/implemented/bug-fix/2026-08-06-token-surface-unpriced-replace-compatibility.zh.md

@@ -0,0 +1,35 @@
+# Agent Note: 未计价的表层替换以中性方式折叠
+
+Status: implemented
+
+[English](2026-08-06-token-surface-unpriced-replace-compatibility.md) | 中文
+
+## 问题
+
+`contextPressure` 与 `contextBreakdown` 两个投影只维护一份滚动累计的表层 token 总量，外加至多一条待结算的影子价格（shadow price）声明，因此其持久化检查点在会话整个生命周期内保持 O(1)。当前的替换生产方会紧贴在替换之前追加一条 `compact/summary` 或 `compact/prune` 计量事件；其 `shadowedTokenCount` 对被替换区间精确计价，`foldSurfaceProjection` 再把它换算成有符号增量。
+
+影子价格协议引入之前录制的会话，其日志中的替换没有相邻的计量事件。O(1) 状态无法重建被替换区间的价格，而折叠此前把每一次未计价替换都当作契约违规并抛出异常，于是回放这类会话会在第一处替换就中断（`token surface: replace at seq … has no adjacent shadow price`），会话从此永远无法打开。
+
+## 决策
+
+到达时没有已就位声明的替换以价格中性的方式折叠：`foldSurfaceProjection` 返回 `deltaTokens: 0`，相当于把被替换区间计价为恰好等于其替换内容的成本，回放随即继续。因中间插入的事件而过期的声明也走同一条中性路径，因为折叠无法把它与从未计量过的日志区分开。
+
+已就位但指向**另一个**区间的声明仍会抛出异常。此时计量事件确实相邻，说明生产方写入了互相矛盾的相邻事件：这是现行影子价格契约的违规，不是历史数据，必须响亮失败，而不能任由总量悄然漂移。
+
+两个投影共用同一个折叠，因此二者都不新增状态字段，也不提升 `stateVersion`。`surface-fold.ts` 与 `ctx.tokenMeter.measure()` 不受影响：它们持有逐节点的已计价表层，本来就不需要声明协议。
+
+## 备选方案
+
+**维持抛出异常。**保住了严格的生产方契约，但协议之前的每个会话都将永远无法回放，而投影本就是为服务回放而存在的。
+
+**在投影状态中持久化完整的已计价表层。**可以对任意被替换区间精确计价，但检查点会随每条模型可见消息各增加一个节点、无上限地增长，恰恰破坏了影子价格协议所要守住的 O(1) 约束（见[上下文仪表的 Agent Note](2026-08-05-context-meter-blind-to-compaction.md)）。
+
+## 影响
+
+未计价的替换让总量保持不动而不是缩小，因此被压缩（compaction）掉的区段仍被计入：`contextBreakdown.messageTokens` 保留这部分多计的量；`contextPressure.projectedTokens` 会高估占用率，但只持续到下一个用量样本重新锚定为止，因为该数字追踪的是自样本以来的增减，而非绝对水平。误差方向是安全的：高估占用率最坏不过是招致一次更早的压缩。
+
+响亮失败保留在它仍有意义的地方：区间不匹配的相邻声明是现行生产方的缺陷，仍会抛出异常。
+
+## 测试
+
+`packages/llm/token-meter/tests/context-breakdown-projection.spec.ts` 钉住了无声明与声明过期两种替换的中性折叠、声明区间不匹配时的抛出异常，以及声明匹配时的精确计价。`packages/llm/token-meter/tests/token-usage-projection.spec.ts` 钉住了 `contextPressure` 在一次未计价替换前后保持不动。

packages/llm/token-meter/src/breakdown-projection.ts

@@ -33,10 +33,11 @@ const breakdownSchema = z.object({
  *
  * Envelope figures are last-wins per `request/header`; the message figure
  * rides {@link foldSurfaceProjection} — the same O(1) fold the occupancy
- * projection uses — so it equals `measure().surfaceTokens` at every event
- * boundary and compaction shrinks it by its logged shadow price, the way it
- * shrinks the next request. The state is a fixed handful of numbers, so the
- * persisted checkpoint stays O(1) over the session's life.
+ * projection uses — so fully metered logs equal `measure().surfaceTokens` at
+ * every event boundary and compaction shrinks the figure by its logged shadow
+ * price. A replacement without a claim preserves the previous total. The
+ * state is a fixed handful of numbers, so the persisted checkpoint stays
+ * O(1) over the session's life.
  */
 export const contextBreakdownProjectionDefinition:
 ProjectionDefinition<'contextBreakdown', ContextBreakdownState> = {

packages/llm/token-meter/src/surface-fold.ts

@@ -3,9 +3,10 @@
  * surface `measure()` serves and compaction plans against. The projection
  * units deliberately do NOT share this fold — their state must stay O(1)
  * for the persisted checkpoint, so they ride `surface-projection.ts`'s
- * shadow-price protocol instead. The two stay in agreement by construction:
- * both price through `estimate.ts`, and every logged shadow price is derived
- * from THIS fold's nodes by the replace producer.
+ * shadow-price protocol instead. Fully metered logs stay in agreement by
+ * construction: both price through `estimate.ts`, and every logged shadow
+ * price is derived from THIS fold's nodes by the replace producer. A
+ * projection replacement without a claim deliberately folds with zero delta.
  *
  * @module @deepseek-ai/dsh-token-meter/surface-fold
  */

packages/llm/token-meter/src/surface-projection.ts

@@ -10,7 +10,9 @@
  * heuristic price of the exact replaced range, so the fold keeps a running
  * total plus at most one pending claim and never retains per-node prices.
  * The counts are exact by construction: producers derive them from the same
- * fixed estimator this module prices appends with.
+ * fixed estimator this module prices appends with. A replacement without an
+ * armed claim folds with zero delta because bounded state cannot reconstruct
+ * the replaced range; this preserves replay at the cost of possible drift.
  *
  * @module @deepseek-ai/dsh-token-meter/surface-projection
  */
@@ -47,16 +49,19 @@ export interface SurfaceTokensFold {
  * Fold one committed event onto a running surface-token total.
  *
  * A shadow-price event arms a claim; any other event expires it, and a
- * surface `replace` must consume a claim naming its exact range — the
+ * surface `replace` consumes the claim naming its exact range — the
  * producers append the metering event and the replacement synchronously
  * adjacent, so a surviving claim always prices the very next event.
+ * A replace with no claim folds with zero delta because the bounded state
+ * cannot reconstruct the replaced range. An armed claim for another range
+ * still fails because the adjacent events contradict each other.
  * @param claim - the claim armed by the immediately preceding event, if any.
  * @param event - the next committed session event.
  * @returns the signed token delta and the claim state after this event.
- * @throws when a replacement arrives without a claim for its exact range —
- *   every in-repo replace producer meters its replacement, so an unpriced
- *   replacement is a shadow-price contract violation and must fail loud
- *   rather than let the total drift.
+ * @throws when a replacement arrives with an armed claim for a different
+ *   range — the metering event was adjacent, so this is a live producer's
+ *   shadow-price contract violation, not historical data, and must fail
+ *   loud rather than let the total drift.
  */
 export function foldSurfaceProjection(
   claim: ShadowPriceClaim | undefined,
@@ -74,10 +79,15 @@ export function foldSurfaceProjection(
   const tokens = message === null ? 0 : estimateMessage(message)
   const op = event.surfaceOp
   if (op === 'append') return { deltaTokens: tokens, claim: undefined }
-  if (claim === undefined || claim.start !== op.start || claim.end !== op.end) {
+  // Sessions recorded before the shadow-price protocol log replacements with
+  // no adjacent metering event; the bounded state cannot reconstruct the
+  // replaced range's price, so fold those neutrally — historical replay
+  // degrades to drift instead of failing.
+  if (claim === undefined) return { deltaTokens: 0, claim: undefined }
+  if (claim.start !== op.start || claim.end !== op.end) {
     throw new Error(
       `token surface: replace at seq ${event.seq} over range ${op.start}-${op.end} has no adjacent shadow price`
-      + (claim === undefined ? '' : ` (armed claim covers ${claim.start}-${claim.end})`),
+      + ` (armed claim covers ${claim.start}-${claim.end})`,
     )
   }
   return { deltaTokens: tokens - claim.tokens, claim: undefined }

packages/llm/token-meter/src/usage-projection.ts

@@ -155,9 +155,10 @@ ProjectionDefinition<'tokenUsage', TokenUsageState> = {
  * `projectedTokens` — the sample plus the surface's signed movement since it
  * was taken — so occupancy answers for the next request rather than the last
  * one. The total rides {@link foldSurfaceProjection}, so the state stays O(1)
- * and a replacement shrinks it by its logged shadow price. A usage sample is
- * stamped BEFORE the same event joins the surface, so an `assistant/message`
- * anchors against the surface its own request saw.
+ * and a replacement shrinks it by its logged shadow price. A replacement
+ * without a claim preserves the previous total. A usage sample is stamped
+ * BEFORE the same event joins the surface, so an `assistant/message` anchors
+ * against the surface its own request saw.
  */
 export const contextPressureProjectionDefinition:
 ProjectionDefinition<'contextPressure', ContextPressureState> = {

packages/llm/token-meter/tests/context-breakdown-projection.spec.ts

@@ -180,7 +180,7 @@ describe('contextBreakdown session projection', () => {
     expect(agree()).toBeLessThan(grown)
   })
 
-  it('fails loud on a replacement without an adjacent matching shadow price', () => {
+  it('folds a replacement without a claim at zero and fails on a mismatched claim', () => {
     const definition = contextBreakdownProjectionDefinition
     const replace = (start: number, end: number): SessionEvent => ({
       type: 'user/message',
@@ -206,15 +206,17 @@ describe('contextBreakdown session projection', () => {
     let state = definition.init()
     state = definition.apply(state, append(1))
     state = definition.apply(state, append(3))
-    // No metering event at all.
-    expect(() => definition.apply(state, replace(1, 3))).toThrow('no adjacent shadow price')
-    // A claim for a different range does not price this replacement.
+    // No metering event: the replacement contributes zero instead of throwing.
+    expect(definition.view(definition.apply(state, replace(1, 3))).messageTokens)
+      .toBe(definition.view(state).messageTokens)
+    // An adjacent claim for another range contradicts the replacement.
     const mismatched = definition.apply(state, meter(1, 1, 8))
     expect(() => definition.apply(mismatched, replace(1, 3))).toThrow('no adjacent shadow price')
-    // A claim expires after one intervening event instead of lingering.
+    // A claim expires after one intervening event, so replacement delta is zero.
     let expired = definition.apply(state, meter(1, 3, 8))
     expired = definition.apply(expired, { type: 'todo/write', seq: 9, time: 0, data: { todos: [] } } as unknown as SessionEvent)
-    expect(() => definition.apply(expired, replace(1, 3))).toThrow('no adjacent shadow price')
+    expect(definition.view(definition.apply(expired, replace(1, 3))).messageTokens)
+      .toBe(definition.view(state).messageTokens)
     // The armed claim prices exactly the next event's matching replacement.
     const armed = definition.apply(state, meter(1, 3, 8))
     expect(definition.view(definition.apply(armed, replace(1, 3))).messageTokens)

packages/llm/token-meter/tests/token-usage-projection.spec.ts

@@ -419,6 +419,25 @@ describe('contextPressure session projection', () => {
     expect(compacted.projectedTokens).toBeLessThan(beforeCompaction!)
   })
 
+  it('folds a replacement without a claim at zero', async () => {
+    const { ctx, session } = await harness()
+    const question = appendUser(session, 'a question from an unmetered log')
+    startStep(session, 1, 1)
+    usageChunk(session, { inputTokens: 100, outputTokens: 1 }, 1, 1)
+    session.append('step/end', { turn: 1, step: 1 })
+    const before = pressure(ctx, session)
+
+    session.append('user/message', createUserMessage({
+      content: [{ type: 'text', text: 'summary without a preceding claim' }],
+      source: { kind: 'plugin', plugin: 'test' },
+    }), {
+      surfaceOp: { op: 'replace', start: question, end: question },
+      sourceEventSeqs: [question],
+    })
+
+    expect(pressure(ctx, session)).toEqual(before)
+  })
+
   it('clamps a projection that heuristic error drove below zero', async () => {
     const { ctx, session } = await harness()
     recordContext(session, 'large', 128_000)