Merge remote-tracking branch 'origin/compact-tool-pairing' into token-meter-service

docs/core-data-structures/session.md

@@ -200,7 +200,7 @@ export interface SurfaceNode {
 }
 ```
 
-`SurfaceNode` is positional state, not durable identity. A replacement can remove a caller-retained node or make a copied `next` stale; consumers that cross a surface mutation validate membership and resolve successors from `Session.surface.nodes`. `SurfaceManager.replaceGeneration` increments for each replacement so incremental consumers can distinguish pure tail growth from a rewrite.
+`SurfaceNode` is positional state, not durable identity. A replacement can remove a caller-retained node or make a copied `next` stale; consumers that cross a surface mutation validate membership and answer positional queries from `Session.surface.nodes`. `SurfaceManager.replaceGeneration` increments for each replacement so incremental consumers can distinguish pure tail growth from a rewrite.
 
 ### `SurfaceFoldReplacement` and `SurfaceFoldResult` — a complete surface replay
 

docs/rfc/implemented/feature/2026-06-18-compaction-capability-seam.md

@@ -52,7 +52,7 @@ The loop derives messages once after `agent/pre-step`. Running before `step/star
 
 Auto-compaction fires before **every** step, not once per turn. This is **load-bearing for runaway-turn survival**: a tool-heavy ReAct turn appends an `assistant/message` + a `tool/result` per step, so the surface grows *within* a turn. A single turn can grow past the window on its own (a "runaway turn") — and the only moment to rescue it before the next model call overflows is the next step's `pre-step` checkpoint. Gating compaction to a turn's first step (or, worse, retaining the whole in-flight turn verbatim) re-opens exactly the hole compaction exists to close: the harness would die when compaction is most needed.
 
-`compactIfNeeded` retains the smallest tail of whole surface units whose estimated size reaches `retainTokens` and compacts older nodes. A unit is a complete closed step or one no-step message. If the token cutoff lands inside a step, retention expands until the cut is tool-pairing balanced. Balance is checked on surface order, not log sequence, because replacement summaries have new sequence numbers at old surface positions. `dsh-compact` exports the before/after edge helpers; their per-session cache folds only appended surface-tail nodes while `replaceGeneration` is unchanged, does no event reads for log-only growth, and rebuilds current membership, positional successors, and balances after replacement. `compactRegion` rejects boundaries that split a tool call from its result. The in-flight turn receives no special retention.
+`compactIfNeeded` retains the smallest tail of whole surface units whose estimated size reaches `retainTokens` and compacts older nodes. A unit is a complete closed step or one no-step message. If the token cutoff lands inside a step, retention expands until the cut is tool-pairing balanced. Balance is checked on surface order, not log sequence, because replacement summaries have new sequence numbers at old surface positions. `dsh-compact` exports the before/after edge helpers; their per-session cache folds only appended surface-tail nodes while `replaceGeneration` is unchanged, does no event reads for log-only growth, and rebuilds current membership and balances after replacement. `compactRegion` rejects boundaries that split a tool call from its result. The in-flight turn receives no special retention.
 
 A runaway turn thus compacts exactly like any other history: its early *closed* steps get summarized while its recent steps stay verbatim. When the only compactable content left is an un-splittable open tail step (its tool-calls have no results yet), compaction declines (`null`) and retries once that step closes.
 
@@ -113,7 +113,7 @@ Two failure paths, both documented:
 - **Packages**: `packages/compact/compact` supplies the interface and `compact-basic` supplies the backend. `packages/llm/token-meter` owns replay-aware measurement independently. The consumer tier is deferred.
 - **New loop seam**: `agent/pre-step` (`@mode serial`) declared in `dsh-agent` and emitted by `dsh-agent-loop` after system assembly and before `step/start`. This is a documented change to the loop — `docs/architecture.md` records it and the generated cordis catalog carries its signature.
 - **`SessionEventMap`** gains `compact/start` / `compact/summary` / `compact/end` by declaration merging (merge-extensible); `SurfaceEventType` is **not** touched. These are session events, not cordis `Events`, so the event-taxonomy gate needs no entry.
-- **`dsh-compact`** owns `toolPairingBalancedBefore(session, node)` and `toolPairingBalancedAfter(session, node)`, the cached surface-edge checks that `compactRegion` and `compactIfNeeded` use to avoid splitting a tool-call/result pair. The cache validates current membership by seq and resolves after-edges from its positional successor map instead of trusting a caller-retained `node.next`; stale or missing seqs and orphan results reject. `dsh-session` continues to own the surface `replace` operation, positional nodes, and rewrite generation.
+- **`dsh-compact`** owns `toolPairingBalancedBefore(session, node)` and `toolPairingBalancedAfter(session, node)`, the cached surface-edge checks that `compactRegion` and `compactIfNeeded` use to avoid splitting a tool-call/result pair. The cache validates current membership by seq and answers both edges from one per-cut balance sequence instead of trusting a caller-retained `node.next`; stale or missing seqs and orphan results reject. `dsh-session` continues to own the surface `replace` operation, positional nodes, and rewrite generation.
 - **`dsh-invariants`** drops its `surface replace: start must be <= end` assertion: a head-anchored compaction lands a high-seq replacement node at an older range's *position*, so `start > end` numerically is normal and valid (the range is positional, validated by the surface's `indexOf` checks that remain). The turn-enclosure invariant is reused unchanged.
 - **Wiring**: `examples/coding-agent/cordis.yml` loads zero-config `dsh-token-meter` before `dsh-compact-basic`; the service-wide window and compact defaults make the pair usable without repeated numeric policy.
 

docs/rfc/proposed/simplification/2026-07-12-simplify-session-log-representation.md

@@ -6,7 +6,7 @@ Status: proposed
 
 The session log maintains two representations that cost more machinery than their consumers require: a pseudo-linked surface and custom request-header deltas.
 
-`SurfaceManager` stores the same order in an array, a seq map, and mutable `prev`/`next` links. Production never reads `prev`; compact's sole `next` read is the successor of an array position. Replacement already uses `indexOf`, so the links do not make its dominant operation constant-time. A seq array with linear replacement lookup has the same asymptotic replacement cost and one representation to validate.
+`SurfaceManager` stores the same order in an array, a seq map, and mutable `prev`/`next` links. Production never reads either link: compact's tool-pairing balance answers from per-cut balances cached in surface order. Replacement already uses `indexOf`, so the links do not make its dominant operation constant-time. A seq array with linear replacement lookup has the same asymptotic replacement cost and one representation to validate.
 
 The request-header subsystem implements a custom system/tool delta codec and transmission-decision layer even though its contract says deltas are an encoding optimization, not a reconstructability requirement. Retaining the initial/resume full snapshot at each loop-instance boundary, then writing a canonical full `request/header` whenever that instance's assembled header changes, preserves replay while deleting `SystemDelta`, `ToolsDelta`, round-trip fallback, and the durable `request/header-delta` variant. Codec-only vocabulary disappears with the codec, not because its individual arms were invalid.
 

packages/compact/compact/README.md

@@ -25,9 +25,9 @@ Both methods are **abstract** — the backend owns trigger policy, retention, ev
 
 ## Tool-pairing boundaries
 
-The interface exports `toolPairingBalancedBefore(session, node)` and `toolPairingBalancedAfter(session, node)` for snapping and validating compaction edges. A safe edge has no unanswered assistant tool call crossing it. Each helper validates the node's seq against current surface membership and resolves the trailing edge from its cached positional successor, so a stale caller-held `node.next` cannot choose the cut.
+The interface exports `toolPairingBalancedBefore(session, node)` and `toolPairingBalancedAfter(session, node)` for snapping and validating compaction edges. A safe edge has no unanswered assistant tool call crossing it. Each helper identifies the node by seq alone and answers from balances cached per cut in current surface order, so a stale caller-held `node.next` cannot choose the cut.
 
-The private per-session cache is keyed by `session.surface.replaceGeneration` and the processed surface-node count. An unchanged generation extends the fold with unseen tail nodes only; a log-only append with no new surface node does no event reads, while a replacement generation rebuilds current membership, successors, and balances. Missing event seqs and a `tool/result` without a preceding open call reject as corrupt surface state.
+The private per-session cache is keyed by `session.surface.replaceGeneration` and the processed surface-node count. An unchanged generation extends the fold with unseen tail nodes only; a log-only append with no new surface node does no event reads, while a replacement generation rebuilds current membership and balances. Missing event seqs and a `tool/result` without a preceding open call reject as corrupt surface state.
 
 ## Surface contract
 

packages/compact/compact/src/tool-pairing.ts

@@ -12,19 +12,21 @@ import type { Session, SessionEvent, SurfaceNode } from '@deepseek-ai/dsh-sessio
 interface BalanceCache {
   /** Surface rewrite generation this state describes. */
   generation: number
-  /** Number of surface nodes already folded into the state. */
-  processedNodes: number
-  /** Balance of the cut immediately before each current surface node. */
-  beforeSeq: Map<number, boolean>
-  /** Current positional successor of each surface node. */
-  successorBySeq: Map<number, number | null>
-  /** Unanswered tool-call count after the processed surface tail. */
-  depth: number
+  /**
+   * Balance of every surface cut in current order: a surface of N nodes has
+   * N + 1 cuts, entry `i` being the cut before node `i` and the final entry
+   * the cut after the surface tail.
+   */
+  cutBalanced: readonly boolean[]
+  /** Current surface position of each node seq, indexing {@link cutBalanced}. */
+  indexBySeq: Map<number, number>
+  /** In-progress tool-call count after the processed surface tail. */
+  inProgressToolCalls: number
 }
 
 const balanceCacheBySession = new WeakMap<Session, BalanceCache>()
 
-/** Return how one surface event changes the unanswered tool-call count. */
+/** Return how one surface event changes the in-progress tool-call count. */
 function nodeDelta(event: SessionEvent): number {
   switch (event.type) {
     case 'assistant/message':
@@ -45,61 +47,30 @@ function eventForNode(events: readonly SessionEvent[], node: SurfaceNode): Sessi
   return event
 }
 
-/** Build balance state for a complete current surface. */
-function rebuildCache(
-  session: Session,
-  nodes: readonly SurfaceNode[],
-  generation: number,
-): BalanceCache {
-  const beforeSeq = new Map<number, boolean>()
-  const successorBySeq = new Map<number, number | null>()
-  const events = session.events
-  let depth = 0
-  let previousSeq: number | undefined
-
-  for (const node of nodes) {
-    beforeSeq.set(node.seq, depth === 0)
-    successorBySeq.set(node.seq, null)
-    if (previousSeq !== undefined) successorBySeq.set(previousSeq, node.seq)
-    depth += nodeDelta(eventForNode(events, node))
-    if (depth < 0) {
-      throw new Error(`tool-pairing balance: tool/result at surface seq ${node.seq} has no matching tool-call (corrupt surface)`)
-    }
-    previousSeq = node.seq
-  }
-
-  return { generation, processedNodes: nodes.length, beforeSeq, successorBySeq, depth }
-}
-
-/** Fold a pure surface tail append into existing balance state. */
+/** Fold surface nodes not yet in the cache into its balance state. */
 function extendCache(
   session: Session,
   cache: BalanceCache,
   nodes: readonly SurfaceNode[],
 ): BalanceCache {
-  const tail = nodes.slice(cache.processedNodes)
+  const processed = cache.cutBalanced.length - 1
+  const tail = nodes.slice(processed)
   // Validate the unseen tail before mutating the live cache, so a corrupt
   // append cannot leave a partially advanced state behind.
   const events = session.events
-  const pending: Array<{ seq: number; before: boolean }> = []
-  let depth = cache.depth
+  const pendingCuts: boolean[] = []
+  let inProgressToolCalls = cache.inProgressToolCalls
   for (const node of tail) {
-    pending.push({ seq: node.seq, before: depth === 0 })
-    depth += nodeDelta(eventForNode(events, node))
-    if (depth < 0) {
+    inProgressToolCalls += nodeDelta(eventForNode(events, node))
+    if (inProgressToolCalls < 0) {
       throw new Error(`tool-pairing balance: tool/result at surface seq ${node.seq} has no matching tool-call (corrupt surface)`)
     }
+    pendingCuts.push(inProgressToolCalls === 0)
   }
 
-  let previousSeq = nodes[cache.processedNodes - 1]?.seq
-  for (const entry of pending) {
-    if (previousSeq !== undefined) cache.successorBySeq.set(previousSeq, entry.seq)
-    cache.beforeSeq.set(entry.seq, entry.before)
-    cache.successorBySeq.set(entry.seq, null)
-    previousSeq = entry.seq
-  }
-  cache.processedNodes = nodes.length
-  cache.depth = depth
+  tail.forEach((node, offset) => cache.indexBySeq.set(node.seq, processed + offset))
+  cache.cutBalanced = cache.cutBalanced.concat(pendingCuts)
+  cache.inProgressToolCalls = inProgressToolCalls
   return cache
 }
 
@@ -110,15 +81,32 @@ function balanceCache(session: Session): BalanceCache {
   const generation = surface.replaceGeneration
   const cached = balanceCacheBySession.get(session)
 
-  if (cached === undefined || cached.generation !== generation || cached.processedNodes > nodes.length) {
-    const rebuilt = rebuildCache(session, nodes, generation)
+  if (cached === undefined || cached.generation !== generation || cached.cutBalanced.length - 1 > nodes.length) {
+    // A rebuild is the same fold started from the empty-surface state, whose
+    // single leading cut is trivially balanced.
+    const rebuilt = extendCache(session, {
+      generation,
+      cutBalanced: [true],
+      indexBySeq: new Map(),
+      inProgressToolCalls: 0,
+    }, nodes)
     balanceCacheBySession.set(session, rebuilt)
     return rebuilt
   }
-  if (cached.processedNodes < nodes.length) return extendCache(session, cached, nodes)
+  if (cached.cutBalanced.length - 1 < nodes.length) return extendCache(session, cached, nodes)
   return cached
 }
 
+/** Balance of the cut at a node's position plus offset, rejecting seqs outside current membership. */
+function cutBalance(cache: BalanceCache, seq: number, offset: 0 | 1): boolean {
+  const index = cache.indexBySeq.get(seq)
+  const balanced = index === undefined ? undefined : cache.cutBalanced[index + offset]
+  if (balanced === undefined) {
+    throw new Error(`tool-pairing balance: surface seq ${seq} not found`)
+  }
+  return balanced
+}
+
 /**
  * Whether the cut immediately before a current surface node is tool-pairing balanced.
  * @param session - session whose surface is checked.
@@ -128,12 +116,7 @@ function balanceCache(session: Session): BalanceCache {
  * matching log event, or a tool result has no preceding open call.
  */
 export function toolPairingBalancedBefore(session: Session, node: SurfaceNode): boolean {
-  const cache = balanceCache(session)
-  const balanced = cache.beforeSeq.get(node.seq)
-  if (balanced === undefined) {
-    throw new Error(`tool-pairing balance: surface seq ${node.seq} not found`)
-  }
-  return balanced
+  return cutBalance(balanceCache(session), node.seq, 0)
 }
 
 /**
@@ -145,16 +128,5 @@ export function toolPairingBalancedBefore(session: Session, node: SurfaceNode):
  * matching log event, or a tool result has no preceding open call.
  */
 export function toolPairingBalancedAfter(session: Session, node: SurfaceNode): boolean {
-  const cache = balanceCache(session)
-  const successor = cache.successorBySeq.get(node.seq)
-  if (successor === undefined) {
-    throw new Error(`tool-pairing balance: surface seq ${node.seq} not found`)
-  }
-  if (successor === null) return cache.depth === 0
-  // Current membership and positional successors are cache-owned. A caller may
-  // retain a node across surface changes, so its mutable-looking `next` field is
-  // never authoritative for this query.
-  // The successor map and balance map are committed together.
-  // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
-  return cache.beforeSeq.get(successor)!
+  return cutBalance(balanceCache(session), node.seq, 1)
 }

packages/compact/compact/tests/tool-pairing.spec.ts

@@ -131,7 +131,7 @@ describe('tool-pairing surface identity', () => {
     expect(() => toolPairingBalancedAfter(session, staleTail)).toThrow(/surface seq .* not found/)
   })
 
-  it('uses the cached positional successor instead of a caller node next field', () => {
+  it('ignores a caller-held node next field and answers from cached balances', () => {
     const session = closedToolStep()
     const assistant = nodeAt(session, seqOf(session, 'assistant/message'))
     expect(toolPairingBalancedAfter(session, { ...assistant, next: null })).toBe(false)