|
|
@@ -4,13 +4,13 @@ Status: implemented
|
|
|
|
|
|
English | [中文](2026-06-21-bounded-llm-request-recovery.zh.md)
|
|
|
|
|
|
-The [per-provider request retry policy](../feature/2026-07-24-provider-retry-policies.md) extends this foundation with exact-provider configuration and an explicit unbounded mode. This note continues to own structured failure facts, the closed-step recovery boundary, normal mode's transient defaults, visible single attempts, and durable retry status. [Terminal LLM stream failures](2026-07-29-terminal-llm-stream-failures.md) supersedes its thrown-error identity and stream-sidecar mechanism.
|
|
|
+The [per-provider request retry policy](../feature/2026-07-24-provider-retry-policies.md) extends this foundation with exact-provider configuration and an explicit unbounded mode. This note continues to own structured failure facts, the failed-attempt recovery boundary, normal mode's transient defaults, visible single attempts, and durable retry status. [Terminal LLM stream failures](2026-07-29-terminal-llm-stream-failures.md) supersedes its thrown-error identity and stream-sidecar mechanism.
|
|
|
|
|
|
## Problem
|
|
|
|
|
|
Provider adapters can fail by throwing during dispatch or iteration or by ending with `finish { kind: 'error' | 'aborted' }`. The final adapter boundary normalizes thrown values to that terminal finish protocol before `dsh-agent-loop` receives them; middleware and result-processing defects remain thrown. The loop offers a terminal model-request failure to `agent/request-error`. An unhandled failure is terminal; a handling listener repairs policy-owned state, returns `{ kind: 'retry' }`, and stops waterfall delegation. The [retry-action decision](../simplification/2026-07-27-request-error-retry-action.md) owns this return contract.
|
|
|
|
|
|
-That boundary is already safe for another request attempt. Raw `assistant/chunk` events carry the failed `turn` and `step`, message derivation ignores them unless a successful `assistant/message` cites them, tool calls are dispatched only after a successful terminal finish and assembly, and a retry opens a new numbered turn from the durable log. The harness therefore does not need a second response lifecycle or tentative-output protocol to keep two attempts separate.
|
|
|
+That boundary is already safe for another request attempt. Raw `assistant/chunk` events carry the failed `turn` and `step`, message derivation ignores them unless a successful `assistant/message` cites them, tool calls are dispatched only after a successful terminal finish and assembly, and a retry reconstructs its next attempt from the durable log. The harness therefore does not need a second response lifecycle or tentative-output protocol to keep two attempts separate.
|
|
|
|
|
|
The prior boundary left three narrower gaps.
|
|
|
|
|
|
@@ -52,7 +52,7 @@ The shared transient-code set is intentionally small: adapter mappings for `RATE
|
|
|
|
|
|
`@deepseek-ai/dsh-llm-retry` is a function plugin that listens to `agent/request-error`. It introduces no service or new loop branch; the agent-loop package changes only the data carried through its existing failed-step recovery control flow.
|
|
|
|
|
|
-The `agent/request-error` waterfall carries the current `LlmFailure`, an immutable list of prior failures that authorized retry turns in the consecutive recovery sequence, and the serving registration's immutable retry policy. The loop transports but does not interpret that policy, owns the consecutive failure history, and clears it after a successful model request. Normal `dsh-llm-retry` policy counts durable retry records scheduled by the same exact-provider policy, while `dsh-compaction-basic` keeps its own context-overflow budget. Alternating transient and context-overflow failures therefore consume their owning finite budgets independently; the maximum request count is one plus the sum of the loaded finite budgets.
|
|
|
+The `agent/request-error` waterfall carries the current `LlmFailure`, an immutable list of prior failures that authorized retries in the consecutive recovery sequence, and the serving registration's immutable retry policy. The loop transports but does not interpret that policy, owns the consecutive failure history, and clears it after a successful model request. Normal `dsh-llm-retry` policy counts durable retry records scheduled by the same exact-provider policy, while `dsh-compaction-basic` keeps its own context-overflow budget. Alternating transient and context-overflow failures therefore consume their owning finite budgets independently; the maximum request count is one plus the sum of the loaded finite budgets.
|
|
|
|
|
|
The [provider-policy decision](../feature/2026-07-24-provider-retry-policies.md) owns the current configuration shape. Provider adapters register their nested `retryPolicy`; omission uses normal defaults: two transient retries, a 500 millisecond initial delay, a 10 second delay cap, 10 percent jitter, and the five transient codes above. The count and delay bounds match the conservative edge of the inspected implementations: [OpenCode uses two request retries with 500 ms/10 s bounds](https://github.com/anomalyco/opencode/blob/9976269ab1accfc9f9dc98a4a688c516934de422/%70ackages/llm/src/route/executor.ts#L36-L39), [Pi separates three agent-level retries from provider retries and defaults provider retries to zero](https://github.com/earendil-works/pi/blob/3da591ab74ab9ab407e72ed882600b2c851fae21/%70ackages/coding-agent/docs/settings.md#L139-L147), and [Codex uses finite request/stream budgets plus a five-minute idle timeout](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/model-provider-info/src/lib.rs#L25-L33). Ten percent follows [Codex's bounded jitter](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/codex-client/src/retry.rs#L40-L47).
|
|
|
|
|
|
@@ -68,7 +68,7 @@ The agent-spine demo bundle loads the plugin so the shared stdio/TUI, one-shot C
|
|
|
|
|
|
### Make one layer own visible attempts
|
|
|
|
|
|
-Adapters perform one provider request per `stream()` call. The pi-ai adapter removes public `maxRetries` and `maxRetryDelayMs` profile fields and disables library retries; the hand-written adapter keeps its current single-attempt behavior. This prevents an SDK budget from multiplying the agent budget and ensures every transient retry is represented by a closed failed step plus `llm/retry`.
|
|
|
+Adapters perform one provider request per `stream()` call. The pi-ai adapter removes public `maxRetries` and `maxRetryDelayMs` profile fields and disables library retries; the hand-written adapter keeps its current single-attempt behavior. This prevents an SDK budget from multiplying the agent budget and ensures every transient retry is represented by its recorded failed attempt plus `llm/retry`.
|
|
|
|
|
|
`ctx.llm.stream()` remains the raw one-attempt waterfall. Direct callers such as compaction summarization receive the structured failure but do not gain automatic retry, because they have no agent step boundary or general durable place to separate attempts. A future direct-call consumer may justify a buffering helper that retries only before emitting a chunk; this decision adds no such helper.
|
|
|
|
|
|
@@ -82,9 +82,9 @@ Boundary tests prove termination at both actual transports. The hand-written ada
|
|
|
|
|
|
### Keep attempts separate in the existing log
|
|
|
|
|
|
-A failed attempt may leave `assistant/chunk` events in its closed step, but it never appends `assistant/message` and never dispatches a tool. A retry closes the failed turn, opens the next numbered turn, reconstructs the request from the durable surface, and produces its own chunks. UIs may render live chunks while a step is open, then mark or clear that transient view when `llm/retry` identifies the failed step or `turn/end` records failure. Web validates the complete retry payload contract, clears the failed partial at `llm/retry`, projects consecutive retry-turn events into one stable row updated to the latest attempt, and derives scheduled, started, or cancelled status from subsequent turn facts. Its countdown anchors the scheduled delay to browser receipt rather than the Host event clock, uses ceiling-rounded seconds with a one-second floor, animates only while unresolved, and keeps exact latest failure details collapsed behind the row. Retry nodes anchor their own trajectory turn even when the failed attempt has no assistant node. Message derivation continues to ignore the failed chunks, and Web applies the same projection during history rebuild so refreshing cannot resurrect discarded partials or duplicate retry rows.
|
|
|
+A failed attempt may leave `assistant/chunk` events in its step, but it never appends `assistant/message` and never dispatches a tool. A retry continues inside the failing turn and step, reconstructs the request from the durable surface, and produces its own chunks; only the final outcome closes the turn. UIs may render live chunks while a step is open, then mark or clear that transient view when `llm/retry` identifies the failed attempt or `turn/end` records failure. Web validates the complete retry payload contract, clears the failed partial at `llm/retry`, projects each producer-correlated `retryId` chain into one stable row updated to the latest attempt, and derives scheduled, started, or cancelled status from `llm/retry-started` and the owning turn and step boundaries' closure. Its countdown anchors the scheduled delay to browser receipt rather than the Host event clock, uses ceiling-rounded seconds with a one-second floor, animates only while unresolved, and keeps exact latest failure details collapsed behind the row. Retry nodes anchor their own trajectory turn even when the failed attempt has no assistant node. Message derivation continues to ignore the failed chunks, and Web applies the same projection during history rebuild so refreshing cannot resurrect discarded partials or duplicate retry rows.
|
|
|
|
|
|
-If recovery is exhausted, the final failure is stored once on `turn/end.reason` with the structured facts. Web derives one `turn-error` node at that sequence position and renders its display-safe message and optional code inline; AUTH projections replace provider copy that may echo credential fragments with `API key is invalid`, while the raw diagnostic remains in the session log. The same fold runs for live events and history replay. If transient recovery continues, `llm/retry` is the durable home for that attempt's failure and delay, so its failed turn does not also gain a terminal error row. No standalone final-error event or response-id vocabulary is added.
|
|
|
+If recovery is exhausted, the final failure is stored once on `turn/end.reason` with the structured facts. Web derives one `turn-error` node at that sequence position and renders its display-safe message and optional code inline; AUTH projections replace provider copy that may echo credential fragments with `API key is invalid`, while the raw diagnostic remains in the session log. The same fold runs for live events and history replay. While transient recovery continues, `llm/retry` is the durable home for each intermediate failure and delay; the terminal row exists only once `turn/end` records the error, and because exhausted recovery shares the failing turn, the turn's retry history never suppresses that row — the settled retry chain and the terminal error render side by side. No standalone final-error event or response-id vocabulary is added.
|
|
|
|
|
|
## Out of scope
|
|
|
|
|
|
@@ -114,15 +114,15 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
|
|
|
- Each provider adapter validates its nested retry policy at Loader startup, and `ctx.llm` captures it with the route; normal mode delegates ineligible paths and makes at most `maxRetries + 1` provider requests when no other policy applies.
|
|
|
- HMR-during-backoff tests prove disposal unregisters the listener, aborts and awaits its captured callbacks, emits no retry decision after disposal, and leaves no timer or promise alive.
|
|
|
- Pure unit tests cover transient-code selection, exponential backoff and jitter bounds, valid and over-cap `Retry-After`, exhausted budgets, deterministic timer/random hooks, and abort during backoff.
|
|
|
-- Real agent-loop tests cover failure before chunks, partial chunks then failure, thrown and in-band failures, retry to success in a new turn, exhaustion to structured `turn/end.reason`, and composition with `dsh-compaction-basic` context-overflow recovery.
|
|
|
+- Real agent-loop tests cover failure before chunks, partial chunks then failure, thrown and in-band failures, retry to success inside the same turn, exhaustion to structured `turn/end.reason`, and composition with `dsh-compaction-basic` context-overflow recovery.
|
|
|
- The partial-chunk integration test proves failed chunks remain attributed to the failed step, no assistant message or tool side effect is committed for that step, and the successful retry records its own chunk seqs and provider/model route.
|
|
|
-- The plugin-owned `llm/retry` event is non-surface, survives JSONL and SQLite round trips, is ignored by message derivation, and drives TUI and Web retraction plus scheduled-retry rendering. Client tests cover complete wire validation, clock-independent countdown, cancellation versus completed retry labels, and trajectory attribution; keyless UI snapshots cover Web scheduling and success, a real Web composition test covers partial transport failure through recovery, and ACP automation snapshots confirm that a discarded attempt stays off the wire while the recovered reply is emitted.
|
|
|
+- The plugin-owned `llm/retry` event is non-surface, survives JSONL and SQLite round trips, is ignored by message derivation, and drives TUI and Web retraction plus scheduled-retry rendering. Client tests cover complete wire validation, clock-independent countdown, cancellation versus completed retry labels, and trajectory attribution; keyless UI snapshots cover Web scheduling and success, real Web composition tests cover partial transport failure through recovery and exhausted recovery's terminal error row beside the settled retry chain, and ACP automation snapshots confirm that a discarded attempt stays off the wire while the recovered reply is emitted.
|
|
|
- Idle-watchdog tests prove the stable signal is rearmed only while `next()` is outstanding, disarmed during consumer think time and in `finally`, and classified separately from a total-call deadline and an earlier caller abort; adapter tests prove the signal stops the underlying request rather than merely detaching it.
|
|
|
- Direct `ctx.llm.stream()` callers remain single-attempt and receive the same structured failure facts.
|
|
|
|
|
|
## Consequences
|
|
|
|
|
|
-- Every retry attempt is visible as a closed failed turn plus `llm/retry`, and adapter-level single-attempt behavior prevents hidden SDK retries from multiplying policy decisions. A retry can still duplicate provider billing even when no chunk arrived; normal mode limits that risk, while explicit always mode accepts it until cancellation or success.
|
|
|
+- Every retry attempt is visible inside its owning turn as the failed attempt's chunks plus `llm/retry`, and adapter-level single-attempt behavior prevents hidden SDK retries from multiplying policy decisions. A retry can still duplicate provider billing even when no chunk arrived; normal mode limits that risk, while explicit always mode accepts it until cancellation or success.
|
|
|
- Provider SDKs may hide status or retry headers. Those adapters retain the stable facts they expose and otherwise use a coarse code rather than letting recovery policy parse fragile text.
|
|
|
- Durable retry events expand the session protocol and UI state machine. Shipping the event and its consumer together prevents an unused telemetry vocabulary, but later schema changes still require persistence and replay work.
|
|
|
- Clearing a failed step's live chunks can visibly retract output. That is preferable to presenting discarded text or partial tool JSON as committed history, and snapshots pin the transition.
|
|
|
@@ -136,3 +136,4 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
|
|
|
- [Timeout deadline library](../../implemented/architecture/2026-07-06-timeout-deadline-library.md) separates shared deadline classification from capability-owned termination.
|
|
|
- [After-call compaction pressure and context-overflow recovery](../../implemented/architecture/2026-07-10-after-call-compaction-pressure-and-overflow-recovery.md) owns the current closed-step request-recovery extension point and bounded overflow retry.
|
|
|
- [Provider-routed LLM adapters](../../implemented/architecture/2026-07-14-provider-routed-llm-adapters.md) owns explicit provider/model routing and the one-adapter-per-provider invariant.
|
|
|
+- [Terminal turn errors survive same-turn retry history](../bug-fix/2026-08-20-turn-error-survives-same-turn-retry-history.md) owns the removal of the Web retry-history suppression that hid exhausted recovery's terminal error row.
|