|
|
@@ -1,6 +1,7 @@
|
|
|
# Agent Note: SQLite physical chunk-row compression
|
|
|
|
|
|
Status: implemented
|
|
|
+Archived: 2026-08-30
|
|
|
|
|
|
English | [中文](2026-08-18-sqlite-physical-chunk-row-compression.zh.md)
|
|
|
|
|
|
@@ -12,15 +13,15 @@ A physical row that represents several events affects append contiguity, crash r
|
|
|
|
|
|
## Decision
|
|
|
|
|
|
-`@deepseek-ai/dsh-session-persistence-sqlite` uses the packed schema-18 implementation. It is the only SQLite persistence package and provider; the predecessor scalar layout and the temporary versioned sibling are not retained. SQLite remains an opt-in switch, while shipped default compositions continue to use JSONL. Both backends implement the same `SessionPersistence` service through `PersistenceCoordinator`, so physical packing changes neither live event delivery nor the logical session API.
|
|
|
+`@deepseek-ai/dsh-session-persistence-sqlite` uses the packed schema-20 implementation. It is the only SQLite persistence package and provider; the predecessor scalar layout and the temporary versioned sibling are not retained. SQLite remains an opt-in switch, while shipped default compositions continue to use JSONL. Both backends implement the same `SessionPersistence` service through `PersistenceCoordinator`, so physical packing changes neither live event delivery nor the logical session API.
|
|
|
|
|
|
-Schema 18 keeps ordinary ROWID tables and the composite `events(session_id, seq)` primary-key index. Scalar rows represent one logical event. Packed rows use the storage tags `text-chunks`, `reasoning-chunks`, and `tool-call-chunks`; the SQL `seq` and `time` columns hold the first logical member, and `data` holds the packed payload. Packed rows set `is_packed=1`, while scalar rows set `is_packed=0`; the explicit discriminator prevents a scalar event whose type matches a storage tag from being decoded as packed. The tags are storage vocabulary, not `SessionEventMap` members.
|
|
|
+Schema 20 keeps ordinary ROWID tables and the composite `events(session_id, seq)` primary-key index. Scalar rows represent one logical event. Packed rows use the storage tags `text-chunks`, `reasoning-chunks`, and `tool-call-chunks`; the SQL `seq` and `time` columns hold the first logical member, and `data` holds the packed payload. Packed rows set `ignorable=0` as a physical discriminator and leave `source_event_seqs` and `surface_op` as `NULL`; scalar rows use `ignorable=1` only for logical ignorable events and `NULL` otherwise. A future ignorable logical event may therefore reuse a storage-tag name without being decoded as a packed row. The tags are storage vocabulary, not `SessionEventMap` members.
|
|
|
|
|
|
-SQLite owns chunk encoding and validation inside the schema-18 package. Exact-field whitelisting means unknown fields, surface metadata, incompatible chunk identity, sequence gaps, and unsafe timestamps remain scalar rather than losing information. One packed row represents at most 1,024 events and 1 MiB of uncompressed UTF-8 `data`; the encoder partitions longer runs, and the decoder rejects rows outside those format limits.
|
|
|
+SQLite owns chunk encoding and validation inside the schema-20 package. Exact-field whitelisting means unknown fields, surface metadata, incompatible chunk identity, sequence gaps, and unsafe timestamps remain scalar rather than losing information. One packed row represents at most 1,024 events and 1 MiB of uncompressed UTF-8 `data`; the encoder partitions longer runs, and the decoder rejects rows outside those format limits.
|
|
|
|
|
|
The `data` column accepts `TEXT` or `BLOB`. Serialized values below 4 KiB remain text. At or above the threshold, the writer uses Zstandard level 3 and retains the frame only when it is smaller than the text; the reader decompresses the blob before strict UTF-8 decoding and JSON parsing. The fixed moderate level and threshold limit frame overhead and synchronous CPU work while capturing the repeated payloads that dominate retained bytes.
|
|
|
|
|
|
-`source_event_seqs` remains the complete ordered list of earlier events cited by a surface node, including every streamed chunk behind an assembled assistant message. Schema 18 stores the first sequence as an unsigned varint and every subsequent signed difference as a ZigZag varint. This preserves arbitrary order and every sequence while exploiting the overwhelmingly consecutive lists produced by streaming. An empty list is an empty non-null blob, distinct from absent provenance.
|
|
|
+`source_event_seqs` remains the complete ordered list of earlier events cited by a surface node, including every streamed chunk behind an assembled assistant message. Schema 20 stores the first sequence as an unsigned varint and every subsequent signed difference as a ZigZag varint. This preserves arbitrary order and every sequence while exploiting the overwhelmingly consecutive lists produced by streaming. An empty list is an empty non-null blob, distinct from absent provenance.
|
|
|
|
|
|
### Transactional append packing
|
|
|
|
|
|
@@ -32,11 +33,11 @@ Normal append never deletes or replaces an earlier event row. Fixed write-behind
|
|
|
|
|
|
Full reads decode each physical row as one all-or-nothing logical span and validate contiguous logical sequences. A reverse pass identifies the last valid `turn/end` without retaining a second decoded copy of the full physical scan; the forward pass decodes one row at a time into the required logical result. A malformed row or gap before that committed boundary is corruption; a malformed final physical row becomes the opaque repair marker at that row's base sequence. Recovery re-reads and validates that marker while holding the write lock, then deletes the whole physical row and any later rows before binding synthetic closers as scalar events. A stale repair cannot delete a newer writer's valid suffix.
|
|
|
|
|
|
-`readFrom(id, fromSeq)` examines packed predecessors only within the maximum schema-18 row span, then reads from the earliest candidate that may contain `fromSeq`. The decoder filters reconstructed members below `fromSeq`, so a suffix may begin inside a packed row without parsing an unrelated earlier scalar row. Reading from that candidate also exposes an overlapping scalar row to contiguity validation instead of letting it hide the packed member. Packed data exceeding the uncompressed format byte limit rejects before JSON parsing.
|
|
|
+`readFrom(id, fromSeq)` examines packed predecessors only within the maximum schema-20 row span, then reads from the earliest candidate that may contain `fromSeq`. The decoder filters reconstructed members below `fromSeq`, so a suffix may begin inside a packed row without parsing an unrelated earlier scalar row. Reading from that candidate also exposes an overlapping scalar row to contiguity validation instead of letting it hide the packed member. Packed data exceeding the uncompressed format byte limit rejects before JSON parsing.
|
|
|
|
|
|
### Schema ownership
|
|
|
|
|
|
-A pristine database initializes at schema 18. Older physical schemas, foreign application identities, non-pristine unversioned databases, and incompatible schema objects reject; the pre-release package supplies no migration. Every connection disables trusted schemas and memory-mapped I/O before inspecting durable schema, then reads both settings back. After selecting and verifying the journal mode, the provider pins `synchronous=FULL` and verifies it so SQLite build defaults cannot weaken committed-append durability. Package code loads every statement and fixed pragma from closed-name `.sql` resources and binds runtime values as parameters.
|
|
|
+A pristine database initializes at schema 20. Older physical schemas, foreign application identities, non-pristine unversioned databases, and incompatible schema objects reject; the pre-release package supplies no migration. Every connection disables trusted schemas and memory-mapped I/O before inspecting durable schema, then reads both settings back. After selecting and verifying the journal mode, the provider pins `synchronous=FULL` and verifies it so SQLite build defaults cannot weaken committed-append durability. Package code loads every statement and fixed pragma from closed-name `.sql` resources and binds runtime values as parameters.
|
|
|
|
|
|
### Physical-write regression
|
|
|
|
|
|
@@ -58,11 +59,11 @@ The repository regression guard writes 1,000 streamed deltas in 40-event durable
|
|
|
|
|
|
**Compress every payload.** Rejected because small independent Zstandard frames add headers and synchronous CPU work while losing the cross-record dictionary opportunity of a whole-file stream. On the 105-session comparison corpus, a threshold sweep produced 75.01 MB at 4 KiB, versus 93.87 MB at 16 KiB and 60.92 MB at 1 KiB. The writer fixes level 3 rather than inheriting a library default, matching the moderate level used by [Codex cold-rollout compression](https://github.com/openai/codex/blob/main/codex-rs/rollout/src/compression.rs) while retaining independent row access.
|
|
|
|
|
|
-The final frozen comparison used 105 sessions, 2,507,860 logical events, 512-event durable batches, three independent builds per backend, and three read passes per build. SQLite used 75.01 MB, wrote in 8.58 s, read complete sessions at 3.95/21.58 ms p50/p95, read 50-event tails at 0.253/0.378 ms, and forked every session in 13.10 s. Zstandard JSONL used 30.65 MB and measured 28.21 s, 4.49/23.36 ms, 10.58/80.90 ms, and 14.48 s. The predecessor scalar SQLite layout used 709.57 MB and measured 10.64 s, 9.02/69.16 ms, 0.189/0.293 ms, and 19.30 s. The packed layout is 89.4% smaller than the predecessor, writes 19.4% faster, improves complete-read p50/p95 by 56.2%/68.8%, and reduces 2,507,860 physical event rows to 65,810. Scalar tail-50 and list micro-latency are lower, but the packed provider remains materially faster than JSONL on those paths and wins the dominant size, write, full-read, and fork costs. The 4 KiB threshold is the accepted balance rather than a strict dominance claim. This comparison measured schema 17; schema 18 retains the chunk codec and bounds but changes the row discriminator, so the exact size and timing values remain schema-17 evidence until schema 18 is remeasured.
|
|
|
+The final frozen comparison used 105 sessions, 2,507,860 logical events, 512-event durable batches, three independent builds per backend, and three read passes per build. SQLite used 75.01 MB, wrote in 8.58 s, read complete sessions at 3.95/21.58 ms p50/p95, read 50-event tails at 0.253/0.378 ms, and forked every session in 13.10 s. Zstandard JSONL used 30.65 MB and measured 28.21 s, 4.49/23.36 ms, 10.58/80.90 ms, and 14.48 s. The predecessor scalar SQLite layout used 709.57 MB and measured 10.64 s, 9.02/69.16 ms, 0.189/0.293 ms, and 19.30 s. The packed layout is 89.4% smaller than the predecessor, writes 19.4% faster, improves complete-read p50/p95 by 56.2%/68.8%, and reduces 2,507,860 physical event rows to 65,810. Scalar tail-50 and list micro-latency are lower, but the packed provider remains materially faster than JSONL on those paths and wins the dominant size, write, full-read, and fork costs. The 4 KiB threshold is the accepted balance rather than a strict dominance claim. This comparison measured schema 17; its exact values are evidence for the original packed-row decision, not schema-20 measurements. The [persistence latency and page-size decision](2026-08-25-persistence-latency-and-page-size.md) owns the schema-19 benchmark and current encoding refinements.
|
|
|
|
|
|
**Store packed payloads under the logical `assistant/chunk` type.** Rejected because payload heuristics make malformed rows ambiguous and couple physical decoding to future logical payload fields. Explicit tags fail loudly.
|
|
|
|
|
|
-**Store `SessionHeader` fields in an extensible metadata blob.** Rejected for schema 18 because `agentPreset` is a typed core resume invariant shared by JSONL and SQLite, not provider extension metadata. Persisting validated core fields directly keeps both backends aligned; an untyped catch-all would add another compatibility mechanism without a current producer. Revisit this only with a core-owned, namespaced `SessionHeader` extension protocol implemented by every backend.
|
|
|
+**Store `SessionHeader` fields in an extensible metadata blob.** Rejected for schema 20 because `agentPreset` is a typed core resume invariant shared by JSONL and SQLite, not provider extension metadata. Persisting validated core fields directly keeps both backends aligned; an untyped catch-all would add another compatibility mechanism without a current producer. Revisit this only with a core-owned, namespaced `SessionHeader` extension protocol implemented by every backend.
|
|
|
|
|
|
**Expose compression rules through configuration or a live registry.** Rejected because same-version databases must be readable independently of runtime topology. The codec is modular source code, but the durable rule set is fixed by schema version.
|
|
|
|