|
|
**Compress every payload.** Rejected because small independent Zstandard frames add headers and synchronous CPU work while losing the cross-record dictionary opportunity of a whole-file stream. On the 105-session comparison corpus, a threshold sweep produced 75.01 MB at 4 KiB, versus 93.87 MB at 16 KiB and 60.92 MB at 1 KiB. The writer fixes level 3 rather than inheriting a library default, matching the moderate level used by [Codex cold-rollout compression](https://github.com/openai/codex/blob/main/codex-rs/rollout/src/compression.rs) while retaining independent row access.
|
|
|
-The final frozen comparison used 105 sessions, 2,507,860 logical events, 512-event durable batches, three independent builds per backend, and three read passes per build. SQLite used 75.01 MB, wrote in 8.58 s, read complete sessions at 3.95/21.58 ms p50/p95, read 50-event tails at 0.253/0.378 ms, and forked every session in 13.10 s. Zstandard JSONL used 30.65 MB and measured 28.21 s, 4.49/23.36 ms, 10.58/80.90 ms, and 14.48 s. The predecessor scalar SQLite layout used 709.57 MB and measured 10.64 s, 9.02/69.16 ms, 0.189/0.293 ms, and 19.30 s. The packed layout is 89.4% smaller than the predecessor, writes 19.4% faster, improves complete-read p50/p95 by 56.2%/68.8%, and reduces 2,507,860 physical event rows to 65,810. Scalar tail-50 and list micro-latency are lower, but the packed provider remains materially faster than JSONL on those paths and wins the dominant size, write, full-read, and fork costs. The 4 KiB threshold is the accepted balance rather than a strict dominance claim.
|
|
|
+The final frozen comparison used 105 sessions, 2,507,860 logical events, 512-event durable batches, three independent builds per backend, and three read passes per build. SQLite used 75.01 MB, wrote in 8.58 s, read complete sessions at 3.95/21.58 ms p50/p95, read 50-event tails at 0.253/0.378 ms, and forked every session in 13.10 s. Zstandard JSONL used 30.65 MB and measured 28.21 s, 4.49/23.36 ms, 10.58/80.90 ms, and 14.48 s. The predecessor scalar SQLite layout used 709.57 MB and measured 10.64 s, 9.02/69.16 ms, 0.189/0.293 ms, and 19.30 s. The packed layout is 89.4% smaller than the predecessor, writes 19.4% faster, improves complete-read p50/p95 by 56.2%/68.8%, and reduces 2,507,860 physical event rows to 65,810. Scalar tail-50 and list micro-latency are lower, but the packed provider remains materially faster than JSONL on those paths and wins the dominant size, write, full-read, and fork costs. The 4 KiB threshold is the accepted balance rather than a strict dominance claim. This comparison measured schema 17; its exact values are evidence for the original packed-row decision, not schema-20 measurements. The [persistence latency and page-size decision](2026-08-25-persistence-latency-and-page-size.md) owns the schema-19 benchmark and current encoding refinements.
|
Tianyi Cui