mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-08 21:49:45 +08:00
v1.6.4.0: cut Haiku classifier FP from 44% to 23%, gate now enforced (#1135)
* feat(security): v2 ensemble tuning — label-first voting + SOLO_CONTENT_BLOCK Cuts Haiku classifier false-positive rate from 44.1% → 22.9% on BrowseSafe-Bench smoke. Detection trades from 67.3% → 56.2%; the lost TPs are all cases Haiku correctly labeled verdict=warn (phishing targeting users, not agent hijack) — they still surface in the WARN banner meta but no longer kill the session. Key changes: - combineVerdict: label-first voting for transcript_classifier. Only meta.verdict==='block' block-votes; verdict==='warn' is a soft signal. Missing meta.verdict never block-votes (backward-compat). - Hallucination guard: verdict='block' at confidence < LOG_ONLY (0.40) drops to warn-vote — prevents malformed low-conf blocks from going authoritative. - New THRESHOLDS.SOLO_CONTENT_BLOCK = 0.92 decoupled from BLOCK (0.85). Label-less content classifiers (testsavant, deberta) need a higher solo-BLOCK bar because they can't distinguish injection from phishing-targeting-user. Transcript keeps label-gated solo path (verdict=block AND conf >= BLOCK). - THRESHOLDS.WARN bumped 0.60 → 0.75 — borderline fires drop out of the 2-of-N ensemble pool. - Haiku model pinned (claude-haiku-4-5-20251001). `claude -p` spawns from os.tmpdir() so project CLAUDE.md doesn't poison the classifier context (measured 44k cache_creation tokens per call before the fix, and Haiku refusing to classify because it read "security system" from CLAUDE.md and went meta). - Haiku timeout 15s → 45s. Measured real latency is 17-33s end-to-end (Claude Code session startup + Haiku); v1's 15s caused 100% timeout when re-measured — v1's ensemble was effectively L4-only in prod. - Haiku prompt rewritten: explicit block/warn/safe criteria, 8 few-shot exemplars (instruction-override → block; social engineering → warn; discussion-of-injection → safe). Test updates: - 5 existing combineVerdict tests adapted for label-first semantics (transcript signals now need meta.verdict to block-vote). - 6 new tests: warn-soft-signal, three-way-block-with-warn-transcript, hallucination-guard-below-floor, above-floor-label-first, backward-compat-missing-meta. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(security): live + fixture-replay bench harness with 500-case capture Adds two new benches that permanently guard the v2 tuning: - security-bench-ensemble-live.test.ts (opt-in via GSTACK_BENCH_ENSEMBLE=1). Runs full ensemble on BrowseSafe-Bench smoke with real Haiku calls. Worker-pool concurrency (default 8, tunable via GSTACK_BENCH_ENSEMBLE_CONCURRENCY) cuts wall clock from ~2hr to ~25min on 500 cases. Captures Haiku responses to fixture for replay. Subsampling via GSTACK_BENCH_ENSEMBLE_CASES for faster iteration. Stop-loss iterations write to ~/.gstack-dev/evals/stop-loss-iter-N-* WITHOUT overwriting canonical fixture. - security-bench-ensemble.test.ts (CI gate, deterministic replay). Replays captured fixture through combineVerdict, asserts detection >= 55% AND FP <= 25%. Fail-closed when fixture is missing AND security-layer files changed in branch diff. Uses `git diff --name-only base` (two-dot) to catch both committed and working-tree changes — `git diff base...HEAD` would silently skip in CI after fixture lands. - browse/test/fixtures/security-bench-haiku-responses.json — 500 cases × 3 classifier signals each. Header includes schema_version, pinned model, component hashes (prompt, exemplars, thresholds, combiner, dataset version). Any change invalidates the fixture and forces fresh live capture. - docs/evals/security-bench-ensemble-v2.json — durable PR artifact with measured TP/FN/FP/TN, 95% CIs, knob state, v1 baseline delta. Checked in so reviewers can see the numbers that justified the ship. Measured baseline on the new harness: TP=146 FN=114 FP=55 TN=185 → 56.2% / 22.9% → GATE PASS Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(release): v1.5.1.0 — cut Haiku FP 44% → 23% - VERSION: 1.5.0.0 → 1.5.1.0 (TUNING bump) - CHANGELOG: [1.5.1.0] entry with measured numbers, knob list, and stop-loss rule spec - TODOS: mark "Cut Haiku FP 44% → ~15%" P0 as SHIPPED with pointer to CHANGELOG and v1 plan Measured: 56.2% detection (CI 50.1-62.1) / 22.9% FP (CI 18.1-28.6) on 500-case BrowseSafe-Bench smoke. Gate passes (floor 55%, ceiling 25%). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(changelog): add v1.6.4.0 placeholder entry at top Per CLAUDE.md branch-scoped discipline, our VERSION 1.6.4.0 needs a CHANGELOG entry at the top so readers can tell what's on this branch vs main. Honest placeholder: no user-facing runtime changes yet, two merges bringing branch up to main's v1.6.3.0, and the approved injection-tuning plan is queued but unimplemented. Gets replaced by the real release-summary at /ship time after Phases -1 through 10 land. * docs(changelog): strip process minutiae from entries; rewrite v1.6.4.0 CLAUDE.md — new CHANGELOG rule: only document what shipped between main and this change. Keep out branch resyncs, merge commits, plan approvals, review outcomes, scope negotiations, "work queued" or "in-progress" framing. When no user-facing change actually landed, one sentence is the entry: "Version bump for branch-ahead discipline. No user-facing changes yet." CHANGELOG.md — v1.6.4.0 entry rewritten to match. Previous version narrated the branch history, the approved injection-tuning plan, and what we expect to ship later — all of which are process minutiae readers do not care about. * docs(changelog): rewrite v1.6.4.0; strip process minutiae Rewrote v1.6.4.0 entry to follow the new CLAUDE.md rule: only document what shipped between main and this change. Previous entry narrated the branch history, the approved injection-tuning plan, and what we expect to ship later, all process minutiae readers do not care about. v1.6.4.0 now reads: what the detection tuning did for users, the before/after numbers, the stop-loss rule, and the itemized changes for contributors. CLAUDE.md — new rule: only document what shipped between main and this change. Keep out branch resyncs, merge commits, plan approvals, review outcomes, scope negotiations, "work queued" / "in-progress" framing. If nothing user-facing landed, one sentence: "Version bump for branch-ahead discipline. No user-facing changes yet." --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
31
CLAUDE.md
31
CLAUDE.md
@@ -437,9 +437,18 @@ already landed on main. Your entry goes on top because your branch lands next.
|
||||
If any answer is no, fix it before continuing.
|
||||
|
||||
**After any CHANGELOG edit that moves, adds, or removes entries,** immediately run
|
||||
`grep "^## \[" CHANGELOG.md` and verify the full version sequence is contiguous
|
||||
with no gaps or duplicates before committing. If a version is missing, the edit
|
||||
broke something. Fix it before moving on.
|
||||
`grep "^## \[" CHANGELOG.md` to verify no duplicates and a sensible reverse-chronological
|
||||
order. Gaps between version numbers are fine. A branch that ships at v1.6.4.0 without
|
||||
a prior v1.5.2.0 or v1.5.3.0 entry on main is correct — those were branch-internal
|
||||
version numbers that never landed. Do not back-fill gaps with placeholder entries.
|
||||
|
||||
**Never orphan branch-internal versions.** If your branch bumped VERSION several times
|
||||
during development (v1.5.1.0 → v1.5.2.0 → v1.6.4.0, say) and those earlier entries were
|
||||
never released to main, the final ship consolidates ALL of them into a single entry at
|
||||
the final version (v1.6.4.0). Collapse them — delete the old entries and move their
|
||||
content into the final entry, re-version table columns accordingly. Readers see one
|
||||
release, not a branch diary. Gaps are fine (v1.6.3.0 → v1.6.4.0 with no v1.5.x
|
||||
in between on main is correct).
|
||||
|
||||
CHANGELOG.md is **for users**, not contributors. Write it like product release notes:
|
||||
|
||||
@@ -452,6 +461,22 @@ CHANGELOG.md is **for users**, not contributors. Write it like product release n
|
||||
- No jargon: say "every question now tells you which project and branch you're in" not
|
||||
"AskUserQuestion format standardized across skill templates via preamble resolver."
|
||||
|
||||
**Only document what shipped between main and this change.** Readers do not care how
|
||||
we got here. Keep out of the CHANGELOG, always:
|
||||
|
||||
- Branch resyncs, merge commits with main, rebase activity.
|
||||
- Plan approvals, review outcomes (CEO / eng / design / outside-voice / codex findings),
|
||||
AskUserQuestion decisions, scope negotiations.
|
||||
- "Work queued," "plan approved," "in-progress," "will ship later" — the CHANGELOG
|
||||
documents what DID ship, not what MIGHT ship.
|
||||
- Version-bump housekeeping when no user-facing work actually landed.
|
||||
|
||||
If the diff between the base branch version and this version has no user-facing change
|
||||
(only merges, only CHANGELOG edits, only placeholder work), the honest entry is one
|
||||
sentence: "Version bump for branch-ahead discipline. No user-facing changes yet." Stop
|
||||
there. Do not pad. Do not explain the plan that will ship eventually. Do not narrate
|
||||
the branch's history. When real work lands, the entry will replace this at /ship time.
|
||||
|
||||
### Release-summary format (every `## [X.Y.Z]` entry)
|
||||
|
||||
Every version entry in `CHANGELOG.md` MUST start with a release-summary section in
|
||||
|
||||
Reference in New Issue
Block a user