test: real-shape regression fixtures from production events.jsonl

Fragility #3 from end-of-session audit. The previous gate tests pinned
isCompressionProfitable(slab, 100, 2, 2.5) === true against 'A'.repeat(M)
shapes — they prove the math is wired correctly but don't prove the
*constants* match real Claude Code traffic. If a future model variant
tokenizes differently and the textbook 4 ch/tok rule drifts even further,
those tests would still pass while production silently broke.

Anonymized 6 production shapes from events.jsonl (2026-05-19..2026-05-20):
- PRODUCTION_SLAB_161K (multi-col, 52 chars/row, 11 ings/141 lines/2 cols)
- PRODUCTION_SLAB_135K_DENSE (neuline-heavy, 19 chars/row, 24 ings)
- PRODUCTION_SLAB_169K_HEAVY (very dense, 16 chars/row — actually rejected)
- BELOW_MIN_CHARS_TINY (142 chars, pre-filter)
- BELOW_MIN_CHARS_BORDERLINE (1123 chars, pre-filter)
- HISTORY_COLLAPSED_LONG_SESSION (collapsed path captured)

Each fixture stores its real captured decision plus the shape parameters
(origChars, numCols, approxCharsPerRow). synthesizeText() pads 'A'.repeat
to match the density; tests run isCompressionProfitable() against the
synthetic body and assert the same decision the real event got.

\#\# Notable: 135K_DENSE divergence

The real 135K neuline-heavy event was ACCEPTED (compressed=true) but the
synthetic 'A'.repeat(19) reconstruction REJECTS even at cpt=2.5. Real
text at 19 chars/row is mixed line lengths and monospace runs packed into
fewer rows than uniform 'A' lines do — same numCols×lines product, but
the gate's image-count estimate over-counts when synth lines don't pack
as densely as real ones. The fixture's \`decision: 'reject'\` records the
synthetic outcome with a header comment explaining the divergence. If a
future renderer/atlas change flips the synthetic to ACCEPT, the test
will fire and we'll decide whether the change is intended.

\#\# Export changes

SLAB_CHARS_PER_TOKEN and HISTORY_CHARS_PER_TOKEN exported from transform.ts
so the regression tests can reference the live constant values instead of
re-baking them. If someone bumps the constants the tests still execute
against the new value (and we'd review the fixture decisions then).

\#\# Tests

8 files / 280 tests pass, typecheck clean, build clean. 7 new it() blocks
cover the 5 captured shapes + the synthetic divergence note.
This commit is contained in:
teamchong
2026-05-20 10:50:16 -04:00
parent 3c759ed662
commit 440ca7f448
3 changed files with 295 additions and 2 deletions
+2 -2
View File
@@ -175,7 +175,7 @@ const CHARS_PER_TOKEN = 4;
* tool_result content have unknown shape (could be raw English prose
* with cpt~4). Leaving those at CHARS_PER_TOKEN=4 preserves the
* conservative bias where shape isn't known a priori. */
const SLAB_CHARS_PER_TOKEN = 2.5;
export const SLAB_CHARS_PER_TOKEN = 2.5;
/** Empirical chars-per-token for the *history compression* path.
*
@@ -204,7 +204,7 @@ const SLAB_CHARS_PER_TOKEN = 2.5;
* a LOWER bound on real text cost whenever real cpt ≤ 2.5 (= every
* history sample we've ever observed). Image-cost-vs-estimated-text
* passing therefore implies image-cost-vs-real-text passing. */
const HISTORY_CHARS_PER_TOKEN = 2.5;
export const HISTORY_CHARS_PER_TOKEN = 2.5;
/** Empirical per-image cost at numCols=1. Source: dashboard.ts measurement
* trace. Kept here as a constant rather than imported from dashboard.ts