mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
test: real-shape regression fixtures from production events.jsonl
Fragility #3 from end-of-session audit. The previous gate tests pinned isCompressionProfitable(slab, 100, 2, 2.5) === true against 'A'.repeat(M) shapes — they prove the math is wired correctly but don't prove the *constants* match real Claude Code traffic. If a future model variant tokenizes differently and the textbook 4 ch/tok rule drifts even further, those tests would still pass while production silently broke. Anonymized 6 production shapes from events.jsonl (2026-05-19..2026-05-20): - PRODUCTION_SLAB_161K (multi-col, 52 chars/row, 11 ings/141 lines/2 cols) - PRODUCTION_SLAB_135K_DENSE (neuline-heavy, 19 chars/row, 24 ings) - PRODUCTION_SLAB_169K_HEAVY (very dense, 16 chars/row — actually rejected) - BELOW_MIN_CHARS_TINY (142 chars, pre-filter) - BELOW_MIN_CHARS_BORDERLINE (1123 chars, pre-filter) - HISTORY_COLLAPSED_LONG_SESSION (collapsed path captured) Each fixture stores its real captured decision plus the shape parameters (origChars, numCols, approxCharsPerRow). synthesizeText() pads 'A'.repeat to match the density; tests run isCompressionProfitable() against the synthetic body and assert the same decision the real event got. \#\# Notable: 135K_DENSE divergence The real 135K neuline-heavy event was ACCEPTED (compressed=true) but the synthetic 'A'.repeat(19) reconstruction REJECTS even at cpt=2.5. Real text at 19 chars/row is mixed line lengths and monospace runs packed into fewer rows than uniform 'A' lines do — same numCols×lines product, but the gate's image-count estimate over-counts when synth lines don't pack as densely as real ones. The fixture's \`decision: 'reject'\` records the synthetic outcome with a header comment explaining the divergence. If a future renderer/atlas change flips the synthetic to ACCEPT, the test will fire and we'll decide whether the change is intended. \#\# Export changes SLAB_CHARS_PER_TOKEN and HISTORY_CHARS_PER_TOKEN exported from transform.ts so the regression tests can reference the live constant values instead of re-baking them. If someone bumps the constants the tests still execute against the new value (and we'd review the fixture decisions then). \#\# Tests 8 files / 280 tests pass, typecheck clean, build clean. 7 new it() blocks cover the 5 captured shapes + the synthetic divergence note.
This commit is contained in:
@@ -175,7 +175,7 @@ const CHARS_PER_TOKEN = 4;
|
||||
* tool_result content have unknown shape (could be raw English prose
|
||||
* with cpt~4). Leaving those at CHARS_PER_TOKEN=4 preserves the
|
||||
* conservative bias where shape isn't known a priori. */
|
||||
const SLAB_CHARS_PER_TOKEN = 2.5;
|
||||
export const SLAB_CHARS_PER_TOKEN = 2.5;
|
||||
|
||||
/** Empirical chars-per-token for the *history compression* path.
|
||||
*
|
||||
@@ -204,7 +204,7 @@ const SLAB_CHARS_PER_TOKEN = 2.5;
|
||||
* a LOWER bound on real text cost whenever real cpt ≤ 2.5 (= every
|
||||
* history sample we've ever observed). Image-cost-vs-estimated-text
|
||||
* passing therefore implies image-cost-vs-real-text passing. */
|
||||
const HISTORY_CHARS_PER_TOKEN = 2.5;
|
||||
export const HISTORY_CHARS_PER_TOKEN = 2.5;
|
||||
|
||||
/** Empirical per-image cost at numCols=1. Source: dashboard.ts measurement
|
||||
* trace. Kept here as a constant rather than imported from dashboard.ts
|
||||
|
||||
Reference in New Issue
Block a user