mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
teamchong
c1d111a083
Adds the data collection half of the adaptive chars-per-token plan
without touching the gate. Every isCompressionProfitable call site
now credits its chars to a named bucket on TransformInfo, and those
counts flow through toTrackEvent into the persisted JSONL.
transform.ts:
- New BucketName union: 'static_slab' | 'reminder' |
'tool_result_{structured,log,prose}' | 'history'
- bumpBucket() helper lazily allocates info.bucketChars
- toolResultBucket() routes by classifyContent shape so the post-
compaction text we credit matches the shape the gate evaluated
- Bump sites: static slab (combinedRaw), reminder (reminderRaw),
tool_result single-string (innerRaw), tool_result multi-block
per TextBlock (innerTextRaw), history (collapsedChars)
- Surfaces historyTextChars separately so history's no-collapse
rejections still contribute a denominator
tracker.ts:
- TrackEvent gains bucket_chars?: Partial<Record<BucketName, number>>
and history_text_chars?: number
- toTrackEvent copies them when populated, drops empty bucket maps
so happy-path events stay small
tests/tracker.test.ts:
- Asserts snake_case field names and nested shape land correctly
- Asserts absent buckets and pass-through events stay clean
No gate logic changed. CHARS_PER_TOKEN / SLAB_CHARS_PER_TOKEN /
HISTORY_CHARS_PER_TOKEN still drive isCompressionProfitable. Phase 2
will fit per-bucket slopes from the captured events and wire a
bucket-aware override.
docs/ADAPTIVE_CPT_PLAN.md: full plan + empirical basis (slab-only
regression on 87 events recovered cpt≈1.50 vs the baked 2.5
constant — 40% on non-slab compressions ship today).
Verified: tsc --noEmit clean; 290/290 tests pass (+2 new);
build emits dist/node.js.