fix(gate): history call site uses empirically-grounded cpt=2.5 (was 4)

Production telemetry (N=10 events with history_reason="not_profitable",
2026-05-18..05-20) shows body-level chars-per-token on rejected history
events: median 1.09, max 1.10. The gate at cpt=4 was estimating text as
3.7× cheaper than reality on every history-collapse decision in the
sampled traffic — same shape of bug as the slab path fixed in c36ec3f.

Why this matters more than the slab fix:
  • The slab is cached (cache_create×1.25 once, cache_read×0.10 after).
    Slab compression saves bytes mostly on cold misses + 2k tok/warm hit.
  • History content is NOT cached. Once turns leave the cache breakpoint
    budget (4 max per request), every subsequent request pays 100% for
    them. A long agentic session re-bills the same closed-prefix turns
    on every turn — exactly the workload Variant C was designed to
    compress, and exactly the path the gate was silently rejecting.

Apply same fix as c36ec3f: HISTORY_CHARS_PER_TOKEN=2.5 (upper bound of
empirical data, identical safety property to SLAB_CHARS_PER_TOKEN since
history content shape ⊆ slab content shape in tool-heavy sessions).

Host can still override via TransformOptions.charsPerToken. The
override-gate `o.charsPerToken !== CHARS_PER_TOKEN` means unspecified
requests (DEFAULTS.charsPerToken=4) get the empirical cpt while explicit
non-default host values win — same intentional pattern as the slab path.

Test: regression fixture pinning a 14-turn × 1500 chars/turn conversation
where:
  • cpt=4 path → 'not_profitable' rejection (image cost 5k > text 4k)
  • cpt=2.5 path → 'collapsed' with 10 turns merged (text 6.4k > image 5k)
Counterfactual half uses cpt=4.5 to bypass the override-gate (passing
exactly 4 collides with the DEFAULTS sentinel).

273 tests pass, typecheck clean, build clean.
This commit is contained in:
teamchong
2026-05-20 07:52:58 -04:00
parent c36ec3fd39
commit b563751cf6
2 changed files with 78 additions and 1 deletions
+37 -1
View File
@@ -177,6 +177,35 @@ const CHARS_PER_TOKEN = 4;
* conservative bias where shape isn't known a priori. */ * conservative bias where shape isn't known a priori. */
const SLAB_CHARS_PER_TOKEN = 2.5; const SLAB_CHARS_PER_TOKEN = 2.5;
/** Empirical chars-per-token for the *history compression* path.
*
* Source: same N=354 production telemetry as SLAB_CHARS_PER_TOKEN, plus
* a tighter sub-sample of N=10 events where the history collapse was
* rejected as `not_profitable` (history_reason="not_profitable" in
* events.jsonl, 2026-05-18 → 2026-05-20):
* body cpt on rejected history events — min 1.08, median 1.09, MAX 1.10.
*
* History content shape: collapsed-prefix turns from the conversation =
* past tool_use args (JSON, very dense), tool_result blocks (variable),
* assistant/user text (prose, less dense). In tool-heavy Claude Code
* sessions the JSON dominates and the observed body-cpt sits at ~1.09 —
* even denser than the slab path's 1.17 median.
*
* Why 2.5 (not lower despite tighter empirical data): a chat-heavy
* session with predominantly user/assistant prose could push history
* content closer to the English-prose cpt~4 range. We have no telemetry
* for those sessions yet (chat-only Claude Code is rare). 2.5 = slab's
* empirically-justified upper bound, applied here to keep both call
* sites symmetric and to absorb chat-shape uncertainty. If a real
* chat-only session ever drives observed cpt above 2.5 we'll see it
* in the dashboard before any net-loss compression lands.
*
* Safety: at cpt=2.5 the gate's text-token estimate (textLen / 2.5) is
* a LOWER bound on real text cost whenever real cpt ≤ 2.5 (= every
* history sample we've ever observed). Image-cost-vs-estimated-text
* passing therefore implies image-cost-vs-real-text passing. */
const HISTORY_CHARS_PER_TOKEN = 2.5;
/** Empirical per-image cost at numCols=1. Source: dashboard.ts measurement /** Empirical per-image cost at numCols=1. Source: dashboard.ts measurement
* trace. Kept here as a constant rather than imported from dashboard.ts * trace. Kept here as a constant rather than imported from dashboard.ts
* to keep `src/core/` free of dashboard imports — that's a one-way edge. */ * to keep `src/core/` free of dashboard imports — that's a one-way edge. */
@@ -1670,8 +1699,15 @@ export async function transformRequest(
// the renderer will use. History is single-col; pinning numCols=1 // the renderer will use. History is single-col; pinning numCols=1
// here makes the gate decision identical to the renderer's image // here makes the gate decision identical to the renderer's image
// count after wrapping. // count after wrapping.
// History cpt is empirically ~1.09 (N=10 rejected-events sample) — JSON-
// dense like the slab, so use the same conservative upper-bound cpt
// baked into HISTORY_CHARS_PER_TOKEN=2.5. Host override (o.charsPerToken)
// wins if the dashboard ever feeds back a live empirical fit.
const historyCpt = o.charsPerToken !== undefined && o.charsPerToken !== CHARS_PER_TOKEN
? o.charsPerToken
: HISTORY_CHARS_PER_TOKEN;
const historyProfitable = (text: string, cols: number): boolean => const historyProfitable = (text: string, cols: number): boolean =>
isCompressionProfitable(text, cols, undefined, 1, o.charsPerToken); isCompressionProfitable(text, cols, undefined, 1, historyCpt);
const { messages: newMessages, info: histInfo } = await collapseHistory( const { messages: newMessages, info: histInfo } = await collapseHistory(
req.messages, req.messages,
historyProfitable, historyProfitable,
+41
View File
@@ -448,6 +448,47 @@ describe('transformRequest history compression (always-on)', () => {
expect(info.historyReason).toBe('no_closed_prefix'); expect(info.historyReason).toBe('no_closed_prefix');
}); });
it('borderline fixture: collapses with built-in HISTORY_CHARS_PER_TOKEN=2.5 where cpt=4 would reject', async () => {
// Empirical 2026-05-20: N=10 production "history rejected as
// not_profitable" events have body cpt 1.08-1.10. The gate using
// cpt=4 was estimating text as 3.7× cheaper than reality and
// rejecting compressions that real billing would have approved.
//
// This fixture pins the wiring of HISTORY_CHARS_PER_TOKEN=2.5 into the
// transformRequest → collapseHistory gate. It sits in the band where
// image cost (5000 tok ≈ 2 images) is:
// • > text-tokens at cpt=4 → REJECT under stale default
// • < text-tokens at cpt=2.5 → ACCEPT under the empirical fix
//
// 10 collapsed turns × 1500 chars/turn ≈ 15-17k chars after framing.
// Single-col packs ≈ 14k chars/image → 2 images. At cpt=4: text=4k tok
// vs image=5k tok → reject. At cpt=2.5: text=6.4k tok > image=5k tok
// → accept with ≈1.4k tok headroom.
const msgs: Message[] = [];
for (let i = 0; i < 14; i++) {
const body = `turn ${i}: ` + bigPlain(1500);
msgs.push(i % 2 === 0 ? usr(body) : asst(body));
}
const { info } = await transformRequest(mkBody(msgs, bigPlain(80_000)));
// Under HISTORY_CHARS_PER_TOKEN=2.5 this fixture collapses cleanly.
expect(info.historyReason).toBe('collapsed');
expect(info.collapsedTurns).toBe(10);
// Counterfactual: a host override slightly above 4 (= "English-prose
// worse than the static default") forces the gate back into rejection
// territory. We use 4.5 rather than 4 because passing exactly the
// DEFAULTS value (4) collides with the `!== CHARS_PER_TOKEN` branch
// and gets re-rewritten to HISTORY_CHARS_PER_TOKEN — by design, so
// unspecified-host requests still get the empirical cpt. 4.5 bypasses
// that override gate and confirms the fix actually flows through the
// cpt argument, not some other side-effect.
const stale = await transformRequest(mkBody(msgs, bigPlain(80_000)), {
charsPerToken: 4.5,
});
expect(stale.info.historyReason).toBe('not_profitable');
expect(stale.info.collapsedTurns).toBeUndefined();
});
it('history-image blocks carry NO cache_control (conservative first-cut)', async () => { it('history-image blocks carry NO cache_control (conservative first-cut)', async () => {
const msgs: Message[] = []; const msgs: Message[] = [];
for (let i = 0; i < 14; i++) { for (let i = 0; i < 14; i++) {