mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
fix(gate): history call site uses empirically-grounded cpt=2.5 (was 4)
Production telemetry (N=10 events with history_reason="not_profitable", 2026-05-18..05-20) shows body-level chars-per-token on rejected history events: median 1.09, max 1.10. The gate at cpt=4 was estimating text as 3.7× cheaper than reality on every history-collapse decision in the sampled traffic — same shape of bug as the slab path fixed inc36ec3f. Why this matters more than the slab fix: • The slab is cached (cache_create×1.25 once, cache_read×0.10 after). Slab compression saves bytes mostly on cold misses + 2k tok/warm hit. • History content is NOT cached. Once turns leave the cache breakpoint budget (4 max per request), every subsequent request pays 100% for them. A long agentic session re-bills the same closed-prefix turns on every turn — exactly the workload Variant C was designed to compress, and exactly the path the gate was silently rejecting. Apply same fix asc36ec3f: HISTORY_CHARS_PER_TOKEN=2.5 (upper bound of empirical data, identical safety property to SLAB_CHARS_PER_TOKEN since history content shape ⊆ slab content shape in tool-heavy sessions). Host can still override via TransformOptions.charsPerToken. The override-gate `o.charsPerToken !== CHARS_PER_TOKEN` means unspecified requests (DEFAULTS.charsPerToken=4) get the empirical cpt while explicit non-default host values win — same intentional pattern as the slab path. Test: regression fixture pinning a 14-turn × 1500 chars/turn conversation where: • cpt=4 path → 'not_profitable' rejection (image cost 5k > text 4k) • cpt=2.5 path → 'collapsed' with 10 turns merged (text 6.4k > image 5k) Counterfactual half uses cpt=4.5 to bypass the override-gate (passing exactly 4 collides with the DEFAULTS sentinel). 273 tests pass, typecheck clean, build clean.
This commit is contained in:
+37
-1
@@ -177,6 +177,35 @@ const CHARS_PER_TOKEN = 4;
|
|||||||
* conservative bias where shape isn't known a priori. */
|
* conservative bias where shape isn't known a priori. */
|
||||||
const SLAB_CHARS_PER_TOKEN = 2.5;
|
const SLAB_CHARS_PER_TOKEN = 2.5;
|
||||||
|
|
||||||
|
/** Empirical chars-per-token for the *history compression* path.
|
||||||
|
*
|
||||||
|
* Source: same N=354 production telemetry as SLAB_CHARS_PER_TOKEN, plus
|
||||||
|
* a tighter sub-sample of N=10 events where the history collapse was
|
||||||
|
* rejected as `not_profitable` (history_reason="not_profitable" in
|
||||||
|
* events.jsonl, 2026-05-18 → 2026-05-20):
|
||||||
|
* body cpt on rejected history events — min 1.08, median 1.09, MAX 1.10.
|
||||||
|
*
|
||||||
|
* History content shape: collapsed-prefix turns from the conversation =
|
||||||
|
* past tool_use args (JSON, very dense), tool_result blocks (variable),
|
||||||
|
* assistant/user text (prose, less dense). In tool-heavy Claude Code
|
||||||
|
* sessions the JSON dominates and the observed body-cpt sits at ~1.09 —
|
||||||
|
* even denser than the slab path's 1.17 median.
|
||||||
|
*
|
||||||
|
* Why 2.5 (not lower despite tighter empirical data): a chat-heavy
|
||||||
|
* session with predominantly user/assistant prose could push history
|
||||||
|
* content closer to the English-prose cpt~4 range. We have no telemetry
|
||||||
|
* for those sessions yet (chat-only Claude Code is rare). 2.5 = slab's
|
||||||
|
* empirically-justified upper bound, applied here to keep both call
|
||||||
|
* sites symmetric and to absorb chat-shape uncertainty. If a real
|
||||||
|
* chat-only session ever drives observed cpt above 2.5 we'll see it
|
||||||
|
* in the dashboard before any net-loss compression lands.
|
||||||
|
*
|
||||||
|
* Safety: at cpt=2.5 the gate's text-token estimate (textLen / 2.5) is
|
||||||
|
* a LOWER bound on real text cost whenever real cpt ≤ 2.5 (= every
|
||||||
|
* history sample we've ever observed). Image-cost-vs-estimated-text
|
||||||
|
* passing therefore implies image-cost-vs-real-text passing. */
|
||||||
|
const HISTORY_CHARS_PER_TOKEN = 2.5;
|
||||||
|
|
||||||
/** Empirical per-image cost at numCols=1. Source: dashboard.ts measurement
|
/** Empirical per-image cost at numCols=1. Source: dashboard.ts measurement
|
||||||
* trace. Kept here as a constant rather than imported from dashboard.ts
|
* trace. Kept here as a constant rather than imported from dashboard.ts
|
||||||
* to keep `src/core/` free of dashboard imports — that's a one-way edge. */
|
* to keep `src/core/` free of dashboard imports — that's a one-way edge. */
|
||||||
@@ -1670,8 +1699,15 @@ export async function transformRequest(
|
|||||||
// the renderer will use. History is single-col; pinning numCols=1
|
// the renderer will use. History is single-col; pinning numCols=1
|
||||||
// here makes the gate decision identical to the renderer's image
|
// here makes the gate decision identical to the renderer's image
|
||||||
// count after wrapping.
|
// count after wrapping.
|
||||||
|
// History cpt is empirically ~1.09 (N=10 rejected-events sample) — JSON-
|
||||||
|
// dense like the slab, so use the same conservative upper-bound cpt
|
||||||
|
// baked into HISTORY_CHARS_PER_TOKEN=2.5. Host override (o.charsPerToken)
|
||||||
|
// wins if the dashboard ever feeds back a live empirical fit.
|
||||||
|
const historyCpt = o.charsPerToken !== undefined && o.charsPerToken !== CHARS_PER_TOKEN
|
||||||
|
? o.charsPerToken
|
||||||
|
: HISTORY_CHARS_PER_TOKEN;
|
||||||
const historyProfitable = (text: string, cols: number): boolean =>
|
const historyProfitable = (text: string, cols: number): boolean =>
|
||||||
isCompressionProfitable(text, cols, undefined, 1, o.charsPerToken);
|
isCompressionProfitable(text, cols, undefined, 1, historyCpt);
|
||||||
const { messages: newMessages, info: histInfo } = await collapseHistory(
|
const { messages: newMessages, info: histInfo } = await collapseHistory(
|
||||||
req.messages,
|
req.messages,
|
||||||
historyProfitable,
|
historyProfitable,
|
||||||
|
|||||||
@@ -448,6 +448,47 @@ describe('transformRequest history compression (always-on)', () => {
|
|||||||
expect(info.historyReason).toBe('no_closed_prefix');
|
expect(info.historyReason).toBe('no_closed_prefix');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('borderline fixture: collapses with built-in HISTORY_CHARS_PER_TOKEN=2.5 where cpt=4 would reject', async () => {
|
||||||
|
// Empirical 2026-05-20: N=10 production "history rejected as
|
||||||
|
// not_profitable" events have body cpt 1.08-1.10. The gate using
|
||||||
|
// cpt=4 was estimating text as 3.7× cheaper than reality and
|
||||||
|
// rejecting compressions that real billing would have approved.
|
||||||
|
//
|
||||||
|
// This fixture pins the wiring of HISTORY_CHARS_PER_TOKEN=2.5 into the
|
||||||
|
// transformRequest → collapseHistory gate. It sits in the band where
|
||||||
|
// image cost (5000 tok ≈ 2 images) is:
|
||||||
|
// • > text-tokens at cpt=4 → REJECT under stale default
|
||||||
|
// • < text-tokens at cpt=2.5 → ACCEPT under the empirical fix
|
||||||
|
//
|
||||||
|
// 10 collapsed turns × 1500 chars/turn ≈ 15-17k chars after framing.
|
||||||
|
// Single-col packs ≈ 14k chars/image → 2 images. At cpt=4: text=4k tok
|
||||||
|
// vs image=5k tok → reject. At cpt=2.5: text=6.4k tok > image=5k tok
|
||||||
|
// → accept with ≈1.4k tok headroom.
|
||||||
|
const msgs: Message[] = [];
|
||||||
|
for (let i = 0; i < 14; i++) {
|
||||||
|
const body = `turn ${i}: ` + bigPlain(1500);
|
||||||
|
msgs.push(i % 2 === 0 ? usr(body) : asst(body));
|
||||||
|
}
|
||||||
|
const { info } = await transformRequest(mkBody(msgs, bigPlain(80_000)));
|
||||||
|
// Under HISTORY_CHARS_PER_TOKEN=2.5 this fixture collapses cleanly.
|
||||||
|
expect(info.historyReason).toBe('collapsed');
|
||||||
|
expect(info.collapsedTurns).toBe(10);
|
||||||
|
|
||||||
|
// Counterfactual: a host override slightly above 4 (= "English-prose
|
||||||
|
// worse than the static default") forces the gate back into rejection
|
||||||
|
// territory. We use 4.5 rather than 4 because passing exactly the
|
||||||
|
// DEFAULTS value (4) collides with the `!== CHARS_PER_TOKEN` branch
|
||||||
|
// and gets re-rewritten to HISTORY_CHARS_PER_TOKEN — by design, so
|
||||||
|
// unspecified-host requests still get the empirical cpt. 4.5 bypasses
|
||||||
|
// that override gate and confirms the fix actually flows through the
|
||||||
|
// cpt argument, not some other side-effect.
|
||||||
|
const stale = await transformRequest(mkBody(msgs, bigPlain(80_000)), {
|
||||||
|
charsPerToken: 4.5,
|
||||||
|
});
|
||||||
|
expect(stale.info.historyReason).toBe('not_profitable');
|
||||||
|
expect(stale.info.collapsedTurns).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
it('history-image blocks carry NO cache_control (conservative first-cut)', async () => {
|
it('history-image blocks carry NO cache_control (conservative first-cut)', async () => {
|
||||||
const msgs: Message[] = [];
|
const msgs: Message[] = [];
|
||||||
for (let i = 0; i < 14; i++) {
|
for (let i = 0; i < 14; i++) {
|
||||||
|
|||||||
Reference in New Issue
Block a user