dashboard: split CV threshold — 5% joint / 2% constrained

Live-traffic diagnosis: typical single-session warm-cache load has the
same cached image across turns (pixel CV = 0%) AND text variance of
3-4% (verified 2026-05-19: 117k/127k/128k chars across three turns =
3.84% CV). Both columns failed the original 5%-everywhere gate, so the
constrained fallback I just shipped never actually activated — the
dashboard fell straight through to wandering stale constants.

The 5% gate exists to stop joint OLS from splitting two indistinguishable
parameters. In constrained mode β is pinned (no split to make), so we're
fitting one parameter through the origin — 1-D OLS is numerically stable
at much lower variance. 2% is conservative; even 1% would converge.

New thresholds:
  MIN_CV_JOINT       = 0.05  (unchanged — both columns must vary)
  MIN_CV_CONSTRAINED = 0.02  (NEW — text only, β-pinned through origin)

Real-world impact: live single-session traffic now activates constrained
mode after 3 compressed turns instead of waiting for cross-session
variance that may never arrive. Dashboard label flips from amber 'stale
constants' to blue 'constrained · α-only' — operator sees a measured α
within ~30 seconds of starting a fresh proxy.

Tests:
  + new: live-traffic-shape sample activates constrained mode (3.84% CV)
  + new: below-2% floor still rejects (no garbage 1-D fits)
  unchanged: 2-D collinear cases still null, joint-fit at high CV still
  returns mode='joint'
This commit is contained in:
teamchong
2026-05-19 13:34:24 -04:00
parent 9d3984fd03
commit 91c969b593
2 changed files with 42 additions and 4 deletions
+13 -3
View File
@@ -418,7 +418,17 @@ export class DashboardState {
const varP = Math.max(0, syy / n - meanP * meanP);
const cvX = meanX > 0 ? Math.sqrt(varX) / meanX : 0;
const cvP = meanP > 0 ? Math.sqrt(varP) / meanP : 0;
const MIN_CV = 0.05;
// Joint mode threshold: 5% on BOTH columns. Below that, OLS can't tell
// whether token cost is coming from text or pixels and the α/β split
// wanders ±15 pp per sample (HANDOFF "What's blocking calibration").
const MIN_CV_JOINT = 0.05;
// Constrained mode threshold: 2% on TEXT only. β is pinned (no split
// to disambiguate), so we're fitting one parameter through the origin.
// 1-D OLS is numerically stable at much lower variance — typical live
// single-session traffic has 3-4% text CV (verified 2026-05-19 live:
// 117k/127k/128k chars over three turns = 3.84% CV). The 2% floor
// still rejects "literally identical text three times in a row".
const MIN_CV_CONSTRAINED = 0.02;
// Anthropic's published per-pixel rate for the default image tiling:
// 1 token per ~750 pixels (≈ 0.001333). Used by the constrained fallback
@@ -427,7 +437,7 @@ export class DashboardState {
const ANTHROPIC_BETA = 1 / 750;
// ---- Mode 1: joint OLS (both columns vary) ----
if (cvX >= MIN_CV && cvP >= MIN_CV) {
if (cvX >= MIN_CV_JOINT && cvP >= MIN_CV_JOINT) {
const det = sxx * syy - sxy * sxy;
if (det !== 0) {
const alpha = (syy * sxt - sxy * syt) / det;
@@ -462,7 +472,7 @@ export class DashboardState {
//
// (derivation: minimize Σ(α·x r)² → dL/dα = 0 → α = Σxr / Σx²
// where r_i = tokens_i β · pixels_i, so Σx·r = sxt β·sxy.)
if (cvX < MIN_CV || sxx === 0) return null;
if (cvX < MIN_CV_CONSTRAINED || sxx === 0) return null;
const alphaConstrained = (sxt - ANTHROPIC_BETA * sxy) / sxx;
if (alphaConstrained <= 0) return null;
return {
+29 -1
View File
@@ -101,13 +101,41 @@ describe('DashboardState.fitCosts() — empirical α/β regression', () => {
it('returns null when text_chars column is constant', () => {
// Mirror case — same body shape across samples, only cache state varies.
// Without text variance, α is unidentifiable.
// Without text variance, α is unidentifiable in EITHER joint or
// constrained mode (constrained still needs text to vary).
dash.update(ev({ textChars: 130_000, pixels: 21_000_000, input: 5, cacheCreate: 500, cacheRead: 141_680 }));
dash.update(ev({ textChars: 130_000, pixels: 23_000_000, input: 5, cacheCreate: 300, cacheRead: 142_447 }));
dash.update(ev({ textChars: 130_000, pixels: 25_000_000, input: 5, cacheCreate: 200, cacheRead: 143_119 }));
expect(dash.fitCosts()).toBeNull();
});
it('constrained-mode CV threshold (2%) accommodates real single-session text variance', () => {
// Live single-session-warm-cache traffic from 2026-05-19: three turns
// hit the proxy with text chars 117,927 / 127,512 / 128,588 and the
// SAME cached image (pixels constant at 22,792,816). Text CV = 3.84%.
// Under the original joint-mode-only 5% gate this returned null and the
// dashboard fell back to wandering stale constants. The split-threshold
// version activates constrained mode here (β pinned, α measured) so
// the operator gets a measured-α answer instead of stale constants.
dash.update(ev({ textChars: 117_927, pixels: 22_792_816, input: 6, cacheCreate: 1116, cacheRead: 124_236 }));
dash.update(ev({ textChars: 127_512, pixels: 22_792_816, input: 476, cacheCreate: 6103, cacheRead: 125_352 }));
dash.update(ev({ textChars: 128_588, pixels: 22_792_816, input: 1, cacheCreate: 2432, cacheRead: 131_455 }));
const fit = dash.fitCosts();
expect(fit).not.toBeNull();
expect(fit!.mode).toBe('constrained');
expect(fit!.alpha).toBeGreaterThan(0);
});
it('still rejects below the 2% floor (literally-identical-text case)', () => {
// Below 2% text CV the residual signal is too thin to measure α even
// with β pinned. Make sure we still return null instead of producing a
// garbage 1-D fit.
dash.update(ev({ textChars: 130_000, pixels: 22_792_816, input: 6, cacheCreate: 1116, cacheRead: 124_236 }));
dash.update(ev({ textChars: 130_500, pixels: 22_792_816, input: 6, cacheCreate: 1116, cacheRead: 124_236 }));
dash.update(ev({ textChars: 130_800, pixels: 22_792_816, input: 6, cacheCreate: 1116, cacheRead: 124_236 }));
expect(dash.fitCosts()).toBeNull();
});
it('returns mode="joint" when both columns vary (full empirical fit)', () => {
// Both columns vary > 5% — joint OLS is identifiable. mode tags the
// headline number as fully empirical so the operator can distinguish