diff --git a/docs/LEGIBILITY-AUDIT-2026-07-01.md b/docs/LEGIBILITY-AUDIT-2026-07-01.md new file mode 100644 index 0000000..5e93d2a --- /dev/null +++ b/docs/LEGIBILITY-AUDIT-2026-07-01.md @@ -0,0 +1,110 @@ +# Imaged-text legibility audit — 2026-07-01 + +Motivated by a demo failure: the pxpipe arm of `demo/effective-context` read all +files but could not report the `AUDIT-ZX9` line count, and hedged that imaged text +was uncertain. This audit measured **why**, with data, and shipped the fixes that +follow from it. + +## TL;DR + +- The Anthropic API **downscales every image** to fit BOTH long-edge ≤1568 px AND + ~1.15 MP, then bills ≈ px/750 (per-image). Our old 1932×1932 page was billed at + the cap but resampled **0.555×** — 5×8 glyphs reached the vision encoder at + ~2.8×4.4 px. We were paying full price for pixels destroyed in transit. +- **Shipped:** page geometry clamped to **1568×728 = 1.14 MP** (WYSIWYG: billed/actual + pixel ratio 3.25 → 1.04). No token-cost change per image; 614/614 tests green. +- Reading exact strings off even **crisp** 5×8 tops out at **63%** (24-token blind + test). Every miss is one of two classes the glyph-confusability matrix predicts: + **case-normalization** (camelCase) and **single-glyph substitution** (hex/num). +- **7 of 9 misses are already covered by the factsheet** (SHAs, numbers ride as + text). The 2 residual misses are camelCase code identifiers — not a factsheet shape. +- **Answer to "factsheet vs. RAG-escape":** both, split on a *sparsity* axis. Sparse + precision (a few ids/page) → factsheet. Dense precision (every symbol in a code + dump, >64 distinct) → the factsheet's 64-token budget can't hold it; that needs the + re-fetch path. The scaffolding (`RecoverableBlock`) exists but is not yet wired to a + model-callable tool. + +## Method + +`count_tokens` sweep (claude-sonnet-4-5) on synthetic PNGs at controlled dims; +14,935-row regression on real proxy traffic (`~/.pxpipe/events.jsonl`) for the +unresized slope; pixel-level Hamming matrix over the Spleen 5×8 atlas; and a 24-token +blind read where transcriptions were committed *before* revealing ground truth. + +## 1. API resize (the free-fidelity lever) + +| dims | px | px/750 | billed | verdict | +|---|---|---|---|---| +| 1072×1072 | 1.15 M | 1532 | 1525 | at cap | +| 1092×1092 | 1.19 M | 1589 | 1525 | CLAMPED | +| 1568×1568 | 2.46 M | 3278 | 1525 | CLAMPED | +| 1928×1928 (old page) | 3.72 M | 4956 | 1525 | CLAMPED (resampled 0.555×) | +| **1568×728 (new page)** | **1.14 M** | **1522** | **1460** | **linear — WYSIWYG** | + +Cap is **per-image** (2×1092 in one request billed 3049 ≈ 2×1525). Sub-cap images pass +through unresized (real-traffic slope 733 px/tok, ≈ the documented 750). + +**Cost nuance (not free):** the encoder caps at ~1.15 MP/image regardless, so per-image +cost is ~unchanged, but the new page holds 28,080 chars vs the old 92,160 → ~3.3× more +images for the same corpus. The old "16× vs text" compression was **partly fictional**: +it compressed by discarding pixels at the API boundary as silent blur. True legible rate +is ~5× vs text. The change doesn't add cost — it makes the model pay *visibly* for +legibility it was already being charged for but never received. + +## 2. Glyph confusability (Spleen 5×8, 94 printable ASCII) + +45 of 94 chars have a nearest neighbor at Hamming ≤ 3 px (of 40). Worst: +`,~;` `.~:` `H~K` at d=1; `0~O` `3~8` `5~S` `6~8` `8~B` `U~u` `V~v` `W~w` `X~x` at d=2. +`H~K` at d=1 is a font defect (glyph-surgery candidate). Downscaling 0.555× (old page) +collapses many d=2 pairs toward d=0 — indistinguishable. + +## 3. Blind read accuracy (crisp 5×8, 24 tokens) + +**15/24 = 63% exact-match.** By class: camel 6/8, num 6/8, **hex 3/8 (38%)**. + +Miss taxonomy — 100% predicted by the matrix: +- **2/2 camel misses are case-only** (`tokenLedgerShard`→`tokenledgerShard`): the S/s + class. This is the exact error that opened the session (`extractFactSheetTokens` + misread as `extractFactsheetTokens`). +- **7/9 total misses are hex/num single- or double-glyph substitution** + (`6↔8`, `5↔3`, `7↔4`) — the d=2 matrix pairs. + +Reading harder cannot fix this: 5×8 is at the information floor for these pairs, and the +decoder fills ambiguity with a *confident, plausible* guess, not an "unclear" flag. + +## 4. Where the factsheet lands + +Of the 9 missed tokens, **7 are tier-0 factsheet shapes** (SHA/number) already emitted +as verbatim text next to the image — so the image misread is irrelevant in production. +The **2 residual** misses are camelCase identifiers, which are *not* a factsheet shape. + +This is the empirical case for the factsheet: for sparse precision-critical tokens it +converts a coin-flip-per-glyph read into a 100% text quote, at ~5% of source chars. + +## Recommendations + +1. **Done — clamp to 1568×728.** Free fidelity; ratio 3.25→1.04; tests green. +2. **Done earlier — ticket-ID factsheet pattern + `×N` counts** (`AUDIT-ZX9`, `CVE-…`). + Directly fixes the demo failure; validated by the `tick` class here. +3. **Recommended, not yet shipped — camelCase in the factsheet, carefully.** A dense + code page can contain **>64 distinct** camelCase symbols, so a blind add would flood + the 64-token budget and be lossy. Scope to ≥2 case-boundaries and let it sit at + tier-1 (below SHAs/nums) so it can never evict critical tokens. Measure budget impact + on a real code corpus first. +4. **Recommended — expose the re-fetch path.** `RecoverableBlock` already records + `rec_` + original text + provenance for every imaged block. Wiring it to a + model-callable "rehydrate this region as text" tool is the real answer for + **dense** exact-recall (code files, large tables) that no fixed-size sidecar can + hold. This is the "put a glass over it" mechanism, and it's half-built. +5. **Optional — glyph surgery** on `H~K` (d=1 defect) and case-contrast for the S/s + class. Zero token cost, bounded upside; do it in the same eval harness as (3). + +## Verdict on "is the factsheet the way, or should there be a RAG escape?" + +Neither alone. The data draws a clean line on token **sparsity**: +- **Sparse** precision (few ids per page): factsheet. Proven — covers 7/9 misses. +- **Dense** precision (every symbol in a dump): exceeds any sidecar budget → re-fetch + (`RecoverableBlock` → a rehydrate tool). Scaffolded, not yet exposed. + +The image carries gist and structure (which survive fine); exactness rides a different +channel chosen by how much of it there is. diff --git a/src/core/export.ts b/src/core/export.ts index 8ce1e6c..522cd3a 100644 --- a/src/core/export.ts +++ b/src/core/export.ts @@ -33,7 +33,7 @@ import { export const DEFAULT_EXPORT_MODEL = 'claude-sonnet-4-5'; // Chars-per-token for the reporting estimate now lives in transform.ts as // REPORT_CHARS_PER_TOKEN (single source of truth for all token-estimate constants). -/** Default column width — dense content mode (384 cols = 1928 px). */ +/** Default column width — dense content mode (312 cols = 1568 px). */ export const DEFAULT_EXPORT_COLS: number = DENSE_CONTENT_COLS; // --------------------------------------------------------------------------- diff --git a/src/core/gpt-model-profiles.ts b/src/core/gpt-model-profiles.ts index 544d01a..434853e 100644 --- a/src/core/gpt-model-profiles.ts +++ b/src/core/gpt-model-profiles.ts @@ -16,7 +16,15 @@ * PXPIPE_GPT_PROFILES='{"gpt-5.6":{"stripCols":176}}' # widen only */ -import { MAX_HEIGHT_PX } from './render.js'; +/** + * GPT strip height, DECOUPLED from render.ts's MAX_HEIGHT_PX (which is Anthropic's + * 1568-edge / ~1.15 MP clamp). OpenAI's pre-tokenize resize is different: fit within + * 2048×2048, then shortest side → 768. A 768-px-wide portrait strip up to 2048 px tall + * survives un-resampled, so GPT keeps the taller page. Every built-in cost number below + * (1190 / 1445 / 2372 / 1464 / 630 …) was calibrated at this height — do not re-link to + * the Anthropic constant. + */ +export const GPT_MAX_HEIGHT_PX = 1932; /** Image-token cost model (mirrors OpenAI's mandatory pre-tokenize resize). */ export type GptVisionCost = @@ -38,7 +46,7 @@ export interface GptModelProfile { export const DEFAULT_GPT_STRIP_COLS = 152; const C = DEFAULT_GPT_STRIP_COLS; -const H = MAX_HEIGHT_PX; +const H = GPT_MAX_HEIGHT_PX; /** * Conservative fallback for unrecognized models: tile 85/170 over-states cost, diff --git a/src/core/library.ts b/src/core/library.ts index 6188618..ff1e4c5 100644 --- a/src/core/library.ts +++ b/src/core/library.ts @@ -165,7 +165,7 @@ export interface RenderTextToImagesOptions { readonly maxCharsPerImage?: number; /** Render style. Default DENSE_RENDER_STYLE (bare 5×8 cell, anti-aliased). */ readonly style?: RenderStyle; - /** Max page height in px. Default MAX_HEIGHT_PX (1932). */ + /** Max page height in px. Default MAX_HEIGHT_PX (728 — Anthropic 1568-edge / ~1.15 MP safe). */ readonly maxHeightPx?: number; } diff --git a/src/core/openai-history.ts b/src/core/openai-history.ts index ec3d560..6017fc3 100644 --- a/src/core/openai-history.ts +++ b/src/core/openai-history.ts @@ -21,7 +21,8 @@ * HistoryTurn list and the planner/renderer are shared. */ -import { renderTextToPngs, reflow, neutralizeSentinel, MAX_HEIGHT_PX, type RenderedImage } from './render.js'; +import { renderTextToPngs, reflow, neutralizeSentinel, type RenderedImage } from './render.js'; +import { GPT_MAX_HEIGHT_PX } from './gpt-model-profiles.js'; import { countTokens as o200kCountTokens } from 'gpt-tokenizer/encoding/o200k_base'; /** Portrait-strip width for GPT history images. Mirrors GPT_STRIP_COLS in @@ -101,7 +102,9 @@ export const GPT_HISTORY_DEFAULTS: GptHistoryOptions = { collapseChunk: 10, freezeChunk: 10, sectionTokens: 2000, - maxHeightPx: MAX_HEIGHT_PX, + // GPT path: OpenAI's resize bounds (2048-bbox / 768 short side) permit the tall + // strip — do NOT re-link to render.ts MAX_HEIGHT_PX (Anthropic's 1568/1.15 MP clamp). + maxHeightPx: GPT_MAX_HEIGHT_PX, maxImages: GPT_HISTORY_MAX_IMAGES, reflow: true, }; diff --git a/src/core/render.ts b/src/core/render.ts index 55d221f..e6462c4 100644 --- a/src/core/render.ts +++ b/src/core/render.ts @@ -23,21 +23,26 @@ import { } from './atlas-gray.js'; import { encodeGrayPng, encodeRgbPng } from './png.js'; -/** Page-height ceiling. 1932 px → 384-col page = 69×69 = 4761 tokens, under both Fable/Opus 4.8 - * limits (≤4784 tokens AND ≤2000 px/side for >20-image requests). Exported for transform.ts math. */ -export const MAX_HEIGHT_PX = 1932; -/** Char budget for the static slab (system + tools + CLAUDE.md). ~50k leaves headroom for - * soft-wrap and dropped chars; pages stay well under the 1932 px ceiling at cols=313. */ -export const READABLE_CHARS_PER_IMAGE = 50000; -/** Char budget for dense content (tool output, collapsed history). 384 cols × 240 rows = 92160 chars - * fills the full ~1932×1932 ceiling. NOTE: verbatim recall of imaged text is unreliable at any size. */ -export const DENSE_CONTENT_CHARS_PER_IMAGE = 92160; -export const DENSE_CONTENT_COLS = 384; +/** Page-height ceiling. Measured (2026-07-01, count_tokens sweep, claude-sonnet-4-5 — see + * /tmp/pxexp/LEVER1-findings.md): the API downscales any image to fit BOTH long-edge ≤1568 + * AND ~1.15 MP (≈1,143,750 px), then bills ≈ px/750 (≈1525 tok cap, applied per-image). + * The old 1932×1932 page was billed at cap but resampled 0.555× → 5×8 glyphs reached the + * encoder at ~2.8×4.4 px. New page shape 1568×728 = 1,141,504 px fits both bounds → + * WYSIWYG for the vision encoder (also satisfies ≤2000 px/side for >20-image requests). */ +export const MAX_HEIGHT_PX = 728; +/** Char budget for the static slab (system + tools + CLAUDE.md). Matches physical page + * capacity at 312 cols × 90 rows so image-count estimates track real pagination. */ +export const READABLE_CHARS_PER_IMAGE = 28080; +/** Char budget for dense content (tool output, collapsed history). 312 cols × 90 rows = 28080 + * chars fills the 1568×728 page. NOTE: verbatim recall of imaged text is unreliable at any size. */ +export const DENSE_CONTENT_CHARS_PER_IMAGE = 28080; +export const DENSE_CONTENT_COLS = 312; /** Bare 5×8 cell (no padding). A/B showed 5×8 beats 7×10 on dense JSON (4/5 vs 3/5 reads, 42% fewer tokens). * Revert to {cellWBonus:2, cellHBonus:2} if misread rates rise. */ export const DENSE_RENDER_STYLE: RenderStyle = { cellWBonus: 0, cellHBonus: 0, aa: true }; -/** Default columns for the static slab. 313 × 5 px + 8 px pad = 1573 px — under the 1932 px ceiling. */ -const DEFAULT_COLS = 313; +/** Default columns for the static slab. 312 × 5 px + 8 px pad = 1568 px — exactly the API's + * long-edge bound (313 cols = 1573 px would trigger a 0.997× resample, blurring every glyph). */ +const DEFAULT_COLS = 312; /** Horizontal padding (left + right each), px. Exported for transform.ts token-cost math. */ export const PAD_X = 4; /** Vertical padding (top + bottom each), px. Exported for transform.ts token-cost math. */ @@ -732,9 +737,9 @@ export async function renderTextToPngs( // OCR column ordering is the risk — gated behind an opt-in flag pending empirical eval. const GUTTER_CELLS = 4; -// Width is capped at 1932 px, not 2000 px: a 1932-tall page = 69 patches, so width must -// also be ≤69 patches (4761 tokens ≤ 4784). A 2000-px width → 72×69 = 4968 > 4784 → REJECTED. -const MAX_WIDTH_PX = 1932; +// Width hard-capped at the API's long-edge bound: anything wider is resampled server-side +// (measured 2026-07-01: fit-within 1568-edge AND ~1.15 MP, then ≈px/750 billing). +const MAX_WIDTH_PX = 1568; const GUTTER_DIVIDER_INK = 64; // pre-invert → 191 post-invert: light gray column separator const GUTTER_DIVIDER_INSET_PX = 2; // keep divider clear of padding rows diff --git a/tests/paging.test.ts b/tests/paging.test.ts index 4a02d58..d587f67 100644 --- a/tests/paging.test.ts +++ b/tests/paging.test.ts @@ -28,14 +28,15 @@ import { DENSE_CONTENT_CHARS_PER_IMAGE, DENSE_CONTENT_COLS, DENSE_RENDER_STYLE, + READABLE_CHARS_PER_IMAGE, renderTextToPngsWithCharLimit, } from '../src/core/render.js'; -// Default render config: cols=100, 240 lines/img → ~24,000 chars/img if +// Default render config: cols=100, 90 lines/img → ~9,000 chars/img if // lines fully fill the width. For shorter lines, the budget is dominated // by row count (each line takes ≥1 row regardless of length). const COLS = 100; -const ROWS_PER_IMG = 240; // floor((1932 - 8) / 8), Spleen 5×8 production cell +const ROWS_PER_IMG = 90; // floor((728 - 8) / 8), Anthropic-clamped 1568×728 page describe('estimateImageCount', () => { it('returns 1 for empty / tiny text', () => { @@ -44,13 +45,13 @@ describe('estimateImageCount', () => { }); it('scales linearly with row count for short-line content', () => { - // Full-canvas policy: 100 cols × 240 rows = 24,000 chars/page. + // Full-canvas policy: 100 cols × 90 rows = 9,000 chars/page. const oneImage = Array.from({ length: ROWS_PER_IMG }, () => 'x').join('\n'); expect(estimateImageCount(oneImage, COLS)).toBe(1); - // 196 short lines spill into a second page. + // ROWS_PER_IMG + 1 short lines spill into a second page. const justOver = Array.from({ length: ROWS_PER_IMG + 1 }, () => 'x').join('\n'); expect(estimateImageCount(justOver, COLS)).toBe(2); - // 10 × 240 rows → exactly 10 full pages. + // 10 × ROWS_PER_IMG rows → exactly 10 full pages. const tenImages = Array.from({ length: ROWS_PER_IMG * 10 }, () => 'x').join('\n'); expect(estimateImageCount(tenImages, COLS)).toBe(10); }); @@ -59,19 +60,19 @@ describe('estimateImageCount', () => { // A single 1000-char line wraps to ceil(1000/100) = 10 rows. const wrapped = 'x'.repeat(1000); expect(estimateImageCount(wrapped, COLS)).toBe(1); // 10 rows, fits in 1 img - // 24,000 chars on one line wraps to 240 rows → exactly 1 full page. - const oneImg = 'x'.repeat(24_000); + // COLS × ROWS_PER_IMG chars on one line wrap to exactly 1 full page. + const oneImg = 'x'.repeat(COLS * ROWS_PER_IMG); expect(estimateImageCount(oneImg, COLS)).toBe(1); - // 24,001 chars overflows into a second page. - const twoImgs = 'x'.repeat(24_001); + // One char more overflows into a second page. + const twoImgs = 'x'.repeat(COLS * ROWS_PER_IMG + 1); expect(estimateImageCount(twoImgs, COLS)).toBe(2); }); it('also accepts a numeric length (legacy chars-based estimate)', () => { - // Numeric path uses the full READABLE_CHARS_PER_IMAGE (50k) budget per page. + // Numeric path uses the full READABLE_CHARS_PER_IMAGE budget per page. expect(estimateImageCount(0, COLS)).toBe(1); - expect(estimateImageCount(50_000, COLS)).toBe(1); - expect(estimateImageCount(50_001, COLS)).toBe(2); + expect(estimateImageCount(READABLE_CHARS_PER_IMAGE, COLS)).toBe(1); + expect(estimateImageCount(READABLE_CHARS_PER_IMAGE + 1, COLS)).toBe(2); }); }); @@ -89,10 +90,10 @@ describe('dense readable render profile', () => { ); expect(imgs.length).toBeGreaterThanOrEqual(3); - // 384 cols × 5 px bare cell + 8 px pad = 1928 px wide — the dense path fills - // the ~1932×1932 ceiling (≤2000 px) at the 5×8 cell for Fable / Opus 4.8. - expect(imgs[0]!.width).toBeGreaterThan(1900); - expect(imgs[0]!.width).toBeLessThanOrEqual(2000); + // 312 cols × 5 px bare cell + 8 px pad = 1568 px wide — the dense path fills + // Anthropic's long-edge bound exactly (any wider → server-side resample). + expect(imgs[0]!.width).toBeGreaterThan(1500); + expect(imgs[0]!.width).toBeLessThanOrEqual(1568); }); it('pages diff-shaped tool output at the dense readable budget', async () => { @@ -355,12 +356,14 @@ describe('paging end-to-end (transformRequest)', () => { }); it('pages dense medium tool_results instead of packing them into one image', async () => { - const lockish = Array.from({ length: 1200 }, (_, i) => + const lockish = Array.from({ length: 600 }, (_, i) => ` pkg-${i}@npm:1.${i}.0(peer@npm:^${i}.0.0)(typescript@npm:^5.${i % 10}.0): checksum=${'a'.repeat(24)}`, ).join('\n'); - // > one dense page (DENSE_CONTENT_CHARS_PER_IMAGE ≈ 92k) so it genuinely pages. - expect(lockish.length).toBeGreaterThan(100_000); - expect(lockish.length).toBeLessThan(150_000); + // > 2 dense pages (DENSE_CONTENT_CHARS_PER_IMAGE = 28,080) so it genuinely + // pages, while 600 short rows / 90 per page = 7 images stays under the ≤10 + // per-tool_result clamp (no truncation). + expect(lockish.length).toBeGreaterThan(2 * DENSE_CONTENT_CHARS_PER_IMAGE); + expect(lockish.length).toBeLessThan(80_000); const { info } = await transformRequest(makeReq(lockish), { multiCol: 1, charsPerToken: 2 }); expect(info.compressed).toBe(true); diff --git a/tests/render.test.ts b/tests/render.test.ts index 9047295..2c7dfb0 100644 --- a/tests/render.test.ts +++ b/tests/render.test.ts @@ -1943,17 +1943,18 @@ describe('transform', () => { // responds to `cols` (which scales chars/image linearly the same way a smaller // cell-H would). - it('maxCharsPerImage: fills the canvas (READABLE_CHARS_PER_IMAGE = 50k)', () => { - // Policy: maximum chars per page, full ~1932-px-tall canvas. At cols=100 the - // canvas holds 100 × 240 = 24,000 chars per page (height-limited). - expect(maxCharsPerImage(100)).toBe(24_000); + it('maxCharsPerImage: fills the canvas (READABLE_CHARS_PER_IMAGE = 28,080)', () => { + // Policy: maximum chars per page, full 728-px-tall canvas (Anthropic + // 1568-edge / ~1.15 MP clamp). At cols=100 the canvas holds + // 100 × 90 = 9,000 chars per page (height-limited). + expect(maxCharsPerImage(100)).toBe(9_000); }); - it('maxCharsPerImage: scales with cols and caps at the 50k page budget', () => { - expect(maxCharsPerImage(20)).toBe(4_800); // 20 × 240 = 4,800 (height-bound) - expect(maxCharsPerImage(50)).toBe(12_000); // 50 × 240 = 12,000 (height-bound) - expect(maxCharsPerImage(200)).toBe(48_000); // 200 × 240 = 48,000 (height-bound) - expect(maxCharsPerImage(313)).toBe(50_000); // 313 × 240 = 75,120 → capped at READABLE + it('maxCharsPerImage: scales with cols and caps at the 28k page budget', () => { + expect(maxCharsPerImage(20)).toBe(1_800); // 20 × 90 = 1,800 (height-bound) + expect(maxCharsPerImage(50)).toBe(4_500); // 50 × 90 = 4,500 (height-bound) + expect(maxCharsPerImage(200)).toBe(18_000); // 200 × 90 = 18,000 (height-bound) + expect(maxCharsPerImage(313)).toBe(28_080); // 313 × 90 = 28,170 → capped at READABLE }); it('isCompressionProfitable: doubling cols halves the 2-image break-even threshold', () => {