A (IDS rows + factsheet) 14/24, B (factsheet only) 14/24,
C (no guards) 8/24. Factsheet delivers the recall lift; IDS rows
show no measurable effect. Evidence for dropping the guards
paragraph from MODEL_RENDER_PROFILES.md.
Ship Grok 5×8 white/IDS packing and universal IDS/factsheet plumbing while
keeping DEFAULT_MODEL_BASES Fable-only: live pure-image Grok is not
Fable-level; image+factsheet is the working exact path.
Chart shows Grok as a text-window series only (no Grok+pxpipe overlay) and
Fable as the measured pxpipe density point. Add Codex-path quality harnesses
and receipts (multi-seed, fix-matrix, shipped smoke).
Grok was imaging on the Responses path with no readability data. Add
eval/grok-density: density ladder, climb receipts, and a live
image-vs-factsheet comparison on grok-4.5.
Findings locked into the receipts:
- Production 5x8 image-only confabulates exact IDs (0/4).
- Image + factsheet clears 4/4 exact, 0 confab, gist+guard ok (~70%
fixture savings).
- Looser pure-image packing can clear exact recall but only ~30%
savings — not worth the default.
Supports keeping Grok opt-in with dense 5x8 + factsheet when enabled.
Ran eval/opus-density/run.mjs against the API (harness shipped without committed
results). Issue #6 answer: Opus 4.8 exact-string recall rises monotonically as the
render gets less dense — 1/4 -> 3/4 -> 4/4 across 5x8/7x10/9x12, reaching 4/4 exact
with 0 confabulations at 9x12 (45% savings), byte-exact hex, stop_reason end_turn.
Density is the lever; not a DEFAULT_MODEL_BASES change.
Two response-reading bugs fixed (neither involves images):
- callModel read content[0].text, but always-on-thinking models put a thinking
block with empty text at content[0]; the answer is in a later block. Every Fable
answer read as "" and scored miss/confab. Fixed: select the text block;
max_tokens 128 -> 512.
- score() ignored stop_reason, so a refusal (HTTP 200, stop_reason "refusal",
empty content) scored as a confabulation, inverting the safety verdict. Fixed:
treat refusal as its own state; a refused guard is safe.
Adds results.json (live run), RESULTS.md (write-up), verify.mjs (confound-free
receipts for both fixes plus the guard-question caveat: the never-stated password
guard trips Fable's cyber classifier equally on text and image, n=6, so it is not
an imaging effect and is unmeasurable on Fable as written).
Issue #6 asks whether Opus 4.8 reads exact strings reliably if the text is
rendered less dense (bigger cells / fewer chars per page). This adds a
measurement harness — no production code, no allowlist change.
- eval/opus-density/run.mjs renders one synthetic session at three densities
(5×8 production, 7×10, 9×12), each with cols dropped so the page stays ≤1568×728
(the linear-billing window), then — when ANTHROPIC_API_KEY is set — asks
claude-opus-4-8 and claude-fable-5 a fixed battery (exact hex / camelCase / path
/ port, a gist question, and a never-stated guard) and scores exact accuracy,
confabulation, abstention, gist, image tokens, and savings.
- eval/opus-density/README.md documents the matrix, metrics, and the acceptance
bar: no silent wrong exact strings, gist == text baseline, savings still
positive — otherwise Opus stays opt-in (never added to DEFAULT_MODEL_BASES here).
Dry-run verified (no key needed): 5×8→79%, 7×10→62%, 9×12→45% savings at
1568×{128,228,344} px. Results are not committed — run on a machine with API
access.
Both demos' a.sh/b.sh now request claude-fable-5[1m] for the default `fable`
arm, consistent with the opus/sonnet arms which already use [1m].
setup.sh PXPIPE_MODELS stays the BASE claude-fable-5 on purpose: the proxy
strips [variant] tags from the incoming model before matching against the
allow-list verbatim (src/core/applicability.ts), so base claude-fable-5
already covers claude-fable-5[1m]. Writing [1m] into the allow-list would
make the stripped incoming base no longer match -> pxpipe silently stops
compressing. Added a comment in each setup.sh pinning that.
Holds the glyph cell at prod 5x8 and varies ONLY render style
(prod/onebit/color/grid/cgrid) at identical pixel dims => identical
image-token cost. Answers what the size sweep could not: at fixed cost,
does any style beat the ~10% exact-read baseline? Same content, grader,
and claude-CLI read method as eval/glyph-matrix/sweep.
eval:
- Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY)
replacing headless `claude -p`. Drives the real TUI on Max-subscription auth
(no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is
argv/stdin/--output-format json compatible with the old call sites.
- Wire every call site to cci.py: gist-recall (run/run2/run3),
needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot),
swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs.
- Isolation per call: --setting-sources project --strict-mcp-config
--dangerously-skip-permissions. Recall evals keep --allowedTools Read;
swe-bench keeps the full agentic toolset. Each trial is a fresh process.
demo:
- Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in
cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the
base id: the proxy scope gate strips [1m] from the request before matching,
so [1m]-on-launch + base-in-scope is the correct pairing.
docs:
- FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.
- Remove eval/corpus/{sessions,text-blocks}.json from the repo — they are real captured transcripts (file paths, project names, message content). gitignore eval/corpus/; local copies kept for the eval scripts.
- gen_sweep.mjs: replace a hardcoded /Users/<name> absolute import with a relative path.
- README: 'holds ~92k readable chars' -> 'up to ~92k chars' (don't overclaim legibility at the new page density).
- CHANGELOG: note OCR legibility at the larger page size is not independently re-eval'd.
Both demos default to Fable; pass a model as the first arg (a.sh opus, sonnet,
haiku, or a full claude-... id) to override -- run both columns on the same
model for a fair A/B. Adds eval/ab token-savings scripts.
Fable 5 / Opus 4.8 accept up to 2576px / 4784 visual tokens per image, but a
request with >20 images (pxpipe always sends many) is held to the stricter
<=2000px/side rule, so the real ceiling is ~1932x1932 (1928x1928 = 69x69 = 4761
tokens). MAX_HEIGHT_PX 1568->1932; dense tool/history pages now render at
DENSE_CONTENT_COLS=384 / DENSE_CONTENT_CHARS_PER_IMAGE=92160 (1928x1928 full
page) -- fewer image blocks at the same OCR-validated 5x8 cell. The static slab
is unchanged (313 cols / 1573x1280).
Also fixes a regression the code review caught: the break-even gate,
truncateForBudget, and estimateImageCount predicted against the slab geometry
(313 cols / 159 rows) while the dense renderer emits 384 cols / 240 rows, so
large tool_results were truncated far earlier than the 10-image cap required --
silently dropping output that would have rendered. A per-page char budget is
threaded through the prediction chain (defaults to READABLE so the slab and unit
tests are byte-identical) plus a denseGateGeometry helper, so the gate prices the
same page the renderer actually produces.
- applicability: isPxpipeSupportedModel reads PXPIPE_MODELS (comma-sep,
default claude-fable-5 only) and strips bracket/variant tags like [1m];
Opus becomes opt-in with no code change. Default behavior unchanged.
- FINDINGS: Opus 4.8 re-test - arithmetic 98/100 (-2pp), verbatim 6/15
(was 93/100, 0/15): improved but still taxed -> default stays Fable-only.
- FINDINGS + eval/glyph-matrix/sweep: root cause of the read tax is per-glyph
RESOLUTION, not font shape (confusions scale with px and vanish by 10x16).
Opus needs ~4x glyph area to read reliably; the 1568px ceiling makes that
break-even, so there is no profitable Opus operating point.
- FINDINGS.md: dated 2026-06-12 entry logging in-the-wild verbatim
misreads (7/10 exact, ~3.3% per-char, 2/3 silent) alongside flawless
gist-tier recall in the same session. Explicitly anecdotal, n=1,
non-reproducible — motivation only.
- eval/glyph-matrix/PLAN.md: public plan for the paused confusion-matrix
experiment (Task #7) so the pending work is visible outside README,
per the measured-numbers-only README rule.
- ON 47823 / OFF 47824 with own PXPIPE_LOG files: operator session can
never pollute bench measurement (separation by construction)
- resume = skip existing patch files; clean stop on quota errors
- Docker grading documented as proxy-free (no model calls in containers)
- README: mention Pro pilot alongside Lite
Does the model actually read pixelpipe's render, or coast on memorized answers?
- novel random-number problems (un-memorizable): text 100% vs image 93% (-7pp real reading tax)
- GSM8K (in training data): 97% vs 96% -- inflated ~3pp by recall
- verbatim recall from a dense render: 0/15
README 'Benchmarks' section leads with the clean 93% and reports the failure
rows too (no cherry-pick). Harness in eval/gsm8k/: gen_novel.py, render_cfg.mjs
(pixelpipe's real renderTextToPngs), bench.py -- claude -p, exact-match grading.
The prior commit captured only the rename (a stale pathspec aborted the
content staging). This applies the actual edits: retitle to FINDINGS,
'## Verdict' + superseded TL;DR, and the two cross-reference updates.
Widen the model gate to /^claude-opus-4-(?:[7-9]|[1-9]\d)(?:-|$)/ (Opus 4.7
and newer; previously 4.6/4.7). Wire isPixelpipeSupportedModel into the proxy
boundary in proxy.ts — the proxy previously gated on economics only and
ignored the model, so it compressed 4.8 traffic despite the docs claiming a
4.6/4.7 scope. Unsupported models now pass through with reason
'unsupported_model'.
Update public-api tests to the 4.7+ spec (320/320 pass).
Correct the docs to live measurement: reverse the POSTMORTEM "dead" verdict
(pixelpipe is a lossy gist-compressor saving ~68% on real dense Claude Code
traffic; the verbatim 0/15 needle finding stands as a caveat, not the
verdict), rewrite README Status/Limitations, and add a correction pointer to
the eval README. Original POSTMORTEM body preserved below the correction.
BREAKING CHANGE: Opus 4.6 and older are no longer supported.
Verbatim-risk guard (skip imaging unique IDs/hashes/exact values) still pending.
Co-render the OCR instruction into the same PNG as the content,
delimited by '===…===' bands, with the API system field dropped.
L1 OCR fidelity, Opus 4.7, 20 production blocks, 7×10 cell:
- baseline (text-only): 97.91% mean / 96.25% min
- reflow (separate system): 91.99% mean / 82.59% min (-5.93pp)
- reflow-inimage: 98.95% mean / 96.42% min (+1.04pp)
reflow-inimage wins on all 20/20 blocks vs reflow, and beats the
text-only baseline on 17/20 blocks. Three blocks hit 100%. The
-5.93pp reflow regression that the cell-pitch sweep partially
recovered disappears entirely when the instruction is co-rendered.
Mechanism: when system carries the instruction and the image
carries the content, the model does cross-modal binding to figure
out what the image is for. Co-rendering reduces it to a
single-modal task with an unambiguous parse rule.
Files:
- eval/eval-l1-ocr.mjs: add reflow-inimage variant + prompt
- README.md: new section before history compression
- eval/EXPERIMENT_LOG.md: attempt #2 writeup
- eval/results/{l1-report.md, l1-results.json}: regenerated
Pack text into a continuous sentinel-delimited stream (↵ = U+21B5) so
wrapLines fills every row to `cols` instead of leaving dead right-margin.
Adds reflow/dereflow, renderTextToPngsReflow{,MultiCol} variants, a full
test suite, and an A/B eval harness with L1 OCR + L2 session results.
teamchong