eval:
- Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY)
replacing headless `claude -p`. Drives the real TUI on Max-subscription auth
(no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is
argv/stdin/--output-format json compatible with the old call sites.
- Wire every call site to cci.py: gist-recall (run/run2/run3),
needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot),
swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs.
- Isolation per call: --setting-sources project --strict-mcp-config
--dangerously-skip-permissions. Recall evals keep --allowedTools Read;
swe-bench keeps the full agentic toolset. Each trial is a fresh process.
demo:
- Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in
cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the
base id: the proxy scope gate strips [1m] from the request before matching,
so [1m]-on-launch + base-in-scope is the correct pairing.
docs:
- FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.
- Remove eval/corpus/{sessions,text-blocks}.json from the repo — they are real captured transcripts (file paths, project names, message content). gitignore eval/corpus/; local copies kept for the eval scripts.
- gen_sweep.mjs: replace a hardcoded /Users/<name> absolute import with a relative path.
- README: 'holds ~92k readable chars' -> 'up to ~92k chars' (don't overclaim legibility at the new page density).
- CHANGELOG: note OCR legibility at the larger page size is not independently re-eval'd.
Both demos default to Fable; pass a model as the first arg (a.sh opus, sonnet,
haiku, or a full claude-... id) to override -- run both columns on the same
model for a fair A/B. Adds eval/ab token-savings scripts.
Fable 5 / Opus 4.8 accept up to 2576px / 4784 visual tokens per image, but a
request with >20 images (pxpipe always sends many) is held to the stricter
<=2000px/side rule, so the real ceiling is ~1932x1932 (1928x1928 = 69x69 = 4761
tokens). MAX_HEIGHT_PX 1568->1932; dense tool/history pages now render at
DENSE_CONTENT_COLS=384 / DENSE_CONTENT_CHARS_PER_IMAGE=92160 (1928x1928 full
page) -- fewer image blocks at the same OCR-validated 5x8 cell. The static slab
is unchanged (313 cols / 1573x1280).
Also fixes a regression the code review caught: the break-even gate,
truncateForBudget, and estimateImageCount predicted against the slab geometry
(313 cols / 159 rows) while the dense renderer emits 384 cols / 240 rows, so
large tool_results were truncated far earlier than the 10-image cap required --
silently dropping output that would have rendered. A per-page char budget is
threaded through the prediction chain (defaults to READABLE so the slab and unit
tests are byte-identical) plus a denseGateGeometry helper, so the gate prices the
same page the renderer actually produces.
- applicability: isPxpipeSupportedModel reads PXPIPE_MODELS (comma-sep,
default claude-fable-5 only) and strips bracket/variant tags like [1m];
Opus becomes opt-in with no code change. Default behavior unchanged.
- FINDINGS: Opus 4.8 re-test - arithmetic 98/100 (-2pp), verbatim 6/15
(was 93/100, 0/15): improved but still taxed -> default stays Fable-only.
- FINDINGS + eval/glyph-matrix/sweep: root cause of the read tax is per-glyph
RESOLUTION, not font shape (confusions scale with px and vanish by 10x16).
Opus needs ~4x glyph area to read reliably; the 1568px ceiling makes that
break-even, so there is no profitable Opus operating point.
- FINDINGS.md: dated 2026-06-12 entry logging in-the-wild verbatim
misreads (7/10 exact, ~3.3% per-char, 2/3 silent) alongside flawless
gist-tier recall in the same session. Explicitly anecdotal, n=1,
non-reproducible — motivation only.
- eval/glyph-matrix/PLAN.md: public plan for the paused confusion-matrix
experiment (Task #7) so the pending work is visible outside README,
per the measured-numbers-only README rule.
- ON 47823 / OFF 47824 with own PXPIPE_LOG files: operator session can
never pollute bench measurement (separation by construction)
- resume = skip existing patch files; clean stop on quota errors
- Docker grading documented as proxy-free (no model calls in containers)
- README: mention Pro pilot alongside Lite
Does the model actually read pixelpipe's render, or coast on memorized answers?
- novel random-number problems (un-memorizable): text 100% vs image 93% (-7pp real reading tax)
- GSM8K (in training data): 97% vs 96% -- inflated ~3pp by recall
- verbatim recall from a dense render: 0/15
README 'Benchmarks' section leads with the clean 93% and reports the failure
rows too (no cherry-pick). Harness in eval/gsm8k/: gen_novel.py, render_cfg.mjs
(pixelpipe's real renderTextToPngs), bench.py -- claude -p, exact-match grading.
The prior commit captured only the rename (a stale pathspec aborted the
content staging). This applies the actual edits: retitle to FINDINGS,
'## Verdict' + superseded TL;DR, and the two cross-reference updates.
Widen the model gate to /^claude-opus-4-(?:[7-9]|[1-9]\d)(?:-|$)/ (Opus 4.7
and newer; previously 4.6/4.7). Wire isPixelpipeSupportedModel into the proxy
boundary in proxy.ts — the proxy previously gated on economics only and
ignored the model, so it compressed 4.8 traffic despite the docs claiming a
4.6/4.7 scope. Unsupported models now pass through with reason
'unsupported_model'.
Update public-api tests to the 4.7+ spec (320/320 pass).
Correct the docs to live measurement: reverse the POSTMORTEM "dead" verdict
(pixelpipe is a lossy gist-compressor saving ~68% on real dense Claude Code
traffic; the verbatim 0/15 needle finding stands as a caveat, not the
verdict), rewrite README Status/Limitations, and add a correction pointer to
the eval README. Original POSTMORTEM body preserved below the correction.
BREAKING CHANGE: Opus 4.6 and older are no longer supported.
Verbatim-risk guard (skip imaging unique IDs/hashes/exact values) still pending.
Co-render the OCR instruction into the same PNG as the content,
delimited by '===…===' bands, with the API system field dropped.
L1 OCR fidelity, Opus 4.7, 20 production blocks, 7×10 cell:
- baseline (text-only): 97.91% mean / 96.25% min
- reflow (separate system): 91.99% mean / 82.59% min (-5.93pp)
- reflow-inimage: 98.95% mean / 96.42% min (+1.04pp)
reflow-inimage wins on all 20/20 blocks vs reflow, and beats the
text-only baseline on 17/20 blocks. Three blocks hit 100%. The
-5.93pp reflow regression that the cell-pitch sweep partially
recovered disappears entirely when the instruction is co-rendered.
Mechanism: when system carries the instruction and the image
carries the content, the model does cross-modal binding to figure
out what the image is for. Co-rendering reduces it to a
single-modal task with an unambiguous parse rule.
Files:
- eval/eval-l1-ocr.mjs: add reflow-inimage variant + prompt
- README.md: new section before history compression
- eval/EXPERIMENT_LOG.md: attempt #2 writeup
- eval/results/{l1-report.md, l1-results.json}: regenerated
Pack text into a continuous sentinel-delimited stream (↵ = U+21B5) so
wrapLines fills every row to `cols` instead of leaving dead right-margin.
Adds reflow/dereflow, renderTextToPngsReflow{,MultiCol} variants, a full
test suite, and an A/B eval harness with L1 OCR + L2 session results.
teamchong