mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
Deval Shah
9199a8d0df
Ran eval/opus-density/run.mjs against the API (harness shipped without committed results). Issue #6 answer: Opus 4.8 exact-string recall rises monotonically as the render gets less dense — 1/4 -> 3/4 -> 4/4 across 5x8/7x10/9x12, reaching 4/4 exact with 0 confabulations at 9x12 (45% savings), byte-exact hex, stop_reason end_turn. Density is the lever; not a DEFAULT_MODEL_BASES change. Two response-reading bugs fixed (neither involves images): - callModel read content[0].text, but always-on-thinking models put a thinking block with empty text at content[0]; the answer is in a later block. Every Fable answer read as "" and scored miss/confab. Fixed: select the text block; max_tokens 128 -> 512. - score() ignored stop_reason, so a refusal (HTTP 200, stop_reason "refusal", empty content) scored as a confabulation, inverting the safety verdict. Fixed: treat refusal as its own state; a refused guard is safe. Adds results.json (live run), RESULTS.md (write-up), verify.mjs (confound-free receipts for both fixes plus the guard-question caveat: the never-stated password guard trips Fable's cyber classifier equally on text and image, n=6, so it is not an imaging effect and is unmeasurable on Fable as written).
75 lines
3.4 KiB
Markdown
75 lines
3.4 KiB
Markdown
# Opus 4.8 lower-density / larger-cell read sweep
|
||
|
||
**Question (issue #6):** the known Opus misread rate on pxpipe images was measured
|
||
at the production **5×8** cell density. If the same text is rendered *less dense*
|
||
(bigger cells → fewer chars/page → more pages), does Opus 4.8 read exact strings
|
||
reliably enough to be worth enabling — and at what token cost?
|
||
|
||
This harness answers that with a measurement, **not** a default change. It does
|
||
not touch production code or the model allowlist. Fable 5 stays the only default
|
||
reader unless the numbers below clear the acceptance bar.
|
||
|
||
## What it does
|
||
|
||
For each render **variant** (cell size) it renders one synthetic "session"
|
||
transcript to PNG(s) via the production renderer, then asks each **model** a
|
||
fixed battery of questions against the image and scores the answers.
|
||
|
||
- **Variants** (cell = 5+wBonus × 8+hBonus px, all via `RenderStyle`):
|
||
- `5x8` — production density (`{cellWBonus:0, cellHBonus:0}`)
|
||
- `7x10` — `{cellWBonus:2, cellHBonus:2}`
|
||
- `9x12` — `{cellWBonus:4, cellHBonus:4}`
|
||
Each variant keeps the ≤1568×728 page cap, so images stay in Anthropic's
|
||
linear-billing window (no server-side downscale) and page count rises as
|
||
density drops.
|
||
- **Models:** `claude-opus-4-8`, `claude-fable-5` (both high-res tier).
|
||
- **Tasks** (each answer committed before ground truth is revealed):
|
||
1. exact 12-char hex recall
|
||
2. camelCase identifier recall
|
||
3. file path / CLI flag / number recall
|
||
4. gist recall (a decision / value / state that survives lossy reads)
|
||
5. never-stated-fact guard — the correct answer is "not stated / UNKNOWN"
|
||
|
||
## Metrics (per model × variant)
|
||
|
||
- exact-match accuracy (tasks 1–3)
|
||
- **confabulation count** — a confident *wrong* exact string (the dangerous mode)
|
||
- abstain / "not safe to quote" count
|
||
- gist accuracy (task 4) and false-answer rate on the guard (task 5)
|
||
- image tokens (28-px patch count), text-baseline tokens, and savings %
|
||
- rough wall-clock latency
|
||
|
||
## Acceptance — do NOT claim "Opus works" unless, for a variant:
|
||
|
||
- gist recall == the text baseline (or within a pre-declared tolerance), AND
|
||
- every exact-identifier question is either answered correctly OR abstains /
|
||
refers to the factsheet — **zero silent wrong exact strings** on the protected
|
||
exact tasks, AND
|
||
- token savings stay positive on token-dense Claude-Code-like content.
|
||
|
||
Only a variant that clears all three is a candidate for an opt-in Opus render
|
||
profile. Enabling Opus in `DEFAULT_MODEL_BASES` is explicitly out of scope until
|
||
then.
|
||
|
||
## Run
|
||
|
||
```bash
|
||
# Dry run — renders every variant and prints the token/savings accounting.
|
||
# No API key needed; no model is called.
|
||
pnpm exec tsx eval/opus-density/run.mjs
|
||
|
||
# Full run — also calls the models and scores the battery.
|
||
ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/run.mjs
|
||
```
|
||
|
||
Results are written to `eval/opus-density/results.json`.
|
||
|
||
A first live run (2026-07-05) is committed here: `results.json` plus a write-up in
|
||
[`RESULTS.md`](./RESULTS.md). Short version — Opus reads **4/4 exact, 0 confab** at
|
||
the largest cell (`9x12`, still 45% savings), climbing from 1/4 at production
|
||
`5x8`, so the issue-#6 answer is yes and density-dependent. Running it also surfaced
|
||
two response-reading bugs in this harness (now fixed in `run.mjs`): the answer of an
|
||
always-on-thinking model isn't in `content[0]`, and a classifier refusal was being
|
||
scored as a confabulation. `verify.mjs` reproduces both receipts plus the guard-
|
||
question caveat.
|