Ran eval/opus-density/run.mjs against the API (harness shipped without committed
results). Issue #6 answer: Opus 4.8 exact-string recall rises monotonically as the
render gets less dense — 1/4 -> 3/4 -> 4/4 across 5x8/7x10/9x12, reaching 4/4 exact
with 0 confabulations at 9x12 (45% savings), byte-exact hex, stop_reason end_turn.
Density is the lever; not a DEFAULT_MODEL_BASES change.
Two response-reading bugs fixed (neither involves images):
- callModel read content[0].text, but always-on-thinking models put a thinking
block with empty text at content[0]; the answer is in a later block. Every Fable
answer read as "" and scored miss/confab. Fixed: select the text block;
max_tokens 128 -> 512.
- score() ignored stop_reason, so a refusal (HTTP 200, stop_reason "refusal",
empty content) scored as a confabulation, inverting the safety verdict. Fixed:
treat refusal as its own state; a refused guard is safe.
Adds results.json (live run), RESULTS.md (write-up), verify.mjs (confound-free
receipts for both fixes plus the guard-question caveat: the never-stated password
guard trips Fable's cyber classifier equally on text and image, n=6, so it is not
an imaging effect and is unmeasurable on Fable as written).
Issue #6 asks whether Opus 4.8 reads exact strings reliably if the text is
rendered less dense (bigger cells / fewer chars per page). This adds a
measurement harness — no production code, no allowlist change.
- eval/opus-density/run.mjs renders one synthetic session at three densities
(5×8 production, 7×10, 9×12), each with cols dropped so the page stays ≤1568×728
(the linear-billing window), then — when ANTHROPIC_API_KEY is set — asks
claude-opus-4-8 and claude-fable-5 a fixed battery (exact hex / camelCase / path
/ port, a gist question, and a never-stated guard) and scores exact accuracy,
confabulation, abstention, gist, image tokens, and savings.
- eval/opus-density/README.md documents the matrix, metrics, and the acceptance
bar: no silent wrong exact strings, gist == text baseline, savings still
positive — otherwise Opus stays opt-in (never added to DEFAULT_MODEL_BASES here).
Dry-run verified (no key needed): 5×8→79%, 7×10→62%, 9×12→45% savings at
1568×{128,228,344} px. Results are not committed — run on a machine with API
access.
Deval Shah