Deval Shah
9199a8d0df
Ran eval/opus-density/run.mjs against the API (harness shipped without committed results). Issue #6 answer: Opus 4.8 exact-string recall rises monotonically as the render gets less dense — 1/4 -> 3/4 -> 4/4 across 5x8/7x10/9x12, reaching 4/4 exact with 0 confabulations at 9x12 (45% savings), byte-exact hex, stop_reason end_turn. Density is the lever; not a DEFAULT_MODEL_BASES change. Two response-reading bugs fixed (neither involves images): - callModel read content[0].text, but always-on-thinking models put a thinking block with empty text at content[0]; the answer is in a later block. Every Fable answer read as "" and scored miss/confab. Fixed: select the text block; max_tokens 128 -> 512. - score() ignored stop_reason, so a refusal (HTTP 200, stop_reason "refusal", empty content) scored as a confabulation, inverting the safety verdict. Fixed: treat refusal as its own state; a refused guard is safe. Adds results.json (live run), RESULTS.md (write-up), verify.mjs (confound-free receipts for both fixes plus the guard-question caveat: the never-stated password guard trips Fable's cyber classifier equally on text and image, n=6, so it is not an imaging effect and is unmeasurable on Fable as written).
3.2 KiB
Opus 4.8 lower-density read sweep — results (answers issue #6)
Live run of run.mjs, 2026-07-05. results.json is this run. Two harness bugs
found and fixed along the way (see below), so fixes and numbers are reported together.
Issue #6: Opus vs density
Exact-string recall on the production renderer, one run per cell:
| variant | page px | img tok | savings | Opus exact | Opus confab | gist | guard |
|---|---|---|---|---|---|---|---|
5x8 (production) |
1568×128 | 280 | 79% | 1/4 | 3 | ok | ok |
7x10 |
1562×228 | 504 | 62% | 3/4 | 1 | ok | ok |
9x12 |
1565×344 | 728 | 45% | 4/4 | 0 | ok | ok |
- Answer: yes, density-dependent.
9x12clears the acceptance bar inREADME.md— 4/4 exact, 0 confabulations, byte-exact hex, gist == baseline, savings positive (45%),stop_reason: end_turn. Production5x8confabulates 3/4 exact strings. - Monotonic and stable across three runs. Candidate: an opt-in lower-density Opus
render profile at
9x12(~1.75× the image tokens of5x8); not aDEFAULT_MODEL_BASESchange. - n=1 per cell (harness design). An independent sweep with a TrueType mono font and Levenshtein-graded probes reproduces the same monotonic cliff — byte-exact recall at low density, collapse as density rises.
Bugs fixed in run.mjs (neither involves images)
| # | Symptom before fix | Root cause | Fix |
|---|---|---|---|
| 1 (dominant) | every Fable answer, correct ones included, scored miss/confab (Fable 0/4) |
reads content[0].text; on always-on-thinking models content[0] is a thinking block with empty text, answer is in a later block |
select the text block; max_tokens 128 → 512 |
| 2 | refusals scored as confabulations (5 confab, guard FAIL) |
score() ignored stop_reason; a refusal (HTTP 200, stop_reason:"refusal", empty content) is neither the expected string nor an abstention |
branch on refusal as its own state; a refused guard is safe |
Bug 2 inverts the harness's safety verdict: a refusal is the safe no-answer, a
confabulation the dangerous one. After both fixes Fable shows 0 confabulations
at every density. Confound-free receipt for bug 1 (verify.mjs, benign prompt):
[bug2] fable, benign prose, stop=end_turn
[0]thinking="" [1]text="33"
OLD content[0].text="" FIXED text-block="33"
Caveat: the guard question, not the rendering, drives Fable's refusals
The never-stated password guard trips Fable's cyber classifier at the same
rate on text and image — modality-independent, so not an imaging effect
(verify.mjs, n=6):
| guard context | Fable refusals |
|---|---|
| text | 6/6 |
| image | 6/6 |
| no context | 1/6 |
Consequences: the guard is unmeasurable on Fable as written, and this is not evidence that rendering exposes Fable to refusals. Suggest a non-credential never-stated guard (e.g. "what was the reviewer's middle name?").
Reproduce
pnpm run build
ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/run.mjs # -> results.json
ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/verify.mjs # bug-1 + guard receipts