mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
teamchong
37 lines
2.2 KiB
Markdown
37 lines
2.2 KiB
Markdown
# needle-haystack eval
|
||
|
||
Receipts for the needle eval. It measures the **worst case** for a lossy
|
||
compressor (exact recovery of a random fact from imaged content), not the whole
|
||
product. Its "dead" conclusion was later **reversed** on live measurement —
|
||
see the correction in [`/FINDINGS.md`](../../FINDINGS.md).
|
||
|
||
**Question:** if you put a unique fact *only* inside content rendered to PNG (never
|
||
in the text tail), can Opus retrieve it? If no, *verbatim* recall from images is unsafe.
|
||
|
||
**Finding (stands):** verbatim 0/15 (both `opus-4-5` and `opus-4-8`); semantic 27–40%
|
||
(≈ prior-guessing, p≈0.45). So imaged content is **unreliable for verbatim recall**
|
||
and can be silently confabulated. This is the caveat, not the verdict: on real
|
||
(dense) traffic pxpipe still saves ~68% as a *gist* tier — see FINDINGS.md correction.
|
||
|
||
| file | phase | what it does |
|
||
|---|---|---|
|
||
| `run3.sh` | 1–2 | 2×2 N=15 harness: {verbatim, semantic} × {compression ON, OFF}, via the live proxy. `MODEL=claude-opus-4-8` (Phase 1 was the same script on `-4-5`). |
|
||
| `results2.tsv` | 2 | raw per-trial output of the `opus-4-8` run. |
|
||
| `crux.py` | 3 | feeds custom images straight to Opus (bypasses pxpipe's renderer) to separate *encoder can't read hex* from *render too dense*. Billboard 120pt → 8/8; clean 30pt → 8/8. The encoder reads hex fine when it's big. |
|
||
| `sweep.py` | 4 | font-size density sweep at fixed ~2,668-image-token dimensions. Finds the cliff: 22pt 100%, 16pt 17% (near-miss corruption), 12pt unreadable. |
|
||
|
||
## Run
|
||
|
||
1. Start the pxpipe proxy on `127.0.0.1:47821`.
|
||
2. `./run3.sh` (needs the `claude` CLI on a MAX plan — no API key).
|
||
3. Python tiers need PIL + a monospace TTF (`crux.py`, `sweep.py`).
|
||
|
||
## Open follow-up — partially answered
|
||
|
||
The sweep used English-prose filler (~3.5 chars/token). Real CC traffic is ~1.17
|
||
chars/token, where 22pt could hit ~1.4× compression at 100% read. The **live proxy
|
||
record** (FINDINGS.md correction, 2026-05-29) confirmed the direction: on real dense
|
||
traffic pxpipe measured ~68% fewer input tokens. Still worth re-running `sweep.py`
|
||
with dense JSON/tool-output filler to pin the controlled per-font-size cliff. See
|
||
FINDINGS.md §"open question" and the correction block.
|