teamchong
7af6c200c3
eval: - Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY) replacing headless `claude -p`. Drives the real TUI on Max-subscription auth (no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is argv/stdin/--output-format json compatible with the old call sites. - Wire every call site to cci.py: gist-recall (run/run2/run3), needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot), swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs. - Isolation per call: --setting-sources project --strict-mcp-config --dangerously-skip-permissions. Recall evals keep --allowedTools Read; swe-bench keeps the full agentic toolset. Each trial is a fresh process. demo: - Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the base id: the proxy scope gate strips [1m] from the request before matching, so [1m]-on-launch + base-in-scope is the correct pairing. docs: - FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.
needle-haystack eval
Receipts for the needle eval. It measures the worst case for a lossy
compressor (exact recovery of a random fact from imaged content), not the whole
product. Its "dead" conclusion was later reversed on live measurement —
see the correction in /FINDINGS.md.
Question: if you put a unique fact only inside content rendered to PNG (never in the text tail), can Opus retrieve it? If no, verbatim recall from images is unsafe.
Finding (stands): verbatim 0/15 (both opus-4-5 and opus-4-8); semantic 27–40%
(≈ prior-guessing, p≈0.45). So imaged content is unreliable for verbatim recall
and can be silently confabulated. This is the caveat, not the verdict: on real
(dense) traffic pxpipe still saves ~68% as a gist tier — see FINDINGS.md correction.
| file | phase | what it does |
|---|---|---|
run3.sh |
1–2 | 2×2 N=15 harness: {verbatim, semantic} × {compression ON, OFF}, via the live proxy. MODEL=claude-opus-4-8 (Phase 1 was the same script on -4-5). |
results2.tsv |
2 | raw per-trial output of the opus-4-8 run. |
crux.py |
3 | feeds custom images straight to Opus (bypasses pxpipe's renderer) to separate encoder can't read hex from render too dense. Billboard 120pt → 8/8; clean 30pt → 8/8. The encoder reads hex fine when it's big. |
sweep.py |
4 | font-size density sweep at fixed ~2,668-image-token dimensions. Finds the cliff: 22pt 100%, 16pt 17% (near-miss corruption), 12pt unreadable. |
Run
- Start the pxpipe proxy on
127.0.0.1:47821. ./run3.sh(needs theclaudeCLI on a MAX plan — no API key).- Python tiers need PIL + a monospace TTF (
crux.py,sweep.py).
Open follow-up — partially answered
The sweep used English-prose filler (~3.5 chars/token). Real CC traffic is ~1.17
chars/token, where 22pt could hit ~1.4× compression at 100% read. The live proxy
record (FINDINGS.md correction, 2026-05-29) confirmed the direction: on real dense
traffic pxpipe measured ~68% fewer input tokens. Still worth re-running sweep.py
with dense JSON/tool-output filler to pin the controlled per-font-size cliff. See
FINDINGS.md §"open question" and the correction block.