Adds the 2026-06-23 entry: pxpipe's right comparison is /compact (itself lossy),
not perfect recall. compact-vs-gist analysis (same job, opposite loss profiles —
pxpipe wins on Fable, peer-on-Opus with abstention, recoverability as the
structural edge); code-grounded notes on the open-source harnesses (codex,
opencode); ties to the agent-memory (reducer/channel-routing) direction. n=1
behavioral parts flagged; eval numbers cross-checked against existing entries.
Docs only — not shipped in the npm package.
eval:
- Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY)
replacing headless `claude -p`. Drives the real TUI on Max-subscription auth
(no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is
argv/stdin/--output-format json compatible with the old call sites.
- Wire every call site to cci.py: gist-recall (run/run2/run3),
needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot),
swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs.
- Isolation per call: --setting-sources project --strict-mcp-config
--dangerously-skip-permissions. Recall evals keep --allowedTools Read;
swe-bench keeps the full agentic toolset. Each trial is a fresh process.
demo:
- Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in
cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the
base id: the proxy scope gate strips [1m] from the request before matching,
so [1m]-on-launch + base-in-scope is the correct pairing.
docs:
- FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.
- applicability: isPxpipeSupportedModel reads PXPIPE_MODELS (comma-sep,
default claude-fable-5 only) and strips bracket/variant tags like [1m];
Opus becomes opt-in with no code change. Default behavior unchanged.
- FINDINGS: Opus 4.8 re-test - arithmetic 98/100 (-2pp), verbatim 6/15
(was 93/100, 0/15): improved but still taxed -> default stays Fable-only.
- FINDINGS + eval/glyph-matrix/sweep: root cause of the read tax is per-glyph
RESOLUTION, not font shape (confusions scale with px and vanish by 10x16).
Opus needs ~4x glyph area to read reliably; the 1568px ceiling makes that
break-even, so there is no profitable Opus operating point.
- FINDINGS.md: dated 2026-06-12 entry logging in-the-wild verbatim
misreads (7/10 exact, ~3.3% per-char, 2/3 silent) alongside flawless
gist-tier recall in the same session. Explicitly anecdotal, n=1,
non-reproducible — motivation only.
- eval/glyph-matrix/PLAN.md: public plan for the paused confusion-matrix
experiment (Task #7) so the pending work is visible outside README,
per the measured-numbers-only README rule.
The prior commit captured only the rename (a stale pathspec aborted the
content staging). This applies the actual edits: retitle to FINDINGS,
'## Verdict' + superseded TL;DR, and the two cross-reference updates.
The project isn't dead (verdict reversed to working ~68% gist-compressor),
so 'postmortem' was a misnomer. Rename to FINDINGS, retitle, promote the
correction to '## Verdict', and mark the old TL;DR as superseded. Update the
two cross-references (eval README link, public-api test comment).
teamchong