- README: Fable clip as headline demo (Drive link + committed thumb),
verified numbers from the recording: plain $42.21 / 96% context vs
pxpipe $4.51; honest caveat on single-reply format compliance
- ATTEMPTS.md: attempt 1 (invalid run, model-switch pitfall) and
attempt 2 (legibility PASS on fable, format miss documented)
- a.sh/b.sh: prompt fix — numbering starts at filler-000
Anthropic's safety classifier OCRs rendered images. A *legible* image of the
system prompt + tool docs (bash / permissions / credential / file-write
vocabulary) trips its prompt-injection heuristic and forces a model downgrade
to Opus, which costs far more than imaging the (already prompt-cached) slab
ever saves. Empirically the caption wording is NOT the trigger — the OCR'd
content is — so no caption rewrite fixes it; the slab must stay text.
Gate at the slab profitability check with PXPIPE_TEXT_SLAB=1. Default is
unchanged (legacy imaged slab) so the cache-align / cache-stability suites
stay valid. Both demos now launch the pxpipe proxy with the toggle set.
Verified: identical ~300KB-slab request images 8 PNGs by default, 0 with the
toggle (reason=slab_text). Full suite: 614 passed.
Both demos' a.sh/b.sh now request claude-fable-5[1m] for the default `fable`
arm, consistent with the opus/sonnet arms which already use [1m].
setup.sh PXPIPE_MODELS stays the BASE claude-fable-5 on purpose: the proxy
strips [variant] tags from the incoming model before matching against the
allow-list verbatim (src/core/applicability.ts), so base claude-fable-5
already covers claude-fable-5[1m]. Writing [1m] into the allow-list would
make the stripped incoming base no longer match -> pxpipe silently stops
compressing. Added a comment in each setup.sh pinning that.
a.sh/b.sh -> claude-sonnet-5[1m] (was the stale claude-sonnet-4-6),
matching the opus alias's [1m] pattern. setup.sh -> claude-sonnet-5
(bare id, feeds PXPIPE_MODELS). claude-sonnet-5 was already in the dashboard
model catalog (fragments.ts); old 4.6 still reachable via the verbatim-id
fallback. No src/ changes.
eval:
- Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY)
replacing headless `claude -p`. Drives the real TUI on Max-subscription auth
(no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is
argv/stdin/--output-format json compatible with the old call sites.
- Wire every call site to cci.py: gist-recall (run/run2/run3),
needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot),
swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs.
- Isolation per call: --setting-sources project --strict-mcp-config
--dangerously-skip-permissions. Recall evals keep --allowedTools Read;
swe-bench keeps the full agentic toolset. Each trial is a fresh process.
demo:
- Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in
cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the
base id: the proxy scope gate strips [1m] from the request before matching,
so [1m]-on-launch + base-in-scope is the correct pairing.
docs:
- FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.
setup.sh resolves a model arg (default fable) and sets PXPIPE_MODELS to Fable plus the chosen model, so the demo matches the production default (Opus OFF) and 'setup.sh opus' enables Opus end-to-end across setup/a.sh/b.sh. READMEs updated to pass the model to all three.
Both demos default to Fable; pass a model as the first arg (a.sh opus, sonnet,
haiku, or a full claude-... id) to override -- run both columns on the same
model for a fair A/B. Adds eval/ab token-savings scripts.
teamchong