Both demos' a.sh/b.sh now request claude-fable-5[1m] for the default `fable`
arm, consistent with the opus/sonnet arms which already use [1m].
setup.sh PXPIPE_MODELS stays the BASE claude-fable-5 on purpose: the proxy
strips [variant] tags from the incoming model before matching against the
allow-list verbatim (src/core/applicability.ts), so base claude-fable-5
already covers claude-fable-5[1m]. Writing [1m] into the allow-list would
make the stripped incoming base no longer match -> pxpipe silently stops
compressing. Added a comment in each setup.sh pinning that.
eval:
- Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY)
replacing headless `claude -p`. Drives the real TUI on Max-subscription auth
(no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is
argv/stdin/--output-format json compatible with the old call sites.
- Wire every call site to cci.py: gist-recall (run/run2/run3),
needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot),
swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs.
- Isolation per call: --setting-sources project --strict-mcp-config
--dangerously-skip-permissions. Recall evals keep --allowedTools Read;
swe-bench keeps the full agentic toolset. Each trial is a fresh process.
demo:
- Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in
cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the
base id: the proxy scope gate strips [1m] from the request before matching,
so [1m]-on-launch + base-in-scope is the correct pairing.
docs:
- FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.
teamchong