mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
teamchong
7af6c200c3
eval: - Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY) replacing headless `claude -p`. Drives the real TUI on Max-subscription auth (no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is argv/stdin/--output-format json compatible with the old call sites. - Wire every call site to cci.py: gist-recall (run/run2/run3), needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot), swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs. - Isolation per call: --setting-sources project --strict-mcp-config --dangerously-skip-permissions. Recall evals keep --allowedTools Read; swe-bench keeps the full agentic toolset. Each trial is a fresh process. demo: - Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the base id: the proxy scope gate strips [1m] from the request before matching, so [1m]-on-launch + base-in-scope is the correct pairing. docs: - FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.
pxpipe demos
Two demos, two questions, two honest verdicts.
| folder | question | status |
|---|---|---|
cost-ab/ |
Does pxpipe cost less on a real task? | ~break-even. Compression is real but lands in cache_read (cheap at $, rumored-free at the subscription cap); session divergence dominates the totals. |
effective-context/ |
Does pxpipe stay sharp in a context that overloads the plain column? | unvalidated — modern long-context Claude may not "drown," so run it before claiming a win. |
What is validated: in the cost A/B's Rust-rewrite task, both columns ported a whole library to Rust and both passed all 5 tests with the exact expected integers — pxpipe through imaged spec + source. So compression didn't break a precision task. pxpipe's case rests on capability (not breaking real work, and effective context), not on saving tokens.
Shared tooling: ../eval/ab/analyze.mjs reads the two proxy logs
(ab-on.jsonl / ab-off.jsonl) and reports the honest cost comparison.