3 Commits

Author	SHA1	Message	Date
teamchong	7af6c200c3	eval+demo: route harnesses through interactive cci.py; opus → claude-opus-4-8[1m] ... eval: - Add eval/lib/cci.py: drop-in interactive Claude session shim (pexpect/PTY) replacing headless `claude -p`. Drives the real TUI on Max-subscription auth (no ANTHROPIC_API_KEY), scrapes the reply + /context + /cost, and is argv/stdin/--output-format json compatible with the old call sites. - Wire every call site to cci.py: gist-recall (run/run2/run3), needle-haystack (sweep/crux), gsm8k (bench), swe-bench (run_pilot), swe-bench-pro (rep_nav/run_pro), and eval/lib/anthropic-client.mjs. - Isolation per call: --setting-sources project --strict-mcp-config --dangerously-skip-permissions. Recall evals keep --allowedTools Read; swe-bench keeps the full agentic toolset. Each trial is a fresh process. demo: - Resolve the `opus` shorthand to the full claude-opus-4-8[1m] model in cost-ab and effective-context a.sh/b.sh. setup.sh intentionally keeps the base id: the proxy scope gate strips [1m] from the request before matching, so [1m]-on-launch + base-in-scope is the correct pairing. docs: - FINDINGS.md: drop the 2026-06-16 Opus 4.8 re-test sections.	2026-06-18 23:43:37 -04:00
teamchong	8e8cb09223	docs: SWE-bench tables lead with per-request -65%, demote confounded run totals	2026-06-10 23:49:38 -04:00
teamchong	76542b4a3c	eval(swe-bench): Lite pilot 10 paired instances — 10/10 resolved both arms, ON $27.27 vs OFF $53.61 (-49%) ... - run_pilot.py harness (paired generation via proxy ON 47821 / OFF 47822) - official swebench Docker grading via colima, reports + patches committed - README.md results table, FINDINGS.md task-completion parity update - repo README benchmark table gains SWE-bench row	2026-06-10 23:25:24 -04:00