teamchongDemo 1 — cost A/B
What it measures: does pxpipe cost less on a real coding task? Honest verdict:
~break-even on cost. The compression is real (~55% fewer real tokens, verified)
but it lands in cache_read — cheap at $ (0.1×), and its weight against a Pro/Max
weekly cap is unpublished. The capability story is in
../effective-context/.
Two columns fix the same failing test suite in isolated working copies — one
plain, one through pxpipe — both behind a proxy so both arms are logged. It's a
real project (template/): the fix hinges on precise SPEC.md rules (volume tiers by
total quantity, loyalty applied after the discount, banker's rounding), so it
doubles as a recall test. node --test is built in (no install).
Run it — 3 scripts, 3 terminals
# Terminal 1 — set up: kills old proxies, builds, starts BOTH proxies, seeds copies
bash demo/cost-ab/setup.sh # Fable only (Opus off); 'setup.sh opus' to enable Opus
# Terminal 2 — LEFT = normal (interactive Claude — you watch the CLI)
bash demo/cost-ab/a.sh # defaults to Fable; `a.sh opus` to use Opus
# Terminal 3 — RIGHT = pxpipe (interactive Claude)
bash demo/cost-ab/b.sh # use the SAME model as a.sh for a fair A/B
a.sh / b.sh launch a real interactive Claude session with the task prompt
already submitted — you see the CLI work, nothing headless. They run two real
sessions (uses plan usage). claude is usually a shell alias; the scripts resolve
the real binary, or set CLAUDE_BIN=/path/to/claude. To redo a run, re-run
setup.sh (it resets the working copies + fresh logs), then a.sh / b.sh.
Model: everything defaults to Fable 5 — Opus is off by default, matching
production. To run the A/B on another model, pass it to all three scripts:
setup.sh opus (adds Opus to the :47824 proxy's compress scope), then
a.sh opus / b.sh opus (also sonnet, haiku, or a full claude-… id) —
same model on both columns for a fair A/B. setup.sh is what enables compression
for that model; you can also toggle it live on the dashboard "compress models" chips.
See the result — just open the dashboard
Each proxy serves a live dashboard in your browser — no commands, no extra window:
| open in browser | shows |
|---|---|
http://127.0.0.1:47824/ (pxpipe → b.sh) |
THIS SESSION — N% fewer tokens (… total) |
http://127.0.0.1:47823/ (plain → a.sh) |
~0% — the passthrough control |
It updates as the run goes. The headline is the honest, rate-free number: real
server tokens (input+cache_create+cache_read+output) vs the same body as text
(count_tokens) — two real numbers, one division, no rate/cap assumptions. The
plain dashboard reading ~0% proves the method doesn't invent savings.
Optional CLI (same numbers, if you prefer a terminal):
node eval/ab/savings.mjs # token compression, both arms
node eval/ab/analyze.mjs ~/.pxpipe/ab-on.jsonl ~/.pxpipe/ab-off.jsonl # $ / cap?? (divergence-confounded)
The $/cap?? deltas from analyze.mjs compare two different runs, so they're
muddied by divergence — trust the per-arm token % (dashboard or savings.mjs).
What the token cut saves depends on pricing (cache_read ×0.1 at $; weekly-cap
weight unknown).
The other demo
This is the cost demo. The capability demo ("does pxpipe stay sharp where plain
drowns?") is in ../effective-context/ — the more
promising story, since cost is ~break-even.