diff --git a/README.md b/README.md index 215f654..7fe4a4a 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ local proxy that exploits that gap: it rewrites the bulky middle of your conversation into compact PNGs before the request leaves your machine. Running against real Claude Code sessions, the production log (13,709 -requests) shows: a **$100 total bill becomes ~$41** — full dollar math, input +requests) shows: a **$100 total bill becomes ~$41**, full dollar math, input + cache writes at 1.25× + cache reads at 0.1× + output at 5×, including the ~6k small requests pxpipe correctly leaves untouched. On the large requests it actually compresses (7,756 of them), $100 of spend becomes ~$28. Quote the @@ -50,8 +50,8 @@ need back byte-exact (IDs, hashes, secrets, exact numbers) must stay text. Recent turns do; a dedicated verbatim-risk guard is not built yet. **Does it break real work?** Parity in what we measured: a 10-instance -SWE-bench Lite pilot (the easy subset) resolved **10/10 on both arms** — -pxpipe ON at $27 vs OFF at $54 token-equivalent — and 19 SWE-bench Pro +SWE-bench Lite pilot (the easy subset) resolved **10/10 on both arms**, +pxpipe ON at $27 vs OFF at $54 token-equivalent, and 19 SWE-bench Pro pairs (harder, long-horizon) resolved **14/19 ON vs 15/19 OFF** at **-60% per-request**: verdicts agree on 18/19, and the single split (one ON fail) re-resolved 3/3 when replicated, i.e. run-to-run agentic @@ -105,7 +105,7 @@ arms), same setup, official `SWE-bench_Pro-os` Docker harness: | resolved | 14/19 | 15/19 | | request size vs own uncompressed body | **−60%** | ±0 | -Verdicts agree on 18/19 (three instances failed both arms — one with +Verdicts agree on 18/19 (three instances failed both arms, one with byte-identical patches across arms). The single split (navidrome, ON fail) was replicated 3x on the ON arm: all three runs produced an identical patch and **resolved**, so the original loss was run-to-run @@ -119,6 +119,42 @@ lead with the clean novel-number eval instead. Reproduce: [`eval/gist-recall/`](eval/gist-recall/) · full analysis in [`FINDINGS.md`](FINDINGS.md). +## FAQ + +**Is the 59% end-to-end, or only on the requests you touched?** +End-to-end, the whole bill. Most compression tools report savings only on +the input slice they touched, which flatters the number. The 59% denominator +is every one of the 13,709 production requests: the ~6,000 small ones pxpipe +correctly left untouched, all cache writes and reads, and all output tokens +(which the proxy never compresses). $100 of real spend becomes ~$41. +Touched-requests-only is -72% and is quoted separately, never as the +headline. + +**How is the math measured?** +Both sides of the same request, at the same moment. For every `/v1/messages` +POST the proxy fires a free `count_tokens` probe on the original uncompressed +body (the counterfactual) in parallel with the real forward, and reads +Anthropic's actually-billed usage block off the response. Both land in the +same row of `~/.pxpipe/events.jsonl`, so there is no turn-count or +run-to-run confound. Dollar conversion uses Fable 5 list ratios: input ×1.0, +cache write ×1.25, cache read ×0.1, output ×5. Cache pricing is applied +identically to both sides, so the caching discount cancels and cannot be +double-counted as "savings". Re-derive it yourself from the events log: the +formula and field names are documented in `src/core/baseline.ts`. + +**What does it actually compress?** +Three kinds of *input* blocks, each behind a profitability gate: + +1. large `tool_result` bodies (file reads, command output, logs) above + ~6k chars of token-dense content +2. older collapsed history: turns behind the live tail get re-rendered as + image pages, recent turns always stay text +3. the static system prompt + tool docs slab + +Everything else passes through byte-identical: your messages, recent turns, +the model's output (it is the response, the proxy never touches it), sparse +prose, and anything too small to win. Non-Fable models pass through entirely. + ## How it works ```