docs: add FAQ - end-to-end bill math vs touched-only, measurement method, compression scope

This commit is contained in:
teamchong
2026-06-11 16:53:18 -04:00
parent d990faf646
commit 62790e7371
+40 -4
View File
@@ -9,7 +9,7 @@ local proxy that exploits that gap: it rewrites the bulky middle of your
conversation into compact PNGs before the request leaves your machine.
Running against real Claude Code sessions, the production log (13,709
requests) shows: a **$100 total bill becomes ~$41** full dollar math, input
requests) shows: a **$100 total bill becomes ~$41**, full dollar math, input
+ cache writes at 1.25× + cache reads at 0.1× + output at 5×, including the
~6k small requests pxpipe correctly leaves untouched. On the large requests it
actually compresses (7,756 of them), $100 of spend becomes ~$28. Quote the
@@ -50,8 +50,8 @@ need back byte-exact (IDs, hashes, secrets, exact numbers) must stay text.
Recent turns do; a dedicated verbatim-risk guard is not built yet.
**Does it break real work?** Parity in what we measured: a 10-instance
SWE-bench Lite pilot (the easy subset) resolved **10/10 on both arms**
pxpipe ON at $27 vs OFF at $54 token-equivalent and 19 SWE-bench Pro
SWE-bench Lite pilot (the easy subset) resolved **10/10 on both arms**,
pxpipe ON at $27 vs OFF at $54 token-equivalent, and 19 SWE-bench Pro
pairs (harder, long-horizon) resolved **14/19 ON vs 15/19 OFF** at
**-60% per-request**: verdicts agree on 18/19, and the single split
(one ON fail) re-resolved 3/3 when replicated, i.e. run-to-run agentic
@@ -105,7 +105,7 @@ arms), same setup, official `SWE-bench_Pro-os` Docker harness:
| resolved | 14/19 | 15/19 |
| request size vs own uncompressed body | **60%** | ±0 |
Verdicts agree on 18/19 (three instances failed both arms one with
Verdicts agree on 18/19 (three instances failed both arms, one with
byte-identical patches across arms). The single split (navidrome, ON
fail) was replicated 3x on the ON arm: all three runs produced an
identical patch and **resolved**, so the original loss was run-to-run
@@ -119,6 +119,42 @@ lead with the clean novel-number eval instead. Reproduce:
[`eval/gist-recall/`](eval/gist-recall/) ·
full analysis in [`FINDINGS.md`](FINDINGS.md).</sub>
## FAQ
**Is the 59% end-to-end, or only on the requests you touched?**
End-to-end, the whole bill. Most compression tools report savings only on
the input slice they touched, which flatters the number. The 59% denominator
is every one of the 13,709 production requests: the ~6,000 small ones pxpipe
correctly left untouched, all cache writes and reads, and all output tokens
(which the proxy never compresses). $100 of real spend becomes ~$41.
Touched-requests-only is -72% and is quoted separately, never as the
headline.
**How is the math measured?**
Both sides of the same request, at the same moment. For every `/v1/messages`
POST the proxy fires a free `count_tokens` probe on the original uncompressed
body (the counterfactual) in parallel with the real forward, and reads
Anthropic's actually-billed usage block off the response. Both land in the
same row of `~/.pxpipe/events.jsonl`, so there is no turn-count or
run-to-run confound. Dollar conversion uses Fable 5 list ratios: input ×1.0,
cache write ×1.25, cache read ×0.1, output ×5. Cache pricing is applied
identically to both sides, so the caching discount cancels and cannot be
double-counted as "savings". Re-derive it yourself from the events log: the
formula and field names are documented in `src/core/baseline.ts`.
**What does it actually compress?**
Three kinds of *input* blocks, each behind a profitability gate:
1. large `tool_result` bodies (file reads, command output, logs) above
~6k chars of token-dense content
2. older collapsed history: turns behind the live tail get re-rendered as
image pages, recent turns always stay text
3. the static system prompt + tool docs slab
Everything else passes through byte-identical: your messages, recent turns,
the model's output (it is the response, the proxy never touches it), sparse
prose, and anything too small to win. Non-Fable models pass through entirely.
## How it works
```