mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
docs: add FAQ - end-to-end bill math vs touched-only, measurement method, compression scope
This commit is contained in:
@@ -9,7 +9,7 @@ local proxy that exploits that gap: it rewrites the bulky middle of your
|
||||
conversation into compact PNGs before the request leaves your machine.
|
||||
|
||||
Running against real Claude Code sessions, the production log (13,709
|
||||
requests) shows: a **$100 total bill becomes ~$41** — full dollar math, input
|
||||
requests) shows: a **$100 total bill becomes ~$41**, full dollar math, input
|
||||
+ cache writes at 1.25× + cache reads at 0.1× + output at 5×, including the
|
||||
~6k small requests pxpipe correctly leaves untouched. On the large requests it
|
||||
actually compresses (7,756 of them), $100 of spend becomes ~$28. Quote the
|
||||
@@ -50,8 +50,8 @@ need back byte-exact (IDs, hashes, secrets, exact numbers) must stay text.
|
||||
Recent turns do; a dedicated verbatim-risk guard is not built yet.
|
||||
|
||||
**Does it break real work?** Parity in what we measured: a 10-instance
|
||||
SWE-bench Lite pilot (the easy subset) resolved **10/10 on both arms** —
|
||||
pxpipe ON at $27 vs OFF at $54 token-equivalent — and 19 SWE-bench Pro
|
||||
SWE-bench Lite pilot (the easy subset) resolved **10/10 on both arms**,
|
||||
pxpipe ON at $27 vs OFF at $54 token-equivalent, and 19 SWE-bench Pro
|
||||
pairs (harder, long-horizon) resolved **14/19 ON vs 15/19 OFF** at
|
||||
**-60% per-request**: verdicts agree on 18/19, and the single split
|
||||
(one ON fail) re-resolved 3/3 when replicated, i.e. run-to-run agentic
|
||||
@@ -105,7 +105,7 @@ arms), same setup, official `SWE-bench_Pro-os` Docker harness:
|
||||
| resolved | 14/19 | 15/19 |
|
||||
| request size vs own uncompressed body | **−60%** | ±0 |
|
||||
|
||||
Verdicts agree on 18/19 (three instances failed both arms — one with
|
||||
Verdicts agree on 18/19 (three instances failed both arms, one with
|
||||
byte-identical patches across arms). The single split (navidrome, ON
|
||||
fail) was replicated 3x on the ON arm: all three runs produced an
|
||||
identical patch and **resolved**, so the original loss was run-to-run
|
||||
@@ -119,6 +119,42 @@ lead with the clean novel-number eval instead. Reproduce:
|
||||
[`eval/gist-recall/`](eval/gist-recall/) ·
|
||||
full analysis in [`FINDINGS.md`](FINDINGS.md).</sub>
|
||||
|
||||
## FAQ
|
||||
|
||||
**Is the 59% end-to-end, or only on the requests you touched?**
|
||||
End-to-end, the whole bill. Most compression tools report savings only on
|
||||
the input slice they touched, which flatters the number. The 59% denominator
|
||||
is every one of the 13,709 production requests: the ~6,000 small ones pxpipe
|
||||
correctly left untouched, all cache writes and reads, and all output tokens
|
||||
(which the proxy never compresses). $100 of real spend becomes ~$41.
|
||||
Touched-requests-only is -72% and is quoted separately, never as the
|
||||
headline.
|
||||
|
||||
**How is the math measured?**
|
||||
Both sides of the same request, at the same moment. For every `/v1/messages`
|
||||
POST the proxy fires a free `count_tokens` probe on the original uncompressed
|
||||
body (the counterfactual) in parallel with the real forward, and reads
|
||||
Anthropic's actually-billed usage block off the response. Both land in the
|
||||
same row of `~/.pxpipe/events.jsonl`, so there is no turn-count or
|
||||
run-to-run confound. Dollar conversion uses Fable 5 list ratios: input ×1.0,
|
||||
cache write ×1.25, cache read ×0.1, output ×5. Cache pricing is applied
|
||||
identically to both sides, so the caching discount cancels and cannot be
|
||||
double-counted as "savings". Re-derive it yourself from the events log: the
|
||||
formula and field names are documented in `src/core/baseline.ts`.
|
||||
|
||||
**What does it actually compress?**
|
||||
Three kinds of *input* blocks, each behind a profitability gate:
|
||||
|
||||
1. large `tool_result` bodies (file reads, command output, logs) above
|
||||
~6k chars of token-dense content
|
||||
2. older collapsed history: turns behind the live tail get re-rendered as
|
||||
image pages, recent turns always stay text
|
||||
3. the static system prompt + tool docs slab
|
||||
|
||||
Everything else passes through byte-identical: your messages, recent turns,
|
||||
the model's output (it is the response, the proxy never touches it), sparse
|
||||
prose, and anything too small to win. Non-Fable models pass through entirely.
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user