From 09e4f635ea951cfa8796746d2462f588dcfc6335 Mon Sep 17 00:00:00 2001 From: Steven Chong <25894545+teamchong@users.noreply.github.com> Date: Thu, 21 May 2026 12:26:26 -0400 Subject: [PATCH] docs(readme): stop claiming '$ saved'; report token deltas only The README headlined a '~76% fewer tokens' framing, a worked example with a 'savings' column, and an Opus pricing table that used the old Opus-4-base rates ($5/$25/MTok input/output) instead of Opus 4.7's actual rates ($2.50/$12.50). Combined, it read like an aggregate $-savings claim that the data does not yet support. What's actually true and stays: - Pixelpipe ships fewer input tokens on cold-miss requests, measured by Anthropic's own count_tokens endpoint. The 173,783 -> 41,321 anecdote is one real request, not a session average. - Image-tokens-vs-text is a real per-request encoding-density observation. What was misleading and is removed/softened: - Opening tagline no longer claims '~76% fewer tokens, 100% quality' as if it generalizes. Replaced with an experimental-status note. - Worked-example 'savings' column renamed 'delta' and contextualized. - 'How we report numbers' section added that explains the baseline_probe_status gate, says we don't currently make an aggregate $ saved claim, and warns the reader to treat any host dashboard that surfaces $ saved without the gate as marketing. - Pricing table corrected to Opus 4.7 rates. --- README.md | 68 +++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 46 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 04f027c..2f97609 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,15 @@ A proxy for Claude Code that intercepts `POST /v1/messages` and renders the bulky static inputs (system prompt + tool docs + closed-prefix history) as grayscale PNGs, letting Opus 4.7's vision stack OCR them on -the way in. Same information, ~76% fewer tokens, 100% reasoning quality -preserved, byte-identical fixed text every turn for a clean prompt-cache -hit. +the way in. The hypothesis: pixels are a denser encoding for the same +information. + +> **Status:** research / experimental. Pixelpipe demonstrably ships +> fewer input tokens on cold-miss requests (measured by Anthropic's own +> `count_tokens` endpoint). Whether that translates into a real +> end-to-end dollar saving on a multi-turn session depends on cache +> behavior, output / thinking tokens, and break-even math we are still +> measuring. We don't currently make a "$ saved" claim. **Opus 4.7 only.** Pre-4.7 vision wasn't accurate enough on dense monospace glyphs — OCR errors would corrupt the prompt before the model @@ -44,11 +50,12 @@ conversation history as designed timelines, tables as actual tables. Concatenating text is how 2024-era prompt engineering works; designing a visual surface is what context-as-UI looks like. -One measured cold-miss event in `events.jsonl` went from **173,783** -input tokens (what Anthropic's `count_tokens` endpoint measures on the -unproxied body) to **41,321** `cache_create` tokens — a **~76% denser -encoding of the same information**, with 100% reasoning quality preserved -and byte-identical fixed text every turn for a clean prompt-cache hit. +On one measured cold-miss request in `events.jsonl`, Anthropic's +`count_tokens` reported **173,783** input tokens for the unproxied body +and **41,321** `cache_create` tokens for the proxied body — a ~76% +reduction in tokens shipped on that single request. That's an +*encoding-density* observation about one request, not an aggregate +$-savings claim. See "How we report numbers" below for the difference. ## What this is NOT @@ -149,12 +156,12 @@ Rates from [Anthropic's pricing page](https://www.anthropic.com/pricing) for Opus 4.7 (input pricing has been flat across 4.5/4.6/4.7). Image tokens are billed at the input rate. -| line item | rate | -| -------------------- | ------------- | -| input | $5.00 / MTok | -| output | $25.00 / MTok | -| cache_create (5 min) | $6.25 / MTok | -| cache_read | $0.50 / MTok | +| line item | rate | +| -------------------- | -------------- | +| input | $2.50 / MTok | +| output | $12.50 / MTok | +| cache_create (5 min) | $3.125 / MTok | +| cache_read | $0.25 / MTok | ### Worked example — one real cold-miss event @@ -180,17 +187,34 @@ cache_create_tokens 111 only the per-turn dynamic delta cache_read_tokens 140,786 paid at 0.1× the input rate ``` -The cold-miss savings are large because the proxy shrinks what gets -cached. Warm-hit savings look small in tokens but are actually larger in -dollars because the 132k-token delta avoids the 1.25× cache_create -premium and instead pays the 0.1× cache_read price. - -| metric | original | via proxy | savings | +| metric | original | via proxy | delta | | ---------------------------- | -------- | --------- | -------- | -| Cold input tokens (per call) | ~174k | ~41k | ~76% | -| Cache-warm input tokens | ~169k | ~141k | ~17% | +| Cold input tokens (per call) | ~174k | ~41k | ~76% fewer | +| Cache-warm input tokens | ~169k | ~141k | ~17% fewer | | Per-image OCR quality vs txt | - | - | ~99.5% | +These are per-request token-count deltas, not a session-level cost +claim. A real session interleaves cold-miss and cache-warm calls, +includes output / thinking tokens we don't touch, and depends on cache +TTL and Claude Code's usage shape — none of which a single-request +delta captures. + +### How we report numbers + +Pixelpipe instruments every proxied request with two free +`count_tokens` probes on the original uncompressed body — one full, +one truncated at the last cache marker — and persists them alongside +Anthropic's billing `usage` block. The bundled dashboard uses those +to show **token deltas per request** and **aggregate token counts**, +and refuses to display a "$ saved" headline unless both probes +succeeded *and* the host has wired in pricing. On real traffic to +date, the honest aggregate is closer to break-even than the +cold-miss number above suggests; the value proposition is still +under measurement. + +If you see a "$ saved" number coming out of a host integration that +doesn't expose this gating, treat it as marketing, not measurement. + --- ## Why it's hard (the parts that bite)