teamchong f1688e9f46 transform+render: R2 multi-col default-on + per-request pixel/char instrumentation
Two pieces of work landing together because they're interleaved in
transform.ts and need to ship at the same time for the dashboard
fit-wiring in the next commit to work.

# R2 multi-column rendering (default `multiCol=2`)

Single-col packing wastes ~68% of Anthropic's 1568×1568 image budget
on real Claude Code tool docs (~38 chars/row). The row-aware break-
even gate correctly rejects compression on those slabs — proxy ends
up doing nothing on the heaviest requests. Multi-col packs N text
columns side-by-side per image so each image covers `N×LINES_PER_IMAGE`
wrapped lines, dropping image count by ~N and crossing break-even.

- render.ts: new `renderTextToPngsMultiCol(text, cols, numCols)` with
  column-major layout (col 0 fills top-to-bottom, then col 1, …).
  Plus `multiColWidth` / `maxFittingCols` helpers. CLI auto-clamps
  numCols so the canvas can never exceed the 1568 px width cap.
- transform.ts: `multiCol` option threaded through; gate, image-count
  estimator, paging budget, and all render paths take `numCols`.
  Intro text now describes column reading order when numCols>1 so
  the OCR pass doesn't read row-wise across columns.
- Host wiring: node.ts (--multi-col / MULTI_COL=N, default 2),
  worker.ts (MULTI_COL env, default 2). Startup log shows multi_col=N.
- numCols=1 stays a byte-identical passthrough to renderTextToPngs
  so flipping back can't regress the cache_control hit rate.

# Empirical-cost instrumentation

The dashboard's `OPUS_IMAGE_TOKEN_COST = 2500` and `CHARS_PER_TOKEN = 4`
are stale pre-Opus-4.7 constants. Fact-checked: 4.7 changed the TEXT
tokenizer only (~1.20× inflation on English, ~1.36× on TypeScript);
image cost is unchanged for same-size images. Direction of error in
the dashboard headline: it UNDERSTATES text savings.

Ditched the "guess from one cold-miss" approach in favor of real
instrumentation that pins both axes at once:

- TransformInfo.imagePixels — Σ width×height across every image
  this request (static slab + reminders + tool_results + history
  collapse). Threaded through textToImageBlocks, collapseHistory.
- TransformInfo.outgoingTextChars — every char of TEXT remaining
  in the transformed outgoing body, computed in one walk before
  serialization (countOutgoingTextChars).
- Tracker event schema picks up image_pixels + outgoing_text_chars.
- history.ts: HistoryCollapseInfo.collapsedImagePixels accumulates.

The regression `tokens ≈ α·outgoing_text_chars + β·image_pixels`
over N cold-miss events solves for empirical chars/token (α) and
pixels/token (β) under the live model — no assumed constants.

# Tests

- 8 new render tests covering the multi-col passthrough contract
  (numCols=1 byte-identical to renderTextToPngs), canvas dimensions,
  image-count halving, determinism, charsRendered sum, estimator
  parity, CLI clamp, and CJK wide-glyph wrap.
- paging.test.ts "counts multiple tool_results" threshold loosened
  from >770k to >700k to cover both single-col and multi-col budgets
  at the same maxImagesPerToolResult cap.
- All 241 tests pass, typecheck + build clean.
2026-05-19 11:39:40 -04:00
2026-05-18 11:52:02 -04:00

pixelpipe

A token-saving proxy for Claude Code that renders the system prompt + tool definitions as images, so Claude OCRs them instead of paying for them as text. 65-73% input-token savings on Opus 4.7, 100% reasoning quality preserved, identical fixed text every turn for a clean prompt-cache.

Runs on Node 18+ and Cloudflare Workers from the same source.


How it works

                                  ┌─ original ────────────────────┐
                                  │ ~68K input tok                │
Claude Code  ──►  pixelpipe  ──►  │  (system + tools as text)     │  ──►  Anthropic
                       │          └───────────────────────────────┘
                       └──────►   ┌─ via proxy ───────────────────┐
                                  │ ~3.5K input tok               │
                                  │  (system + tools as PNG +     │
                                  │   prompt-cache breakpoint)    │
                                  └───────────────────────────────┘
                                          ↓ Anthropic vision OCR
                                          100% reasoning quality retained

The proxy intercepts POST /v1/messages, pulls the system prompt + tool documentation out of the JSON body, renders it into one or more grayscale PNGs using a build-time-generated GNU Unifont glyph atlas (covers ~35k BMP codepoints by default — Latin, Cyrillic, Greek, CJK, Hiragana, Katakana, Hangul, Hebrew, Arabic, math symbols, box drawing, decorative symbols), and substitutes those PNGs back in as image content blocks with an ephemeral cache_control breakpoint.

Token math (Opus 4.7, real Claude Code workflow):

metric original via proxy savings
Cold input tokens ~68K ~3.5K 95%
Cache-warm input tokens ~7.5K ~3.5K 53%
Per-call median (mixed) - - 65-73%
Per-image OCR quality vs txt - - ~99.5%

Quick start (Node)

npm install
npm run build           # produces dist/node.js
node bin/cli.js         # listens on 127.0.0.1:47821 by default

After editing code, restart in one step:

pnpm run restart                              # graceful SIGTERM of any running
                                               # instance → rebuild → fresh start
pnpm run restart -- --no-build                # skip rebuild (dist/ is fresh)
pnpm run restart -- --port 47822 --no-tools   # forward CLI flags to the proxy

pnpm run restart does, in order:

  1. Lists every running pixelpipe PID (via pgrep) and SIGTERMs them all. Orphans from prior crashed sessions are cleaned up too.
  2. Waits up to 5s for graceful exit (the SIGTERM handler flushes the JSONL tracker). Escalates to SIGKILL only if anything's still alive.
  3. Runs pnpm run build. Build failures abort the restart — the script refuses to start a stale binary. Pass --no-build to skip when you know dist/ is fresh.
  4. Checks the target port is free. If it isn't, names the holding process and refuses to start (cheaper than a crashed Node stacktrace).
  5. execs node bin/cli.js "$@" in the foreground so Ctrl-C reaches Node.

Point Claude Code at it:

ANTHROPIC_BASE_URL=http://127.0.0.1:47821 \
  claude --exclude-dynamic-system-prompt-sections

That's it. Use Claude Code normally.

The --exclude-dynamic-system-prompt-sections flag suppresses the small per-turn variable section so the rendered image stays byte-identical across turns — that's what makes the prompt cache actually hit.


Quick start (Cloudflare Workers)

npx wrangler dev        # local dev on :8787
npx wrangler deploy     # ship to *.workers.dev

Then in Claude Code:

ANTHROPIC_BASE_URL=https://pixelpipe.<your-account>.workers.dev \
  claude --exclude-dynamic-system-prompt-sections

You can attach a custom hostname and route in wrangler.toml.


Configuration

Both runtimes read the same options — Node from CLI flags or env, Worker from wrangler.toml [vars].

flag / var default meaning
--port PORT 47821 Node only — listen port
--upstream ANTHROPIC_UPSTREAM https://api.anthropic.com where to forward
--no-compress COMPRESS=0 on master switch
--no-tools COMPRESS_TOOLS=0 on fold tool docs into the image
--no-schemas COMPRESS_SCHEMAS=0 on include input_schema JSON in the image
--min-chars MIN_COMPRESS_CHARS 2000 skip compression below this many chars
--placement PLACEMENT system system or user — where image lands
--cols COLS 100 soft-wrap column count

In Workers, set the optional upstream API key with:

npx wrangler secret put ANTHROPIC_API_KEY

If unset, the proxy forwards whatever x-api-key the client sent.


Architecture

src/
├── core/              100% runtime-agnostic (Web Standard APIs only)
│   ├── atlas.ts         (generated) sparse Unicode atlas, base64-inlined
│   ├── png.ts           minimal grayscale PNG encoder
│   ├── render.ts        text → PNG bytes
│   ├── transform.ts     request body rewriter
│   ├── proxy.ts         the fetch handler
│   └── types.ts         Anthropic API types
├── node.ts            node:http adapter + CLI
└── worker.ts          export default { fetch }

scripts/
├── gen-atlas.ts       build-time: OTF → atlas.ts (uses @napi-rs/canvas)
└── build.mjs          esbuild bundler for Node target

assets/
├── Unifont-16.0.04.otf       primary font (~35k BMP codepoints w/ full-bmp profile)
├── UNIFONT_LICENSE.txt       OFL + GPL-with-font-exception
└── JetBrainsMono-Regular.ttf legacy / ASCII-only fallback (kept on disk)

The atlas is generated at build time from Unifont-16.0.04.otf, base64-inlined into a .ts file with sparse codepoint + offset tables (binary-packed), and shipped with the bundle. At runtime there are zero external files to read and zero non-Web-Standard imports — that's the only way this works in Workers without per-request asset fetches.

Regenerate the atlas (after swapping fonts, sizes, or codepoint profile):

pnpm run build:atlas                          # default: full-bmp (~35k cp, all BMP Unifont covers)
ATLAS_PROFILE=practical pnpm run build:atlas  # drops Hangul (~24k cp; for Workers free-tier)
FONT_PX=12 pnpm run build:atlas               # nondefault size; verify cells

Limitations

  • The bundled GNU Unifont at 10px (cell 5×11 px Latin, 10×11 CJK) is Anthropic-OCR-clean for ~35k BMP codepoints by default (full-bmp profile): Latin, Cyrillic, Greek, CJK Unified Ideographs, Hiragana, Katakana, Hangul, Hebrew, Arabic, math symbols, box-drawing, arrows, Dingbats, Letterlike Symbols, Enclosed Alphanumerics, etc. Drops for codepoints outside the profile (e.g. emoji 😀 — supplementary plane) get counted in events.jsonl#dropped_chars (with the top-20 broken out as dropped_codepoints_top) so you can spot patterns. For Workers free-tier deployments under the 1 MB compressed-bundle cap, switch to ATLAS_PROFILE=practical pnpm run build:atlas (~24k cp; drops Hangul). Right-to-left scripts render left-to-right in source order (no bidi shaping); Devanagari / Thai / similar complex shaping is also unsupported.
  • Compression sets a 5-minute prompt-cache TTL. Adding cache_control: ephemeral causes warm-cache rotation, not eviction.
  • A 5KB break-even point: if input is < MIN_COMPRESS_CHARS chars we skip compression entirely (overhead would exceed savings).
  • Per-machine font: regenerate the atlas if you swap fonts. The generated src/core/atlas.ts is checked in so consumers don't need @napi-rs/canvas to install.
  • Workers CPU limit: this is fine for free-tier (10ms CPU) on small prompts; large prompts (>30K chars) may need the paid tier.

Development

npm install
npm run dev:node              # tsx watch on src/node.ts
npm run dev:worker            # wrangler dev
npm run test                  # vitest
npm run test:watch
npm run typecheck             # tsc --noEmit
pnpm run build:atlas          # regenerate src/core/atlas.ts from OTF
npm run build                 # build dist/node.js
npm run deploy:worker         # wrangler deploy

License

MIT.

S
Description
cut Fable 5 token usage by rendering text context as images
Readme MIT 24 MiB
Languages
TypeScript 86.7%
JavaScript 9.5%
Shell 2.1%
Python 1.7%