pxpipe/eval

Reflow Eval Harness

Evaluation harness for the reflow image-rendering mode in pxpipe.

Reflow re-packs text densely and marks original newlines with the ↵ glyph (U+21B5) before rendering to PNG. It reduces image count by ~30–50% on typical Claude Code history. This harness verifies that the model still understands reflowed text — telemetry measures tokens/bytes, not comprehension.

---

Prerequisites

# Build the compiled dist/ output (required by the eval scripts)
pnpm run build

# Extract corpus from your local Claude conversations
node eval/extract-corpus.mjs

The corpus is written to eval/corpus/:

• text-blocks.json — text blocks for L1 OCR eval (~20 by default)
• sessions.json — conversation sessions for L2 session replay (~10 by default)

---

L0 — Unit Tests (reference only)

L0 is a vitest unit test file (tests/render.test.ts plus a forthcoming tests/reflow.test.ts owned by the core team). It tests the reflow / dereflow / renderTextToPngsReflow functions in isolation.

# Run all unit tests including L0
pnpm test

L0 does not require an API key and runs in CI. It verifies:

• dereflow(reflow(t)) === minifyForRender(t) for all t
• Sentinel collision fallback
• Image dimensions and chunk counts

---

L1 — OCR Fidelity

Renders text blocks to PNG two ways (baseline and reflow), sends each image set to the Anthropic API asking for verbatim transcription, then diffs the result against the source using character-level Levenshtein edit distance.

The reflow system prompt includes: "↵ denotes a line break".

Dry run (no API key, no cost)

node eval/eval-l1-ocr.mjs --dry-run
# or via orchestrator:
node eval/run-eval.mjs --level 1 --dry-run

Cost estimate only

node eval/run-eval.mjs --estimate-only

Real run

export ANTHROPIC_API_KEY=sk-ant-...
node eval/eval-l1-ocr.mjs --confirm
# or:
node eval/run-eval.mjs --level 1 --confirm

Estimated cost (L1 only, 20 blocks, claude-sonnet-4-5)

Item	Estimate
API calls	40 (2 per block × 20 blocks)
Input tokens	~70,000–90,000 (dominated by image tiles)
Output tokens	~10,000–20,000 (transcriptions)
USD	~$0.50–$1.00

The actual cost depends on rendered image sizes. Most blocks produce 1–2 PNGs, costing ~1,600 tokens per image tile at Sonnet pricing.

Output

• eval/results/l1-report.md — markdown report with per-block scores
• eval/results/l1-results.json — raw JSON for programmatic use

Interpreting L1 results

Metric	Threshold	Meaning
Mean char accuracy delta	≥ −2pp	Acceptable; within noise
Mean char accuracy delta	< −5pp	Reflow OCR materially worse
Macro accuracy (reflow)	≥ 95%	High overall character fidelity
Image count savings	30–50%	Expected typical range

A negative delta means reflow is slightly less accurate. Up to −2pp is acceptable because: (a) the reflow system prompt compensates for ↵ rendering, (b) real accuracy is bounded by the model's vision encoder, not the sentinel.

---

L2 — Task-level A/B Session Replay

This is the real gate. Extracts real conversation sessions from ~/.claude/projects/**/*.jsonl, renders the history both ways, asks the model to produce the next turn in the conversation, then uses a model-judge to score whether the reflow-history answer is as good as the baseline.

Dry run

node eval/eval-l2-session.mjs --dry-run
# or:
node eval/run-eval.mjs --level 2 --dry-run

Real run

export ANTHROPIC_API_KEY=sk-ant-...
node eval/eval-l2-session.mjs --confirm
# or full pipeline:
node eval/run-eval.mjs --level all --confirm

Estimated cost (L2 only, 10 sessions, claude-sonnet-4-5)

Item	Estimate
API calls	30 (2 replay + 1 judge per session × 10)
Input tokens	~200,000–400,000 (history images are large)
Output tokens	~10,000–30,000
USD	~$1.50–$4.00

History rendering dominates: each session history is 2,000–8,000 chars → 1–5 images → 1,600–8,000 input tokens per image call. Use --max-sessions to control scope.

Full run cost (L1 + L2, 20 blocks + 10 sessions)

~$2.00–$5.00 USD with claude-sonnet-4-5.

Use --model claude-haiku-4-5 to reduce cost by ~4× at some accuracy trade-off.

Output

• eval/results/l2-report.md — markdown report with per-session judge scores
• eval/results/l2-results.json — raw JSON
• eval/results/summary.md — combined summary with shipping gate checklist

Interpreting L2 results

Metric	Threshold	Action
Mean judge score	≥ 0.80	✅ Ship reflow
Mean judge score	0.65–0.79	⚠️ Investigate failing sessions
Mean judge score	< 0.65	❌ Do not ship reflow
Pass rate (≥ 0.75)	≥ 80%	✅ Consistent quality
Pass rate	60–79%	⚠️ Some sessions fail
Pass rate	< 60%	❌ Widespread comprehension loss

The judge uses this scoring rubric:

• 1.0 — semantically equivalent to baseline
• 0.8 — mostly equivalent, minor differences
• 0.6 — partially equivalent, some content missing
• 0.4 — substantially worse
• 0.2 — mostly unrelated
• 0.0 — completely wrong

---

Full Pipeline

# 1. Build dist/
pnpm run build

# 2. Extract corpus (writes eval/corpus/)
node eval/extract-corpus.mjs --max-blocks 20 --max-sessions 10

# 3. Cost estimate
node eval/run-eval.mjs --estimate-only

# 4. Dry run (verify end-to-end without spend)
node eval/run-eval.mjs --dry-run

# 5. Real run (requires API key + explicit --confirm)
export ANTHROPIC_API_KEY=sk-ant-...
node eval/run-eval.mjs --confirm

# 6. View results
cat eval/results/summary.md

---

Options Reference

extract-corpus.mjs

Flag	Default	Description
--max-blocks N	20	Max text blocks for L1
--max-sessions N	10	Max sessions for L2
--out-dir DIR	eval/corpus	Output directory
--projects-dir DIR	~/.claude/projects	Claude projects directory
--verbose	false	Verbose progress

eval-l1-ocr.mjs / eval-l2-session.mjs / run-eval.mjs

Flag	Default	Description
--dry-run	false	No API calls; fake scores
--confirm	false	Required for real API spend
--max-blocks N	20	L1 block count
--max-sessions N	10	L2 session count
--model NAME	claude-sonnet-4-5	Replay + transcription model
--judge-model NAME	(same as model)	L2 judge model
--corpus-dir DIR	eval/corpus	Corpus input directory
--out-dir DIR	eval/results	Results output directory
--estimate-only	false	Print cost estimate and exit
--skip-extract	false	Skip corpus extraction step
--level 1|2|all	all	Which levels to run (run-eval only)
--verbose	false	Verbose per-block/session output

---

File Layout

eval/
├── README.md                 ← this file
├── extract-corpus.mjs        ← corpus extraction from ~/.claude/projects
├── eval-l1-ocr.mjs           ← L1 OCR fidelity eval
├── eval-l2-session.mjs       ← L2 session replay eval
├── run-eval.mjs              ← top-level orchestrator
├── lib/
│   ├── anthropic-client.mjs  ← minimal Anthropic API client (fetch-based)
│   ├── cost.mjs              ← token/USD cost estimator
│   ├── diff.mjs              ← Levenshtein + character accuracy scorer
│   └── render-bridge.mjs     ← imports render functions from dist/
├── corpus/                   ← generated by extract-corpus.mjs
│   ├── text-blocks.json
│   └── sessions.json
└── results/                  ← generated by eval runs
    ├── l1-report.md
    ├── l1-results.json
    ├── l2-report.md
    ├── l2-results.json
    └── summary.md

---

Notes

• The eval imports from dist/core/render.js. Run pnpm run build first.
• No extra npm packages are required. The Anthropic client uses Node's built-in fetch.
• Dry-run mode simulates ~3% OCR error rate to produce non-trivial diff scores.
• The corpus extractor gracefully falls back to a synthetic corpus when ~/.claude/projects is empty or absent.
• .claude/ is globally gitignored — corpus and results are in eval/ and should be added to .gitignore if you don't want to commit them.