diff --git a/FINDINGS.md b/FINDINGS.md index c7ceecb..7521f07 100644 --- a/FINDINGS.md +++ b/FINDINGS.md @@ -1,13 +1,52 @@ # FINDINGS — pxpipe (text→PNG token compression) **Status:** ⚠️ **VERDICT REVERSED — see correction below.** Originally ruled "dead"; live measurement shows pxpipe is a working *lossy gist-compressor* saving ~68% on real (dense) Claude Code traffic, with a known verbatim-recall gap. -**Date:** 2026-05-28 (original) · 2026-05-29 (correction) · 2026-06-09 (Fable 5 update) · 2026-06-10 (gist-recall A/B, SWE-bench pilot) +**Date:** 2026-05-28 (original) · 2026-05-29 (correction) · 2026-06-09 (Fable 5 update) · 2026-06-10 (gist-recall A/B, SWE-bench pilot) · 2026-06-12 (field observation, n=1) **Models tested:** `claude-opus-4-5` (original run), `claude-opus-4-8` (re-test after a model bump), `claude-fable-5` (2026-06-09) **Model scope (current):** Fable 5 only, enforced in library + proxy (Opus disabled 2026-06-09 — see update below). **Harness:** `eval/needle-haystack/` (receipts preserved from `/tmp/needle_eval`) --- +## Update (2026-06-12) — field observation: live verbatim misreads in a real session (n=1, anecdotal) + +**Not a controlled eval.** Logged because it is the first *unprompted, +in-the-wild* capture of the exact boundary the harnesses bracket, with +ground truth available for post-hoc grading. + +Setup: a working session was resumed from imaged history (older context +rendered to PNG pages, recent turns text — the production architecture). +Mid-session, the assistant was challenged to transcribe, from the imaged +history alone, ten 12-char hex IDs that had appeared in an earlier tool +output. The transcription was committed *before* re-reading the source +files on disk, which then served as ground truth. + +| tier | result | +|---|---| +| gist (resume a paused run: plan, counts, file states, next steps) | fully functional — zero observed errors, consistent with gist-recall 98/98 | +| verbatim (10 × 12-char hex through imaged history) | **7/10 exact** · 4 char-level errors / 120 chars (~3.3%) | + +Misread detail: `0→8`, `e→4`, `e→8` substitutions plus one dropped `d` — +the substitutions are the closed-counterform confusion classes the planned +glyph-matrix experiment targets (see `eval/glyph-matrix/PLAN.md`). **2 of +the 3 misses were silent** — stated confidently, no hedge — matching the +needle-haystack failure mode (*silent confabulation*: plausible wrong hex, +not an error). The one read the assistant self-flagged as low-confidence +was indeed wrong (a weak calibration signal, n=1). + +**What this adds:** a live demonstration that flawless gist-tier recall and +silent verbatim-tier corruption co-occur *in the same session, on the same +imaged pages* — the boundary holds in the field exactly where the harness +drew it (gist = functional, exact-bytes = unsafe, failure = silent). + +**Caveats:** n=1 session; the reader was the interactive session model, not +the fable-5 production gate; render parameters of the imaged history were +not recorded; session-internal images make this non-reproducible. Treat as +illustration/motivation only — the numbers that matter come from +`eval/glyph-matrix/` when it completes. + +--- + ## Update (2026-06-10) — gist-recall A/B: zero information loss measured at the gist tier The README's weakest claim ("useful when you know where to blur") had no diff --git a/eval/glyph-matrix/PLAN.md b/eval/glyph-matrix/PLAN.md new file mode 100644 index 0000000..c5ae2c4 --- /dev/null +++ b/eval/glyph-matrix/PLAN.md @@ -0,0 +1,46 @@ +# Glyph confusion matrix + render-style A/B (Task #7) — PLANNED, paused for usage budget + +**Status:** paused 2026-06-12 at 05:45am · resume after weekly limit reset (Jun 18 ~5:59pm Toronto) +**Working state:** `~/glyphmx-paused/` (canonical backup, 330 files, 21MB) · scratch `/tmp/glyphmx` · resume via `run.sh` (limit-aware, skips banked trials) +**Detailed handoff:** `HANDOFF.md` (repo root) + +## What this measures + +Per-character confusion matrix for 12-char hex IDs read back through the +pixelpipe render path, across render-style arms — the precision-tier +failure mode that gist-recall (98/98) structurally cannot surface and +needle-haystack only bracketed (~87% exact-match on worst-case hex at +prod style). + +- **40 trials/arm × 5 arms** (prod 500px baseline + style A/B first pass: onebit, color, grid, + winner expansion to 30 later) +- Each trial: 1 page, ~222 image tokens, 40 dense JSON lines, 5 labeled 12-char hex IDs (LCG-seeded, seed 20260612) + 8-hex distractors +- Reader: `claude-fable-5` only (production gate model), direct API route +- Grading: exact-hit rate per trial (12/12 chars), per-char confusion matrix, pooled misses; hard-fails on empty/poisoned output graded as misses + +## Banked so far + +- 2/40 prod-arm trials valid (`out_prod_0.txt`, `out_prod_1.txt` in `~/glyphmx-paused/`), not poisoned; run halted on session limit (`HALTED-ON-LIMIT` in `run.log`) +- All 120 rendered pages for the first pass are pre-rendered and backed up — resume only spends read calls, not render work + +## Why it's not in README yet + +README rule for this repo: measured numbers + n only, no marketing tone. +This experiment ships to README **after** the matrix completes — as +"at render style X, content carries at ~N× fewer input tokens with +measured per-character read error E% on worst-case zero-redundancy +content." Until then it lives here so the plan is public without an +unearned headline. + +## Decision criteria (when trials finish) + +1. Per-arm exact-hit rate (5 labels vs golds), per-char confusion on misses, which hex glyph pairs confuse (0↔8, e↔4, e↔8, f/t suspects). +2. If any zero-cost style beats prod meaningfully (and prod ≈ known ~87%), recommend flipping default in `src/core/render.ts` (e.g. `DENSE_RENDER_STYLE` gains `colorCycle:true`) — token-cost-neutral, but re-run harness to confirm before changing defaults. +3. Update README verbatim row + FINDINGS.md dated entry; copy receipts (`golds.json`, outs, `grade.py`) into `eval/glyph-matrix/` and commit. + +## Related field evidence + +See FINDINGS.md "Update (2026-06-12) — field observation": an n=1 +in-the-wild session captured 7/10 exact / ~3.3% per-char misreads on the +same confusion classes (0→8, e→4/8), 2 of 3 misses silent. Anecdotal, +non-reproducible — motivation only; this harness is what produces the +publishable numbers.