mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
docs: FINDINGS field observation (n=1) + glyph-matrix PLAN.md
- FINDINGS.md: dated 2026-06-12 entry logging in-the-wild verbatim misreads (7/10 exact, ~3.3% per-char, 2/3 silent) alongside flawless gist-tier recall in the same session. Explicitly anecdotal, n=1, non-reproducible — motivation only. - eval/glyph-matrix/PLAN.md: public plan for the paused confusion-matrix experiment (Task #7) so the pending work is visible outside README, per the measured-numbers-only README rule.
This commit is contained in:
+40
-1
@@ -1,13 +1,52 @@
|
||||
# FINDINGS — pxpipe (text→PNG token compression)
|
||||
|
||||
**Status:** ⚠️ **VERDICT REVERSED — see correction below.** Originally ruled "dead"; live measurement shows pxpipe is a working *lossy gist-compressor* saving ~68% on real (dense) Claude Code traffic, with a known verbatim-recall gap.
|
||||
**Date:** 2026-05-28 (original) · 2026-05-29 (correction) · 2026-06-09 (Fable 5 update) · 2026-06-10 (gist-recall A/B, SWE-bench pilot)
|
||||
**Date:** 2026-05-28 (original) · 2026-05-29 (correction) · 2026-06-09 (Fable 5 update) · 2026-06-10 (gist-recall A/B, SWE-bench pilot) · 2026-06-12 (field observation, n=1)
|
||||
**Models tested:** `claude-opus-4-5` (original run), `claude-opus-4-8` (re-test after a model bump), `claude-fable-5` (2026-06-09)
|
||||
**Model scope (current):** Fable 5 only, enforced in library + proxy (Opus disabled 2026-06-09 — see update below).
|
||||
**Harness:** `eval/needle-haystack/` (receipts preserved from `/tmp/needle_eval`)
|
||||
|
||||
---
|
||||
|
||||
## Update (2026-06-12) — field observation: live verbatim misreads in a real session (n=1, anecdotal)
|
||||
|
||||
**Not a controlled eval.** Logged because it is the first *unprompted,
|
||||
in-the-wild* capture of the exact boundary the harnesses bracket, with
|
||||
ground truth available for post-hoc grading.
|
||||
|
||||
Setup: a working session was resumed from imaged history (older context
|
||||
rendered to PNG pages, recent turns text — the production architecture).
|
||||
Mid-session, the assistant was challenged to transcribe, from the imaged
|
||||
history alone, ten 12-char hex IDs that had appeared in an earlier tool
|
||||
output. The transcription was committed *before* re-reading the source
|
||||
files on disk, which then served as ground truth.
|
||||
|
||||
| tier | result |
|
||||
|---|---|
|
||||
| gist (resume a paused run: plan, counts, file states, next steps) | fully functional — zero observed errors, consistent with gist-recall 98/98 |
|
||||
| verbatim (10 × 12-char hex through imaged history) | **7/10 exact** · 4 char-level errors / 120 chars (~3.3%) |
|
||||
|
||||
Misread detail: `0→8`, `e→4`, `e→8` substitutions plus one dropped `d` —
|
||||
the substitutions are the closed-counterform confusion classes the planned
|
||||
glyph-matrix experiment targets (see `eval/glyph-matrix/PLAN.md`). **2 of
|
||||
the 3 misses were silent** — stated confidently, no hedge — matching the
|
||||
needle-haystack failure mode (*silent confabulation*: plausible wrong hex,
|
||||
not an error). The one read the assistant self-flagged as low-confidence
|
||||
was indeed wrong (a weak calibration signal, n=1).
|
||||
|
||||
**What this adds:** a live demonstration that flawless gist-tier recall and
|
||||
silent verbatim-tier corruption co-occur *in the same session, on the same
|
||||
imaged pages* — the boundary holds in the field exactly where the harness
|
||||
drew it (gist = functional, exact-bytes = unsafe, failure = silent).
|
||||
|
||||
**Caveats:** n=1 session; the reader was the interactive session model, not
|
||||
the fable-5 production gate; render parameters of the imaged history were
|
||||
not recorded; session-internal images make this non-reproducible. Treat as
|
||||
illustration/motivation only — the numbers that matter come from
|
||||
`eval/glyph-matrix/` when it completes.
|
||||
|
||||
---
|
||||
|
||||
## Update (2026-06-10) — gist-recall A/B: zero information loss measured at the gist tier
|
||||
|
||||
The README's weakest claim ("useful when you know where to blur") had no
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# Glyph confusion matrix + render-style A/B (Task #7) — PLANNED, paused for usage budget
|
||||
|
||||
**Status:** paused 2026-06-12 at 05:45am · resume after weekly limit reset (Jun 18 ~5:59pm Toronto)
|
||||
**Working state:** `~/glyphmx-paused/` (canonical backup, 330 files, 21MB) · scratch `/tmp/glyphmx` · resume via `run.sh` (limit-aware, skips banked trials)
|
||||
**Detailed handoff:** `HANDOFF.md` (repo root)
|
||||
|
||||
## What this measures
|
||||
|
||||
Per-character confusion matrix for 12-char hex IDs read back through the
|
||||
pixelpipe render path, across render-style arms — the precision-tier
|
||||
failure mode that gist-recall (98/98) structurally cannot surface and
|
||||
needle-haystack only bracketed (~87% exact-match on worst-case hex at
|
||||
prod style).
|
||||
|
||||
- **40 trials/arm × 5 arms** (prod 500px baseline + style A/B first pass: onebit, color, grid, + winner expansion to 30 later)
|
||||
- Each trial: 1 page, ~222 image tokens, 40 dense JSON lines, 5 labeled 12-char hex IDs (LCG-seeded, seed 20260612) + 8-hex distractors
|
||||
- Reader: `claude-fable-5` only (production gate model), direct API route
|
||||
- Grading: exact-hit rate per trial (12/12 chars), per-char confusion matrix, pooled misses; hard-fails on empty/poisoned output graded as misses
|
||||
|
||||
## Banked so far
|
||||
|
||||
- 2/40 prod-arm trials valid (`out_prod_0.txt`, `out_prod_1.txt` in `~/glyphmx-paused/`), not poisoned; run halted on session limit (`HALTED-ON-LIMIT` in `run.log`)
|
||||
- All 120 rendered pages for the first pass are pre-rendered and backed up — resume only spends read calls, not render work
|
||||
|
||||
## Why it's not in README yet
|
||||
|
||||
README rule for this repo: measured numbers + n only, no marketing tone.
|
||||
This experiment ships to README **after** the matrix completes — as
|
||||
"at render style X, content carries at ~N× fewer input tokens with
|
||||
measured per-character read error E% on worst-case zero-redundancy
|
||||
content." Until then it lives here so the plan is public without an
|
||||
unearned headline.
|
||||
|
||||
## Decision criteria (when trials finish)
|
||||
|
||||
1. Per-arm exact-hit rate (5 labels vs golds), per-char confusion on misses, which hex glyph pairs confuse (0↔8, e↔4, e↔8, f/t suspects).
|
||||
2. If any zero-cost style beats prod meaningfully (and prod ≈ known ~87%), recommend flipping default in `src/core/render.ts` (e.g. `DENSE_RENDER_STYLE` gains `colorCycle:true`) — token-cost-neutral, but re-run harness to confirm before changing defaults.
|
||||
3. Update README verbatim row + FINDINGS.md dated entry; copy receipts (`golds.json`, outs, `grade.py`) into `eval/glyph-matrix/` and commit.
|
||||
|
||||
## Related field evidence
|
||||
|
||||
See FINDINGS.md "Update (2026-06-12) — field observation": an n=1
|
||||
in-the-wild session captured 7/10 exact / ~3.3% per-char misreads on the
|
||||
same confusion classes (0→8, e→4/8), 2 of 3 misses silent. Anecdotal,
|
||||
non-reproducible — motivation only; this harness is what produces the
|
||||
publishable numbers.
|
||||
Reference in New Issue
Block a user