mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
teamchong
4774aeb14c
Co-render the OCR instruction into the same PNG as the content, delimited by '===…===' bands, with the API system field dropped. L1 OCR fidelity, Opus 4.7, 20 production blocks, 7×10 cell: - baseline (text-only): 97.91% mean / 96.25% min - reflow (separate system): 91.99% mean / 82.59% min (-5.93pp) - reflow-inimage: 98.95% mean / 96.42% min (+1.04pp) reflow-inimage wins on all 20/20 blocks vs reflow, and beats the text-only baseline on 17/20 blocks. Three blocks hit 100%. The -5.93pp reflow regression that the cell-pitch sweep partially recovered disappears entirely when the instruction is co-rendered. Mechanism: when system carries the instruction and the image carries the content, the model does cross-modal binding to figure out what the image is for. Co-rendering reduces it to a single-modal task with an unambiguous parse rule. Files: - eval/eval-l1-ocr.mjs: add reflow-inimage variant + prompt - README.md: new section before history compression - eval/EXPERIMENT_LOG.md: attempt #2 writeup - eval/results/{l1-report.md, l1-results.json}: regenerated
1.8 KiB
1.8 KiB
L1 OCR Fidelity Report
Generated: 2026-05-23T01:54:01.508Z
Model: opus
Dry run: false
Blocks evaluated: 20
Summary (per variant)
| Variant | Mean Acc | Median Acc | Min Acc | Δ vs baseline | Image savings |
|---|---|---|---|---|---|
| baseline | 97.91% | 98.01% | 96.25% | +0.00pp | 0.0% |
| reflow | 91.99% | 93.44% | 82.59% | -5.93pp | 0.0% |
| reflow-inimage | 98.95% | 99.07% | 96.42% | +1.04pp | 0.0% |
Interpretation
- baseline is the no-reflow reference; reflow is the regression to fix.
- A variant ships if its mean accuracy is within −2pp of baseline AND its image savings are > 0%.
Δ vs baselineofreflowquantifies the damage; the structure-aid variants should claw it back.
Per-Block Accuracy
| Block | Chars | Role | baseline | reflow | reflow-inimage |
|---|---|---|---|---|---|
| 1 | 211 | user | 97.6% | 97.2% | 99.5% |
| 2 | 228 | assistant | 99.1% | 94.7% | 99.6% |
| 3 | 253 | assistant | 96.4% | 93.7% | 98.0% |
| 4 | 284 | assistant | 97.2% | 91.2% | 98.9% |
| 5 | 315 | assistant | 97.5% | 85.7% | 100.0% |
| 6 | 317 | user | 96.8% | 94.2% | 97.4% |
| 7 | 317 | assistant | 98.1% | 82.6% | 98.1% |
| 8 | 321 | assistant | 96.3% | 93.4% | 99.1% |
| 9 | 322 | assistant | 99.7% | 92.9% | 99.1% |
| 10 | 336 | user | 97.3% | 87.8% | 96.4% |
| 11 | 340 | assistant | 99.1% | 97.1% | 99.7% |
| 12 | 351 | user | 98.0% | 97.5% | 100.0% |
| 13 | 362 | assistant | 98.3% | 93.9% | 98.3% |
| 14 | 383 | user | 98.9% | 96.8% | 99.5% |
| 15 | 397 | user | 98.2% | 92.4% | 98.7% |
| 16 | 405 | user | 98.0% | 93.8% | 99.3% |
| 17 | 415 | assistant | 99.5% | 86.5% | 99.8% |
| 18 | 423 | user | 98.3% | 92.3% | 98.8% |
| 19 | 429 | assistant | 96.3% | 83.0% | 98.8% |
| 20 | 436 | assistant | 97.5% | 93.1% | 100.0% |