Files
Steven Chong 056f970881 test(eval): measure Grok image recall and the factsheet contract
Grok was imaging on the Responses path with no readability data. Add
eval/grok-density: density ladder, climb receipts, and a live
image-vs-factsheet comparison on grok-4.5.

Findings locked into the receipts:
- Production 5x8 image-only confabulates exact IDs (0/4).
- Image + factsheet clears 4/4 exact, 0 confab, gist+guard ok (~70%
  fixture savings).
- Looser pure-image packing can clear exact recall but only ~30%
  savings — not worth the default.

Supports keeping Grok opt-in with dense 5x8 + factsheet when enabled.
2026-07-09 12:44:14 -04:00

1.1 KiB

Grok climb — densest packing that matches the Opus bar

Live binary-search climb on grok-4.5, 2026-07-09. Goal: 4/4 exact, 0 confab, gist ok, guard ok, densest packing first. Image savings use measured Grok billing (~1000 tok/MPix), not the GPT tile model.

Result

Best so far: spleen5x8_aa_d4_c84 at ~30% savings.

{
  "stripCols": 84,
  "maxHeightPx": 1932,
  "style": {
    "cellWBonus": 4,
    "cellHBonus": 4,
    "aa": true,
    "grid": false,
    "colorCycle": false
  },
  "font": "spleen5x8"
}

Ladder (Spleen 5x8, AA)

id exact confab save dims
spleen5x8_aa_d0_c152 0/4 4 ~76% 768x360
spleen5x8_aa_d2_c108 2/4 2 ~70% 764x458
spleen5x8_aa_d3_c95 2/4 2 ~37% 768x954
spleen5x8_aa_d4_c84 4/4 0 ~30% 764x1064

Notes

  • Production packing (d0) confabulates exact IDs.
  • d2/d3 improve but still confab hex/port.
  • d4 (cell bonus 4/4, stripCols 84) clears Opus bar at ~30% real savings.
  • Font/atlas swaps are eval-only; production still ships Spleen 5x8.