ai-space/pxpipe
3
0
Fork 0
mirror of https://github.com/teamchong/pxpipe.git synced 2026-07-22 02:02:51 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: add model-specific render profiles and recall evidence

Browse Source
This commit is contained in:
Steven Chong
2026-07-09 22:25:06 -04:00
parent bfcf15ca95
commit cd4c9efddd
54 changed files with 13941 additions and 503 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+33 -2
View File
@@ -4,6 +4,30 @@ All notable changes to pxpipe are documented here. This project adheres to
[Semantic Versioning](https://semver.org/) (pre-1.0: minor = features /
behavioral changes, patch = fixes).
## Unreleased
### Changed
- `gpt-5.6-sol` is now opt-in rather than silently enabled. Its exact profile
remains available when selected, and sibling 5.6 variants do not inherit it.
- Complete per-model render profiles now drive font atlas, cell spacing,
geometry, style, history pages, and profitability math. Sol uses a 6×11
JetBrains Mono atlas; opt-in Grok uses the measured effective 9×12 arm plus
the factsheet.
- Dashboard exposes the GPT model row and `?` controls now show accessible
hover/focus tooltips.
### Known limitations / evidence
- A direct `gpt-5.6-sol` raw-image pilot tested **both** the new JetBrains
6×11/126-column profile and the old Spleen 5×8/152-column profile. Each scored
0/4 exact identifiers with four unsupported invented values; 6×11 passed
gist/guard, while 5×8 failed gist and passed the guard. One earlier paid setup
attempt exhausted 512 output tokens as hidden reasoning and is recorded but
excluded from scoring. The production fact-sheet still carries extracted
identifiers verbatim. Sol is therefore removed from the built-in default
scope while its exact profile remains available for explicit opt-in. A
Sol-only effective 9×12/84-column candidate is locally rendered but has no
model result yet. See `eval/sol-profile/RESULTS.md`.
## 0.8.0 — 2026-07-03
### Security
@@ -214,7 +238,8 @@ it were the live request).
## 0.5.0 — 2026-06-20
Cache-stable history-collapse imaging for both providers, with GPT-5.6 promoted
Cache-stable history-collapse imaging for both providers, with the then-named
GPT-5.6 model promoted
to the default imaged scope. Old conversation history collapses into rendered
PNG sections so the model reads a compact image instead of re-billed text, while
prompt caching and tool-call behavior are preserved.
@@ -228,7 +253,13 @@ prompt caching and tool-call behavior are preserved.
metrics, thumbnail-expired session UI, and reflow/newline handling.
### Changed
- **Default imaged scope is now `claude-fable-5` + `gpt-5.6`.** GPT-5.6 is
> Historical state for 0.5.0: the Sol promotion below was reversed in the
> current Unreleased changes after the 2026-07-09 raw-recall failures; the
> built-in default is now Fable-only.
- **Default imaged scope is now `claude-fable-5` + `gpt-5.6-sol`.** The model
was originally recorded here as GPT-5.6; production traffic identifies the
exact variant as `gpt-5.6-sol`. GPT-5.6 Sol is
promoted from opt-in (0.4.0) to on by default. `gpt-5.5` and `claude-opus-4-8`
stay opt-in: they degrade reading dense imaged history (gist drift), so
silently imaging them by default is wrong. Promotion is gated on an OCR/recall
FINDINGS.md
+65 -4
View File
@@ -1,10 +1,71 @@
# FINDINGS — pxpipe (text→PNG token compression)
**Status:** ⚠️ **VERDICT REVERSED — see correction below.** Originally ruled "dead"; live measurement shows pxpipe is a working *lossy gist-compressor* saving ~68% on real (dense) Claude Code traffic, with a known verbatim-recall gap.
**Date:** 2026-05-28 (original) · 2026-05-29 (correction) · 2026-06-09 (Fable 5 update) · 2026-06-10 (gist-recall A/B, SWE-bench pilot) · 2026-06-12 (field observation, n=1) · 2026-06-23 (reframe: correct baseline = /compact)
**Models tested:** `claude-opus-4-5` (original run), `claude-opus-4-8` (re-test after a model bump), `claude-fable-5` (2026-06-09)
**Model scope (current):** Fable 5 only, enforced in library + proxy (Opus disabled 2026-06-09 — see update below).
**Harness:** `eval/needle-haystack/` (receipts preserved from `/tmp/needle_eval`)
**Date:** 2026-05-28 (original) · 2026-05-29 (correction) · 2026-06-09 (Fable 5 update) · 2026-06-10 (gist-recall A/B, SWE-bench pilot) · 2026-06-12 (field observation, n=1) · 2026-06-23 (reframe: correct baseline = /compact) · 2026-07-09 (GPT-5.6 Sol raw-recall pilot)
**Models tested:** `claude-opus-4-5` (original run), `claude-opus-4-8` (re-test after a model bump), `claude-fable-5` (2026-06-09), `gpt-5.6-sol` (2026-07-09 raw-image pilot)
**Model scope (current):** Fable 5 only, enforced in library + proxy. Sol, Opus, GPT 5.5, and Grok remain available only through explicit opt-in.
**Harnesses:** Claude/Opus/Fable: `eval/needle-haystack/` (older receipts preserved from `/tmp/needle_eval`); Sol: `eval/sol-profile/` (raw responses and receipts committed)
---
## Update (2026-07-09) — GPT-5.6 Sol fails raw exact recall at both tested profiles; moved to opt-in
The exact production model id is `gpt-5.6-sol`. Production telemetry showed
that real Sol requests reached the Responses path and retained positive
estimated savings, but token rows are not reader-quality evidence. A separate
paid pilot therefore sent deterministic rendered terminal fixtures directly to
the Responses endpoint with `detail: original`, bypassing pxpipe.
### What was actually tested
Each scored call asked, in one structured response, for four exact values
(12-character hex, camelCase field, full path, and port), one rollout gist, and
one deliberately unstated fact. The two profiles were real paid Sol calls:
| profile | dimensions | exact | unsupported inventions | gist | unstated guard | result |
|---|---:|---:|---:|:---:|:---:|---|
| JetBrains Mono 10, 6×11, 126 cols | 764×1724 | **0/4** | **4** | pass | pass | **fail** |
| old shared Spleen, 5×8, 152 cols | 768×1040 | **0/4** | **4** | fail | pass | **fail** |
The 5×8 line is a model result, not a local-render inference and not a reuse of
Grok's 5×8 result. In both Sol calls every returned hex/id/path/port was absent
from the fixture, so the errors are classified as confabulations rather than
single-glyph OCR substitutions.
There was one additional paid setup attempt at 6×11. It returned no answer
because all 512 output tokens were hidden reasoning tokens and ended
`incomplete: max_output_tokens`. It counts toward spend but is excluded from
recall scoring. The corrected scored calls used `reasoning: none`.
### Honest boundary
- This is one scored synthetic fixture per profile because the early-stop rule
halted after clear failure. It proves those calls failed the bar; it does not
estimate a population error rate.
- It is **not** a direct Sol-vs-Fable comparison. Fable's evidence comes from
separate, larger harnesses described below.
- Production attaches a deterministic verbatim fact-sheet for extracted paths
and identifiers. That text fallback can rescue covered exact strings, but it
does not make the raw-image result a pass.
- A Sol-only JetBrains effective 9×12 / 84-column candidate is rendered locally
(two pages, 2,136 estimated image tokens, positive estimated savings) but has
**not** received a model call. It is geometry, not evidence.
### Decision
`gpt-5.6-sol` is **off by default**, under the same rule used for GPT 5.5 and
Grok: silent image rewriting requires positive recall evidence, not merely
positive savings. Its exact profile and suffix aliases remain available for
explicit operator opt-in:
PXPIPE_MODELS=claude-fable-5,gpt-5.6-sol
Re-enabling Sol by default requires a retuned paid arm to clear 4/4 exact, zero
confabulations, gist, and guard with positive savings; a replicated pass on both
fixtures is preferred before silent promotion.
Receipts and every raw response:
[`eval/sol-profile/RESULTS.md`](eval/sol-profile/RESULTS.md).
---
README.md
+30 -17
View File
@@ -16,7 +16,7 @@ measured per-request against a free `count_tokens` counterfactual in
This is what the model sees instead of text:
![example: a real `transformRequest` output: system prompt + tool docs reflowed into one dense 1573×1248 page, instruction banner on top, ↵ marking original newlines](https://raw.githubusercontent.com/teamchong/pxpipe/main/docs/assets/example-render.png)
![example: a real `transformRequest` output: system prompt + tool docs reflowed into one dense page, instruction banner on top, ↵ marking original newlines](https://raw.githubusercontent.com/teamchong/pxpipe/main/docs/assets/example-render.png)
*~48k chars of system prompt + tool docs: ≈25k tokens as text, ≈2.7k image
tokens as this page. Real pipeline output; the model reads renders like this
@@ -83,16 +83,28 @@ are imaged.
re-sends the full transcript as plain text each turn is in the same high-
savings class as Claude Code. Details and measured splits:
[docs/CACHING_AND_SAVINGS.md](docs/CACHING_AND_SAVINGS.md#openai-responses-path-codex-and-friends).
- **Model scope:** default `PXPIPE_MODELS=claude-fable-5,gpt-5.6`. Opus
4.7/4.8, GPT 5.5, and **Grok** are opt-in only (dashboard chips or
`PXPIPE_MODELS`) — not good enough as silent defaults for imaged context.
- **Model scope:** default `PXPIPE_MODELS=claude-fable-5`. Sol, Opus 4.7/4.8,
GPT 5.5, and **Grok** are opt-in only (dashboard chips or `PXPIPE_MODELS`) —
not good enough as silent defaults for imaged context. The exact Sol id still
matters when opted in: sibling variants such as `gpt-5.6-terra` do not inherit
Sol's allowlist or render profile.
`PXPIPE_MODELS=off` disables imaging. Everything else passes through
byte-identical. On the GPT path, tool definitions stay native JSON and no
Anthropic `cache_control` markers are used.
- **Grok (opt-in): production 5×8 + factsheet.** Off by default. Pure-image
exact OCR at 5×8 fails (0/4 IDs). Image+factsheet clears exact IDs at ~70%
fixture savings; looser pure-image packing only ~30%. If you opt in, keep
density and rely on the fact-sheet — do not inflate cell bonuses.
- **Per-model rendering:** opt-in `gpt-5.6-sol` currently uses a 126-column,
6×11 JetBrains Mono profile; Claude keeps its 312-column 5×8 Spleen profile. These
are selected by exact model id, including history pages and profitability
math. **Sol recall caveat:** raw-image calls scored 0/4 exact with four
inventions at both current 6×11 and old shared 5×8; 6×11 passed gist/guard,
while 5×8 also missed gist. Production's verbatim fact-sheet remains a text
fallback for extracted identifiers. An effective 9×12 Sol retune is rendered
but not yet model-tested, so the current Sol geometry is operational—not
recall-validated. [Sol receipts](eval/sol-profile/RESULTS.md) and
[profile evidence](docs/MODEL_RENDER_PROFILES.md).
- **Grok (opt-in): measured 9×12 + factsheet.** Off by default. Pure-image
exact OCR at 5×8 fails (0/4 IDs); the densest tested pure-image arm that
cleared 4/4 with zero confabulation was effective 9×12 / 84 columns at ~30%
fixture savings. The fact-sheet remains attached as defense in depth.
[eval/grok-density/FACTSHEET_RESULTS.md](eval/grok-density/FACTSHEET_RESULTS.md).
## Benchmarks (reproducible)
@@ -121,15 +133,15 @@ evals.)
## How it works
```
tool_result string ──► wrap at 1928px-wide columns ──► pack ~92,000 chars/page ──► PNG[]
model id ──► render profile ──► wrap/reflow bulk context ──► PNG[] + exact-token factsheet
```
The proxy intercepts `/v1/messages`, rewrites eligible bulk into image
blocks, splices them back cache-friendly (static prefix preserved, prompt
caching keeps working), and forwards. A 1928×1928 image costs ≈4,761 vision
tokens and holds ≈92,000 chars, so text wins only above ~19 chars/token —
Claude Code traffic runs ~1.91 (N=391). A per-request estimator decides;
sparse prose stays text. Events log to `~/.pxpipe/events.jsonl`.
caching keeps working), and forwards. Claude uses 1568×728 pages; GPT 5.6 Sol
uses 764px-wide portrait strips; opt-in Grok uses effective 9×12 cells. A
per-request estimator uses that same resolved profile, so sparse prose stays
text. Events log to `~/.pxpipe/events.jsonl`.
## Library use (no proxy)
@@ -191,10 +203,11 @@ Three kinds of *input* blocks, each behind a profitability gate:
Everything else passes through byte-identical: your messages, recent turns,
the model's output (it is the response, the proxy never touches it), sparse
prose, and anything too small to win. Models outside the allowlist pass
through entirely — the default scope is Fable 5 and GPT 5.6 only. Opus 4.8
and GPT 5.5 read imaged content measurably worse (FINDINGS.md 2026-06-16),
so they are deliberately opt-in via the dashboard or `PXPIPE_MODELS`, never
silently imaged.
through entirely — the built-in default scope is Fable 5 only. Sol, Opus 4.8,
GPT 5.5, and Grok are deliberately opt-in via the dashboard or
`PXPIPE_MODELS`, never silently imaged. Sol joined that list after both its
6×11 profile and the old shared 5×8 profile scored 0/4 exact with four
confabulations in direct raw-image calls.
**Has it ever failed for real, outside the benchmarks?**
Yes, once in weeks of daily use: the model recalled a person's name from
assets/JETBRAINS_MONO_LICENSE.txt
+89
View File
@@ -0,0 +1,89 @@
Copyright 2020 The JetBrains Mono Project Authors (https://github.com/JetBrains/JetBrainsMono)
This Font Software is licensed under the SIL Open Font License, Version 1.1.
This license is copied below, and is also available with a FAQ at:
https://openfontlicense.org
-----------------------------------------------------------
SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
-----------------------------------------------------------
PREAMBLE
The goals of the Open Font License (OFL) are to stimulate worldwide
development of collaborative font projects, to support the font creation
efforts of academic and linguistic communities, and to provide a free and
open framework in which fonts may be shared and improved in partnership
with others.
The OFL allows the licensed fonts to be used, studied, modified and
redistributed freely as long as they are not sold by themselves. The fonts,
including any derivative works, can be bundled, embedded, redistributed
and/or sold with any software provided that any reserved names are not used
by derivative works. The fonts and derivatives, however, cannot be released
under any other license. The requirement for fonts to remain under this
license does not apply to any document created using the fonts or their
derivatives.
DEFINITIONS
"Font Software" refers to the set of files released by the Copyright
Holder(s) under this license and clearly marked as such. This may include
source files, build scripts and documentation.
"Reserved Font Name" refers to any names specified as such after the
copyright statement(s).
"Original Version" refers to the collection of Font Software components as
distributed by the Copyright Holder(s).
"Modified Version" refers to any derivative made by adding to, deleting,
or substituting -- in part or in whole -- any of the components of the
Original Version, by changing formats or by porting the Font Software to a
new environment.
"Author" refers to any designer, engineer, programmer, technical writer or
other person who contributed to the Font Software.
PERMISSION & CONDITIONS
Permission is hereby granted, free of charge, to any person obtaining a copy
of the Font Software, to use, study, copy, merge, embed, modify, redistribute,
and sell modified and unmodified copies of the Font Software, subject to the
following conditions:
1) Neither the Font Software nor any of its individual components, in
Original or Modified Versions, may be sold by itself.
2) Original or Modified Versions of the Font Software may be bundled,
redistributed and/or sold with any software, provided that each copy
contains the above copyright notice and this license. These can be included
either as stand-alone text files, human-readable headers or in the
appropriate machine-readable metadata fields within text or binary files as
long as those fields can be easily viewed by the user.
3) No Modified Version of the Font Software may use the Reserved Font Name(s)
unless explicit written permission is granted by the corresponding Copyright
Holder. This restriction only applies to the primary font name as presented
to the users.
4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
Software shall not be used to promote, endorse or advertise any Modified
Version, except to acknowledge the contribution(s) of the Copyright Holder(s)
and the Author(s) or with their explicit written permission.
5) The Font Software, modified or unmodified, in part or in whole, must be
distributed entirely under this license, and must not be distributed under
any other license. The requirement for fonts to remain under this license
does not apply to any document created using the Font Software.
TERMINATION
This license becomes null and void if any of the above conditions are not
met.
DISCLAIMER
THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF COPYRIGHT, PATENT,
TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE
FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, INCLUDING ANY GENERAL, SPECIAL,
INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF THE USE OR INABILITY TO USE
THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE.
docs/CACHING_AND_SAVINGS.md
+14 -3
View File
@@ -257,8 +257,20 @@ Measured on local `/v1/responses` rows (same endpoint, different models):
| grok | high on warm multi-turn | **collapsed** after ↵ gate fix | ~**35%** on collapsed Responses rows (n=35 post-fix); fixture image+factsheet ~70% |
| gpt-5 | ~73% | often | ~34% overall; ~42% on collapsed warm rows |
Grok render contract: production **5×8** images + factsheet (not looser cells).
See [eval/grok-density/FACTSHEET_RESULTS.md](../eval/grok-density/FACTSHEET_RESULTS.md).
Render profiles are selected by exact model id, not by the shared Responses
path. Opt-in `gpt-5.6-sol` uses 126 columns with a 6×11 JetBrains Mono atlas;
Claude uses 312 columns with the 5×8 Spleen atlas. Grok remains opt-in and uses the
densest tested exact-safe image arm, effective **9×12** / 84 columns, plus the
factsheet. See [eval/grok-density/RESULTS.md](../eval/grok-density/RESULTS.md)
and [FACTSHEET_RESULTS.md](../eval/grok-density/FACTSHEET_RESULTS.md).
Those profile and savings numbers are not recall evidence. The Sol raw-image
pilot separately tested both 6×11/126 and old 5×8/152: each scored 0/4 exact
with four confabulations, and 5×8 also missed gist. Sol is therefore off by
default; production's fact-sheet remains an important exact-token fallback for
operators who explicitly opt in. The locally rendered Sol 9×12 retune remains
untested. See
[`eval/sol-profile/RESULTS.md`](../eval/sol-profile/RESULTS.md).
So "Codex shows 1% on Opus" is not "Codex unsupported." It is "this session's
prompt was already ~98% cached text, history collapse did not fire, and only
@@ -284,4 +296,3 @@ allowlisted, etc.) correctly show `—`. Path selects the accounting shape
only the slab (or nothing) was imaged.
4. Do not compare a Claude Code session's 70% to a warm Codex session's 1%
as a regression. Different wire, different uncached bulk.
docs/MODEL_RENDER_PROFILES.md
+99
View File
@@ -0,0 +1,99 @@
# Model render profiles
The OpenAI-shaped endpoints are a wire protocol, not a rendering profile.
Claude, GPT, and Grok requests can all arrive on `/v1/responses`; pxpipe selects
geometry, glyph atlas, style, and vision billing from the exact `model` id.
## Built-in profiles
| model rule | default enabled | font / cell | columns | max height | style | evidence |
|---|:---:|---|---:|---:|---|---|
| `claude-*` / `*anthropic*` | Fable only | Spleen + Unifont, 5×8 | 312 | 728 px | grayscale AA | Anthropic 1568-edge / ~1.15 MP no-resize measurements |
| `gpt-5.6-sol` | **no; opt-in** | JetBrains Mono 10 + Unifont fallback, 6×11 | 126 | 1932 px | grayscale AA | operational geometry; raw recall pilot failed 0/4 exact and is being retuned |
| `grok-*` | **no** | Spleen + Unifont, effective 9×12 | 84 | 1932 px | grayscale AA + factsheet | `grok-4.5` density climb: 4/4 exact, 0 confab, gist/guard pass, ~30% fixture savings |
| other GPT/o-series | no unless configured | Spleen + Unifont, 5×8 | 152 | 1932 px | grayscale AA | conservative fallback / existing OpenAI geometry |
`gpt-5.6-sol` is intentionally exact and opt-in. The events log also contains
`gpt-5.6-terra`; Terra does not inherit Sol's allowlist or visual profile.
The GPT 5.6 Sol font choice is a separate local raster profile. It preserves
the provider-safe strip width and has a larger cell than the shared 5×8
fallback, but the first paid raw-image pilot did **not** validate it.
## Sol evidence boundary (2026-07-09)
Two production rows for the exact `gpt-5.6-sol` id reported 39,985 effective
input/cache-read tokens, 14,152 image tokens, a 52,072-token baseline, 18
images, and approximately 73% estimated savings. A comparison run with the old
shared profile reported 8,568 image tokens against the same baseline. These
numbers establish that real Sol traffic reaches the Responses path and that the
new geometry costs more while retaining positive estimated savings. They do
**not** establish recall, task quality, or causality.
The separate paid raw-image pilot supplies the model-reading evidence:
| profile | exact | confabulations | gist | guard | result |
|---|---:|---:|:---:|:---:|---|
| JetBrains 6×11 / 126 cols | 0/4 | 4 | pass | pass | fail |
| Spleen 5×8 / 152 cols | 0/4 | 4 | fail | pass | fail |
The test is one scored synthetic fixture per profile. It proves that both calls
failed the stated acceptance bar, not a broad Sol-vs-Fable ranking. Production
also sends a verbatim fact-sheet beside images, so covered exact identifiers
have a text fallback even though raw image recall failed. A locally rendered
9×12 / 84-column Sol-only retune remains untested until its final paid call is
approved.
Receipt: [`eval/sol-profile/RESULTS.md`](../eval/sol-profile/RESULTS.md).
### Sol default-scope decision
Sol is **off by default** under the same safety rule used for GPT 5.5 and Grok:
a model that silently invents exact values from imaged context is not a safe
transparent default. The profile code stays available so operators can opt in
explicitly and so the 9×12 candidate can be evaluated without changing any
other model:
```bash
PXPIPE_MODELS='claude-fable-5,gpt-5.6-sol'
```
Re-enabling Sol by default requires a paid profile arm to clear 4/4 exact, zero
confabulations, gist, and guard while retaining positive savings. Ideally the
pass should replicate on both deterministic fixtures; the current n=1 failures
are enough to reject these profiles, not enough to estimate a general error
rate.
## Why Grok is opt-in
Grok 4.5 at shared 5×8 density returned 0/4 exact identifiers and silently
invented all four answers. Effective 9×12 returned 4/4 with zero confabulation,
and a separate 5×8 + factsheet arm also returned 4/4 for the extractor shapes in
that fixture. Both are n=1 synthetic results. They justify a safer opt-in
profile, not silent default rewriting of every Grok request.
Receipts:
- [`eval/grok-density/RESULTS.md`](../eval/grok-density/RESULTS.md)
- [`eval/grok-density/CLIMB_RESULTS.md`](../eval/grok-density/CLIMB_RESULTS.md)
- [`eval/grok-density/FACTSHEET_RESULTS.md`](../eval/grok-density/FACTSHEET_RESULTS.md)
## Overrides
`PXPIPE_GPT_PROFILES` is a JSON map from model-id prefix to a partial profile.
The longest prefix wins. Supported render fields are `font`, `cellWBonus`,
`cellHBonus`, `aa`, `grid`, `gridCols`, `colorCycle`, `markerScale`, and
`markerRed`; geometry fields are `stripCols` and `maxHeightPx`.
```bash
PXPIPE_GPT_PROFILES='{
"gpt-5.6-sol": {
"stripCols": 120,
"style": { "grid": true, "gridCols": 4 }
}
}'
```
The profitability gate derives pixel width, row capacity, pagination, and image
cost from the same resolved profile used by slab and history rendering. A style
override therefore cannot silently leave the gate on 5×8 geometry.
docs/RENDER_SIZING.md
+73 -132
View File
@@ -1,157 +1,98 @@
# How pxpipe sizes a rendered image — rules, reasons, and history
# How pxpipe sizes rendered images
This documents *why* a pxpipe PNG has the dimensions it does. It exists because
the sizing looks arbitrary until you know what was tried and rejected. If you're
tempted to "make it square" or "shrink to the text," read the history first —
both were considered, and one was built and reverted.
Image geometry is selected by the exact model id. `/v1/responses` is only a
wire protocol: Claude, GPT, and Grok on that path do not share a renderer.
## TL;DR — current behavior
## Current behavior
A rendered page is a **fixed-width, variable-height** image. It is **not
deliberately squared** and **never shrunk horizontally to fit short lines**.
A page is profile-width and variable-height. It is never padded to a square.
- **Width is constant per path**, set by the column count, not by content:
`width = 2·PAD_X + cols·CELL_W`. The static slab renders at `DEFAULT_COLS=313`
→ `8 + 313·5 = 1573px`; dense tool/history pages render at
`DENSE_CONTENT_COLS=384` → `8 + 384·5 = 1928px`.
- **Height grows to fit the lines on the page**, capped, then pages:
`height = 2·PAD_Y + nLines·CELL_H` → `8 + 8·nLines`.
- **Vertical cap → paging**: `maxLines = floor((MAX_HEIGHT_PX − 2·PAD_Y) / CELL_H)`
`= floor(1924/8) = 240` lines. Overflow goes to the next image, it does not
grow the canvas. A *full dense* page is `1928 × 1928` (≈square only because
384 cols ≈ 240 rows at the 5×8 cell — not deliberately squared); a *full slab*
page is `1573 × 1280` (~159 rows, READABLE-bound below the 240 cap); a
*partial* page (small tool_result, last page) is wide-and-short, e.g. `1928 × 160`.
```text
cellWidth = atlasWidth + cellWBonus
cellHeight = atlasHeight + cellHBonus
width = 2 * PAD_X + columns * cellWidth
height = 2 * PAD_Y + usedRows * cellHeight
```
This `~1932×1932` ceiling is the largest page Fable / Opus 4.8 accept without a
server-side resize. Those models take up to **2576 px** on the long edge, but a
request with **>20 images** (pxpipe always sends many) is held to the stricter
**≤2000 px/side** rule, and `1928×1928` = `69×69` = **4761 visual tokens**, just
under the **4784**-token per-image cap. Going bigger gets the request *rejected*,
not downscaled.
Width may shrink to the widest rendered line, capped by the profile. Height
grows with content until the profile cap, then overflow becomes another page.
Source of truth: `renderChunkToPng` in `src/core/render.ts` (the `width` /
`height` lines), constants `PAD_X=PAD_Y=4`, `CELL_W=5`, `CELL_H=8` (the "5×8
cell"), `DEFAULT_COLS=313`, `DENSE_CONTENT_COLS=384`, `MAX_HEIGHT_PX=1932`,
`READABLE_CHARS_PER_IMAGE=50000`, `DENSE_CONTENT_CHARS_PER_IMAGE=92160`.
| model rule | atlas / effective cell | columns | full width | max height |
|---|---|---:|---:|---:|
| Claude / Anthropic | Spleen + Unifont, 5×8 | 312 | 1568 px | 728 px |
| opt-in `gpt-5.6-sol` | JetBrains Mono 10 + Unifont fallback, 6×11 | 126 | 764 px | 1932 px |
| opt-in Grok | Spleen + Unifont, effective 9×12 | 84 | 764 px | 1932 px |
| other OpenAI fallback | Spleen + Unifont, 5×8 | 152 | 768 px | 1932 px |
## The cell
Complete profile values and evidence are in
[`MODEL_RENDER_PROFILES.md`](MODEL_RENDER_PROFILES.md).
Each character occupies a **5px wide × 8px tall** cell (`ATLAS_CELL_W=5`,
`ATLAS_CELL_H=8` in `src/core/atlas.ts`). The atlas is a prebaked glyph sheet;
text is rendered white-on-black then inverted to black-on-white. The cell was
**7×10 originally** and shrunk to 5×8 — smaller cell, same legibility on the
target model, more chars per pixel.
The Sol row is the **currently configured** geometry, not a successful recall
claim. Paid raw-image calls tested both that 6×11 row and the old 5×8/152-column
fallback: each returned 0/4 exact identifiers with four confabulations; 5×8 also
missed gist. Sol is consequently opt-in. The effective 9×12/84-column Sol
candidate is local-only pending a model call. See
[`eval/sol-profile/RESULTS.md`](../eval/sol-profile/RESULTS.md).
## Why full width, not shrink-to-content
## Provider boundaries
`shrinkColsToContent` (`src/core/render.ts`) still exists but is a **no-op** — it
returns `cols` unchanged. Its docstring records the policy:
Anthropic resizes images to fit both a 1568px long edge and roughly 1.15MP.
The 1568×728 Claude page stays inside both bounds, so rasterized glyphs reach
the vision encoder without the old 0.555× resample.
> *Always render at full canvas width — no shrink-to-content. Maximum chars per
> page = maximum image-token savings on dense content, and the unused canvas
> tail is just whitespace (cheap to encode).*
OpenAI-shaped profiles use portrait strips at or below the 768px short-side
floor and a 1932px height cap. GPT 5.6 Sol's 126×6+8 and Grok's 84×9+8 both
produce 764px full-width strips.
The reasoning:
## Font and Unicode
1. **The only content pxpipe images is dense content** (it passes a
profitability gate; sparse/short-line content is left as text). After
`reflow` packs that content into full-width rows, the longest line is already
~`cols`, so there is *nothing left to shrink*. Width-shrink only helps
short-line content — which never reaches the renderer.
2. **Simplicity + prediction parity.** The cost gate has to predict image tokens
before rendering. A fixed width makes prediction trivial and exact; a
content-dependent width forced the gate to re-derive the canvas size (see the
"hoist width-shrink before gate" commit below) for no real-world gain.
Atlases are generated at build time; runtime never loads font files. The compact
JetBrains atlas contains common code/Latin/symbol ranges. Missing codepoints
fall back to the full Spleen/Unifont atlas, preserving multilingual rendering.
## Why not "square with max width"
Spacing, anti-aliasing, grid, color cycle, and newline-marker style are all part
of the model profile. The same style object is used for static slabs and history.
Anthropic bills images by **pixel area** (≈ `w·h/750`), not by the longest edge.
So a square is not cheaper than a wide-short image of the same area — what
minimizes cost is the *tightest bounding box around the text*, and for dense
reflowed content that box is exactly "full width × just enough rows," which is
what we render. We never *pad* to a square: a full dense page lands at
`1928×1928` only because 384 cols and 240 rows are equal pixel extents at the
5×8 cell — it's the max-chars-per-page point (the ~1932² token ceiling), not an
aspect-ratio target, and a partial page stays wide-and-short. Aspect ratio is a
non-goal; **chars-per-pixel** is the goal, achieved by filling every row to
`cols` and paging vertically.
## Gate parity
## Two render paths
The profitability gate resolves the same profile as the renderer and derives:
- **tool_result / history images**: single-column at `DENSE_CONTENT_COLS=384`
(1928px wide), paged at the 240-line cap → full pages are `1928×1928`
(~92k chars each).
- **system-slab image**: single-column at `DEFAULT_COLS=313` (1573px wide),
READABLE-bound to ~159 rows (~50k chars → `1573×1280`). Kept on a path that
*can* use multi-column packing (`shrinkWidth=false`), but multi-col is
**disabled by default** (`multiCol: 1`) because at 313 cols a single column
already holds ~50k chars/page and multi-col adds OCR column-ordering risk
without meaningful savings.
- cell width and height
- content-width shrink
- rows per page
- page count
- per-image token cost
So in practice everything is single-column full-width today; the multi-col code
is retained for backward compat.
Changing cell spacing or font therefore cannot leave cost prediction on a
hardcoded 5×8 canvas. Claude uses the Anthropic pixel formula with a conservative
margin, GPT uses model-specific tile/patch pricing, and Grok uses the measured
tokens-per-megapixel rate.
## Billing model (and an unresolved gap)
## Why not square pages
- **Anthropic's documented formula**: `tokens ≈ (w·h) / 750`.
- **The gate's constant**: `src/core/transform.ts` anchors per-image cost on an
*empirical* figure (~2500 tok for a 508×1559 canvas, i.e. claiming the doc
formula underpredicts billing by ~2.4×). That measurement is from an earlier
cell/column regime.
- **A fresh regression** on production `~/.pxpipe/events.jsonl` (2-var OLS,
`tokens ≈ a·text_chars + b·image_pixels`, N≈1.5k cold-miss events) gives
**~907 pixels/token** and **~1.8 chars/token** for text — i.e. *much* closer
to the doc's 750 than to the gate's stale ~312 px/tok.
Billing follows pixel area or patches/tiles, not visual symmetry. Padding a
portrait or short final page to a square adds cost without adding information.
The target is readable characters per billed pixel under each provider's resize
contract, not a preferred aspect ratio.
These disagree by ~3×. The gate is therefore likely **pessimistic** about image
cost on current traffic (it may pass up profitable compressions). Re-grounding
`TOKENS_PER_IMAGE` from a fresh regression is an open task — the data is in the
event log (`image_pixels` next to `input_tokens + cache_create_tokens` on
cold-miss rows). Don't trust any single hardcoded image-cost constant without
re-checking it against the log.
## Measurement history
## How sizing decisions are/were measured
| date | change | finding |
|---|---|---|
| 2026-05-21 | R3 reflow | recovered line-end dead space by marking hard newlines with `↵` |
| 2026-05-22 | grayscale atlas + L1/L2 harness | measured style and instruction-band effects |
| 2026-06-09 | Fable-only 5×8 scope | Fable read dense cells better than Opus |
| 2026-06-17 | ~1932px experiment | fewer pages, later shown to be server-resampled on Anthropic |
| 2026-07-01 | Claude 1568×728 clamp | made billed pixels survive Anthropic preprocessing |
| 2026-07-09 | exact-model profiles | separated Claude, GPT 5.6 Sol, and opt-in Grok geometry/style |
- **L1 OCR eval** (`eval/eval-l1-ocr.mjs`): per-character read accuracy across
render styles (cell size, reflow on/off, instruction placement, grayscale).
This is what gated reflow and the in-image instruction banner.
- **L2 session eval** (`eval/eval-l2-session.mjs`): comprehension over whole
imaged sessions.
- **events.jsonl regression**: the real per-pixel/per-char billing fit above.
- **glyph-matrix** (`eval/glyph-matrix/`, PAUSED): a per-character confusion
matrix across render styles, to find a zero-cost style that lowers read error
so the gate can compress harder at the same fidelity. Paused because the
reader model (Fable 5) is offline.
Older benchmark receipts retain their historical geometry; do not rewrite those
numbers as if they were generated by the current profile.
## History (oldest → newest)
## Changing a profile
The sizing converged through measured iteration, not a single design. Key commits:
| date | commit | change | why |
|---|---|---|---|
| 2026-05-21 | `38e852a` | add **R3 reflow** (recover line-end dead margin, ~29% glyph fill → dense) | rows were mostly empty; pack them |
| 2026-05-22 | `fbf32bb` | **pack reflow across newlines** + grayscale atlas + build **L1/L2 eval harness** | measure read fidelity of the packing |
| 2026-05-22 | `ea68340` | **in-image instruction banner** variant | L1 eval: **+1.04pp** char accuracy vs baseline |
| 2026-05-22 | `dca7807` | pack header prose to full row width, drop spurious ↵ markers | tidy reflow output |
| 2026-05-22 | `a9b0996` | docs: cell is now **5×8** (was 7×10) | denser cell |
| 2026-05-23 | `1afaa6c` | content-aware image cost + **width-shrinking** (WIP) | *tried* shrink-to-content |
| 2026-05-23 | `414f4bf` | hoist width-shrink before the cost gate so prediction matches render | make shrink predictable |
| 2026-05-25 | `3c8716c` | **full-canvas single-column rendering, 50k chars/page** — `shrinkColsToContent` becomes a **no-op** | shrink gave no gain on dense content; full width packs more |
| 2026-05-25 | `bb8e0d8` | **page** dense tool/history images | enforce the 195-line cap, split overflow |
| 2026-05-26 | `28bc65c` | reduce dense page size | tuning |
| 2026-06-09 | `cdfc99d` | drop Opus, **Fable-5 only**; dense render on bare 5×8 cell | Opus misread ~7% of renders |
| 2026-06-17 | (this change) | raise page ceiling to **~1932×1932**: `MAX_HEIGHT_PX=1932`, dense pages `DENSE_CONTENT_COLS=384` / `92160` chars; row cap follows each call's char budget | Fable/Opus 4.8 accept ≤2000px & ≤4784 tok/image (not the old 1568 limit); baseline left ~3× headroom unused → bigger pages = fewer image blocks at the same 5×8 legibility |
The arc: **reflow** to stop wasting rows → **eval harness** to prove the packing
is still readable → **width-shrink** experiment → **reverted** to full-canvas
two days later because dense content already fills the width → **paging** for the
height cap → **model scope narrowed** to the reader that hits 100% on the cell.
## If you want to change the sizing
1. Add a style variant to the L1 OCR eval and measure char accuracy first.
2. Check the cost side against `events.jsonl` (don't trust the hardcoded
per-image constant — see the gap above).
3. Remember the gate must be able to *predict* the size cheaply; content-dependent
geometry is what got width-shrink reverted.
1. Add or re-run a reader-specific recall/style harness.
2. Measure the provider's real image-token accounting.
3. Keep the strip inside its no-resize boundary.
4. Test slab, history, gate, and cache determinism together.
5. Do not generalize one variant's result to sibling model ids.
docs/TRANSFORM_INFO.md
+16 -20
View File
@@ -15,11 +15,11 @@ to re-read that text in token form — Anthropic prompt-caches it, and image
blocks OCR cleanly at small font sizes. So pxpipe pulls the static prefix
out of the JSON body, renders it as one or more grayscale PNG image blocks,
and pins a single `cache_control` breakpoint on the last image. Anthropic
charges roughly `ceil(W*H/750)` tokens per image; a full static-slab page is
1573×1280, so each slab tile costs ~2684 tokens regardless of how much text it
carries. A 68K-token static slab collapses to ~3.5K image tokens on the first
turn and to a cache-read (billed at 0.10×) on every subsequent turn. The
trade is real text tokens for a few image tokens we cache once.
charges roughly `ceil(W*H/750)` tokens per image; the current Anthropic profile
caps pages at 1568×728 so rasterized pixels survive the provider's resize. A
large static slab becomes a small set of image pages on the first turn and a
cache-read (billed at 0.10×) on subsequent turns. The trade is real text tokens
for image tokens cached under the same stable prefix.
## 2. The static / dynamic split
@@ -258,25 +258,21 @@ The `1.25` and `0.10` multipliers match Anthropic's published cache pricing
for Opus: writing to the cache costs 1.25× the per-token input rate; reading
from the cache costs 0.10×.
**Per-call baseline cost** — what the *same* call would have billed if we had
NOT compressed:
**Per-call baseline cost** — what the same call would have billed if it had not
been compressed. The implementation uses the actual baseline probe and observed
cache state; it does not multiply image count by a fixed historical canvas:
```
text_tokens_we_removed = origChars / 4 # ~4 chars per token, rough
image_tokens_we_added = imageCount * 4761 # dense 1928×1928 ≈ 4761 tokens (slab ~2684)
extra_text_baseline = max(0, text_tokens_we_removed - image_tokens_we_added)
# cache_create dominates the first turn; bias the baseline toward 1.25 in
# that regime. Otherwise assume a fully warm cache (0.10).
cache_total = cache_create + cache_read
baseline_rate = cache_create > 0
? (cache_create / cache_total) * 1.25 + (1 - cache_create / cache_total) * 0.10
: 0.10
baseline = effective + extra_text_baseline * baseline_rate
saved = baseline - effective
baseline_raw = count_tokens(original_request)
baseline_eff = price(baseline_raw, observed_cache_state)
actual_eff = price(actual_usage, observed_cache_state)
saved = baseline_eff - actual_eff
```
On OpenAI-shaped rows, response usage supplies `cached_tokens`; pxpipe records
actual image tokens from real dimensions and the o200k text value of the
replaced content. Rates and render geometry are selected by exact model id.
The dashboard's "tokens saved" and "$ saved" cards (and
`pxpipe stats` for the offline aggregate) both surface these numbers.
The $ figure in `src/dashboard.ts` uses a fixed per-Mtok input rate — note
eval/grok-density/FACTSHEET_RESULTS.md
+11 -13
View File
@@ -6,17 +6,16 @@ Fixture: same precision-critical synthetic session as the density harness
## Question
Production Grok keeps **5×8** packing because pure-image exact OCR fails at
that density. Does the shipping **image + factsheet** path clear the Opus
exact bar without giving up density? Is a denser pure-image packing still
worth considering?
One candidate kept **5×8** packing and relied on an image + factsheet contract.
Does that combination clear this fixture's exact bar? How does it compare with
the lower-density pure-image profile?
## Results
| arm | exact | confab | gist | guard | save≈ | notes |
|-----|------:|-------:|:----:|:-----:|------:|-------|
| `5x8_image_only` | **0/4** | **4** | ok | ok | 76% | confabulates every ID |
| `5x8_image_plus_factsheet` | **4/4** | **0** | ok | ok | **70%** | **shipping contract** |
| `5x8_image_plus_factsheet` | **4/4** | **0** | ok | ok | **70%** | factsheet candidate |
| `5x8_grid_plus_factsheet` | 4/4 | 0 | ok | ok | 70% | style no worse with sheet |
| `5x8_color_plus_factsheet` | 4/4 | 0 | ok | ok | 70% | style no worse with sheet |
| `d4_c84_image_only` | 4/4 | 0 | ok | ok | **30%** | pure-image Opus bar, half the density win |
@@ -36,14 +35,13 @@ and history images (`src/core/openai.ts`).
1. **Grok stays opt-in only** (not in `DEFAULT_MODEL_BASES`). Same bar as Opus:
not good enough as a silent pxpipe default.
2. **Do not change default Grok density when opted in.** 5×8 stays.
3. **Exact IDs are a factsheet problem, not a cell-size problem** at production
density. Image+factsheet passes 4/4 with ~70% fixture savings; pure-image d4
also passes but only ~30% savings.
4. Style knobs (grid, colorCycle) at 5×8 do not replace the factsheet for exact
recall; with the sheet they are fine and optional.
5. Still open only if factsheet **coverage** misses a token class in the wild —
fix extractors, do not bloat cells.
2. The factsheet rescues the four token shapes in this fixture, but n=1 cannot
prove extractor coverage for real sessions.
3. The built-in opt-in profile therefore uses the independently measured
effective **9×12 / 84-column** arm (4/4, zero confab) and keeps the factsheet
as defense in depth. This is more conservative than 5×8 + factsheet.
4. Style knobs (grid, colorCycle) at 5×8 do not replace either larger cells or
the factsheet for exact recall.
Enable with `PXPIPE_MODELS=...,grok-4.5` or the dashboard Grok chip.
eval/grok-density/README.md
+8 -12
View File
@@ -1,9 +1,8 @@
# Grok density / image-recall sweep
**Question:** prior pxpipe recall work covered Fable 5, Opus 4.8, and GPT notes.
Grok is imaging in production through the OpenAI Responses path, but it has no
measured profile. Does Grok read production-density images well enough, or does
it need a lower-density / different strip geometry profile?
If Grok is explicitly enabled on the OpenAI Responses path, does it read the
shared dense profile, or does it need lower-density geometry?
This harness answers that with a measurement. It does **not** change production
defaults. A Grok profile should only be added if the numbers clear the bar
@@ -14,9 +13,7 @@ below.
For each render **variant** (cell size) it:
1. Renders one synthetic session transcript to PNG(s) with the production
renderer, using the OpenAI-path page geometry Grok actually sees today
(strip capped at 768 px wide / 1932 px tall — the default GPT-profile
geometry that unmatched models, including Grok, currently inherit).
renderer, capped at 768 px wide / 1932 px tall.
2. Asks the model a fixed battery of questions against the image(s).
3. Scores exact recall, gist, confabulation guard, and refusal.
@@ -79,16 +76,15 @@ Responses path with different page geometry and vision billing, so it needs
its own client and cost accounting. Shared pieces (fixture, scoring rules)
mirror the Opus harness on purpose so results are comparable.
## Production contract (image + factsheet)
## Factsheet arm
Density sweeps above measure **pure image** reading. Production Grok does not
rely on that for exact IDs: the Responses transform attaches a verbatim
fact-sheet next to the 5×8 images. Live proof that image-only fails and
image+factsheet clears the Opus exact bar:
Density sweeps above measure **pure image** reading. The Responses transform
also attaches a verbatim fact-sheet. The factsheet harness proves that it
rescues this fixture at 5×8, but not that every real token shape is covered.
The built-in opt-in profile therefore uses effective 9×12 and the factsheet:
```bash
GROK_DENSITY_LIVE=1 node eval/grok-density/factsheet-vs-image.mjs
```
See [FACTSHEET_RESULTS.md](./FACTSHEET_RESULTS.md).
eval/grok-density/RESULTS.md
+11 -13
View File
@@ -1,14 +1,14 @@
# Grok density / image-recall sweep — results
Live run of `run.mjs`, 2026-07-09. Model: `grok-4.5`. Geometry: default
Responses-path profile (`stripCols=152`, `maxHeightPx=1932`, vision tile
`85/170`). `results.json` is this run.
Live run of `run.mjs`, 2026-07-09. Model: `grok-4.5`. The original receipt used
the then-shared Responses geometry (`stripCols=152`, `maxHeightPx=1932`) and
the GPT tile estimator. `results.json` is that historical run; later measured
Grok billing is ~1000 tokens/MPix (see `CLIMB_RESULTS.md`).
## Question
Prior recall work covered Fable 5 and Opus 4.8. Grok is already imaging on the
Responses path under the unmatched-model fallback profile. Does it read
production-density images well enough, or does it need a lower-density profile?
Prior recall work covered Fable 5 and Opus 4.8. If Grok is opted in, does it
read the shared dense profile well enough, or need a lower-density profile?
## Exact-string recall
@@ -31,8 +31,8 @@ production-density images well enough, or does it need a lower-density profile?
nibble). Gist + guard ok.
**9x12** — clears the acceptance bar: 4/4 exact (byte-exact hex), 0 confab,
gist ok, guard ok. Savings collapse to ~5% under the current tile cost model
because the taller page bills more vision tokens.
gist ok, guard ok. This original table's 5% used the wrong GPT tile estimator;
measured Grok billing puts the same profile at ~30% fixture savings.
## Verdict
@@ -42,11 +42,9 @@ because the taller page bills more vision tokens.
strings. Candidate: an **opt-in lower-density Grok render profile at
`9x12`** (or at least denser than production). **Not** a silent default
change, and **not** a `DEFAULT_MODEL_BASES` change.
- Current unmatched-model vision profile (`tile 85/170`, 152 cols, 1932 px
height) is only a cost/geometry fallback. It does not measure Grok
readability. A real Grok profile should encode both:
1. render density that clears exact recall (`9x12` candidate)
2. vision billing numbers once Grok's real image-token formula is known
- The built-in opt-in Grok profile now encodes effective 9×12 / 84 columns and
measured ~1000 tokens/MPix billing. It still remains outside the default
allowlist because this is n=1 synthetic evidence.
- n=1 per cell (harness design). Re-run before baking a production profile.
## How this was run
eval/grok-density/brute-live-results.json
+102
View File
@@ -0,0 +1,102 @@
{
"generatedAt": "2026-07-09T11:47:03.225Z",
"model": "grok-4.5",
"textTokens": 1168,
"rows": [
{
"id": "w0h0_c152_H1932",
"cellWBonus": 0,
"cellHBonus": 0,
"stripCols": 152,
"maxHeightPx": 1932,
"cols": 152,
"pages": 1,
"dims": [
"768x360"
],
"imageTokens": 425,
"savingsPct": 64,
"model": {
"exactCorrect": 0,
"exactTotal": 4,
"confab": 3,
"gistOk": true,
"guardOk": true,
"answers": [
{
"id": "hex",
"kind": "exact",
"expected": "a3f9c1e0b7d2",
"got": "NOT STATED",
"status": "completed",
"ok": false,
"abstained": true,
"confab": false,
"refused": false,
"ms": 13550
},
{
"id": "camel",
"kind": "exact",
"expected": "tokenLedgerShard",
"got": "tokenedgeShard",
"status": "completed",
"ok": false,
"abstained": false,
"confab": true,
"refused": false,
"ms": 861
},
{
"id": "path",
"kind": "exact",
"expected": "src/core/anthropic-vision.ts",
"got": "pro/core/anthropic-client.ts",
"status": "completed",
"ok": false,
"abstained": false,
"confab": true,
"refused": false,
"ms": 2916
},
{
"id": "port",
"kind": "exact",
"expected": "47821",
"got": "97821",
"status": "completed",
"ok": false,
"abstained": false,
"confab": true,
"refused": false,
"ms": 2715
},
{
"id": "gist",
"kind": "gist",
"expected": "3",
"got": "3",
"status": "completed",
"ok": true,
"abstained": false,
"confab": false,
"refused": false,
"ms": 918
},
{
"id": "guard",
"kind": "guard",
"expected": "NOT STATED",
"got": "NOT STATED",
"status": "completed",
"ok": true,
"abstained": true,
"confab": false,
"refused": false,
"ms": 4625
}
]
}
}
]
}
eval/grok-density/brute-live.mjs
+213
View File
@@ -0,0 +1,213 @@
// Focused live sweep: density ladder where dry-run still shows real savings.
// Full 225-combo dry-run is in brute.mjs; this only spends model calls on the
// candidates that can still win on cost.
import { writeFileSync, readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { renderTextToPngs } from '../../dist/core/render.js';
import { openAIVisionTokens } from '../../dist/core/openai.js';
const here = dirname(fileURLToPath(import.meta.url));
const MODEL = process.env.GROK_DENSITY_MODEL || 'grok-4.5';
const TIMEOUT_MS = Number(process.env.GROK_DENSITY_TIMEOUT_MS || 180000);
const PAD_X = 4;
const CELL_BASE_W = 5;
const TRUTH = {
hex: 'a3f9c1e0b7d2',
camel: 'tokenLedgerShard',
path: 'src/core/anthropic-vision.ts',
port: '47821',
};
const SESSION = [
'<user t="1">Wire up the retry path. Use a retry budget of 3 attempts, backing off 250ms.</user>',
`<assistant t="2">Done. The token cache key is ${TRUTH.hex}. I renamed the field to ${TRUTH.camel}`,
`and moved the tier math into ${TRUTH.path}. The CLI now takes --max-visual-tokens. Proxy stays on port ${TRUTH.port}.</assistant>`,
'<user t="3">Good. Keep the retry budget as decided; do not change the backoff.</user>',
...Array.from({ length: 40 }, (_, i) =>
`<assistant t="${4 + i}">step ${i}: processed shard ${i} of the ${TRUTH.camel}, ok, continuing the run.</assistant>`),
].join('\n');
const QUESTIONS = [
{ id: 'hex', kind: 'exact', q: 'What is the exact token cache key (12-char hex) mentioned in the transcript?', answer: TRUTH.hex },
{ id: 'camel', kind: 'exact', q: 'What is the exact field name the assistant renamed the field to?', answer: TRUTH.camel },
{ id: 'path', kind: 'exact', q: 'What exact file path did the tier math move into?', answer: TRUTH.path },
{ id: 'port', kind: 'exact', q: 'What port does the proxy stay on?', answer: TRUTH.port },
{ id: 'gist', kind: 'gist', q: 'What retry budget was decided (a number of attempts)?', answer: '3' },
{ id: 'guard', kind: 'guard', q: 'What database password was configured in this session? If it was not stated, say "NOT STATED".', answer: 'NOT STATED' },
];
const TEXT_TOKENS = Math.ceil(SESSION.length / 4);
// Density ladder + geometry knobs that dry-run keeps save>=30% for most cells.
const CELLS = [
[0, 0], // production baseline
[1, 1],
[2, 2], // prior 7x10-ish
[2, 3],
[3, 2],
[3, 3],
[3, 4],
[4, 2],
[4, 3],
[4, 4], // prior 9x12-ish
];
const STRIP_COLS = [152, 128];
const MAX_H = [1932, 1536];
function colsFor(stripCols, wBonus) {
const maxW = 2 * PAD_X + stripCols * CELL_BASE_W;
return Math.max(8, Math.floor((maxW - 2 * PAD_X) / (CELL_BASE_W + wBonus)));
}
function responsesBaseUrl() {
const base = (process.env.OPENAI_BASE_URL || '').replace(/\/$/, '');
if (!base) throw new Error('OPENAI_BASE_URL required');
return base.endsWith('/responses') ? base : `${base}/responses`;
}
async function callModel(dataUrls, question) {
const key = process.env.OPENAI_API_KEY;
if (!key) throw new Error('OPENAI_API_KEY required');
const content = [
...dataUrls.map((u) => ({ type: 'input_image', image_url: u, detail: 'original' })),
{ type: 'input_text', text: question + '\nAnswer with ONLY the exact value, or "NOT STATED" if it is not present. Do not guess.' },
];
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
const t0 = Date.now();
try {
const res = await fetch(responsesBaseUrl(), {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${key}` },
body: JSON.stringify({
model: MODEL,
stream: false,
max_output_tokens: 512,
input: [{ role: 'user', content }],
}),
signal: controller.signal,
});
const raw = await res.text();
const j = JSON.parse(raw);
if (!res.ok) throw new Error(`HTTP ${res.status}: ${j?.error?.message || raw.slice(0, 160)}`);
let text = typeof j.output_text === 'string' ? j.output_text : '';
if (!text && Array.isArray(j.output)) {
for (const item of j.output) {
if (!item || item.type !== 'message' || !Array.isArray(item.content)) continue;
for (const part of item.content) {
if (part && (part.type === 'output_text' || part.type === 'text') && typeof part.text === 'string') text += part.text;
}
}
}
return { text: text.trim(), ms: Date.now() - t0, status: j.status || null };
} finally {
clearTimeout(timer);
}
}
function score(kind, expected, got) {
if (!got) return { ok: false, abstained: false, confab: false, refused: true };
const g = got.toLowerCase();
const abstained = /not stated|unknown|not safe|can't|cannot|not present|i don't know|do not know/.test(g);
if (kind === 'guard') return { ok: abstained, abstained, confab: !abstained, refused: false };
if (kind === 'gist') return { ok: g.includes(String(expected).toLowerCase()), abstained, confab: false, refused: false };
return { ok: got.includes(expected), abstained, confab: !got.includes(expected) && !abstained, refused: false };
}
const combos = [];
for (const stripCols of STRIP_COLS) {
for (const maxHeightPx of MAX_H) {
for (const [cellWBonus, cellHBonus] of CELLS) {
combos.push({
id: `w${cellWBonus}h${cellHBonus}_c${stripCols}_H${maxHeightPx}`,
cellWBonus,
cellHBonus,
stripCols,
maxHeightPx,
cols: colsFor(stripCols, cellWBonus),
style: { cellWBonus, cellHBonus, aa: true },
});
}
}
}
console.log(`Grok focused live brute · model=${MODEL} · combos=${combos.length}`);
const rows = [];
for (const c of combos) {
const imgs = await renderTextToPngs(SESSION, c.cols, c.style, c.maxHeightPx);
const pages = imgs.map((im) => ({ png: im.png, width: im.width, height: im.height }));
const imageTokens = pages.reduce((n, p) => n + openAIVisionTokens(MODEL, p.width, p.height), 0);
const savingsPct = Math.round((1 - imageTokens / TEXT_TOKENS) * 100);
const row = {
id: c.id,
cellWBonus: c.cellWBonus,
cellHBonus: c.cellHBonus,
stripCols: c.stripCols,
maxHeightPx: c.maxHeightPx,
cols: c.cols,
pages: pages.length,
dims: pages.map((p) => `${p.width}x${p.height}`),
imageTokens,
savingsPct,
model: null,
};
console.log(`\n[${c.id}] save=${savingsPct}% pages=${pages.length} dims=${row.dims.join(',')} imgTok=${imageTokens}`);
if (savingsPct < 15) {
console.log(' skip live: savings below 15%');
rows.push(row);
continue;
}
const dataUrls = pages.map((p) => 'data:image/png;base64,' + Buffer.from(p.png).toString('base64'));
const m = { exactCorrect: 0, exactTotal: 0, confab: 0, gistOk: false, guardOk: false, answers: [] };
for (const q of QUESTIONS) {
try {
const { text, ms, status } = await callModel(dataUrls, q.q);
const s = score(q.kind, q.answer, text);
m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: text, status, ...s, ms });
if (q.kind === 'exact') {
m.exactTotal++;
if (s.ok) m.exactCorrect++;
}
if (s.confab) m.confab++;
if (q.kind === 'gist' && !s.refused) m.gistOk = s.ok;
if (q.kind === 'guard' && !s.refused) m.guardOk = s.ok;
const mark = s.ok ? 'OK' : s.refused ? 'REFUSED' : s.abstained ? 'ABSTAIN' : s.confab ? 'CONFAB' : 'MISS';
console.log(` ${q.id.padEnd(6)} ${mark.padEnd(8)} ${JSON.stringify(text).slice(0, 70)} (${ms}ms)`);
} catch (err) {
console.error(` ${q.id.padEnd(6)} ERROR ${err.message}`);
m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: '', error: String(err.message || err), ok: false, confab: false, refused: true, ms: 0 });
if (q.kind === 'exact') m.exactTotal++;
}
}
row.model = m;
console.log(` → exact ${m.exactCorrect}/${m.exactTotal} confab ${m.confab} gist ${m.gistOk ? 'ok' : 'FAIL'} guard ${m.guardOk ? 'ok' : 'FAIL'} save ${savingsPct}%`);
rows.push(row);
// checkpoint after each combo so a mid-run death is not total loss
writeFileSync(join(here, 'brute-live-results.json'), JSON.stringify({
generatedAt: new Date().toISOString(),
model: MODEL,
textTokens: TEXT_TOKENS,
rows,
}, null, 2));
}
const liveRows = rows.filter((r) => r.model);
const perfect = liveRows
.filter((r) => r.model.exactCorrect === 4 && r.model.confab === 0 && r.model.gistOk && r.model.guardOk)
.sort((a, b) => b.savingsPct - a.savingsPct || a.imageTokens - b.imageTokens);
const good = liveRows
.filter((r) => r.model.exactCorrect >= 3 && r.model.confab <= 1 && r.model.gistOk && r.model.guardOk)
.sort((a, b) => b.savingsPct - a.savingsPct || a.imageTokens - b.imageTokens);
const out = {
generatedAt: new Date().toISOString(),
model: MODEL,
textTokens: TEXT_TOKENS,
rows,
perfect,
good,
bestPerfect: perfect[0] || null,
bestGood: good[0] || null,
};
writeFileSync(join(here, 'brute-live-results.json'), JSON.stringify(out, null, 2));
console.log('\nBEST perfect:', perfect[0]?.id, perfect[0]?.savingsPct);
console.log('BEST good:', good[0]?.id, good[0]?.savingsPct, good[0]?.model && `${good[0].model.exactCorrect}/4`);
for (const r of perfect.slice(0, 8)) {
console.log(` perfect ${r.id} save=${r.savingsPct}% dims=${r.dims.join(',')}`);
}
eval/grok-density/brute-results.json
+8153
View File
File diff suppressed because it is too large Load Diff
eval/grok-density/brute.mjs
+259
View File
@@ -0,0 +1,259 @@
// Brute-force Grok render profiles across density, strip geometry, AND style.
// Production is a fixed 5x8 atlas (not a TrueType face), so "font" here means
// the atlas/style arms the renderer actually exposes: AA vs 1-bit, grid, color.
//
// Dry-run default. GROK_BRUTE_LIVE=1 scores recall only for combos with save>=15%.
//
// pnpm run build
// node eval/grok-density/brute.mjs
// GROK_BRUTE_LIVE=1 node eval/grok-density/brute.mjs
import { writeFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { renderTextToPngs } from '../../dist/core/render.js';
import { openAIVisionTokens } from '../../dist/core/openai.js';
const here = dirname(fileURLToPath(import.meta.url));
const MODEL = process.env.GROK_DENSITY_MODEL || 'grok-4.5';
const LIVE = process.env.GROK_BRUTE_LIVE === '1' || process.env.GROK_BRUTE_LIVE === 'true';
const TIMEOUT_MS = Number(process.env.GROK_DENSITY_TIMEOUT_MS || 180000);
const MIN_SAVE = Number(process.env.GROK_BRUTE_MIN_SAVE || 15);
const PAD_X = 4;
const CELL_BASE_W = 5;
const TRUTH = {
hex: 'a3f9c1e0b7d2',
camel: 'tokenLedgerShard',
path: 'src/core/anthropic-vision.ts',
port: '47821',
};
const SESSION = [
'<user t="1">Wire up the retry path. Use a retry budget of 3 attempts, backing off 250ms.</user>',
`<assistant t="2">Done. The token cache key is ${TRUTH.hex}. I renamed the field to ${TRUTH.camel}`,
`and moved the tier math into ${TRUTH.path}. The CLI now takes --max-visual-tokens. Proxy stays on port ${TRUTH.port}.</assistant>`,
'<user t="3">Good. Keep the retry budget as decided; do not change the backoff.</user>',
...Array.from({ length: 40 }, (_, i) =>
`<assistant t="${4 + i}">step ${i}: processed shard ${i} of the ${TRUTH.camel}, ok, continuing the run.</assistant>`),
].join('\n');
const QUESTIONS = [
{ id: 'hex', kind: 'exact', q: 'What is the exact token cache key (12-char hex) mentioned in the transcript?', answer: TRUTH.hex },
{ id: 'camel', kind: 'exact', q: 'What is the exact field name the assistant renamed the field to?', answer: TRUTH.camel },
{ id: 'path', kind: 'exact', q: 'What exact file path did the tier math move into?', answer: TRUTH.path },
{ id: 'port', kind: 'exact', q: 'What port does the proxy stay on?', answer: TRUTH.port },
{ id: 'gist', kind: 'gist', q: 'What retry budget was decided (a number of attempts)?', answer: '3' },
{ id: 'guard', kind: 'guard', q: 'What database password was configured in this session? If it was not stated, say "NOT STATED".', answer: 'NOT STATED' },
];
const TEXT_TOKENS = Math.ceil(SESSION.length / 4);
// Style arms the renderer can actually change (atlas is fixed Unifont 5x8, not a face picker).
const STYLES = [
{ name: 'aa', style: { aa: true } },
{ name: 'onebit', style: { aa: false } },
{ name: 'aa+grid', style: { aa: true, grid: true } },
{ name: 'aa+color', style: { aa: true, colorCycle: true } },
{ name: 'aa+role', style: { aa: true, colorByRole: true } },
{ name: 'onebit+grid', style: { aa: false, grid: true } },
{ name: 'onebit+color', style: { aa: false, colorCycle: true } },
];
// Density ladder: production through large cells.
const CELLS = [
[0, 0],
[1, 1],
[2, 2],
[2, 3],
[3, 2],
[3, 3],
[4, 3],
[4, 4],
];
const STRIP_COLS = [152, 128, 100];
const MAX_H = [1932, 1536, 1024];
function colsFor(stripCols, wBonus) {
const maxW = 2 * PAD_X + stripCols * CELL_BASE_W;
return Math.max(8, Math.floor((maxW - 2 * PAD_X) / (CELL_BASE_W + wBonus)));
}
function responsesBaseUrl() {
const base = (process.env.OPENAI_BASE_URL || '').replace(/\/$/, '');
if (!base) throw new Error('OPENAI_BASE_URL required for live');
return base.endsWith('/responses') ? base : `${base}/responses`;
}
async function callModel(dataUrls, question) {
const key = process.env.OPENAI_API_KEY;
if (!key) throw new Error('OPENAI_API_KEY required for live');
const content = [
...dataUrls.map((u) => ({ type: 'input_image', image_url: u, detail: 'original' })),
{ type: 'input_text', text: question + '\nAnswer with ONLY the exact value, or "NOT STATED" if it is not present. Do not guess.' },
];
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
const t0 = Date.now();
try {
const res = await fetch(responsesBaseUrl(), {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${key}` },
body: JSON.stringify({
model: MODEL,
stream: false,
max_output_tokens: 512,
input: [{ role: 'user', content }],
}),
signal: controller.signal,
});
const raw = await res.text();
const j = JSON.parse(raw);
if (!res.ok) throw new Error(`HTTP ${res.status}: ${j?.error?.message || raw.slice(0, 160)}`);
let text = typeof j.output_text === 'string' ? j.output_text : '';
if (!text && Array.isArray(j.output)) {
for (const item of j.output) {
if (!item || item.type !== 'message' || !Array.isArray(item.content)) continue;
for (const part of item.content) {
if (part && (part.type === 'output_text' || part.type === 'text') && typeof part.text === 'string') text += part.text;
}
}
}
return { text: text.trim(), ms: Date.now() - t0, status: j.status || null };
} finally {
clearTimeout(timer);
}
}
function score(kind, expected, got) {
if (!got) return { ok: false, abstained: false, confab: false, refused: true };
const g = got.toLowerCase();
const abstained = /not stated|unknown|not safe|can't|cannot|not present|i don't know|do not know/.test(g);
if (kind === 'guard') return { ok: abstained, abstained, confab: !abstained, refused: false };
if (kind === 'gist') return { ok: g.includes(String(expected).toLowerCase()), abstained, confab: false, refused: false };
return { ok: got.includes(expected), abstained, confab: !got.includes(expected) && !abstained, refused: false };
}
const combos = [];
for (const stripCols of STRIP_COLS) {
for (const maxHeightPx of MAX_H) {
for (const [cellWBonus, cellHBonus] of CELLS) {
for (const st of STYLES) {
combos.push({
id: `w${cellWBonus}h${cellHBonus}_c${stripCols}_H${maxHeightPx}_${st.name}`,
cellWBonus,
cellHBonus,
stripCols,
maxHeightPx,
styleName: st.name,
cols: colsFor(stripCols, cellWBonus),
style: { ...st.style, cellWBonus, cellHBonus },
});
}
}
}
}
console.log(`Grok brute · model=${MODEL} · combos=${combos.length} · live=${LIVE} · minSave=${MIN_SAVE}`);
console.log(`axes: cells=${CELLS.length} styles=${STYLES.length} stripCols=${STRIP_COLS.length} maxH=${MAX_H.length}`);
console.log(`text baseline ≈ ${TEXT_TOKENS} tok`);
const rows = [];
for (const c of combos) {
const imgs = await renderTextToPngs(SESSION, c.cols, c.style, c.maxHeightPx);
const pages = imgs.map((im) => ({ png: im.png, width: im.width, height: im.height }));
const imageTokens = pages.reduce((n, p) => n + openAIVisionTokens(MODEL, p.width, p.height), 0);
const savingsPct = Math.round((1 - imageTokens / TEXT_TOKENS) * 100);
const row = {
id: c.id,
cellWBonus: c.cellWBonus,
cellHBonus: c.cellHBonus,
stripCols: c.stripCols,
maxHeightPx: c.maxHeightPx,
styleName: c.styleName,
cols: c.cols,
pages: pages.length,
dims: pages.map((p) => `${p.width}x${p.height}`),
imageTokens,
savingsPct,
model: null,
};
const liveThis = LIVE && savingsPct >= MIN_SAVE;
if (!LIVE || liveThis) {
console.log(`[${c.id}] save=${savingsPct}% pages=${pages.length} dims=${row.dims.join(',')} imgTok=${imageTokens}`);
} else {
console.log(`[skip-live ${c.id}] save=${savingsPct}%`);
}
if (liveThis) {
const dataUrls = pages.map((p) => 'data:image/png;base64,' + Buffer.from(p.png).toString('base64'));
const m = { exactCorrect: 0, exactTotal: 0, confab: 0, gistOk: false, guardOk: false, answers: [] };
for (const q of QUESTIONS) {
try {
const { text, ms, status } = await callModel(dataUrls, q.q);
const s = score(q.kind, q.answer, text);
m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: text, status, ...s, ms });
if (q.kind === 'exact') {
m.exactTotal++;
if (s.ok) m.exactCorrect++;
}
if (s.confab) m.confab++;
if (q.kind === 'gist' && !s.refused) m.gistOk = s.ok;
if (q.kind === 'guard' && !s.refused) m.guardOk = s.ok;
const mark = s.ok ? 'OK' : s.refused ? 'REFUSED' : s.abstained ? 'ABSTAIN' : s.confab ? 'CONFAB' : 'MISS';
console.log(` ${q.id.padEnd(6)} ${mark.padEnd(8)} ${JSON.stringify(text).slice(0, 70)} (${ms}ms)`);
} catch (err) {
console.error(` ${q.id.padEnd(6)} ERROR ${err.message}`);
m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: '', error: String(err.message || err), ok: false, confab: false, refused: true, ms: 0 });
if (q.kind === 'exact') m.exactTotal++;
}
}
row.model = m;
console.log(` → exact ${m.exactCorrect}/${m.exactTotal} confab ${m.confab} gist ${m.gistOk ? 'ok' : 'FAIL'} guard ${m.guardOk ? 'ok' : 'FAIL'} save ${savingsPct}%`);
// checkpoint
writeFileSync(join(here, 'brute-results.json'), JSON.stringify({
generatedAt: new Date().toISOString(),
model: MODEL,
live: LIVE,
textTokens: TEXT_TOKENS,
rows: [...rows, row],
}, null, 2));
}
rows.push(row);
}
const liveRows = rows.filter((r) => r.model);
const perfect = liveRows
.filter((r) => r.model.exactCorrect === 4 && r.model.confab === 0 && r.model.gistOk && r.model.guardOk)
.sort((a, b) => b.savingsPct - a.savingsPct || a.imageTokens - b.imageTokens);
const good = liveRows
.filter((r) => r.model.exactCorrect >= 3 && r.model.confab <= 1 && r.model.gistOk && r.model.guardOk)
.sort((a, b) => b.savingsPct - a.savingsPct || a.imageTokens - b.imageTokens);
const out = {
generatedAt: new Date().toISOString(),
model: MODEL,
live: LIVE,
textTokens: TEXT_TOKENS,
comboCount: combos.length,
axes: { cells: CELLS, styles: STYLES.map((s) => s.name), stripCols: STRIP_COLS, maxH: MAX_H },
rows,
perfect,
good,
bestPerfect: perfect[0] || null,
bestGood: good[0] || null,
};
writeFileSync(join(here, 'brute-results.json'), JSON.stringify(out, null, 2));
console.log(`\nWrote ${join(here, 'brute-results.json')}`);
if (!LIVE) {
const profitable = rows.filter((r) => r.savingsPct >= MIN_SAVE);
console.log(`Dry-run: ${profitable.length}/${rows.length} combos save>=${MIN_SAVE}%`);
// best savings per style at fixed production geometry, and best savings overall
const byStyle = new Map();
for (const r of profitable) {
const k = r.styleName;
if (!byStyle.has(k) || r.savingsPct > byStyle.get(k).savingsPct) byStyle.set(k, r);
}
console.log('Best savings per style (any density/geometry):');
for (const [k, r] of [...byStyle.entries()].sort((a, b) => b[1].savingsPct - a[1].savingsPct)) {
console.log(` ${k}: ${r.id} save=${r.savingsPct}% dims=${r.dims.join(',')}`);
}
} else {
console.log('BEST perfect:', perfect[0]?.id, perfect[0]?.savingsPct);
console.log('BEST good:', good[0]?.id, good[0]?.savingsPct, good[0]?.model && `${good[0].model.exactCorrect}/4`);
for (const r of perfect.slice(0, 10)) console.log(` perfect ${r.id} save=${r.savingsPct}%`);
}
eval/grok-density/live-all-results.json
+1658
View File
File diff suppressed because it is too large Load Diff
eval/grok-density/live-all.mjs
+277
View File
@@ -0,0 +1,277 @@
// Live brute: real fonts (eval atlases) × style arms × densest packing.
// Goal: match Opus bar (4/4 exact, 0 confab, gist+guard ok) at the highest
// savings. Uses measured Grok image billing ~1000 tok/MPix for savings.
//
// Temporarily swaps src/core/atlas*.ts for each font, rebuilds dist, scores,
// then restores production atlases from git.
import { spawnSync } from 'node:child_process';
import { copyFileSync, writeFileSync, readFileSync, existsSync } from 'node:fs';
import { resolve, dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
const here = dirname(fileURLToPath(import.meta.url));
const root = resolve(here, '../..');
const MODEL = process.env.GROK_DENSITY_MODEL || 'grok-4.5';
const TIMEOUT_MS = Number(process.env.GROK_DENSITY_TIMEOUT_MS || 180000);
const TOK_PER_MPIX = 1000; // from measure-image-tokens.mjs
const TRUTH = {
hex: 'a3f9c1e0b7d2',
camel: 'tokenLedgerShard',
path: 'src/core/anthropic-vision.ts',
port: '47821',
};
const SESSION = [
'<user t="1">Wire up the retry path. Use a retry budget of 3 attempts, backing off 250ms.</user>',
`<assistant t="2">Done. The token cache key is ${TRUTH.hex}. I renamed the field to ${TRUTH.camel}`,
`and moved the tier math into ${TRUTH.path}. The CLI now takes --max-visual-tokens. Proxy stays on port ${TRUTH.port}.</assistant>`,
'<user t="3">Good. Keep the retry budget as decided; do not change the backoff.</user>',
...Array.from({ length: 40 }, (_, i) =>
`<assistant t="${4 + i}">step ${i}: processed shard ${i} of the ${TRUTH.camel}, ok, continuing the run.</assistant>`),
].join('\n');
const QUESTIONS = [
{ id: 'hex', kind: 'exact', q: 'What is the exact token cache key (12-char hex) mentioned in the transcript?', answer: TRUTH.hex },
{ id: 'camel', kind: 'exact', q: 'What is the exact field name the assistant renamed the field to?', answer: TRUTH.camel },
{ id: 'path', kind: 'exact', q: 'What exact file path did the tier math move into?', answer: TRUTH.path },
{ id: 'port', kind: 'exact', q: 'What port does the proxy stay on?', answer: TRUTH.port },
{ id: 'gist', kind: 'gist', q: 'What retry budget was decided (a number of attempts)?', answer: '3' },
{ id: 'guard', kind: 'guard', q: 'What database password was configured in this session? If it was not stated, say "NOT STATED".', answer: 'NOT STATED' },
];
const TEXT_TOKENS = Math.ceil(SESSION.length / 4);
const FONTS = [
{ name: 'spleen5x8', onebit: 'atlas-spleen5x8.ts', gray: 'atlas-gray-spleen5x8.ts' },
{ name: 'jbmono8', onebit: 'atlas-jbmono8.ts', gray: 'atlas-gray-jbmono8.ts' },
{ name: 'jbmono10', onebit: 'atlas-jbmono10.ts', gray: 'atlas-gray-jbmono10.ts' },
{ name: 'unifont8', onebit: 'atlas-unifont8.ts', gray: 'atlas-gray-unifont8.ts' },
{ name: 'unifont10', onebit: 'atlas-unifont10.ts', gray: 'atlas-gray-unifont10.ts' },
];
// Style arms that do not require a different atlas size (same cell).
const STYLES = [
{ name: 'aa', style: { aa: true, cellWBonus: 0, cellHBonus: 0 } },
{ name: 'onebit', style: { aa: false, cellWBonus: 0, cellHBonus: 0 } },
{ name: 'aa+grid', style: { aa: true, grid: true, gridCols: 1, cellWBonus: 0, cellHBonus: 0 } },
{ name: 'aa+color', style: { aa: true, colorCycle: true, cellWBonus: 0, cellHBonus: 0 } },
{ name: 'aa+role', style: { aa: true, colorByRole: true, cellWBonus: 0, cellHBonus: 0 } },
{ name: 'onebit+grid', style: { aa: false, grid: true, gridCols: 1, cellWBonus: 0, cellHBonus: 0 } },
{ name: 'onebit+color', style: { aa: false, colorCycle: true, cellWBonus: 0, cellHBonus: 0 } },
// small spacing only if needed later — included so we can climb toward Opus bar without jumping to 9x12
{ name: 'aa+w1h1', style: { aa: true, cellWBonus: 1, cellHBonus: 1 } },
{ name: 'aa+w2h2', style: { aa: true, cellWBonus: 2, cellHBonus: 2 } },
];
// Geometry: densest first (max savings). stripCols from cell width at runtime.
const GEOS = [
{ name: 'wide1932', stripPx: 768, maxHeightPx: 1932 },
{ name: 'wide1536', stripPx: 768, maxHeightPx: 1536 },
{ name: 'mid1932', stripPx: 640, maxHeightPx: 1932 },
];
function sh(cmd, env = {}) {
const r = spawnSync(cmd, {
cwd: root,
env: { ...process.env, ...env },
encoding: 'utf8',
shell: true,
});
if (r.stdout) process.stdout.write(r.stdout);
if (r.stderr) process.stderr.write(r.stderr);
return r.status ?? 1;
}
function restoreProdAtlas() {
sh('git checkout -- src/core/atlas.ts src/core/atlas-gray.ts');
sh('pnpm run build');
}
function installFont(font) {
const one = join(here, 'atlases', font.onebit);
const gray = join(here, 'atlases', font.gray);
if (!existsSync(one) || !existsSync(gray)) throw new Error(`missing atlas for ${font.name}`);
copyFileSync(one, resolve(root, 'src/core/atlas.ts'));
// gray atlas export names differ (ATLAS_GRAY_*). The production atlas-gray.ts
// file already uses those names. Our generator writes ATLAS_GRAY_* only in gray mode.
copyFileSync(gray, resolve(root, 'src/core/atlas-gray.ts'));
const st = sh('pnpm run build');
if (st !== 0) throw new Error(`build failed for ${font.name}`);
}
function responsesBaseUrl() {
const base = (process.env.OPENAI_BASE_URL || '').replace(/\/$/, '');
if (!base) throw new Error('OPENAI_BASE_URL required');
return base.endsWith('/responses') ? base : `${base}/responses`;
}
async function callModel(dataUrls, question) {
const key = process.env.OPENAI_API_KEY;
if (!key) throw new Error('OPENAI_API_KEY required');
const content = [
...dataUrls.map((u) => ({ type: 'input_image', image_url: u, detail: 'original' })),
{ type: 'input_text', text: question + '\nAnswer with ONLY the exact value, or "NOT STATED" if it is not present. Do not guess.' },
];
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
const t0 = Date.now();
try {
const res = await fetch(responsesBaseUrl(), {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${key}` },
body: JSON.stringify({
model: MODEL,
stream: false,
max_output_tokens: 512,
input: [{ role: 'user', content }],
}),
signal: controller.signal,
});
const raw = await res.text();
const j = JSON.parse(raw);
if (!res.ok) throw new Error(`HTTP ${res.status}: ${j?.error?.message || raw.slice(0, 160)}`);
let text = typeof j.output_text === 'string' ? j.output_text : '';
if (!text && Array.isArray(j.output)) {
for (const item of j.output) {
if (!item || item.type !== 'message' || !Array.isArray(item.content)) continue;
for (const part of item.content) {
if (part && (part.type === 'output_text' || part.type === 'text') && typeof part.text === 'string') text += part.text;
}
}
}
return {
text: text.trim(),
ms: Date.now() - t0,
status: j.status || null,
usage: j.usage || null,
};
} finally {
clearTimeout(timer);
}
}
function score(kind, expected, got) {
if (!got) return { ok: false, abstained: false, confab: false, refused: true };
const g = got.toLowerCase();
const abstained = /not stated|unknown|not safe|can't|cannot|not present|i don't know|do not know/.test(g);
if (kind === 'guard') return { ok: abstained, abstained, confab: !abstained, refused: false };
if (kind === 'gist') return { ok: g.includes(String(expected).toLowerCase()), abstained, confab: false, refused: false };
return { ok: got.includes(expected), abstained, confab: !got.includes(expected) && !abstained, refused: false };
}
const rows = [];
const outPath = join(here, 'live-all-results.json');
function checkpoint() {
const perfect = rows
.filter((r) => r.model && r.model.exactCorrect === 4 && r.model.confab === 0 && r.model.gistOk && r.model.guardOk)
.sort((a, b) => b.savingsPct - a.savingsPct || a.imageTokensEst - b.imageTokensEst);
writeFileSync(
outPath,
JSON.stringify(
{
generatedAt: new Date().toISOString(),
model: MODEL,
textTokens: TEXT_TOKENS,
goal: 'opus-bar: 4/4 exact, 0 confab, gist+guard ok, maximize savings',
rows,
perfect,
bestPerfect: perfect[0] || null,
},
null,
2,
),
);
}
try {
for (const font of FONTS) {
console.log(`\n######## FONT ${font.name} ########`);
installFont(font);
// dynamic import after rebuild
const { renderTextToPngs } = await import('../../dist/core/render.js?font=' + font.name + '&t=' + Date.now());
const { ATLAS_CELL_W, ATLAS_CELL_H } = await import('../../dist/core/atlas.js?font=' + font.name + '&t=' + Date.now());
console.log(`atlas cell ${ATLAS_CELL_W}x${ATLAS_CELL_H}`);
for (const geo of GEOS) {
for (const st of STYLES) {
const cellW = ATLAS_CELL_W + (st.style.cellWBonus || 0);
const cols = Math.max(8, Math.floor((geo.stripPx - 8) / cellW));
const id = `${font.name}_${st.name}_${geo.name}_c${cols}`;
const imgs = await renderTextToPngs(SESSION, cols, st.style, geo.maxHeightPx);
const pages = imgs.map((im) => ({ png: im.png, width: im.width, height: im.height }));
const pixels = pages.reduce((n, p) => n + p.width * p.height, 0);
const imageTokensEst = Math.round((pixels / 1e6) * TOK_PER_MPIX);
const savingsPct = Math.round((1 - imageTokensEst / TEXT_TOKENS) * 100);
const row = {
id,
font: font.name,
style: st.name,
geo: geo.name,
cell: `${ATLAS_CELL_W}x${ATLAS_CELL_H}`,
cols,
pages: pages.length,
dims: pages.map((p) => `${p.width}x${p.height}`),
pixels,
imageTokensEst,
savingsPct,
model: null,
};
console.log(`\n[${id}] save~${savingsPct}% pages=${pages.length} dims=${row.dims.join(',')} imgTok~${imageTokensEst}`);
// Skip hopeless savings only if worse than a known perfect candidate later;
// still score all first-pass dense candidates. Skip only if negative savings.
if (savingsPct < 0) {
console.log(' skip: negative savings');
rows.push(row);
checkpoint();
continue;
}
const dataUrls = pages.map((p) => 'data:image/png;base64,' + Buffer.from(p.png).toString('base64'));
const m = { exactCorrect: 0, exactTotal: 0, confab: 0, gistOk: false, guardOk: false, answers: [], usageImageDelta: null };
for (const q of QUESTIONS) {
try {
const { text, ms, status, usage } = await callModel(dataUrls, q.q);
const s = score(q.kind, q.answer, text);
m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: text, status, ...s, ms, usage });
if (q.kind === 'exact') {
m.exactTotal++;
if (s.ok) m.exactCorrect++;
}
if (s.confab) m.confab++;
if (q.kind === 'gist' && !s.refused) m.gistOk = s.ok;
if (q.kind === 'guard' && !s.refused) m.guardOk = s.ok;
if (q.id === 'hex' && usage?.input_tokens != null) m.usageInput = usage.input_tokens;
const mark = s.ok ? 'OK' : s.refused ? 'REFUSED' : s.abstained ? 'ABSTAIN' : s.confab ? 'CONFAB' : 'MISS';
console.log(` ${q.id.padEnd(6)} ${mark.padEnd(8)} ${JSON.stringify(text).slice(0, 70)} (${ms}ms)`);
} catch (err) {
console.error(` ${q.id.padEnd(6)} ERROR ${err.message}`);
m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: '', error: String(err.message || err), ok: false, confab: false, refused: true, ms: 0 });
if (q.kind === 'exact') m.exactTotal++;
}
}
row.model = m;
const pass = m.exactCorrect === 4 && m.confab === 0 && m.gistOk && m.guardOk;
console.log(
` → exact ${m.exactCorrect}/${m.exactTotal} confab ${m.confab} gist ${m.gistOk ? 'ok' : 'FAIL'} guard ${m.guardOk ? 'ok' : 'FAIL'} save~${savingsPct}% ${pass ? '*** OPUS BAR ***' : ''}`,
);
rows.push(row);
checkpoint();
}
}
}
} finally {
console.log('\nRestoring production atlas...');
restoreProdAtlas();
}
const perfect = rows
.filter((r) => r.model && r.model.exactCorrect === 4 && r.model.confab === 0 && r.model.gistOk && r.model.guardOk)
.sort((a, b) => b.savingsPct - a.savingsPct || a.imageTokensEst - b.imageTokensEst);
console.log('\n======== SUMMARY ========');
console.log('perfect count', perfect.length);
for (const r of perfect.slice(0, 15)) {
console.log(` ${r.id} save~${r.savingsPct}% imgTok~${r.imageTokensEst} dims=${r.dims.join(',')}`);
}
if (perfect[0]) console.log('BEST', perfect[0].id, perfect[0].savingsPct);
else console.log('No combo reached Opus bar yet.');
checkpoint();
eval/grok-density/run.mjs
+8 -7
View File
@@ -12,7 +12,7 @@ import { fileURLToPath } from 'node:url';
import { dirname, join } from 'node:path';
import { renderTextToPngs } from '../../dist/core/render.js';
import { resolveGptProfile } from '../../dist/core/gpt-model-profiles.js';
import { openAIVisionTokens } from '../../dist/core/openai.js';
import { visionTokensForModel } from '../../dist/core/openai.js';
const here = dirname(fileURLToPath(import.meta.url));
@@ -44,15 +44,17 @@ const QUESTIONS = [
{ id: 'guard', kind: 'guard', q: 'What database password was configured in this session? If it was not stated, say "NOT STATED".', answer: 'NOT STATED' },
];
// Grok currently inherits the default GPT-path geometry (stripCols=152 → 768 px
// wide, maxHeightPx=1932). Keep the strip under that short-side floor so the
// provider does not resample 5 px glyphs away.
// Keep every density arm under the 768px short-side floor so the provider does
// not resample small glyphs away. PROFILE is used for the max-height contract;
// the width baseline stays 152 production 5x8 columns so arms remain comparable
// to the banked receipt even though the current opt-in Grok profile is 9x12/84.
const MODEL = process.env.GROK_DENSITY_MODEL || 'grok-4.5';
const PROFILE = resolveGptProfile(MODEL);
const PAD_X = 4;
const CELL_BASE_W = 5;
const DENSE_BASELINE_COLS = 152;
const colsForSafe = (wBonus) => {
const maxW = 2 * PAD_X + PROFILE.stripCols * CELL_BASE_W; // 768 for default profile
const maxW = 2 * PAD_X + DENSE_BASELINE_COLS * CELL_BASE_W;
return Math.floor((maxW - 2 * PAD_X) / (CELL_BASE_W + wBonus));
};
@@ -67,7 +69,7 @@ const TEXT_TOKENS = Math.ceil(SESSION.length / 4);
function imageTokensForPages(pages) {
// Use the same vision-cost function the Responses transform uses for this model.
return pages.reduce((n, p) => n + openAIVisionTokens(MODEL, p.width, p.height), 0);
return pages.reduce((n, p) => n + visionTokensForModel(MODEL, p.width, p.height), 0);
}
function responsesBaseUrl() {
@@ -258,4 +260,3 @@ console.log(`\nWrote ${outPath}`);
if (!live) {
console.log('Dry-run only. Re-run with GROK_DENSITY_LIVE=1 to score model answers.');
}
eval/sol-profile/.gitignore
+2
View File
@@ -0,0 +1,2 @@
# Locally rendered sources/PNGs. Hashes and dimensions are recorded in preflight.json.
.work/
eval/sol-profile/README.md
+151
View File
@@ -0,0 +1,151 @@
# GPT-5.6 Sol render-profile pilot
This is a deliberately small, paired **raw image-reading** pilot for the exact
`gpt-5.6-sol` model. It compares:
- `old_shared`: Spleen/Unifont grayscale AA, 5×8 cell, 152 columns;
- `current_sol`: the production-resolved JetBrains Mono 10/Unifont grayscale AA,
6×11 cell, 126 columns.
Both arms keep the 1932 px height cap. The harness asserts all of these values
before rendering and never targets Grok.
## Live status (2026-07-09)
The 5×8 arm **was tested**, not merely rendered:
| arm | paid/scored | exact | confabulations | gist | guard | status |
|---|:---:|---:|---:|:---:|:---:|---|
| JetBrains 6×11 / 126 cols | yes | **0/4** | **4** | pass | pass | fail |
| Spleen 5×8 / 152 cols | yes | **0/4** | **4** | fail | pass | fail |
| JetBrains effective 9×12 / 84 cols | no | — | — | — | — | locally rendered; paid call pending |
There was also one paid setup attempt for 6×11 that returned no answer because
all 512 output tokens were hidden reasoning tokens. It counts toward spending
but is excluded from recall scoring. Thus the receipts currently contain three
paid attempts: one invalid setup attempt and two scored profile calls.
Full expected/returned values, usage, latency, image dimensions, scoring, and
receipt links are in [`RESULTS.md`](./RESULTS.md).
**Policy decision:** Sol is now off by default. Its exact profile remains
available through the dashboard or explicit
`PXPIPE_MODELS=claude-fable-5,gpt-5.6-sol`. It should not return to the built-in
default until a retuned paid arm clears 4/4 exact, zero confabulations, gist,
and guard with positive savings.
## Design
Two fixed-seed synthetic terminal fixtures each contain one labelled value for:
1. a 12-character hexadecimal fingerprint;
2. a camelCase runtime field;
3. a full path;
4. a port;
5. a rollout gist (scored as an A/B/C choice);
6. one deliberately unstated fact, which must produce `NOT STATED`.
Each fixture/profile arm sends all six questions in **one JSON response**, so the
maximum is 2 fixtures × 2 profiles = **4 paid calls**. Calls are independent and
counterbalanced after the first pair. The candidate Sol profile runs first so a
candidate failure costs only one call. There are no automatic retries. Any API failure, malformed response,
exact-read miss, gist/guard miss, refusal, or confabulation stops the pilot
early. A manually inspected attempt that returns no answer because hidden
reasoning exhausts `max_output_tokens` is a harness failure, still counts toward
the four-call cap, and may be resumed only with a reduced remaining plan.
A wrong exact value is recorded as a confabulation only when it does not occur
verbatim in the rendered fixture. A wrong but visible distractor is an exact-read
failure, not an invention. Any factual answer to the unstated guard is a
confabulation.
## Raw path, not pxpipe
Live requests post directly to the OpenAI-compatible `/v1/responses` endpoint
from `OPENAI_BASE_URL` (or the eval-only `SOL_PROFILE_BASE_URL`). Image parts use
`detail: "original"`. The harness rejects the known pxpipe listener port 47821,
so it cannot recursively transform its own evaluation input.
## Run
```bash
pnpm run build
# Local render + hashes + dimensions + projected token accounting only.
# This mode performs no fetch and cannot incur model charges.
node eval/sol-profile/run.mjs
```
The dry run writes `preflight.json` and local PNGs under ignored `.work/`.
Review the projected call and token totals, then obtain explicit approval before
live mode. Live mode has a second literal guard:
```bash
SOL_PROFILE_LIVE=1 \
SOL_PROFILE_PAID_APPROVAL=approved-4-sol-profile-calls \
node eval/sol-profile/run.mjs
```
Do not set that acknowledgement merely because it appears in this README; it
represents approval given after reviewing the current preflight. The live run
writes:
- `results.json`: raw response bodies, provider usage, scoring, confabulations,
latency, image dimensions, hashes, and estimated image-vs-text savings;
- `raw/*.response.json`: byte-for-byte response bodies;
- `raw/*.receipt.json`: request metadata and response receipts, without secrets
or embedded image base64.
## Interpretation boundary
This pilot has at most **n=2 fixtures per profile**. Passing supports retaining
Sol's profile as a small pilot; it does not prove general task quality, exact
recall on real transcripts, or superiority over every font/geometry. Local
rendering and production token telemetry are cost/geometry evidence only, not a
readability benchmark. A failure should retune or revert only the Sol profile.
### Output-cap recovery
The initial 2026-07-09 live attempt used low reasoning and returned no answer:
all 512 output tokens were reasoning tokens. After inspecting that receipt, the
manual recovery used `reasoning: none` and only three remaining calls (candidate
alpha, old alpha, candidate beta), preserving the four-attempt cap:
```bash
SOL_PROFILE_LIVE=1 \
SOL_PROFILE_PAID_APPROVAL=approved-4-sol-profile-calls \
SOL_PROFILE_RESUME_AFTER_OUTPUT_CAP=1 \
node eval/sol-profile/run.mjs
```
### Old-profile fallback after candidate failure
If the completed 6×11 candidate call returns 0/4 exact plus four unsupported
values, the already-preflighted old profile is treated as the smallest Sol-only
fallback candidate. It remains inside the original four-attempt and token cap:
```bash
SOL_PROFILE_LIVE=1 \
SOL_PROFILE_PAID_APPROVAL=approved-4-sol-profile-calls \
SOL_PROFILE_RESUME_OLD_AS_RETUNE=1 \
node eval/sol-profile/run.mjs
```
The beta fallback call runs only if alpha passes every acceptance check.
### Final spaced Sol candidate
If both original arms return 0/4 exact with four confabulations, the final
within-cap candidate keeps Sol's JetBrains Mono 10 glyphs but expands them to an
effective 9×12 cell at 84 columns. This changes only the Sol candidate and uses
the fourth and final paid attempt:
```bash
SOL_PROFILE_LIVE=1 \
SOL_PROFILE_PAID_APPROVAL=approved-4-sol-profile-calls \
SOL_PROFILE_RESUME_SPACED_RETUNE=1 \
node eval/sol-profile/run.mjs
```
eval/sol-profile/RESULTS.md
+141
View File
@@ -0,0 +1,141 @@
# GPT-5.6 Sol raw image-recall pilot
Live run: **2026-07-09**
Model: **`gpt-5.6-sol`**
Endpoint: direct OpenAI-compatible Responses path, bypassing pxpipe
Image detail: **`original`**
This pilot tests raw reading of rendered terminal text. It does not include
pxpipe's verbatim fact-sheet, and it does not compare Sol directly with Fable.
## Acceptance bar
A profile clears this small fixture only if all of the following hold in one
structured response:
- 4/4 exact values: 12-character hex, camelCase field, full path, and port;
- correct rollout gist;
- correct `NOT STATED` response for an absent fact;
- zero unsupported invented values.
## Results
| profile | image | estimated image tokens | exact | confabulations | gist | guard | verdict |
|---|---:|---:|---:|---:|:---:|:---:|---|
| JetBrains Mono 10, 6×11, 126 cols | 764×1724 | 1,296 | **0/4** | **4** | pass | pass | **fail** |
| Spleen, 5×8, 152 cols | 768×1040 | 792 | **0/4** | **4** | fail | pass | **fail** |
## Paid-attempt ledger
Every paid attempt is retained, including the unscored setup failure:
| # | arm | reasoning | HTTP/result | input | output (reasoning) | latency | receipt |
|---:|---|---|---|---:|---:|---:|---|
| 1 | alpha / JetBrains 6×11 | low | incomplete: `max_output_tokens`; **not scored** | 1,748 | 512 (512) | 13,857 ms | [body](./raw/01-alpha-current_sol.response.json), [metadata](./raw/01-alpha-current_sol.receipt.json) |
| 2 | alpha / JetBrains 6×11 | none | completed; 0/4 exact, 4 confabulations | 1,748 | 52 (0) | 3,294 ms | [body](./raw/02-alpha-current_sol.response.json), [metadata](./raw/02-alpha-current_sol.receipt.json) |
| 3 | alpha / Spleen 5×8 | none | completed; 0/4 exact, 4 confabulations | 1,143 | 59 (0) | 2,429 ms | [body](./raw/03-alpha-old_shared.response.json), [metadata](./raw/03-alpha-old_shared.receipt.json) |
| 4 | alpha / JetBrains effective 9×12 | none | **not called; approval pending** | ≈2,354 projected | — | — | local hashes in [preflight](./preflight.json) |
The 5×8 line is a real paid model call. It is not an inferred result from local
rendering or from Grok's separate evaluation.
### JetBrains 6×11 response
Expected:
- `c7a1e90b4d2f`
- `retryBudgetSeconds`
- `/srv/sol-pilot/releases/alpha-07/config/runtime-map.json`
- `47831`
Returned:
- `c0ffee0913a7`
- `matrixField`
- `/srv/app/releases/alpha-07/config/runtime.yaml`
- `41702`
All four returned values were absent from the fixture, so they are counted as
confabulations rather than mere OCR substitutions. The model correctly selected
gist `B` and returned `NOT STATED` for the absent region.
Provider usage: 1,748 input tokens, 52 output tokens, zero reasoning tokens;
3,294 ms latency.
### Spleen 5×8 response
Returned:
- `7ac089b47c21`
- `fluxLimiterCeiling`
- `/srv/releases/2025-04-18/edge-router/manifest.json`
- `41820`
Again, all four values were absent from the fixture. The model also selected the
wrong gist (`A` instead of `B`), while correctly returning `NOT STATED` for the
absent region.
Provider usage: 1,143 input tokens, 59 output tokens, zero reasoning tokens;
2,429 ms latency.
## Setup attempt excluded from scoring
The first paid attempt used `reasoning: low`. It returned no answer because all
512 output tokens were hidden reasoning tokens and the response ended
`incomplete` with `max_output_tokens`. It is retained in the receipts and counts
against the four-call spending cap, but it is not evidence about recall.
Across the three paid attempts so far (one setup failure and two scored calls),
provider usage totals 4,639 input tokens and 623 output tokens.
## Interpretation
1. **Neither tested profile clears raw Sol exact recall.** The current 6×11
profile is better on gist than the old 5×8 profile, but 0/4 exact plus four
inventions fails the acceptance bar.
2. This is one synthetic fixture per scored profile. It proves those calls
failed; it does not establish a population failure rate or compare Sol with
Fable.
3. Production Responses transforms attach a verbatim fact-sheet for paths and
identifiers. That defense may preserve covered exact strings in real pxpipe
traffic, but it does not turn this raw-image result into a pass.
4. Local token savings are cost evidence only: approximately 73.5% for 6×11 and
83.8% for 5×8 versus this fixture's chars/4 text estimate.
## Default-scope decision
`gpt-5.6-sol` is **off by default** after these failures, matching the policy for
GPT 5.5 and Grok: silent image rewriting requires positive reader evidence, not
only positive token savings. The exact Sol profile remains available for
explicit operator opt-in:
```bash
PXPIPE_MODELS='claude-fable-5,gpt-5.6-sol'
```
Re-enabling it by default requires a retuned paid arm to pass 4/4 exact, zero
confabulations, gist, and guard while retaining positive savings. A replicated
pass on both fixtures is preferred before silent promotion.
## Pending Sol-only retune
A larger candidate has been rendered locally but not called:
- JetBrains Mono 10 glyphs in effective 9×12 cells;
- 84 columns;
- pages 764×1928 and 764×884;
- 2,136 estimated image tokens, ≈2,354 projected total input tokens;
- ≈56.3% estimated image-vs-text savings.
It is the fourth and final possible paid attempt. Its higher input projection
requires separate approval before the call. Until then, it is local geometry,
not model evidence.
## Receipts
- Machine-readable summary: [`results.json`](./results.json)
- Preflight and image hashes: [`preflight.json`](./preflight.json)
- Byte-for-byte response bodies and request receipts: [`raw/`](./raw/)
eval/sol-profile/preflight.json
+261
View File
@@ -0,0 +1,261 @@
{
"generatedAt": "2026-07-09T22:02:35.780Z",
"live": false,
"model": "gpt-5.6-sol",
"endpointPolicy": "direct Responses endpoint; port 47821 is rejected",
"callCap": 4,
"callOrder": [
{
"fixture": "alpha",
"profile": "current_sol"
},
{
"fixture": "alpha",
"profile": "old_shared"
},
{
"fixture": "beta",
"profile": "old_shared"
},
{
"fixture": "beta",
"profile": "current_sol"
}
],
"profiles": {
"old_shared": {
"cols": 152,
"maxHeightPx": 1932,
"cell": "5x8",
"style": {
"font": "spleen-5x8",
"cellWBonus": 0,
"cellHBonus": 0,
"aa": true,
"grid": false,
"gridCols": 0,
"colorCycle": false,
"markerScale": 1,
"markerRed": false
}
},
"current_sol": {
"cols": 126,
"maxHeightPx": 1932,
"cell": "6x11",
"style": {
"font": "jetbrains-mono-10",
"cellWBonus": 0,
"cellHBonus": 0,
"aa": true,
"grid": false,
"gridCols": 0,
"colorCycle": false,
"markerScale": 1,
"markerRed": false
}
},
"retuned_sol_9x12": {
"cols": 84,
"maxHeightPx": 1932,
"cell": "9x12",
"style": {
"font": "jetbrains-mono-10",
"cellWBonus": 3,
"cellHBonus": 1,
"aa": true,
"grid": false,
"gridCols": 0,
"colorCycle": false,
"markerScale": 1,
"markerRed": false
}
}
},
"arms": [
{
"fixture": "alpha",
"profile": "current_sol",
"expected": {
"fingerprint": "c7a1e90b4d2f",
"camelCase": "retryBudgetSeconds",
"path": "/srv/sol-pilot/releases/alpha-07/config/runtime-map.json",
"port": "47831",
"gist": "B",
"unstated": "NOT STATED"
},
"sourceChars": 19543,
"sourceSha256": "2115e44441e61b5a9e251934f09a9fa305a6307db3f600806d73a75d408d4f7a",
"sourceTokenizerTokens": 5621,
"textTokensCharsPer4": 4886,
"pages": [
{
"file": ".work/alpha/current_sol/page-1.png",
"width": 764,
"height": 1724,
"bytes": 113913,
"sha256": "0caaf82a632c9f4d9486bfbd9db86f714eb7d6111d3c9ac89e9a248c73a3fba7",
"estimatedImageTokens": 1296
}
],
"imageTokens": 1296,
"projectedNonImageInputTokens": 210,
"projectedInputTokens": 1506,
"estimatedSavingsVsTextPct": 73.5
},
{
"fixture": "alpha",
"profile": "old_shared",
"expected": {
"fingerprint": "c7a1e90b4d2f",
"camelCase": "retryBudgetSeconds",
"path": "/srv/sol-pilot/releases/alpha-07/config/runtime-map.json",
"port": "47831",
"gist": "B",
"unstated": "NOT STATED"
},
"sourceChars": 19543,
"sourceSha256": "2115e44441e61b5a9e251934f09a9fa305a6307db3f600806d73a75d408d4f7a",
"sourceTokenizerTokens": 5621,
"textTokensCharsPer4": 4886,
"pages": [
{
"file": ".work/alpha/old_shared/page-1.png",
"width": 768,
"height": 1040,
"bytes": 75258,
"sha256": "71c5f88bcfa3ec196d824221d61563031877b1ea70aa4e37ef50084f8898e043",
"estimatedImageTokens": 792
}
],
"imageTokens": 792,
"projectedNonImageInputTokens": 210,
"projectedInputTokens": 1002,
"estimatedSavingsVsTextPct": 83.8
},
{
"fixture": "beta",
"profile": "old_shared",
"expected": {
"fingerprint": "8d3f6a20c1e7",
"camelCase": "maxVisualTokens",
"path": "/opt/sol-pilot/fixtures/beta/session-ledger.toml",
"port": "18082",
"gist": "A",
"unstated": "NOT STATED"
},
"sourceChars": 19524,
"sourceSha256": "d0ebdef72da526883ec91855635b2cbf548f0b8a661c137324057a9ac230a68f",
"sourceTokenizerTokens": 5621,
"textTokensCharsPer4": 4881,
"pages": [
{
"file": ".work/beta/old_shared/page-1.png",
"width": 768,
"height": 1040,
"bytes": 75542,
"sha256": "98d345bf3fe990ce83466437bb0468b78e2325e4e7245e51e22b6ba618e70899",
"estimatedImageTokens": 792
}
],
"imageTokens": 792,
"projectedNonImageInputTokens": 208,
"projectedInputTokens": 1000,
"estimatedSavingsVsTextPct": 83.8
},
{
"fixture": "beta",
"profile": "current_sol",
"expected": {
"fingerprint": "8d3f6a20c1e7",
"camelCase": "maxVisualTokens",
"path": "/opt/sol-pilot/fixtures/beta/session-ledger.toml",
"port": "18082",
"gist": "A",
"unstated": "NOT STATED"
},
"sourceChars": 19524,
"sourceSha256": "d0ebdef72da526883ec91855635b2cbf548f0b8a661c137324057a9ac230a68f",
"sourceTokenizerTokens": 5621,
"textTokensCharsPer4": 4881,
"pages": [
{
"file": ".work/beta/current_sol/page-1.png",
"width": 764,
"height": 1713,
"bytes": 116406,
"sha256": "e309491c6fd973fcad1951ba473bebe305c7b07f7ee50fee94a269faf548cfdd",
"estimatedImageTokens": 1296
}
],
"imageTokens": 1296,
"projectedNonImageInputTokens": 208,
"projectedInputTokens": 1504,
"estimatedSavingsVsTextPct": 73.4
}
],
"totals": {
"calls": 4,
"imageTokens": 4176,
"projectedNonImageInputTokens": 836,
"projectedInputTokens": 5012,
"byProfile": {
"current_sol": {
"calls": 2,
"imageTokens": 2592,
"projectedNonImageInputTokens": 418,
"projectedInputTokens": 3010
},
"old_shared": {
"calls": 2,
"imageTokens": 1584,
"projectedNonImageInputTokens": 418,
"projectedInputTokens": 2002
}
},
"currentVsOldImageTokenIncreasePct": 63.6
},
"finalRetuneCandidate": {
"fixture": "alpha",
"profile": "retuned_sol_9x12",
"expected": {
"fingerprint": "c7a1e90b4d2f",
"camelCase": "retryBudgetSeconds",
"path": "/srv/sol-pilot/releases/alpha-07/config/runtime-map.json",
"port": "47831",
"gist": "B",
"unstated": "NOT STATED"
},
"sourceChars": 19543,
"sourceSha256": "2115e44441e61b5a9e251934f09a9fa305a6307db3f600806d73a75d408d4f7a",
"sourceTokenizerTokens": 5621,
"textTokensCharsPer4": 4886,
"pages": [
{
"file": ".work/alpha/retuned_sol_9x12/page-1.png",
"width": 764,
"height": 1928,
"bytes": 99191,
"sha256": "f3dedc0e470b8270d2e31e1e894014674330200cb8037217233420c454a77f37",
"estimatedImageTokens": 1464
},
{
"file": ".work/alpha/retuned_sol_9x12/page-2.png",
"width": 764,
"height": 884,
"bytes": 45483,
"sha256": "412ceab2ee6f292217f9136858539a481e65d226e983c1e5a82f527cba8fccc6",
"estimatedImageTokens": 672
}
],
"imageTokens": 2136,
"projectedNonImageInputTokens": 218,
"projectedInputTokens": 2354,
"estimatedSavingsVsTextPct": 56.3
},
"tokenNotes": {
"image": "pxpipe production visionTokensForModel(gpt-5.6-sol, width, height) patch estimate",
"nonImage": "gpt-tokenizer(prompt) plus a small Responses-envelope allowance; provider usage may differ",
"textBaseline": "source chars/4, used only for estimated image-vs-text savings"
}
}
eval/sol-profile/raw/01-alpha-current_sol.receipt.json
+30
View File
@@ -0,0 +1,30 @@
{
"sequence": 1,
"fixture": "alpha",
"profile": "current_sol",
"request": {
"endpoint": "http://127.0.0.1:8082/v1/responses",
"model": "gpt-5.6-sol",
"maxOutputTokens": 512,
"reasoningEffort": "low",
"imageDetail": "original",
"imageSha256": [
"0caaf82a632c9f4d9486bfbd9db86f714eb7d6111d3c9ac89e9a248c73a3fba7"
],
"imageDimensions": [
"764x1724"
],
"prompt": "Read only the attached synthetic terminal image(s). Do not use outside knowledge and do not guess.\nReturn exactly one JSON object with these six string keys and no Markdown:\n- \"fingerprint\": the 12-character DEPLOYMENT_FINGERPRINT value.\n- \"camelCase\": the RUNTIME_FIELD value, preserving case.\n- \"path\": the complete ACTIVE_MANIFEST value.\n- \"port\": the CONTROL_PORT digits.\n- \"gist\": choose only \"A\", \"B\", or \"C\":\n A = the rollout resumed after health checks recovered;\n B = the rollout stayed paused because a checksum mismatch remained;\n C = the transcript does not state the rollout outcome.\n- \"unstated\": answer \"Which cloud region hosted the canary?\"; if absent, return exactly \"NOT STATED\".\nIf an exact field is unreadable, return \"UNREADABLE\" for that field instead of inventing a value."
},
"response": {
"status": 200,
"headers": {
"x-request-id": "5dbb1379-3a2b-49c1-88e6-2ef4baa5aded",
"openai-processing-ms": "13292",
"cf-ray": "a18a94dc3f3efce9-AUS-DOG"
},
"latencyMs": 13857,
"rawBodySha256": "07afb8a3fb7d685a8d480c5efa31697a9b51bb0ecb09720858cdcee7e7dbec4a",
"error": null
}
}
eval/sol-profile/raw/01-alpha-current_sol.response.json
+87
View File
@@ -0,0 +1,87 @@
{
"id": "resp_0413981843da1074016a501928820c8190a2a8c656f84e1c75",
"object": "response",
"created_at": 1783634216,
"status": "incomplete",
"background": false,
"billing": {
"payer": "developer"
},
"completed_at": null,
"error": null,
"frequency_penalty": 0.0,
"incomplete_details": {
"reason": "max_output_tokens"
},
"instructions": null,
"max_output_tokens": 512,
"max_tool_calls": null,
"model": "gpt-5.6-sol",
"moderation": null,
"output": [
{
"id": "rs_0413981843da1074016a50192976748190bb8cf1789c294ae6",
"type": "reasoning",
"content": [],
"summary": []
}
],
"parallel_tool_calls": true,
"presence_penalty": 0.0,
"previous_response_id": null,
"prompt_cache_key": null,
"prompt_cache_retention": "in_memory",
"reasoning": {
"context": "all_turns",
"effort": "low",
"mode": "standard",
"summary": null
},
"safety_identifier": null,
"service_tier": "default",
"store": false,
"temperature": 1.0,
"text": {
"format": {
"type": "text"
},
"verbosity": "medium"
},
"tool_choice": "auto",
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 0
}
},
"tools": [],
"top_logprobs": 0,
"top_p": 0.98,
"truncation": "disabled",
"usage": {
"input_tokens": 1748,
"input_tokens_details": {
"cache_write_tokens": 1745,
"cached_tokens": 0
},
"output_tokens": 512,
"output_tokens_details": {
"reasoning_tokens": 512
},
"total_tokens": 2260
},
"user": null,
"metadata": {}
}
eval/sol-profile/raw/02-alpha-current_sol.receipt.json
+30
View File
@@ -0,0 +1,30 @@
{
"sequence": 2,
"fixture": "alpha",
"profile": "current_sol",
"request": {
"endpoint": "http://127.0.0.1:8082/v1/responses",
"model": "gpt-5.6-sol",
"maxOutputTokens": 512,
"reasoningEffort": "none",
"imageDetail": "original",
"imageSha256": [
"0caaf82a632c9f4d9486bfbd9db86f714eb7d6111d3c9ac89e9a248c73a3fba7"
],
"imageDimensions": [
"764x1724"
],
"prompt": "Read only the attached synthetic terminal image(s). Do not use outside knowledge and do not guess.\nReturn exactly one JSON object with these six string keys and no Markdown:\n- \"fingerprint\": the 12-character DEPLOYMENT_FINGERPRINT value.\n- \"camelCase\": the RUNTIME_FIELD value, preserving case.\n- \"path\": the complete ACTIVE_MANIFEST value.\n- \"port\": the CONTROL_PORT digits.\n- \"gist\": choose only \"A\", \"B\", or \"C\":\n A = the rollout resumed after health checks recovered;\n B = the rollout stayed paused because a checksum mismatch remained;\n C = the transcript does not state the rollout outcome.\n- \"unstated\": answer \"Which cloud region hosted the canary?\"; if absent, return exactly \"NOT STATED\".\nIf an exact field is unreadable, return \"UNREADABLE\" for that field instead of inventing a value."
},
"response": {
"status": 200,
"headers": {
"x-request-id": "fd696db0-302f-4065-9714-209dbcaab732",
"openai-processing-ms": "2948",
"cf-ray": "a18a9751a965fce9-AUS-DOG"
},
"latencyMs": 3294,
"rawBodySha256": "492e540e233eb5dd21caf87a6246cec0c62d866fc8dca6e57b8cca019918dad9",
"error": null
}
}
eval/sol-profile/raw/02-alpha-current_sol.response.json
+94
View File
@@ -0,0 +1,94 @@
{
"id": "resp_020acbb2cc7a89da016a50198d3364819387c192a9f4b370b9",
"object": "response",
"created_at": 1783634317,
"status": "completed",
"background": false,
"billing": {
"payer": "developer"
},
"completed_at": 1783634320,
"error": null,
"frequency_penalty": 0.0,
"incomplete_details": null,
"instructions": null,
"max_output_tokens": 512,
"max_tool_calls": null,
"model": "gpt-5.6-sol",
"moderation": null,
"output": [
{
"id": "msg_020acbb2cc7a89da016a50198edbdc819383c8352f6b908521",
"type": "message",
"status": "completed",
"content": [
{
"type": "output_text",
"annotations": [],
"logprobs": [],
"text": "{\"fingerprint\":\"c0ffee0913a7\",\"camelCase\":\"matrixField\",\"path\":\"/srv/app/releases/alpha-07/config/runtime.yaml\",\"port\":\"41702\",\"gist\":\"B\",\"unstated\":\"NOT STATED\"}"
}
],
"phase": "final_answer",
"role": "assistant"
}
],
"parallel_tool_calls": true,
"presence_penalty": 0.0,
"previous_response_id": null,
"prompt_cache_key": null,
"prompt_cache_retention": "in_memory",
"reasoning": {
"context": "all_turns",
"effort": "none",
"mode": "standard",
"summary": null
},
"safety_identifier": null,
"service_tier": "default",
"store": false,
"temperature": 1.0,
"text": {
"format": {
"type": "text"
},
"verbosity": "low"
},
"tool_choice": "auto",
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 0
}
},
"tools": [],
"top_logprobs": 0,
"top_p": 0.98,
"truncation": "disabled",
"usage": {
"input_tokens": 1748,
"input_tokens_details": {
"cache_write_tokens": 1745,
"cached_tokens": 0
},
"output_tokens": 52,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 1800
},
"user": null,
"metadata": {}
}
eval/sol-profile/raw/03-alpha-old_shared.receipt.json
+30
View File
@@ -0,0 +1,30 @@
{
"sequence": 3,
"fixture": "alpha",
"profile": "old_shared",
"request": {
"endpoint": "http://127.0.0.1:8082/v1/responses",
"model": "gpt-5.6-sol",
"maxOutputTokens": 512,
"reasoningEffort": "none",
"imageDetail": "original",
"imageSha256": [
"71c5f88bcfa3ec196d824221d61563031877b1ea70aa4e37ef50084f8898e043"
],
"imageDimensions": [
"768x1040"
],
"prompt": "Read only the attached synthetic terminal image(s). Do not use outside knowledge and do not guess.\nReturn exactly one JSON object with these six string keys and no Markdown:\n- \"fingerprint\": the 12-character DEPLOYMENT_FINGERPRINT value.\n- \"camelCase\": the RUNTIME_FIELD value, preserving case.\n- \"path\": the complete ACTIVE_MANIFEST value.\n- \"port\": the CONTROL_PORT digits.\n- \"gist\": choose only \"A\", \"B\", or \"C\":\n A = the rollout resumed after health checks recovered;\n B = the rollout stayed paused because a checksum mismatch remained;\n C = the transcript does not state the rollout outcome.\n- \"unstated\": answer \"Which cloud region hosted the canary?\"; if absent, return exactly \"NOT STATED\".\nIf an exact field is unreadable, return \"UNREADABLE\" for that field instead of inventing a value."
},
"response": {
"status": 200,
"headers": {
"x-request-id": "ff15ed67-d378-4e1b-ad9a-d2d3d9c2be9d",
"openai-processing-ms": "2034",
"cf-ray": "a18a9b58c937fce9-AUS-DOG"
},
"latencyMs": 2429,
"rawBodySha256": "5ede01fba8670a28b80d9f7ce3bb7b60ae4d86ef50826afc8c787f59b45dfbdc",
"error": null
}
}
eval/sol-profile/raw/03-alpha-old_shared.response.json
+94
View File
@@ -0,0 +1,94 @@
{
"id": "resp_083fde5dff13ac2c016a501a3242348197882ae33f8b15e29a",
"object": "response",
"created_at": 1783634482,
"status": "completed",
"background": false,
"billing": {
"payer": "developer"
},
"completed_at": 1783634484,
"error": null,
"frequency_penalty": 0.0,
"incomplete_details": null,
"instructions": null,
"max_output_tokens": 512,
"max_tool_calls": null,
"model": "gpt-5.6-sol",
"moderation": null,
"output": [
{
"id": "msg_083fde5dff13ac2c016a501a3331948197afa081cd8f933ab0",
"type": "message",
"status": "completed",
"content": [
{
"type": "output_text",
"annotations": [],
"logprobs": [],
"text": "{\"fingerprint\":\"7ac089b47c21\",\"camelCase\":\"fluxLimiterCeiling\",\"path\":\"/srv/releases/2025-04-18/edge-router/manifest.json\",\"port\":\"41820\",\"gist\":\"A\",\"unstated\":\"NOT STATED\"}"
}
],
"phase": "final_answer",
"role": "assistant"
}
],
"parallel_tool_calls": true,
"presence_penalty": 0.0,
"previous_response_id": null,
"prompt_cache_key": null,
"prompt_cache_retention": "in_memory",
"reasoning": {
"context": "all_turns",
"effort": "none",
"mode": "standard",
"summary": null
},
"safety_identifier": null,
"service_tier": "default",
"store": false,
"temperature": 1.0,
"text": {
"format": {
"type": "text"
},
"verbosity": "low"
},
"tool_choice": "auto",
"tool_usage": {
"image_gen": {
"input_tokens": 0,
"input_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"output_tokens": 0,
"output_tokens_details": {
"image_tokens": 0,
"text_tokens": 0
},
"total_tokens": 0
},
"web_search": {
"num_requests": 0
}
},
"tools": [],
"top_logprobs": 0,
"top_p": 0.98,
"truncation": "disabled",
"usage": {
"input_tokens": 1143,
"input_tokens_details": {
"cache_write_tokens": 1140,
"cached_tokens": 0
},
"output_tokens": 59,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 1202
},
"user": null,
"metadata": {}
}
eval/sol-profile/results.json
+324
View File
@@ -0,0 +1,324 @@
{
"generatedAt": "2026-07-09T21:56:55.765Z",
"model": "gpt-5.6-sol",
"live": true,
"preflightSha256": "78288b351deea4bce2e21f018c255904770e78864040a8771ed6e08f94e58179",
"plannedCalls": 4,
"completedCalls": 3,
"stoppedEarly": true,
"stopReason": "clear acceptance failure on alpha/old_shared",
"calls": [
{
"sequence": 1,
"fixture": "alpha",
"profile": "current_sol",
"imageDimensions": [
"764x1724"
],
"estimatedImageTokens": 1296,
"estimatedSavingsVsTextPct": 73.5,
"latencyMs": 13857,
"httpStatus": 200,
"responseHeaders": {
"x-request-id": "5dbb1379-3a2b-49c1-88e6-2ef4baa5aded",
"openai-processing-ms": "13292",
"cf-ray": "a18a94dc3f3efce9-AUS-DOG"
},
"transportError": null,
"usage": {
"input_tokens": 1748,
"input_tokens_details": {
"cache_write_tokens": 1745,
"cached_tokens": 0
},
"output_tokens": 512,
"output_tokens_details": {
"reasoning_tokens": 512
},
"total_tokens": 2260
},
"responseId": "resp_0413981843da1074016a501928820c8190a2a8c656f84e1c75",
"responseModel": "gpt-5.6-sol",
"rawResponseBody": "{\n \"id\": \"resp_0413981843da1074016a501928820c8190a2a8c656f84e1c75\",\n \"object\": \"response\",\n \"created_at\": 1783634216,\n \"status\": \"incomplete\",\n \"background\": false,\n \"billing\": {\n \"payer\": \"developer\"\n },\n \"completed_at\": null,\n \"error\": null,\n \"frequency_penalty\": 0.0,\n \"incomplete_details\": {\n \"reason\": \"max_output_tokens\"\n },\n \"instructions\": null,\n \"max_output_tokens\": 512,\n \"max_tool_calls\": null,\n \"model\": \"gpt-5.6-sol\",\n \"moderation\": null,\n \"output\": [\n {\n \"id\": \"rs_0413981843da1074016a50192976748190bb8cf1789c294ae6\",\n \"type\": \"reasoning\",\n \"content\": [],\n \"summary\": []\n }\n ],\n \"parallel_tool_calls\": true,\n \"presence_penalty\": 0.0,\n \"previous_response_id\": null,\n \"prompt_cache_key\": null,\n \"prompt_cache_retention\": \"in_memory\",\n \"reasoning\": {\n \"context\": \"all_turns\",\n \"effort\": \"low\",\n \"mode\": \"standard\",\n \"summary\": null\n },\n \"safety_identifier\": null,\n \"service_tier\": \"default\",\n \"store\": false,\n \"temperature\": 1.0,\n \"text\": {\n \"format\": {\n \"type\": \"text\"\n },\n \"verbosity\": \"medium\"\n },\n \"tool_choice\": \"auto\",\n \"tool_usage\": {\n \"image_gen\": {\n \"input_tokens\": 0,\n \"input_tokens_details\": {\n \"image_tokens\": 0,\n \"text_tokens\": 0\n },\n \"output_tokens\": 0,\n \"output_tokens_details\": {\n \"image_tokens\": 0,\n \"text_tokens\": 0\n },\n \"total_tokens\": 0\n },\n \"web_search\": {\n \"num_requests\": 0\n }\n },\n \"tools\": [],\n \"top_logprobs\": 0,\n \"top_p\": 0.98,\n \"truncation\": \"disabled\",\n \"usage\": {\n \"input_tokens\": 1748,\n \"input_tokens_details\": {\n \"cache_write_tokens\": 1745,\n \"cached_tokens\": 0\n },\n \"output_tokens\": 512,\n \"output_tokens_details\": {\n \"reasoning_tokens\": 512\n },\n \"total_tokens\": 2260\n },\n \"user\": null,\n \"metadata\": {}\n}",
"rawResponseSha256": "07afb8a3fb7d685a8d480c5efa31697a9b51bb0ecb09720858cdcee7e7dbec4a",
"score": {
"responseStatus": "incomplete",
"incompleteReason": "max_output_tokens",
"outputText": "",
"refusals": [],
"parseError": "empty output",
"strictJson": false,
"schemaValid": false,
"exact": {
"fingerprint": {
"expected": "c7a1e90b4d2f",
"got": "",
"ok": false
},
"camelCase": {
"expected": "retryBudgetSeconds",
"got": "",
"ok": false
},
"path": {
"expected": "/srv/sol-pilot/releases/alpha-07/config/runtime-map.json",
"got": "",
"ok": false
},
"port": {
"expected": "47831",
"got": "",
"ok": false
}
},
"exactCorrect": 0,
"exactTotal": 4,
"gist": {
"expected": "B",
"got": "",
"ok": false
},
"guard": {
"expected": "NOT STATED",
"got": "",
"ok": false
},
"confabulations": [],
"abstentions": [],
"pass": false
},
"reasoningEffort": "low"
},
{
"sequence": 2,
"fixture": "alpha",
"profile": "current_sol",
"imageDimensions": [
"764x1724"
],
"estimatedImageTokens": 1296,
"estimatedSavingsVsTextPct": 73.5,
"latencyMs": 3294,
"httpStatus": 200,
"responseHeaders": {
"x-request-id": "fd696db0-302f-4065-9714-209dbcaab732",
"openai-processing-ms": "2948",
"cf-ray": "a18a9751a965fce9-AUS-DOG"
},
"transportError": null,
"usage": {
"input_tokens": 1748,
"input_tokens_details": {
"cache_write_tokens": 1745,
"cached_tokens": 0
},
"output_tokens": 52,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 1800
},
"responseId": "resp_020acbb2cc7a89da016a50198d3364819387c192a9f4b370b9",
"responseModel": "gpt-5.6-sol",
"reasoningEffort": "none",
"rawResponseBody": "{\n \"id\": \"resp_020acbb2cc7a89da016a50198d3364819387c192a9f4b370b9\",\n \"object\": \"response\",\n \"created_at\": 1783634317,\n \"status\": \"completed\",\n \"background\": false,\n \"billing\": {\n \"payer\": \"developer\"\n },\n \"completed_at\": 1783634320,\n \"error\": null,\n \"frequency_penalty\": 0.0,\n \"incomplete_details\": null,\n \"instructions\": null,\n \"max_output_tokens\": 512,\n \"max_tool_calls\": null,\n \"model\": \"gpt-5.6-sol\",\n \"moderation\": null,\n \"output\": [\n {\n \"id\": \"msg_020acbb2cc7a89da016a50198edbdc819383c8352f6b908521\",\n \"type\": \"message\",\n \"status\": \"completed\",\n \"content\": [\n {\n \"type\": \"output_text\",\n \"annotations\": [],\n \"logprobs\": [],\n \"text\": \"{\\\"fingerprint\\\":\\\"c0ffee0913a7\\\",\\\"camelCase\\\":\\\"matrixField\\\",\\\"path\\\":\\\"/srv/app/releases/alpha-07/config/runtime.yaml\\\",\\\"port\\\":\\\"41702\\\",\\\"gist\\\":\\\"B\\\",\\\"unstated\\\":\\\"NOT STATED\\\"}\"\n }\n ],\n \"phase\": \"final_answer\",\n \"role\": \"assistant\"\n }\n ],\n \"parallel_tool_calls\": true,\n \"presence_penalty\": 0.0,\n \"previous_response_id\": null,\n \"prompt_cache_key\": null,\n \"prompt_cache_retention\": \"in_memory\",\n \"reasoning\": {\n \"context\": \"all_turns\",\n \"effort\": \"none\",\n \"mode\": \"standard\",\n \"summary\": null\n },\n \"safety_identifier\": null,\n \"service_tier\": \"default\",\n \"store\": false,\n \"temperature\": 1.0,\n \"text\": {\n \"format\": {\n \"type\": \"text\"\n },\n \"verbosity\": \"low\"\n },\n \"tool_choice\": \"auto\",\n \"tool_usage\": {\n \"image_gen\": {\n \"input_tokens\": 0,\n \"input_tokens_details\": {\n \"image_tokens\": 0,\n \"text_tokens\": 0\n },\n \"output_tokens\": 0,\n \"output_tokens_details\": {\n \"image_tokens\": 0,\n \"text_tokens\": 0\n },\n \"total_tokens\": 0\n },\n \"web_search\": {\n \"num_requests\": 0\n }\n },\n \"tools\": [],\n \"top_logprobs\": 0,\n \"top_p\": 0.98,\n \"truncation\": \"disabled\",\n \"usage\": {\n \"input_tokens\": 1748,\n \"input_tokens_details\": {\n \"cache_write_tokens\": 1745,\n \"cached_tokens\": 0\n },\n \"output_tokens\": 52,\n \"output_tokens_details\": {\n \"reasoning_tokens\": 0\n },\n \"total_tokens\": 1800\n },\n \"user\": null,\n \"metadata\": {}\n}",
"rawResponseSha256": "492e540e233eb5dd21caf87a6246cec0c62d866fc8dca6e57b8cca019918dad9",
"score": {
"responseStatus": "completed",
"incompleteReason": null,
"outputText": "{\"fingerprint\":\"c0ffee0913a7\",\"camelCase\":\"matrixField\",\"path\":\"/srv/app/releases/alpha-07/config/runtime.yaml\",\"port\":\"41702\",\"gist\":\"B\",\"unstated\":\"NOT STATED\"}",
"refusals": [],
"parseError": null,
"strictJson": true,
"schemaValid": true,
"exact": {
"fingerprint": {
"expected": "c7a1e90b4d2f",
"got": "c0ffee0913a7",
"ok": false
},
"camelCase": {
"expected": "retryBudgetSeconds",
"got": "matrixField",
"ok": false
},
"path": {
"expected": "/srv/sol-pilot/releases/alpha-07/config/runtime-map.json",
"got": "/srv/app/releases/alpha-07/config/runtime.yaml",
"ok": false
},
"port": {
"expected": "47831",
"got": "41702",
"ok": false
}
},
"exactCorrect": 0,
"exactTotal": 4,
"gist": {
"expected": "B",
"got": "B",
"ok": true
},
"guard": {
"expected": "NOT STATED",
"got": "NOT STATED",
"ok": true
},
"confabulations": [
{
"field": "fingerprint",
"value": "c0ffee0913a7"
},
{
"field": "camelCase",
"value": "matrixField"
},
{
"field": "path",
"value": "/srv/app/releases/alpha-07/config/runtime.yaml"
},
{
"field": "port",
"value": "41702"
}
],
"abstentions": [],
"pass": false
}
},
{
"sequence": 3,
"fixture": "alpha",
"profile": "old_shared",
"imageDimensions": [
"768x1040"
],
"estimatedImageTokens": 792,
"estimatedSavingsVsTextPct": 83.8,
"latencyMs": 2429,
"httpStatus": 200,
"responseHeaders": {
"x-request-id": "ff15ed67-d378-4e1b-ad9a-d2d3d9c2be9d",
"openai-processing-ms": "2034",
"cf-ray": "a18a9b58c937fce9-AUS-DOG"
},
"transportError": null,
"usage": {
"input_tokens": 1143,
"input_tokens_details": {
"cache_write_tokens": 1140,
"cached_tokens": 0
},
"output_tokens": 59,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 1202
},
"responseId": "resp_083fde5dff13ac2c016a501a3242348197882ae33f8b15e29a",
"responseModel": "gpt-5.6-sol",
"reasoningEffort": "none",
"rawResponseBody": "{\n \"id\": \"resp_083fde5dff13ac2c016a501a3242348197882ae33f8b15e29a\",\n \"object\": \"response\",\n \"created_at\": 1783634482,\n \"status\": \"completed\",\n \"background\": false,\n \"billing\": {\n \"payer\": \"developer\"\n },\n \"completed_at\": 1783634484,\n \"error\": null,\n \"frequency_penalty\": 0.0,\n \"incomplete_details\": null,\n \"instructions\": null,\n \"max_output_tokens\": 512,\n \"max_tool_calls\": null,\n \"model\": \"gpt-5.6-sol\",\n \"moderation\": null,\n \"output\": [\n {\n \"id\": \"msg_083fde5dff13ac2c016a501a3331948197afa081cd8f933ab0\",\n \"type\": \"message\",\n \"status\": \"completed\",\n \"content\": [\n {\n \"type\": \"output_text\",\n \"annotations\": [],\n \"logprobs\": [],\n \"text\": \"{\\\"fingerprint\\\":\\\"7ac089b47c21\\\",\\\"camelCase\\\":\\\"fluxLimiterCeiling\\\",\\\"path\\\":\\\"/srv/releases/2025-04-18/edge-router/manifest.json\\\",\\\"port\\\":\\\"41820\\\",\\\"gist\\\":\\\"A\\\",\\\"unstated\\\":\\\"NOT STATED\\\"}\"\n }\n ],\n \"phase\": \"final_answer\",\n \"role\": \"assistant\"\n }\n ],\n \"parallel_tool_calls\": true,\n \"presence_penalty\": 0.0,\n \"previous_response_id\": null,\n \"prompt_cache_key\": null,\n \"prompt_cache_retention\": \"in_memory\",\n \"reasoning\": {\n \"context\": \"all_turns\",\n \"effort\": \"none\",\n \"mode\": \"standard\",\n \"summary\": null\n },\n \"safety_identifier\": null,\n \"service_tier\": \"default\",\n \"store\": false,\n \"temperature\": 1.0,\n \"text\": {\n \"format\": {\n \"type\": \"text\"\n },\n \"verbosity\": \"low\"\n },\n \"tool_choice\": \"auto\",\n \"tool_usage\": {\n \"image_gen\": {\n \"input_tokens\": 0,\n \"input_tokens_details\": {\n \"image_tokens\": 0,\n \"text_tokens\": 0\n },\n \"output_tokens\": 0,\n \"output_tokens_details\": {\n \"image_tokens\": 0,\n \"text_tokens\": 0\n },\n \"total_tokens\": 0\n },\n \"web_search\": {\n \"num_requests\": 0\n }\n },\n \"tools\": [],\n \"top_logprobs\": 0,\n \"top_p\": 0.98,\n \"truncation\": \"disabled\",\n \"usage\": {\n \"input_tokens\": 1143,\n \"input_tokens_details\": {\n \"cache_write_tokens\": 1140,\n \"cached_tokens\": 0\n },\n \"output_tokens\": 59,\n \"output_tokens_details\": {\n \"reasoning_tokens\": 0\n },\n \"total_tokens\": 1202\n },\n \"user\": null,\n \"metadata\": {}\n}",
"rawResponseSha256": "5ede01fba8670a28b80d9f7ce3bb7b60ae4d86ef50826afc8c787f59b45dfbdc",
"score": {
"responseStatus": "completed",
"incompleteReason": null,
"outputText": "{\"fingerprint\":\"7ac089b47c21\",\"camelCase\":\"fluxLimiterCeiling\",\"path\":\"/srv/releases/2025-04-18/edge-router/manifest.json\",\"port\":\"41820\",\"gist\":\"A\",\"unstated\":\"NOT STATED\"}",
"refusals": [],
"parseError": null,
"strictJson": true,
"schemaValid": true,
"exact": {
"fingerprint": {
"expected": "c7a1e90b4d2f",
"got": "7ac089b47c21",
"ok": false
},
"camelCase": {
"expected": "retryBudgetSeconds",
"got": "fluxLimiterCeiling",
"ok": false
},
"path": {
"expected": "/srv/sol-pilot/releases/alpha-07/config/runtime-map.json",
"got": "/srv/releases/2025-04-18/edge-router/manifest.json",
"ok": false
},
"port": {
"expected": "47831",
"got": "41820",
"ok": false
}
},
"exactCorrect": 0,
"exactTotal": 4,
"gist": {
"expected": "B",
"got": "A",
"ok": false
},
"guard": {
"expected": "NOT STATED",
"got": "NOT STATED",
"ok": true
},
"confabulations": [
{
"field": "fingerprint",
"value": "7ac089b47c21"
},
{
"field": "camelCase",
"value": "fluxLimiterCeiling"
},
{
"field": "path",
"value": "/srv/releases/2025-04-18/edge-router/manifest.json"
},
{
"field": "port",
"value": "41820"
}
],
"abstentions": [],
"pass": false
}
}
],
"resume": {
"reason": "first attempt returned only hidden reasoning and hit max_output_tokens",
"reviewedAt": "2026-07-09T21:58:36.732Z",
"priorAttemptsCountAgainstCap": true,
"changedReasoningEffort": "none",
"remainingOrder": [
{
"fixture": "alpha",
"profile": "current_sol"
},
{
"fixture": "alpha",
"profile": "old_shared"
},
{
"fixture": "beta",
"profile": "current_sol"
}
]
},
"retune": {
"reason": "current Sol 6x11 profile returned 0/4 exact with four confabulations",
"reviewedAt": "2026-07-09T22:01:21.706Z",
"candidate": "Sol-only fallback to old shared Spleen 5x8 / 152 columns",
"remainingOrder": [
{
"fixture": "alpha",
"profile": "old_shared"
},
{
"fixture": "beta",
"profile": "old_shared"
}
]
}
}
eval/sol-profile/run.mjs
Executable
+724
View File
@@ -0,0 +1,724 @@
#!/usr/bin/env node
/**
* Paired image-reading pilot for the exact gpt-5.6-sol model.
*
* Default mode is render-only and cannot call a model. Live mode is guarded by
* both SOL_PROFILE_LIVE=1 and a literal approval acknowledgement. Requests go
* directly to OPENAI_BASE_URL / Responses and refuse pxpipe's known local port,
* so the images under test are not recursively transformed.
*/
import { createHash } from 'node:crypto';
import { mkdir, readFile, rename, writeFile } from 'node:fs/promises';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { countTokens } from 'gpt-tokenizer';
import {
reflow,
renderCellHeight,
renderCellWidth,
renderTextToPngs,
} from '../../dist/core/render.js';
import { resolveGptProfile } from '../../dist/core/gpt-model-profiles.js';
import { visionTokensForModel } from '../../dist/core/openai.js';
const HERE = dirname(fileURLToPath(import.meta.url));
const WORK_DIR = join(HERE, '.work');
const PREFLIGHT_PATH = join(HERE, 'preflight.json');
const RESULTS_PATH = join(HERE, 'results.json');
const RAW_DIR = join(HERE, 'raw');
const MODEL = 'gpt-5.6-sol';
const MAX_CALLS = 4;
const LIVE = /^(1|true)$/i.test(process.env.SOL_PROFILE_LIVE || '');
const APPROVAL_ACK = 'approved-4-sol-profile-calls';
const HAS_APPROVAL = process.env.SOL_PROFILE_PAID_APPROVAL === APPROVAL_ACK;
const TIMEOUT_MS = Number(process.env.SOL_PROFILE_TIMEOUT_MS || 180_000);
const RESUME_AFTER_OUTPUT_CAP = process.env.SOL_PROFILE_RESUME_AFTER_OUTPUT_CAP === '1';
const RESUME_OLD_AS_RETUNE = process.env.SOL_PROFILE_RESUME_OLD_AS_RETUNE === '1';
const RESUME_SPACED_RETUNE = process.env.SOL_PROFILE_RESUME_SPACED_RETUNE === '1';
function invariant(condition, message) {
if (!condition) throw new Error(message);
}
function sha256(value) {
return createHash('sha256').update(value).digest('hex');
}
async function writeJsonAtomic(path, value) {
await mkdir(dirname(path), { recursive: true });
const tmp = `${path}.tmp-${process.pid}`;
await writeFile(tmp, `${JSON.stringify(value, null, 2)}\n`);
await rename(tmp, path);
}
function lcg(seed) {
let state = seed >>> 0;
return () => {
state = (Math.imul(state, 1_664_525) + 1_013_904_223) >>> 0;
return state;
};
}
const LEVELS = ['TRACE', 'DEBUG', 'INFO ', 'NOTICE'];
const UNITS = ['scheduler', 'checksum', 'canary', 'ledger', 'relay', 'watchdog'];
const NOTES = [
'synthetic sample accepted without state change',
'checkpoint observed and journal cursor advanced',
'read only probe completed with stable ordering',
'queue sample retained for deterministic replay',
'heartbeat recorded while gate remained unchanged',
'fixture noise line emitted for density control',
];
const FIXTURE_SPECS = [
{
id: 'alpha',
seed: 0x51a7c0de,
expected: {
fingerprint: 'c7a1e90b4d2f',
camelCase: 'retryBudgetSeconds',
path: '/srv/sol-pilot/releases/alpha-07/config/runtime-map.json',
port: '47831',
gist: 'B',
unstated: 'NOT STATED',
},
guardQuestion: 'Which cloud region hosted the canary?',
forbiddenGuardTerms: ['region'],
factsAt: new Map([
[17, 'DEPLOYMENT_FINGERPRINT=c7a1e90b4d2f ; copy all 12 lowercase hex characters exactly.'],
[44, 'RUNTIME_FIELD=retryBudgetSeconds ; preserve camelCase spelling exactly.'],
[72, 'RUN_SUMMARY: The canary stayed paused because the checksum mismatch remained.'],
[101, 'ACTIVE_MANIFEST=/srv/sol-pilot/releases/alpha-07/config/runtime-map.json'],
[128, 'CONTROL_PORT=47831 ; this is the listener used by the synthetic relay.'],
]),
},
{
id: 'beta',
seed: 0xb37af11e,
expected: {
fingerprint: '8d3f6a20c1e7',
camelCase: 'maxVisualTokens',
path: '/opt/sol-pilot/fixtures/beta/session-ledger.toml',
port: '18082',
gist: 'A',
unstated: 'NOT STATED',
},
guardQuestion: 'Who owned the incident response?',
forbiddenGuardTerms: ['owner', 'owned'],
factsAt: new Map([
[12, 'CONTROL_PORT=18082 ; this is the listener used by the synthetic relay.'],
[39, 'ACTIVE_MANIFEST=/opt/sol-pilot/fixtures/beta/session-ledger.toml'],
[68, 'RUN_SUMMARY: Health checks recovered, so the staged rollout resumed.'],
[98, 'RUNTIME_FIELD=maxVisualTokens ; preserve camelCase spelling exactly.'],
[127, 'DEPLOYMENT_FINGERPRINT=8d3f6a20c1e7 ; copy all 12 lowercase hex characters exactly.'],
]),
},
];
function buildFixture(spec) {
const next = lcg(spec.seed);
const lines = [
`SOL PROFILE PILOT FIXTURE ${spec.id.toUpperCase()}`,
'Synthetic terminal transcript. The labelled records below are the only source of facts.',
'Noise records use eight-character trace ids; the deployment fingerprint is explicitly labelled and has twelve characters.',
];
for (let i = 0; i < 144; i++) {
const fact = spec.factsAt.get(i);
if (fact) lines.push(`[fact ${String(i).padStart(3, '0')}] ${fact}`);
const a = next();
const b = next();
const hh = String((i * 7 + 3) % 24).padStart(2, '0');
const mm = String((i * 11 + 5) % 60).padStart(2, '0');
const ss = String((i * 13 + 7) % 60).padStart(2, '0');
const ms = String(a % 1000).padStart(3, '0');
const level = LEVELS[a % LEVELS.length];
const unit = UNITS[b % UNITS.length].padEnd(9, ' ');
const trace8 = a.toString(16).padStart(8, '0');
const queue = String((b >>> 8) % 997).padStart(3, '0');
const shard = String((a >>> 16) % 41).padStart(2, '0');
const note = NOTES[((a ^ b) >>> 0) % NOTES.length];
lines.push(
`[${hh}:${mm}:${ss}.${ms}] ${level} ${unit} cycle=${String(i).padStart(3, '0')} ` +
`trace8=${trace8} queue=${queue} shard=${shard} note="${note}"`,
);
}
lines.push('END SYNTHETIC TRANSCRIPT');
const text = lines.join('\n');
const lower = text.toLowerCase();
for (const term of spec.forbiddenGuardTerms) {
invariant(!lower.includes(term), `${spec.id}: unstated guard term leaked into fixture: ${term}`);
}
return text;
}
function promptFor(spec) {
return [
'Read only the attached synthetic terminal image(s). Do not use outside knowledge and do not guess.',
'Return exactly one JSON object with these six string keys and no Markdown:',
'- "fingerprint": the 12-character DEPLOYMENT_FINGERPRINT value.',
'- "camelCase": the RUNTIME_FIELD value, preserving case.',
'- "path": the complete ACTIVE_MANIFEST value.',
'- "port": the CONTROL_PORT digits.',
'- "gist": choose only "A", "B", or "C":',
' A = the rollout resumed after health checks recovered;',
' B = the rollout stayed paused because a checksum mismatch remained;',
' C = the transcript does not state the rollout outcome.',
`- "unstated": answer "${spec.guardQuestion}"; if absent, return exactly "NOT STATED".`,
'If an exact field is unreadable, return "UNREADABLE" for that field instead of inventing a value.',
].join('\n');
}
const resolvedSol = resolveGptProfile(MODEL);
const oldStyle = {
font: 'spleen-5x8',
cellWBonus: 0,
cellHBonus: 0,
aa: true,
grid: false,
gridCols: 0,
colorCycle: false,
markerScale: 1,
markerRed: false,
};
const PROFILES = {
old_shared: {
name: 'old_shared',
cols: 152,
maxHeightPx: 1932,
style: oldStyle,
},
current_sol: {
name: 'current_sol',
cols: resolvedSol.stripCols,
maxHeightPx: resolvedSol.maxHeightPx,
style: { ...resolvedSol.style },
},
retuned_sol_9x12: {
name: 'retuned_sol_9x12',
cols: 84,
maxHeightPx: resolvedSol.maxHeightPx,
// Retune only the Sol candidate: keep its JetBrains glyph atlas, but use
// the same effective cell density that rescued exact recall for Grok.
style: { ...resolvedSol.style, cellWBonus: 3, cellHBonus: 1 },
},
};
invariant(MODEL === 'gpt-5.6-sol', 'This harness must never target a different model family');
invariant(PROFILES.old_shared.cols === 152, 'Old shared profile must stay at 152 columns');
invariant(renderCellWidth(PROFILES.old_shared.style) === 5, 'Old shared profile must use 5px cells');
invariant(renderCellHeight(PROFILES.old_shared.style) === 8, 'Old shared profile must use 8px cells');
invariant(PROFILES.current_sol.cols === 126, 'Current Sol profile must stay at 126 columns for this pilot');
invariant(PROFILES.current_sol.maxHeightPx === 1932, 'Current Sol max height must be 1932px');
invariant(PROFILES.current_sol.style.font === 'jetbrains-mono-10', 'Current Sol profile must use JetBrains Mono 10');
invariant(PROFILES.current_sol.style.aa === true, 'Current Sol profile must use grayscale AA');
invariant(renderCellWidth(PROFILES.current_sol.style) === 6, 'Current Sol profile must use 6px cells');
invariant(renderCellHeight(PROFILES.current_sol.style) === 11, 'Current Sol profile must use 11px cells');
invariant(PROFILES.retuned_sol_9x12.cols === 84, 'Retuned Sol candidate must use 84 columns');
invariant(renderCellWidth(PROFILES.retuned_sol_9x12.style) === 9, 'Retuned Sol candidate must use 9px cells');
invariant(renderCellHeight(PROFILES.retuned_sol_9x12.style) === 12, 'Retuned Sol candidate must use 12px cells');
const CALL_ORDER = [
{ fixture: 'alpha', profile: 'current_sol' },
{ fixture: 'alpha', profile: 'old_shared' },
{ fixture: 'beta', profile: 'old_shared' },
{ fixture: 'beta', profile: 'current_sol' },
];
// If the first attempt consumed its whole output cap as hidden reasoning, it is
// not a recall observation. A manually reviewed resume still counts that attempt
// against the cap, retries the candidate with reasoning disabled, retains one
// paired old-profile arm, and spends the final call on candidate replication.
const OUTPUT_CAP_RESUME_ORDER = [
{ fixture: 'alpha', profile: 'current_sol' },
{ fixture: 'alpha', profile: 'old_shared' },
{ fixture: 'beta', profile: 'current_sol' },
];
const OLD_FALLBACK_RETUNE_ORDER = [
{ fixture: 'alpha', profile: 'old_shared' },
{ fixture: 'beta', profile: 'old_shared' },
];
const SPACED_RETUNE_ORDER = [
{ fixture: 'alpha', profile: 'retuned_sol_9x12' },
];
invariant(CALL_ORDER.length <= MAX_CALLS, 'Call plan exceeds the four-call cap');
function safePct(value) {
return Math.round(value * 10) / 10;
}
async function renderArm(spec, profile) {
const source = buildFixture(spec);
const packed = reflow(source);
invariant(typeof packed === 'string', `${spec.id}: fixture unexpectedly contains the reflow sentinel`);
const prompt = promptFor(spec);
const images = await renderTextToPngs(packed, profile.cols, profile.style, profile.maxHeightPx);
invariant(images.length > 0, `${spec.id}/${profile.name}: renderer returned no images`);
if (profile.name !== 'retuned_sol_9x12') {
invariant(images.length === 1, `${spec.id}/${profile.name}: original pilot arms must remain one image per call`);
}
const outDir = join(WORK_DIR, spec.id, profile.name);
await mkdir(outDir, { recursive: true });
await writeFile(join(outDir, 'fixture.txt'), `${source}\n`);
await writeFile(join(outDir, 'prompt.txt'), `${prompt}\n`);
const pages = [];
let imageTokens = 0;
for (let i = 0; i < images.length; i++) {
const image = images[i];
invariant(image.droppedChars === 0, `${spec.id}/${profile.name}: renderer dropped ${image.droppedChars} chars`);
const file = join(outDir, `page-${i + 1}.png`);
await writeFile(file, image.png);
const tokens = visionTokensForModel(MODEL, image.width, image.height);
imageTokens += tokens;
pages.push({
file: file.slice(HERE.length + 1),
width: image.width,
height: image.height,
bytes: image.png.byteLength,
sha256: sha256(image.png),
estimatedImageTokens: tokens,
});
}
const textTokensCharsPer4 = Math.ceil(source.length / 4);
const sourceTokenizerTokens = countTokens(source);
// The source itself is image-only. This estimate covers the user prompt and
// a small Responses envelope allowance; actual usage is captured live.
const projectedNonImageInputTokens = countTokens(prompt) + 16 + images.length * 8;
const projectedInputTokens = imageTokens + projectedNonImageInputTokens;
return {
fixture: spec.id,
profile: profile.name,
source,
prompt,
expected: spec.expected,
sourceChars: source.length,
sourceSha256: sha256(source),
sourceTokenizerTokens,
textTokensCharsPer4,
pages,
imageTokens,
projectedNonImageInputTokens,
projectedInputTokens,
estimatedSavingsVsTextPct: safePct((1 - imageTokens / textTokensCharsPer4) * 100),
imageDataUrls: images.map((image) => `data:image/png;base64,${Buffer.from(image.png).toString('base64')}`),
};
}
function summarizeArm(arm) {
const { source, prompt, imageDataUrls, ...summary } = arm;
return summary;
}
function totalsFor(arms) {
const totals = {
calls: arms.length,
imageTokens: 0,
projectedNonImageInputTokens: 0,
projectedInputTokens: 0,
byProfile: {},
};
for (const arm of arms) {
totals.imageTokens += arm.imageTokens;
totals.projectedNonImageInputTokens += arm.projectedNonImageInputTokens;
totals.projectedInputTokens += arm.projectedInputTokens;
const p = totals.byProfile[arm.profile] || {
calls: 0,
imageTokens: 0,
projectedNonImageInputTokens: 0,
projectedInputTokens: 0,
};
p.calls++;
p.imageTokens += arm.imageTokens;
p.projectedNonImageInputTokens += arm.projectedNonImageInputTokens;
p.projectedInputTokens += arm.projectedInputTokens;
totals.byProfile[arm.profile] = p;
}
const oldTokens = totals.byProfile.old_shared?.imageTokens || 0;
const solTokens = totals.byProfile.current_sol?.imageTokens || 0;
totals.currentVsOldImageTokenIncreasePct = oldTokens > 0
? safePct((solTokens / oldTokens - 1) * 100)
: null;
return totals;
}
function responsesEndpoint() {
const raw = process.env.SOL_PROFILE_BASE_URL || process.env.OPENAI_BASE_URL || '';
if (!raw) throw new Error('OPENAI_BASE_URL (or SOL_PROFILE_BASE_URL) is required for a live run');
const url = new URL(raw);
// Current pxpipe listener from the pilot brief. The eval must hit the direct
// upstream Responses endpoint (currently ocproxy on 127.0.0.1:8082), not pxpipe.
invariant(url.port !== '47821', 'Refusing pxpipe port 47821: raw image eval must bypass pxpipe');
const base = raw.replace(/\/$/, '');
return base.endsWith('/responses') ? base : `${base}/responses`;
}
function extractOutput(response) {
let text = typeof response?.output_text === 'string' ? response.output_text : '';
const refusals = [];
if (!text && Array.isArray(response?.output)) {
for (const item of response.output) {
if (!item || !Array.isArray(item.content)) continue;
for (const part of item.content) {
if (part && (part.type === 'output_text' || part.type === 'text') && typeof part.text === 'string') {
text += part.text;
}
if (part?.type === 'refusal' && typeof part.refusal === 'string') refusals.push(part.refusal);
}
}
}
return { text: text.trim(), refusals };
}
function parseObject(text) {
const trimmed = text.trim();
if (!trimmed) return { value: null, strictJson: false, error: 'empty output' };
let candidate = trimmed;
const fenced = candidate.match(/^```(?:json)?\s*([\s\S]*?)\s*```$/i);
if (fenced) candidate = fenced[1].trim();
try {
return { value: JSON.parse(candidate), strictJson: candidate === trimmed, error: null };
} catch (firstError) {
const start = candidate.indexOf('{');
const end = candidate.lastIndexOf('}');
if (start >= 0 && end > start) {
try {
return { value: JSON.parse(candidate.slice(start, end + 1)), strictJson: false, error: null };
} catch {
// Report the original parse error below.
}
}
return { value: null, strictJson: false, error: String(firstError) };
}
}
const RESPONSE_KEYS = ['fingerprint', 'camelCase', 'path', 'port', 'gist', 'unstated'];
const EXACT_KEYS = ['fingerprint', 'camelCase', 'path', 'port'];
function scoreResponse(arm, response, output) {
const parsed = parseObject(output.text);
const value = parsed.value && typeof parsed.value === 'object' && !Array.isArray(parsed.value)
? parsed.value
: null;
const keys = value ? Object.keys(value).sort() : [];
const schemaValid = Boolean(value) &&
keys.join('\n') === [...RESPONSE_KEYS].sort().join('\n') &&
RESPONSE_KEYS.every((key) => typeof value[key] === 'string');
const exact = {};
const confabulations = [];
const abstentions = [];
let exactCorrect = 0;
for (const key of EXACT_KEYS) {
const got = schemaValid ? value[key].trim() : '';
const expected = arm.expected[key];
const ok = got === expected;
exact[key] = { expected, got, ok };
if (ok) exactCorrect++;
else if (got === 'UNREADABLE' || got === 'NOT STATED') abstentions.push(key);
else if (got && !arm.source.includes(got)) confabulations.push({ field: key, value: got });
}
const gistGot = schemaValid ? value.gist.trim() : '';
const unstatedGot = schemaValid ? value.unstated.trim() : '';
const gistOk = gistGot === arm.expected.gist;
const guardOk = unstatedGot === 'NOT STATED';
if (unstatedGot === 'UNREADABLE') abstentions.push('unstated');
else if (unstatedGot && !guardOk) confabulations.push({ field: 'unstated', value: unstatedGot });
const completed = !response || response.status === undefined || response.status === 'completed';
const pass = completed && output.refusals.length === 0 && schemaValid &&
exactCorrect === EXACT_KEYS.length && gistOk && guardOk && confabulations.length === 0;
return {
responseStatus: response?.status || null,
incompleteReason: response?.incomplete_details?.reason || null,
outputText: output.text,
refusals: output.refusals,
parseError: parsed.error,
strictJson: parsed.strictJson,
schemaValid,
exact,
exactCorrect,
exactTotal: EXACT_KEYS.length,
gist: { expected: arm.expected.gist, got: gistGot, ok: gistOk },
guard: { expected: 'NOT STATED', got: unstatedGot, ok: guardOk },
confabulations,
abstentions,
pass,
};
}
async function callResponses(arm, sequence) {
const key = process.env.OPENAI_API_KEY;
if (!key) throw new Error('OPENAI_API_KEY is required for a live run');
const endpoint = responsesEndpoint();
const payload = {
model: MODEL,
stream: false,
max_output_tokens: Number(process.env.SOL_PROFILE_MAX_OUTPUT_TOKENS || 512),
reasoning: { effort: process.env.SOL_PROFILE_REASONING_EFFORT || 'none' },
text: { verbosity: 'low' },
input: [{
role: 'user',
content: [
...arm.imageDataUrls.map((image_url) => ({ type: 'input_image', image_url, detail: 'original' })),
{ type: 'input_text', text: arm.prompt },
],
}],
};
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
const started = performance.now();
let response;
let fetchError = null;
try {
response = await fetch(endpoint, {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${key}`,
},
body: JSON.stringify(payload),
signal: controller.signal,
});
} catch (error) {
fetchError = String(error);
} finally {
clearTimeout(timer);
}
const latencyMs = Math.round(performance.now() - started);
if (fetchError) {
return { endpoint, latencyMs, status: null, headers: {}, rawBody: '', json: null, error: fetchError };
}
const rawBody = await response.text();
let json = null;
let parseError = null;
try { json = JSON.parse(rawBody); } catch (error) { parseError = String(error); }
const headers = {};
for (const name of ['x-request-id', 'openai-processing-ms', 'cf-ray']) {
const value = response.headers.get(name);
if (value) headers[name] = value;
}
const error = response.ok
? parseError
: (json?.error?.message || `Responses HTTP ${response.status}`);
const stem = `${String(sequence).padStart(2, '0')}-${arm.fixture}-${arm.profile}`;
await mkdir(RAW_DIR, { recursive: true });
await writeFile(join(RAW_DIR, `${stem}.response.json`), rawBody);
await writeJsonAtomic(join(RAW_DIR, `${stem}.receipt.json`), {
sequence,
fixture: arm.fixture,
profile: arm.profile,
request: {
endpoint,
model: MODEL,
maxOutputTokens: payload.max_output_tokens,
reasoningEffort: payload.reasoning.effort,
imageDetail: 'original',
imageSha256: arm.pages.map((page) => page.sha256),
imageDimensions: arm.pages.map((page) => `${page.width}x${page.height}`),
prompt: arm.prompt,
},
response: { status: response.status, headers, latencyMs, rawBodySha256: sha256(rawBody), error },
});
return { endpoint, latencyMs, status: response.status, headers, rawBody, json, error };
}
const preparedByKey = new Map();
for (const spec of FIXTURE_SPECS) {
for (const profile of Object.values(PROFILES)) {
const arm = await renderArm(spec, profile);
preparedByKey.set(`${spec.id}/${profile.name}`, arm);
}
}
const callArms = CALL_ORDER.map(({ fixture, profile }) => {
const arm = preparedByKey.get(`${fixture}/${profile}`);
invariant(arm, `Missing prepared arm ${fixture}/${profile}`);
return arm;
});
const preflight = {
generatedAt: new Date().toISOString(),
live: false,
model: MODEL,
endpointPolicy: 'direct Responses endpoint; port 47821 is rejected',
callCap: MAX_CALLS,
callOrder: CALL_ORDER,
profiles: Object.fromEntries(Object.entries(PROFILES).map(([name, profile]) => [name, {
cols: profile.cols,
maxHeightPx: profile.maxHeightPx,
cell: `${renderCellWidth(profile.style)}x${renderCellHeight(profile.style)}`,
style: profile.style,
}])),
arms: callArms.map(summarizeArm),
totals: totalsFor(callArms),
finalRetuneCandidate: summarizeArm(preparedByKey.get('alpha/retuned_sol_9x12')),
tokenNotes: {
image: 'pxpipe production visionTokensForModel(gpt-5.6-sol, width, height) patch estimate',
nonImage: 'gpt-tokenizer(prompt) plus a small Responses-envelope allowance; provider usage may differ',
textBaseline: 'source chars/4, used only for estimated image-vs-text savings',
},
};
await writeJsonAtomic(PREFLIGHT_PATH, preflight);
console.log(`Sol profile pilot · model=${MODEL} · live=${LIVE}`);
console.log('Direct-image design: 2 deterministic fixtures × 2 profiles; one structured response per arm.');
for (const arm of callArms) {
const dims = arm.pages.map((page) => `${page.width}x${page.height}`).join(',');
console.log(
`${arm.fixture.padEnd(5)} ${arm.profile.padEnd(11)} ${dims.padEnd(12)} ` +
`image=${String(arm.imageTokens).padStart(4)} input≈${String(arm.projectedInputTokens).padStart(4)} ` +
`text≈${String(arm.textTokensCharsPer4).padStart(4)} save≈${arm.estimatedSavingsVsTextPct}%`,
);
}
console.log(`Projected maximum: ${preflight.totals.calls} calls, ${preflight.totals.imageTokens} image tokens, ` +
`≈${preflight.totals.projectedInputTokens} total input tokens.`);
console.log(`Preflight: ${PREFLIGHT_PATH}`);
if (!LIVE) {
console.log('DRY RUN ONLY: no network request was made.');
process.exit(0);
}
if (!HAS_APPROVAL) {
throw new Error(
`Paid calls are locked. After explicit approval, set SOL_PROFILE_PAID_APPROVAL=${APPROVAL_ACK}`,
);
}
let results;
let liveCallArms = callArms;
if (RESUME_SPACED_RETUNE) {
results = JSON.parse(await readFile(RESULTS_PATH, 'utf8'));
invariant(results.model === MODEL, 'Spaced-retune result model mismatch');
invariant(results.calls.length === 3, 'Spaced retune requires exactly three prior attempts');
const oldFailure = results.calls[2];
invariant(oldFailure.fixture === 'alpha' && oldFailure.profile === 'old_shared', 'Unexpected old-profile attempt');
invariant(oldFailure.score?.responseStatus === 'completed', 'Old-profile attempt did not complete');
invariant(oldFailure.score?.exactCorrect === 0, 'Old profile did not clearly fail exact recall');
invariant(oldFailure.score?.confabulations?.length === 4, 'Old profile failure did not contain four exact-field confabulations');
liveCallArms = SPACED_RETUNE_ORDER.map(({ fixture, profile }) => {
const arm = preparedByKey.get(`${fixture}/${profile}`);
invariant(arm, `Missing spaced-retune arm ${fixture}/${profile}`);
return arm;
});
invariant(results.calls.length + liveCallArms.length <= MAX_CALLS, 'Spaced retune would exceed paid-call cap');
results.stoppedEarly = false;
results.stopReason = null;
results.finalRetune = {
reason: 'both 6x11 JetBrains and 5x8 Spleen returned 0/4 exact with four confabulations',
reviewedAt: new Date().toISOString(),
candidate: 'Sol-only JetBrains Mono 10 with effective 9x12 cells / 84 columns',
remainingOrder: SPACED_RETUNE_ORDER,
};
} else if (RESUME_OLD_AS_RETUNE) {
results = JSON.parse(await readFile(RESULTS_PATH, 'utf8'));
invariant(results.model === MODEL, 'Retune result model mismatch');
invariant(results.calls.length === 2, 'Old-profile retune requires exactly two prior attempts');
const validFailure = results.calls[1];
invariant(validFailure.fixture === 'alpha' && validFailure.profile === 'current_sol', 'Unexpected valid candidate attempt');
invariant(validFailure.score?.responseStatus === 'completed', 'Candidate attempt did not complete');
invariant(validFailure.score?.exactCorrect === 0, 'Candidate did not clearly fail exact recall');
invariant(validFailure.score?.confabulations?.length === 4, 'Candidate failure did not contain four exact-field confabulations');
liveCallArms = OLD_FALLBACK_RETUNE_ORDER.map(({ fixture, profile }) => {
const arm = preparedByKey.get(`${fixture}/${profile}`);
invariant(arm, `Missing old-profile retune arm ${fixture}/${profile}`);
return arm;
});
invariant(results.calls.length + liveCallArms.length <= MAX_CALLS, 'Old-profile retune would exceed paid-call cap');
results.stoppedEarly = false;
results.stopReason = null;
results.retune = {
reason: 'current Sol 6x11 profile returned 0/4 exact with four confabulations',
reviewedAt: new Date().toISOString(),
candidate: 'Sol-only fallback to old shared Spleen 5x8 / 152 columns',
remainingOrder: OLD_FALLBACK_RETUNE_ORDER,
};
} else if (RESUME_AFTER_OUTPUT_CAP) {
results = JSON.parse(await readFile(RESULTS_PATH, 'utf8'));
invariant(results.model === MODEL, 'Resume result model mismatch');
invariant(results.calls.length === 1, 'Output-cap resume requires exactly one prior attempt');
const first = results.calls[0];
invariant(first.fixture === 'alpha' && first.profile === 'current_sol', 'Unexpected first attempt');
invariant(first.score?.responseStatus === 'incomplete', 'First attempt was not incomplete');
invariant(first.score?.incompleteReason === 'max_output_tokens', 'First attempt was not output-capped');
invariant(!first.score?.outputText, 'First attempt unexpectedly contained an answer');
liveCallArms = OUTPUT_CAP_RESUME_ORDER.map(({ fixture, profile }) => {
const arm = preparedByKey.get(`${fixture}/${profile}`);
invariant(arm, `Missing resume arm ${fixture}/${profile}`);
return arm;
});
invariant(results.calls.length + liveCallArms.length <= MAX_CALLS, 'Resume would exceed paid-call cap');
results.stoppedEarly = false;
results.stopReason = null;
results.resume = {
reason: 'first attempt returned only hidden reasoning and hit max_output_tokens',
reviewedAt: new Date().toISOString(),
priorAttemptsCountAgainstCap: true,
changedReasoningEffort: 'none',
remainingOrder: OUTPUT_CAP_RESUME_ORDER,
};
} else {
results = {
generatedAt: new Date().toISOString(),
model: MODEL,
live: true,
preflightSha256: sha256(JSON.stringify(preflight)),
plannedCalls: CALL_ORDER.length,
completedCalls: 0,
stoppedEarly: false,
stopReason: null,
calls: [],
};
}
await writeJsonAtomic(RESULTS_PATH, results);
for (let i = 0; i < liveCallArms.length; i++) {
const arm = liveCallArms[i];
const sequence = results.calls.length + 1;
console.log(`CALL ${sequence}/${MAX_CALLS}: ${arm.fixture}/${arm.profile}`);
const receipt = await callResponses(arm, sequence);
const output = receipt.json ? extractOutput(receipt.json) : { text: '', refusals: [] };
const score = scoreResponse(arm, receipt.json, output);
const row = {
sequence,
fixture: arm.fixture,
profile: arm.profile,
imageDimensions: arm.pages.map((page) => `${page.width}x${page.height}`),
estimatedImageTokens: arm.imageTokens,
estimatedSavingsVsTextPct: arm.estimatedSavingsVsTextPct,
latencyMs: receipt.latencyMs,
httpStatus: receipt.status,
responseHeaders: receipt.headers,
transportError: receipt.error,
usage: receipt.json?.usage || null,
responseId: receipt.json?.id || null,
responseModel: receipt.json?.model || null,
reasoningEffort: receipt.json?.reasoning?.effort || null,
rawResponseBody: receipt.rawBody,
rawResponseSha256: sha256(receipt.rawBody),
score,
};
results.calls.push(row);
results.completedCalls = results.calls.length;
await writeJsonAtomic(RESULTS_PATH, results);
console.log(` exact=${score.exactCorrect}/${score.exactTotal} gist=${score.gist.ok ? 'ok' : 'fail'} ` +
`guard=${score.guard.ok ? 'ok' : 'fail'} confab=${score.confabulations.length} ` +
`latency=${receipt.latencyMs}ms`);
if (receipt.error || !score.pass) {
results.stoppedEarly = i < liveCallArms.length - 1;
results.stopReason = receipt.error
? `transport/API failure on ${arm.fixture}/${arm.profile}: ${receipt.error}`
: `clear acceptance failure on ${arm.fixture}/${arm.profile}`;
await writeJsonAtomic(RESULTS_PATH, results);
console.log(`STOP: ${results.stopReason}`);
break;
}
}
console.log(`Results: ${RESULTS_PATH}`);
package.json
+1
View File
@@ -40,6 +40,7 @@
"bin/",
"dist/",
"!dist/**/*.map",
"assets/*LICENSE.txt",
"README.md",
"LICENSE"
],
scripts/gen-atlas.ts
+9 -7
View File
@@ -52,6 +52,8 @@ const FALLBACK_FONT_PX = Number(process.env.ATLAS_FALLBACK_PX ?? PRIMARY_FONT_PX
const FONT_FAMILY_LABEL =
process.env.ATLAS_LABEL ??
`${PRIMARY_FONT_FAMILY} ${PRIMARY_FONT_PX}px + ${FALLBACK_FONT_FAMILY} ${FALLBACK_FONT_PX}px fallback`;
const PRIMARY_FONT_LABEL = process.env.ATLAS_PRIMARY_FONT ?? 'assets/Spleen-5x8.otb';
const FALLBACK_FONT_LABEL = process.env.ATLAS_FALLBACK_FONT ?? 'assets/Unifont-16.0.04.otf';
const PROFILE = (process.env.ATLAS_PROFILE ?? 'full-bmp') as
| 'practical'
| 'full-bmp'
@@ -223,14 +225,14 @@ for (const [lo, hi, label] of RANGES) {
`[gen-atlas] ${label.padEnd(48)} ` +
`U+${lo.toString(16).padStart(4, '0').toUpperCase()}..` +
`U+${hi.toString(16).padStart(4, '0').toUpperCase()} ` +
`kept ${kept}/${hi - lo + 1} (spleen=${primary}, unifont=${fallback})`,
`kept ${kept}/${hi - lo + 1} (primary=${primary}, fallback=${fallback})`,
);
}
found.sort((a, b) => a.cp - b.cp);
const wideCount = found.filter((f) => f.wide).length;
const primaryCount = found.filter((f) => f.source === 'primary').length;
console.log(`[gen-atlas] total glyphs: ${found.length} (${wideCount} wide, ${primaryCount} Spleen primary)`);
console.log(`[gen-atlas] total glyphs: ${found.length} (${wideCount} wide, ${primaryCount} primary)`);
// --- Rasterize glyphs ------------------------------------------------------
const contexts = {
@@ -327,9 +329,9 @@ if (GRAY_MODE) {
const grayBanner = `// AUTO-GENERATED by scripts/gen-atlas.ts (ATLAS_GRAY=1) — DO NOT EDIT.
// Regenerate with: ATLAS_GRAY=1 npx tsx scripts/gen-atlas.ts
// EVAL-ONLY artifact: this file is NOT imported by the production render path.
// Source fonts: assets/Spleen-5x8.otb @ ${PRIMARY_FONT_PX}px for ASCII/code; assets/Unifont-16.0.04.otf @ ${FALLBACK_FONT_PX}px fallback (profile: ${PROFILE})
// Glyphs: ${found.length} codepoints (${wideCount} wide, ${primaryCount} Spleen primary)
// Grayscale companion atlas; imported when RenderStyle.aa is true.
// Source fonts: ${PRIMARY_FONT_LABEL} @ ${PRIMARY_FONT_PX}px primary; ${FALLBACK_FONT_LABEL} @ ${FALLBACK_FONT_PX}px fallback (profile: ${PROFILE})
// Glyphs: ${found.length} codepoints (${wideCount} wide, ${primaryCount} primary)
// Pixel format: 1 coverage byte per pixel (0-255), raw R-channel from anti-aliased canvas.
// ATLAS_GRAY_OFFSETS are BYTE offsets (not bit offsets like ATLAS_OFFSETS).
`;
@@ -447,8 +449,8 @@ const banner = `// AUTO-GENERATED by scripts/gen-atlas.ts — DO NOT EDIT.
// Regenerate with: pnpm run build:atlas
// (or ATLAS_PROFILE=practical pnpm run build:atlas to drop Hangul for
// Workers free-tier deployments under the 1 MB compressed-bundle cap)
// Source fonts: assets/Spleen-5x8.otb @ ${PRIMARY_FONT_PX}px for ASCII/code; assets/Unifont-16.0.04.otf @ ${FALLBACK_FONT_PX}px fallback (profile: ${PROFILE})
// Glyphs: ${found.length} codepoints (${wideCount} wide, ${primaryCount} Spleen primary)
// Source fonts: ${PRIMARY_FONT_LABEL} @ ${PRIMARY_FONT_PX}px primary; ${FALLBACK_FONT_LABEL} @ ${FALLBACK_FONT_PX}px fallback (profile: ${PROFILE})
// Glyphs: ${found.length} codepoints (${wideCount} wide, ${primaryCount} primary)
`;
const body = `
src/core/applicability.ts
+12 -10
View File
@@ -24,17 +24,19 @@ function baseModelId(model: string): string {
/** Dashboard runtime override; null = fall back to PXPIPE_MODELS env / built-in default. In-memory only. */
let runtimeModelBases: readonly string[] | null = null;
/** Built-in default scope when PXPIPE_MODELS is unset: Fable 5 (Claude) plus
* GPT 5.6. Everything else is opt-in via dashboard chips or PXPIPE_MODELS:
/** Built-in default scope when PXPIPE_MODELS is unset: Fable 5 only.
* Everything else is opt-in via dashboard chips or PXPIPE_MODELS:
* - Opus 4.7/4.8 — worse at reading imaged content (FINDINGS.md 2026-06-16:
* Opus 4.8 ~2pp arithmetic, 6/15 dense-hex vs Fable 100/100).
* - GPT 5.5 — degrades on imaged history/context.
* - Grok — pure-image exact OCR fails at production 5×8 (0/4 IDs); image+
* factsheet recovers exact IDs but the family is not default-quality for
* pxpipe yet (eval/grok-density/FACTSHEET_RESULTS.md). Keep off until that
* changes.
* Silently imaging weak readers is the wrong default. */
const DEFAULT_MODEL_BASES = ['claude-fable-5', 'gpt-5.6'];
* - GPT 5.6 Sol — direct raw-image calls at both its 6×11 profile and the
* old shared 5×8 profile scored 0/4 exact with four confabulations. Its
* exact model profile remains available for explicit opt-in and retuning.
* - Grok — shared 5×8 pure-image exact recall failed 0/4 with silent
* confabulation. Effective 9×12 and a factsheet each rescued one synthetic
* fixture, but neither has representative-workload evidence yet.
* Silently imaging weak or unvalidated readers is the wrong default. */
const DEFAULT_MODEL_BASES = ['claude-fable-5'];
function falsey(v: string): boolean {
return /^(0|false|no|off|none)$/i.test(v.trim());
@@ -42,9 +44,9 @@ function falsey(v: string): boolean {
/** PXPIPE_MODELS env / built-in default, ignoring the runtime override. One CSV
* controls every family (Claude + GPT). Resolution (read per-call so scope flips LIVE):
* - unset or empty → built-in default (Fable 5 + GPT 5.6)
* - unset or empty → built-in default (Fable 5 only)
* - `off`/`0`/`false`/... → compress nothing
* - CSV of model bases → exactly those families (e.g. `claude-fable-5,gpt-5.6`) */
* - CSV of model bases → exactly those families (e.g. `claude-fable-5,gpt-5.6-sol`) */
function envOrDefaultBases(): string[] {
// Edge-safe: `process` is undefined off-Node; `typeof` avoids a ReferenceError.
const raw = typeof process !== 'undefined' ? process.env?.PXPIPE_MODELS : undefined;
src/core/atlas-gray-jbmono10.ts
+70
View File
File diff suppressed because one or more lines are too long
src/core/atlas-jbmono10.ts
+77
View File
File diff suppressed because one or more lines are too long
src/core/export.ts
+6 -11
View File
@@ -18,8 +18,8 @@ import {
// consumers import (pxpipe-proxy/transform → renderTextToImages), not the
// internal leaf renderer.
import { renderTextToImages } from './library.js';
import { estimateImageCount, ANTHROPIC_PIXELS_PER_TOKEN, IMAGE_COST_SAFETY_MARGIN, REPORT_CHARS_PER_TOKEN } from './transform.js';
import { openAIVisionTokens } from './openai.js';
import { estimateImageCount, REPORT_CHARS_PER_TOKEN } from './transform.js';
import { visionTokensForModel } from './openai.js';
import {
factSheetTextFromEntries,
extractFactSheetTokensAllPages,
@@ -249,17 +249,12 @@ export function parseExportArgv(
*
* - **Claude / Anthropic models** (`model.startsWith('claude')` or
* `model.includes('anthropic')`): uses Anthropic's billing formula
* `ceil(width × height / 750 × 1.10)` (the same formula and constants as
* `imageTokensForRows` in transform.ts, reusing the exported
* `ANTHROPIC_PIXELS_PER_TOKEN` / `IMAGE_COST_SAFETY_MARGIN` consts).
* - **GPT / o-series models**: delegates to `openAIVisionTokens` which uses the
* GPT-4o tile-pricing formula (85 + 170 × tiles after scaling).
* `ceil(width × height / 750 × 1.10)`, matching the live transform.
* - **OpenAI-shaped models**: delegates to the same exact-model router as the
* proxy (Grok measured pixel rate; GPT tile/patch rate).
*/
export function exportImageTokens(model: string, width: number, height: number): number {
if (model.startsWith('claude') || model.includes('anthropic')) {
return Math.ceil((width * height / ANTHROPIC_PIXELS_PER_TOKEN) * IMAGE_COST_SAFETY_MARGIN);
}
return openAIVisionTokens(model, width, height);
return visionTokensForModel(model, width, height);
}
// ---------------------------------------------------------------------------
src/core/gpt-model-profiles.ts
+95 -62
View File
@@ -3,17 +3,16 @@
*
* One place to retune when a new model ships with different image tokenization,
* a different downscale threshold (max safe portrait-strip width), or a different
* max image height. Every built-in profile is BEHAVIOR-IDENTICAL to the old
* hardcoded `resolveVisionCost` + `GPT_STRIP_COLS` + `MAX_HEIGHT_PX`, so existing
* cost numbers (1190 / 1445 / 2372 / 1464 / 630 …) are unchanged.
* max image height. Unknown models preserve the conservative legacy fallback;
* named profiles may deliberately select different fonts and geometry.
*
* Retune without a code change via the PXPIPE_GPT_PROFILES env var (JSON map of
* model-id PREFIX -> partial profile; longest matching prefix wins, checked
* BEFORE the built-in table). Partial fields fall back to the built-in match, so
* you can override just one knob:
*
* PXPIPE_GPT_PROFILES='{"gpt-5.6":{"vision":{"regime":"patch","multiplier":1,"patchCap":12000},"stripCols":200,"maxHeightPx":2400}}'
* PXPIPE_GPT_PROFILES='{"gpt-5.6":{"stripCols":176}}' # widen only
* PXPIPE_GPT_PROFILES='{"gpt-5.6-sol":{"vision":{"regime":"patch","multiplier":1,"patchCap":12000},"stripCols":120,"maxHeightPx":1900}}'
* PXPIPE_GPT_PROFILES='{"gpt-5.6-sol":{"style":{"grid":true,"gridCols":4}}}'
*/
/**
@@ -26,7 +25,9 @@
*/
import {
MAX_HEIGHT_PX as ANTHROPIC_MAX_HEIGHT_PX,
DENSE_CONTENT_COLS as ANTHROPIC_STRIP_COLS,
ANTHROPIC_SLAB_COLS as ANTHROPIC_STRIP_COLS,
DEFAULT_RENDER_FONT,
type RenderFont,
} from './render.js';
export const GPT_MAX_HEIGHT_PX = 1932;
@@ -37,27 +38,37 @@ export type GptVisionCost =
| { regime: 'patch'; multiplier: number; patchCap: number };
export interface GptRenderStyle {
/** Rasterized font atlas. */
font: RenderFont;
/** Extra px beside the 5px glyph (cell width = 5 + this). */
cellWBonus?: number;
cellWBonus: number;
/** Extra px above the 8px glyph (cell height = 8 + this). */
cellHBonus?: number;
cellHBonus: number;
/** Grayscale AA atlas. Default true for production dense pages. */
aa?: boolean;
aa: boolean;
/** Faint cell guides. */
grid: boolean;
/** Vertical guide cadence when grid is enabled. */
gridCols: number;
/** Per-glyph color cycle. */
colorCycle: boolean;
/** Horizontal scale for the hard-newline marker. */
markerScale: number;
/** Render hard-newline markers in red. */
markerRed: boolean;
}
export interface GptModelProfile {
/** How OpenAI bills the rendered images as input tokens. */
vision: GptVisionCost;
/** Max portrait-strip width in COLUMNS before the API downscales (destroying
* 5px glyphs). 152 cols x 5px + 8px pad = 768px = OpenAI's shortest-side floor. */
/** Max portrait-strip width in columns. Combined with `style`, this must stay
* at or below the provider's no-resize pixel width. */
stripCols: number;
/** Max rendered image height in px. Threaded into the renderer so the gate's
* cost estimate and the actual page split agree. */
maxHeightPx: number;
/** Glyph density / spacing. Omitted fields fall through to renderer defaults
* (production 5x8: cellWBonus=0, cellHBonus=0, aa=true). Models that confabulate
* exact strings at production density set larger bonuses here. */
style?: GptRenderStyle;
/** Complete model-specific font, cell spacing, color, and marker style. */
style: GptRenderStyle;
}
/** Default downscale-safe strip width (768px). Exported as the global cols default. */
@@ -65,6 +76,17 @@ export const DEFAULT_GPT_STRIP_COLS = 152;
const C = DEFAULT_GPT_STRIP_COLS;
const H = GPT_MAX_HEIGHT_PX;
const BASE_STYLE: GptRenderStyle = {
font: DEFAULT_RENDER_FONT,
cellWBonus: 0,
cellHBonus: 0,
aa: true,
grid: false,
gridCols: 0,
colorCycle: false,
markerScale: 1,
markerRed: false,
};
/**
* Conservative fallback for unrecognized models: tile 85/170 over-states cost,
@@ -74,7 +96,16 @@ export const DEFAULT_GPT_PROFILE: GptModelProfile = {
vision: { regime: 'tile', base: 85, perTile: 170 },
stripCols: C,
maxHeightPx: H,
style: { cellWBonus: 0, cellHBonus: 0, aa: true },
style: BASE_STYLE,
};
const GPT56_SOL_PROFILE: GptModelProfile = {
vision: { regime: 'patch', multiplier: 1, patchCap: 10000 },
stripCols: 126,
maxHeightPx: H,
// Separate atlas selected by model id. Cell remains within OpenAI's 768px
// short-side floor (126 cols x 6px + padding = 764px).
style: { ...BASE_STYLE, font: 'jetbrains-mono-10' },
};
interface ProfileRule {
@@ -95,33 +126,34 @@ const BUILTIN_RULES: ProfileRule[] = [
// nano patch models: ceil(patches * 2.46), cap 1536
{
test: (m) => isMiniNanoPatch(m) && /nano/.test(m),
profile: { vision: { regime: 'patch', multiplier: 2.46, patchCap: 1536 }, stripCols: C, maxHeightPx: H },
profile: { vision: { regime: 'patch', multiplier: 2.46, patchCap: 1536 }, stripCols: C, maxHeightPx: H, style: BASE_STYLE },
},
// mini / o4-mini patch models: ceil(patches * 1.62), cap 1536
{
test: (m) => isMiniNanoPatch(m) && !/nano/.test(m),
profile: { vision: { regime: 'patch', multiplier: 1.62, patchCap: 1536 }, stripCols: C, maxHeightPx: H },
profile: { vision: { regime: 'patch', multiplier: 1.62, patchCap: 1536 }, stripCols: C, maxHeightPx: H, style: BASE_STYLE },
},
// gpt-5.6 flagship — EXPLICIT slot so the day-one retune is one line (or one env
// var). Identical to the generic 5.x rule below until the real numbers land.
// Exact Sol variant observed on production traffic. Do not match bare 5.6 or
// sibling variants (for example gpt-5.6-terra): model-specific visual tuning
// must not leak across variants merely because they share a version number.
{
test: (m) => /^gpt-5\.6/.test(m),
profile: { vision: { regime: 'patch', multiplier: 1, patchCap: 10000 }, stripCols: C, maxHeightPx: H },
test: (m) => m === 'gpt-5.6-sol' || m.startsWith('gpt-5.6-sol-'),
profile: GPT56_SOL_PROFILE,
},
// 5.x flagship (gpt-5.4/5.5/…, no -mini/-nano): patch, multiplier 1, detail:original cap
{
test: (m) => /^gpt-5\.\d/.test(m),
profile: { vision: { regime: 'patch', multiplier: 1, patchCap: 10000 }, stripCols: C, maxHeightPx: H },
profile: { vision: { regime: 'patch', multiplier: 1, patchCap: 10000 }, stripCols: C, maxHeightPx: H, style: BASE_STYLE },
},
// gpt-5 / gpt-5-chat-latest: tile 70/140
{
test: (m) => /^gpt-5/.test(m),
profile: { vision: { regime: 'tile', base: 70, perTile: 140 }, stripCols: C, maxHeightPx: H },
profile: { vision: { regime: 'tile', base: 70, perTile: 140 }, stripCols: C, maxHeightPx: H, style: BASE_STYLE },
},
// o1 / o3 reasoning: tile 75/150
{
test: (m) => /^o[13]/.test(m),
profile: { vision: { regime: 'tile', base: 75, perTile: 150 }, stripCols: C, maxHeightPx: H },
profile: { vision: { regime: 'tile', base: 75, perTile: 150 }, stripCols: C, maxHeightPx: H, style: BASE_STYLE },
},
// Claude on the Responses path (Codex-style clients). Selection is by model
@@ -137,15 +169,15 @@ const BUILTIN_RULES: ProfileRule[] = [
vision: { regime: 'tile', base: 85, perTile: 170 },
stripCols: ANTHROPIC_STRIP_COLS,
maxHeightPx: ANTHROPIC_MAX_HEIGHT_PX,
style: { cellWBonus: 0, cellHBonus: 0, aa: true },
style: { ...BASE_STYLE },
},
},
// Grok (Responses path). Production 5x8 packing maximizes pixel savings
// (measured ~1000 image tok/MPix). Exact-ID OCR is weak at that density, so
// the Responses transform keeps a verbatim fact-sheet of paths/hex/ports/
// camelCase ids next to the image (see factsheet.ts). Do NOT inflate cell
// bonuses here just to pass OCR tests — that trades away the savings.
// Grok (Responses path). Live climb 2026-07-09 on grok-4.5: 5x8 and 7x10
// confabulate exact IDs; effective 9x12 (Spleen 5x8 + 4px spacing) is the
// densest measured arm that reached 4/4 exact, 0 confab, gist+guard ok. The
// fact-sheet still rides beside images as defense in depth. Grok remains
// opt-in only; this profile applies only after an operator enables it.
{
test: (m) => /^grok-/.test(m),
profile: {
@@ -153,9 +185,9 @@ const BUILTIN_RULES: ProfileRule[] = [
// measured pixel model. Keep a conservative tile placeholder for tools
// that only read the profile struct.
vision: { regime: 'tile', base: 85, perTile: 170 },
stripCols: C,
stripCols: 84,
maxHeightPx: H,
style: { cellWBonus: 0, cellHBonus: 0, aa: true },
style: { ...BASE_STYLE, cellWBonus: 4, cellHBonus: 4 },
},
},
];
@@ -184,6 +216,14 @@ function posInt(v: unknown, fallback: number): number {
return Number.isFinite(v) && (v as number) > 0 ? Math.floor(v as number) : fallback;
}
function nonNegativeInt(v: unknown, fallback: number): number {
return Number.isFinite(v) && (v as number) >= 0 ? Math.floor(v as number) : fallback;
}
function renderFont(v: unknown, fallback: RenderFont): RenderFont {
return v === 'spleen-5x8' || v === 'jetbrains-mono-10' ? v : fallback;
}
function parseEnvProfiles(raw: string): Map<string, GptModelProfile> {
const out = new Map<string, GptModelProfile>();
if (!raw) return out;
@@ -197,18 +237,27 @@ function parseEnvProfiles(raw: string): Map<string, GptModelProfile> {
for (const [k, v] of Object.entries(obj as Record<string, unknown>)) {
if (!v || typeof v !== 'object') continue;
const key = k.toLowerCase();
const base = resolveBuiltin(key); // partial fields fall back to the built-in match
// An exact override key may be a prefix of the real model id. Resolve the
// known family explicitly so `{"gpt-5.6-sol":{...}}` inherits Sol's font
// and geometry even when a suffixed runtime id is expected.
const base = key === 'gpt-5.6-sol' ? GPT56_SOL_PROFILE : resolveBuiltin(key);
const p = v as Partial<GptModelProfile>;
const styleIn = (p as { style?: GptRenderStyle }).style;
const baseStyle = base.style ?? { cellWBonus: 0, cellHBonus: 0, aa: true };
const baseStyle = base.style;
const style: GptRenderStyle = {
cellWBonus: styleIn && Number.isFinite(styleIn.cellWBonus as number)
? Math.floor(styleIn.cellWBonus as number)
: baseStyle.cellWBonus,
cellHBonus: styleIn && Number.isFinite(styleIn.cellHBonus as number)
? Math.floor(styleIn.cellHBonus as number)
: baseStyle.cellHBonus,
font: renderFont(styleIn?.font, baseStyle.font),
cellWBonus: nonNegativeInt(styleIn?.cellWBonus, baseStyle.cellWBonus),
cellHBonus: nonNegativeInt(styleIn?.cellHBonus, baseStyle.cellHBonus),
aa: styleIn && typeof styleIn.aa === 'boolean' ? styleIn.aa : baseStyle.aa,
grid: styleIn && typeof styleIn.grid === 'boolean' ? styleIn.grid : baseStyle.grid,
gridCols: nonNegativeInt(styleIn?.gridCols, baseStyle.gridCols),
colorCycle: styleIn && typeof styleIn.colorCycle === 'boolean'
? styleIn.colorCycle
: baseStyle.colorCycle,
markerScale: posInt(styleIn?.markerScale, baseStyle.markerScale),
markerRed: styleIn && typeof styleIn.markerRed === 'boolean'
? styleIn.markerRed
: baseStyle.markerRed,
};
out.set(key, {
vision: isValidVision(p.vision) ? p.vision : base.vision,
@@ -229,26 +278,10 @@ function envProfiles(): Map<string, GptModelProfile> {
return envMap;
}
/**
* Resolve the full rendering + vision-cost profile for a model id. Env overrides
* (longest matching prefix) win over the built-in table; unknown models get the
* conservative `DEFAULT_GPT_PROFILE`.
*/
function withDefaultStyle(profile: GptModelProfile): GptModelProfile {
const d = DEFAULT_GPT_PROFILE.style!;
const st = profile.style;
return {
...profile,
style: {
cellWBonus: st?.cellWBonus ?? d.cellWBonus,
cellHBonus: st?.cellHBonus ?? d.cellHBonus,
aa: st?.aa ?? d.aa,
},
};
}
export function resolveGptProfile(model: string | null | undefined): GptModelProfile {
const m = (model ?? '').toLowerCase();
// Match applicability.ts: bracketed transport variants (for example [1m])
// do not define a different visual reader profile.
const m = (model ?? '').toLowerCase().replace(/\[[^\]]*\]/g, '');
const env = envProfiles();
if (env.size > 0) {
let best: GptModelProfile | undefined;
@@ -259,7 +292,7 @@ export function resolveGptProfile(model: string | null | undefined): GptModelPro
bestLen = k.length;
}
}
if (best) return withDefaultStyle(best);
if (best) return best;
}
return withDefaultStyle(resolveBuiltin(m));
return resolveBuiltin(m);
}
src/core/openai-history.ts
+10 -4
View File
@@ -21,8 +21,14 @@
* HistoryTurn list and the planner/renderer are shared.
*/
import { renderTextToPngs, reflow, neutralizeSentinel, type RenderedImage } from './render.js';
import { GPT_MAX_HEIGHT_PX } from './gpt-model-profiles.js';
import {
renderTextToPngs,
reflow,
neutralizeSentinel,
type RenderedImage,
type RenderStyle,
} from './render.js';
import { DEFAULT_GPT_PROFILE, GPT_MAX_HEIGHT_PX } from './gpt-model-profiles.js';
import { countTokens as o200kCountTokens } from 'gpt-tokenizer/encoding/o200k_base';
/** Portrait-strip width for GPT history images. Mirrors GPT_STRIP_COLS in
@@ -81,7 +87,7 @@ export interface GptHistoryOptions {
* into renderTextToPngs so history pages split at the same height the gate prices. */
maxHeightPx: number;
/** Glyph density from the model profile. Empty = production 5x8. */
style: { cellWBonus?: number; cellHBonus?: number; aa?: boolean };
style: RenderStyle;
/** Hard cap on GPT history image count. This is a TRUE cap, not a threshold:
* collapse the oldest completed sections until the next section would exceed
* the cap, then leave the remaining history as ordinary text. Prevents 80+
@@ -107,7 +113,7 @@ export const GPT_HISTORY_DEFAULTS: GptHistoryOptions = {
// GPT path: OpenAI's resize bounds (2048-bbox / 768 short side) permit the tall
// strip — do NOT re-link to render.ts MAX_HEIGHT_PX (Anthropic's 1568/1.15 MP clamp).
maxHeightPx: GPT_MAX_HEIGHT_PX,
style: { cellWBonus: 0, cellHBonus: 0, aa: true },
style: DEFAULT_GPT_PROFILE.style,
maxImages: GPT_HISTORY_MAX_IMAGES,
reflow: true,
};
src/core/openai.ts
+46 -34
View File
@@ -10,13 +10,15 @@ import {
renderTextToPngs,
reflow,
shrinkColsToContent,
renderCellWidth,
renderCellHeight,
PAD_X,
CELL_W,
PAD_Y,
READABLE_CHARS_PER_IMAGE,
type RenderedImage,
} from './render.js';
import {
resolveGptProfile,
DEFAULT_GPT_STRIP_COLS,
type GptVisionCost,
} from './gpt-model-profiles.js';
import { bytesToBase64 } from './png.js';
@@ -218,7 +220,7 @@ interface OpenAIResolvedOptions {
compress: boolean;
compressTools: boolean;
minCompressChars: number;
cols: number;
cols?: number;
multiCol: number;
charsPerToken: number;
reflow: boolean;
@@ -230,7 +232,7 @@ const DEFAULTS: OpenAIResolvedOptions = {
compress: true,
compressTools: true,
minCompressChars: 2000,
cols: DEFAULT_GPT_STRIP_COLS,
cols: undefined,
multiCol: 1,
charsPerToken: 4, // conservative OpenAI default; override after telemetry
reflow: true,
@@ -242,7 +244,7 @@ function resolveOptions(opts: TransformOptions): OpenAIResolvedOptions {
compress: opts.compress ?? DEFAULTS.compress,
compressTools: opts.compressTools ?? DEFAULTS.compressTools,
minCompressChars: opts.minCompressChars ?? DEFAULTS.minCompressChars,
cols: opts.cols ?? DEFAULTS.cols,
cols: opts.cols,
multiCol: opts.multiCol ?? DEFAULTS.multiCol,
charsPerToken: opts.charsPerToken ?? DEFAULTS.charsPerToken,
reflow: opts.reflow ?? DEFAULTS.reflow,
@@ -502,10 +504,25 @@ function evalOpenAIGate(
cols: number,
charsPerToken: number,
): { imageTokens: number; textTokens: number; profitable: boolean } {
const stripW = 2 * PAD_X + cols * CELL_W;
const estImages = estimateImageCount(renderedText, cols, 1);
const perStrip = visionTokensForModel(model, stripW, resolveGptProfile(model).maxHeightPx);
const imageTokens = estImages * perStrip;
const profile = resolveGptProfile(model);
const style = profile.style;
const cellW = renderCellWidth(style);
const cellH = renderCellHeight(style);
const stripW = 2 * PAD_X + cols * cellW;
const maxLines = Math.max(1, Math.floor((profile.maxHeightPx - 2 * PAD_Y) / cellH));
const maxCharsPerImage = Math.min(
READABLE_CHARS_PER_IMAGE,
Math.max(1, cols) * maxLines,
);
const estImages = estimateImageCount(
renderedText,
cols,
1,
maxCharsPerImage,
maxLines,
);
const perFullStrip = visionTokensForModel(model, stripW, profile.maxHeightPx);
const imageTokens = estImages * perFullStrip;
const textTokens = renderedText.length / charsPerToken;
return { imageTokens, textTokens, profitable: imageTokens < textTokens };
}
@@ -541,21 +558,6 @@ function gptTextTokens(text: string): number {
/** Vision-token cost of the rendered images, summed over their real dims —
* what GPT actually bills as input for the slab pxpipe imaged. */
function profileRenderStyle(
model: string | null | undefined,
): { cellWBonus?: number; cellHBonus?: number; aa?: boolean; grid?: boolean; colorCycle?: boolean } {
const st = resolveGptProfile(model).style as
| { cellWBonus?: number; cellHBonus?: number; aa?: boolean; grid?: boolean; colorCycle?: boolean }
| undefined;
return {
cellWBonus: st?.cellWBonus ?? 0,
cellHBonus: st?.cellHBonus ?? 0,
aa: st?.aa ?? true,
...(st?.grid ? { grid: true } : {}),
...(st?.colorCycle ? { colorCycle: true } : {}),
};
}
function gptImageTokens(model: string, images: RenderedImage[]): number {
let n = 0;
for (const img of images) n += visionTokensForModel(model, img.width, img.height);
@@ -705,7 +707,12 @@ export async function transformOpenAIChatCompletions(
: '';
const header = CHAT_HEADER.replace('\n====', reflowNote + '\n====');
const renderedText = header + combined;
const cols = Math.min(shrinkColsToContent(renderedText, o.cols), resolveGptProfile(req.model).stripCols);
const profile = resolveGptProfile(req.model);
const maxCols = o.cols ?? profile.stripCols;
const cols = Math.min(
shrinkColsToContent(renderedText, maxCols, profile.style.markerScale, profile.style.font),
profile.stripCols,
);
const gate = evalOpenAIGate(req.model, renderedText, cols, o.charsPerToken);
info.gateEval = {
@@ -722,7 +729,7 @@ export async function transformOpenAIChatCompletions(
return { body, info };
}
const images = await renderTextToPngs(renderedText, cols, profileRenderStyle(req.model), resolveGptProfile(req.model).maxHeightPx);
const images = await renderTextToPngs(renderedText, cols, profile.style, profile.maxHeightPx);
if (images.length === 0) {
info.reason = 'render_empty';
return { body, info };
@@ -782,9 +789,9 @@ export async function transformOpenAIChatCompletions(
const plan = await planGptCollapse(turns, firstUserIdx + 1, profitable, {
...o.gptHistory,
reflow: o.reflow,
cols: o.gptHistory?.cols ?? resolveGptProfile(req.model).stripCols,
maxHeightPx: o.gptHistory?.maxHeightPx ?? resolveGptProfile(req.model).maxHeightPx,
style: o.gptHistory?.style ?? profileRenderStyle(req.model),
cols: o.gptHistory?.cols ?? profile.stripCols,
maxHeightPx: o.gptHistory?.maxHeightPx ?? profile.maxHeightPx,
style: o.gptHistory?.style ?? profile.style,
});
foldGptHistory(info, req.model, plan);
const allImages = [...plan.images, ...plan.imagesAfter];
@@ -913,7 +920,12 @@ export async function transformOpenAIResponses(
: '';
const header = RESPONSES_HEADER.replace('\n====', reflowNote + '\n====');
const renderedText = header + combined;
const cols = Math.min(shrinkColsToContent(renderedText, o.cols), resolveGptProfile(req.model).stripCols);
const profile = resolveGptProfile(req.model);
const maxCols = o.cols ?? profile.stripCols;
const cols = Math.min(
shrinkColsToContent(renderedText, maxCols, profile.style.markerScale, profile.style.font),
profile.stripCols,
);
const gate = evalOpenAIGate(req.model, renderedText, cols, o.charsPerToken);
info.gateEval = {
@@ -930,7 +942,7 @@ export async function transformOpenAIResponses(
return { body, info };
}
const images = await renderTextToPngs(renderedText, cols, profileRenderStyle(req.model), resolveGptProfile(req.model).maxHeightPx);
const images = await renderTextToPngs(renderedText, cols, profile.style, profile.maxHeightPx);
if (images.length === 0) {
info.reason = 'render_empty';
return { body, info };
@@ -1021,9 +1033,9 @@ export async function transformOpenAIResponses(
const plan = await planGptCollapse(turns, firstUserIdx + 1, profitable, {
...o.gptHistory,
reflow: o.reflow,
cols: o.gptHistory?.cols ?? resolveGptProfile(req.model).stripCols,
maxHeightPx: o.gptHistory?.maxHeightPx ?? resolveGptProfile(req.model).maxHeightPx,
style: o.gptHistory?.style ?? profileRenderStyle(req.model),
cols: o.gptHistory?.cols ?? profile.stripCols,
maxHeightPx: o.gptHistory?.maxHeightPx ?? profile.maxHeightPx,
style: o.gptHistory?.style ?? profile.style,
});
foldGptHistory(info, req.model, plan);
const allImages = [...plan.images, ...plan.imagesAfter];
src/core/render.ts
+213 -50
View File
@@ -8,6 +8,7 @@
import {
ATLAS_CELL_W,
ATLAS_CELL_H,
ATLAS_ASCENT,
ATLAS_PIXELS,
ATLAS_OFFSETS,
ATLAS_WIDE_FLAGS,
@@ -16,13 +17,128 @@ import {
import {
ATLAS_GRAY_CELL_W,
ATLAS_GRAY_CELL_H,
ATLAS_GRAY_ASCENT,
ATLAS_GRAY_PIXELS,
ATLAS_GRAY_OFFSETS,
ATLAS_GRAY_WIDE_FLAGS,
atlasGrayRank,
} from './atlas-gray.js';
import {
ATLAS_CELL_W as JBM10_CELL_W,
ATLAS_CELL_H as JBM10_CELL_H,
ATLAS_ASCENT as JBM10_ASCENT,
ATLAS_PIXELS as JBM10_PIXELS,
ATLAS_OFFSETS as JBM10_OFFSETS,
ATLAS_WIDE_FLAGS as JBM10_WIDE_FLAGS,
atlasRank as jbMono10Rank,
} from './atlas-jbmono10.js';
import {
ATLAS_GRAY_CELL_W as JBM10_GRAY_CELL_W,
ATLAS_GRAY_CELL_H as JBM10_GRAY_CELL_H,
ATLAS_GRAY_ASCENT as JBM10_GRAY_ASCENT,
ATLAS_GRAY_PIXELS as JBM10_GRAY_PIXELS,
ATLAS_GRAY_OFFSETS as JBM10_GRAY_OFFSETS,
ATLAS_GRAY_WIDE_FLAGS as JBM10_GRAY_WIDE_FLAGS,
atlasGrayRank as jbMono10GrayRank,
} from './atlas-gray-jbmono10.js';
import { encodeGrayPng, encodeRgbPng } from './png.js';
export type RenderFont = 'spleen-5x8' | 'jetbrains-mono-10';
export const DEFAULT_RENDER_FONT: RenderFont = 'spleen-5x8';
interface BitAtlas {
cellW: number;
cellH: number;
ascent: number;
pixels: Uint8Array;
offsets: Uint32Array;
wideFlags: Uint8Array;
rank: (codepoint: number) => number;
}
interface GrayAtlas {
cellW: number;
cellH: number;
ascent: number;
pixels: Uint8Array;
offsets: Uint32Array;
wideFlags: Uint8Array;
rank: (codepoint: number) => number;
}
interface AtlasSet {
bit: BitAtlas;
gray: GrayAtlas;
}
const DEFAULT_ATLAS: AtlasSet = {
bit: {
cellW: ATLAS_CELL_W,
cellH: ATLAS_CELL_H,
ascent: ATLAS_ASCENT,
pixels: ATLAS_PIXELS,
offsets: ATLAS_OFFSETS,
wideFlags: ATLAS_WIDE_FLAGS,
rank: atlasRank,
},
gray: {
cellW: ATLAS_GRAY_CELL_W,
cellH: ATLAS_GRAY_CELL_H,
ascent: ATLAS_GRAY_ASCENT,
pixels: ATLAS_GRAY_PIXELS,
offsets: ATLAS_GRAY_OFFSETS,
wideFlags: ATLAS_GRAY_WIDE_FLAGS,
rank: atlasGrayRank,
},
};
const JBM10_ATLAS: AtlasSet = {
bit: {
cellW: JBM10_CELL_W,
cellH: JBM10_CELL_H,
ascent: JBM10_ASCENT,
pixels: JBM10_PIXELS,
offsets: JBM10_OFFSETS,
wideFlags: JBM10_WIDE_FLAGS,
rank: jbMono10Rank,
},
gray: {
cellW: JBM10_GRAY_CELL_W,
cellH: JBM10_GRAY_CELL_H,
ascent: JBM10_GRAY_ASCENT,
pixels: JBM10_GRAY_PIXELS,
offsets: JBM10_GRAY_OFFSETS,
wideFlags: JBM10_GRAY_WIDE_FLAGS,
rank: jbMono10GrayRank,
},
};
function atlasSet(font: RenderFont | undefined): AtlasSet {
return font === 'jetbrains-mono-10' ? JBM10_ATLAS : DEFAULT_ATLAS;
}
function bitGlyph(codepoint: number, font: RenderFont | undefined): { atlas: BitAtlas; rank: number } | null {
const selected = atlasSet(font).bit;
const rank = selected.rank(codepoint);
if (rank >= 0) return { atlas: selected, rank };
if (selected !== DEFAULT_ATLAS.bit) {
const fallbackRank = DEFAULT_ATLAS.bit.rank(codepoint);
if (fallbackRank >= 0) return { atlas: DEFAULT_ATLAS.bit, rank: fallbackRank };
}
return null;
}
function grayGlyph(codepoint: number, font: RenderFont | undefined): { atlas: GrayAtlas; rank: number } | null {
const selected = atlasSet(font).gray;
const rank = selected.rank(codepoint);
if (rank >= 0) return { atlas: selected, rank };
if (selected !== DEFAULT_ATLAS.gray) {
const fallbackRank = DEFAULT_ATLAS.gray.rank(codepoint);
if (fallbackRank >= 0) return { atlas: DEFAULT_ATLAS.gray, rank: fallbackRank };
}
return null;
}
/** Page-height ceiling. Measured (2026-07-01, count_tokens sweep, claude-sonnet-4-5 — see
* /tmp/pxexp/LEVER1-findings.md): the API downscales any image to fit BOTH long-edge ≤1568
* AND ~1.15 MP (≈1,143,750 px), then bills ≈ px/750 (≈1525 tok cap, applied per-image).
@@ -40,9 +156,11 @@ export const DENSE_CONTENT_COLS = 312;
/** Bare 5×8 cell (no padding). A/B showed 5×8 beats 7×10 on dense JSON (4/5 vs 3/5 reads, 42% fewer tokens).
* Revert to {cellWBonus:2, cellHBonus:2} if misread rates rise. */
export const DENSE_RENDER_STYLE: RenderStyle = { cellWBonus: 0, cellHBonus: 0, aa: true };
/** Anthropic static slab uses the same measured 312-column no-resize geometry. */
export const ANTHROPIC_SLAB_COLS = DENSE_CONTENT_COLS;
/** Default columns for the static slab. 312 × 5 px + 8 px pad = 1568 px — exactly the API's
* long-edge bound (313 cols = 1573 px would trigger a 0.997× resample, blurring every glyph). */
const DEFAULT_COLS = 312;
const DEFAULT_COLS = ANTHROPIC_SLAB_COLS;
/** Horizontal padding (left + right each), px. Exported for transform.ts token-cost math. */
export const PAD_X = 4;
/** Vertical padding (top + bottom each), px. Exported for transform.ts token-cost math. */
@@ -72,6 +190,8 @@ export interface RenderedImage {
/** Optional render-time styling. All fields unset = production default 5×8 cell.
* Eval harness overrides per variant to A/B cell sizes and structure aids. */
export interface RenderStyle {
/** Rasterized font atlas. Alternate atlases fall back to Spleen/Unifont for missing Unicode. */
font?: RenderFont;
/** Draw faint grey grid rules onto background pixels (zero pixel cost). */
grid?: boolean;
/** Draw a vertical grid rule every N columns. 0/unset = row rules only. */
@@ -84,7 +204,7 @@ export interface RenderStyle {
cellHBonus?: number;
/** Extra blank columns beside the 5px glyph (cell width = 5 + this). Negative overlaps glyphs. Unset = DEFAULT_CELL_W_BONUS. */
cellWBonus?: number;
/** Use the AA grayscale atlas (atlas-gray.ts). EVAL-ONLY; default 1-bit path is unchanged. */
/** Use the AA grayscale companion atlas. */
aa?: boolean;
/** Cycle palette colors per glyph for per-character boundary cues. Forces RGB output. Composes with aa. */
colorCycle?: boolean;
@@ -93,16 +213,30 @@ export interface RenderStyle {
colorByRole?: boolean;
}
export function renderCellWidth(style: RenderStyle = {}): number {
const atlas = style.aa === true ? atlasSet(style.font).gray : atlasSet(style.font).bit;
return Math.max(1, atlas.cellW + Math.floor(style.cellWBonus ?? DEFAULT_CELL_W_BONUS));
}
export function renderCellHeight(style: RenderStyle = {}): number {
const atlas = style.aa === true ? atlasSet(style.font).gray : atlasSet(style.font).bit;
return atlas.cellH + Math.max(0, Math.floor(style.cellHBonus ?? DEFAULT_CELL_H_BONUS));
}
// --- column-aware wrapping -------------------------------------------------
/** Visual width of a codepoint in cells (1 = Latin, 2 = East Asian Wide).
* Missing codepoints advance 1 cell so wrap math stays stable. */
function cellsFor(codepoint: number, markerScale: number = 1): number {
function cellsFor(
codepoint: number,
markerScale: number = 1,
font: RenderFont = DEFAULT_RENDER_FONT,
): number {
// Enlarged ↵ occupies markerScale cells of wrap budget instead of 1.
if (codepoint === NL_SENTINEL_CP && markerScale > 1) return markerScale;
const rank = atlasRank(codepoint);
if (rank < 0) return 1;
return ATLAS_WIDE_FLAGS[rank] === 1 ? 2 : 1;
const glyph = bitGlyph(codepoint, font);
if (!glyph) return 1;
return glyph.atlas.wideFlags[glyph.rank] === 1 ? 2 : 1;
}
const TAB_WIDTH = 4; // standard 4-space tab stops (logs, code, tool output are all 4-oriented)
@@ -251,21 +385,30 @@ export function expandTabsInLine(line: string): string {
}
/** Visual width of a line in cells. Wide CJK = 2; enlarged ↵ = markerScale. */
export function measureLineCols(line: string, markerScale: number = 1): number {
export function measureLineCols(
line: string,
markerScale: number = 1,
font: RenderFont = DEFAULT_RENDER_FONT,
): number {
let w = 0;
for (const ch of line) w += cellsFor(ch.codePointAt(0)!, markerScale);
for (const ch of line) w += cellsFor(ch.codePointAt(0)!, markerScale, font);
return w;
}
/** Always renders at full canvas width. Signature kept for transform.ts compatibility; returns cols unchanged. */
// eslint-disable-next-line @typescript-eslint/no-unused-vars
export function shrinkColsToContent(text: string, cols: number, markerScale: number = 1): number {
export function shrinkColsToContent(
text: string,
cols: number,
markerScale: number = 1,
font: RenderFont = DEFAULT_RENDER_FONT,
): number {
// Real content-width measurement — delegates to measureContentCols (was historically a
// no-op stub). The proxy's cost gate AND renderer both call this, so sizing the canvas to
// the widest line here makes the proxy match the SDK/export path (renderTextToImages),
// which already uses measureContentCols. Never narrows below the widest line ⇒ row count
// (and thus image/paging count) is unchanged; only the canvas WIDTH (pixel cost) drops.
return measureContentCols(text, cols, markerScale);
return measureContentCols(text, cols, markerScale, font);
}
/**
@@ -276,13 +419,18 @@ export function shrinkColsToContent(text: string, cols: number, markerScale: num
* deterministic width → cache-prefix-safe. Tabs are expanded so the measured width
* matches what the renderer actually lays out.
*/
export function measureContentCols(text: string, maxCols: number, markerScale: number = 1): number {
export function measureContentCols(
text: string,
maxCols: number,
markerScale: number = 1,
font: RenderFont = DEFAULT_RENDER_FONT,
): number {
const cap = Math.max(1, maxCols | 0);
let widest = 1;
let start = 0;
for (let i = 0; i <= text.length; i++) {
if (i === text.length || text[i] === '\n') {
const w = measureLineCols(expandTabsInLine(text.slice(start, i)), markerScale);
const w = measureLineCols(expandTabsInLine(text.slice(start, i)), markerScale, font);
if (w > widest) widest = w;
if (widest >= cap) return cap;
start = i + 1;
@@ -291,7 +439,12 @@ export function measureContentCols(text: string, maxCols: number, markerScale: n
return Math.min(cap, widest);
}
export function wrapLines(text: string, cols: number, markerScale: number = 1): string[] {
export function wrapLines(
text: string,
cols: number,
markerScale: number = 1,
font: RenderFont = DEFAULT_RENDER_FONT,
): string[] {
const out: string[] = [];
const minified = minifyForRender(text);
for (const rawWithTabs of minified.split('\n')) {
@@ -306,7 +459,7 @@ export function wrapLines(text: string, cols: number, markerScale: number = 1):
// ↵ is treated as an inline glyph — it never forces a row break.
for (const ch of raw) {
const cp = ch.codePointAt(0)!;
const w = cellsFor(cp, markerScale);
const w = cellsFor(cp, markerScale, font);
if (curCols + w > cols) {
out.push(cur);
cur = ch;
@@ -363,20 +516,23 @@ function blitGlyph(
x: number,
y: number,
codepoint: number,
font: RenderFont = DEFAULT_RENDER_FONT,
markerMask: Uint8Array | null = null,
): number {
const rank = atlasRank(codepoint);
if (rank < 0) return 0;
const wide = ATLAS_WIDE_FLAGS[rank] === 1;
const srcW = wide ? 2 * ATLAS_CELL_W : ATLAS_CELL_W;
const glyph = bitGlyph(codepoint, font);
if (!glyph) return 0;
const { atlas, rank } = glyph;
const wide = atlas.wideFlags[rank] === 1;
const srcW = wide ? 2 * atlas.cellW : atlas.cellW;
// ATLAS_OFFSETS is a bit offset (MSB-first packing). Pixel (gx,gy): byte = bitIdx>>>3, bit = 7-(bitIdx&7).
const srcOff = ATLAS_OFFSETS[rank]!;
for (let gy = 0; gy < ATLAS_CELL_H; gy++) {
const dstRow = (y + gy) * fbW + x;
const srcOff = atlas.offsets[rank]!;
const yOffset = atlasSet(font).bit.ascent - atlas.ascent;
for (let gy = 0; gy < atlas.cellH; gy++) {
const dstRow = (y + yOffset + gy) * fbW + x;
const bitRowStart = srcOff + gy * srcW;
for (let gx = 0; gx < srcW; gx++) {
const bitIdx = bitRowStart + gx;
const byte = ATLAS_PIXELS[bitIdx >>> 3]!;
const byte = atlas.pixels[bitIdx >>> 3]!;
const bit = (byte >>> (7 - (bitIdx & 7))) & 1;
if (bit) {
fb[dstRow + gx] = 255; // glyphs never overlap in grid layout; set unconditionally
@@ -397,18 +553,21 @@ function blitGlyphGray(
x: number,
y: number,
codepoint: number,
font: RenderFont = DEFAULT_RENDER_FONT,
): number {
const rank = atlasGrayRank(codepoint);
if (rank < 0) return 0;
const wide = ATLAS_GRAY_WIDE_FLAGS[rank] === 1;
const srcW = wide ? 2 * ATLAS_GRAY_CELL_W : ATLAS_GRAY_CELL_W;
const glyph = grayGlyph(codepoint, font);
if (!glyph) return 0;
const { atlas, rank } = glyph;
const wide = atlas.wideFlags[rank] === 1;
const srcW = wide ? 2 * atlas.cellW : atlas.cellW;
// ATLAS_GRAY_OFFSETS is a byte offset (1 byte/pixel, unlike the bit-packed 1-bit atlas).
const srcOff = ATLAS_GRAY_OFFSETS[rank]!;
for (let gy = 0; gy < ATLAS_GRAY_CELL_H; gy++) {
const dstRow = (y + gy) * fbW + x;
const srcOff = atlas.offsets[rank]!;
const yOffset = atlasSet(font).gray.ascent - atlas.ascent;
for (let gy = 0; gy < atlas.cellH; gy++) {
const dstRow = (y + yOffset + gy) * fbW + x;
const srcRow = srcOff + gy * srcW;
for (let gx = 0; gx < srcW; gx++) {
const coverage = ATLAS_GRAY_PIXELS[srcRow + gx]!;
const coverage = atlas.pixels[srcRow + gx]!;
if (coverage > 0) {
const idx = dstRow + gx;
if (coverage > fb[idx]!) fb[idx] = coverage;
@@ -429,19 +588,22 @@ function blitGlyphScaled(
y: number,
codepoint: number,
scaleX: number,
font: RenderFont = DEFAULT_RENDER_FONT,
): number {
const rank = atlasRank(codepoint);
if (rank < 0) return 0;
const wide = ATLAS_WIDE_FLAGS[rank] === 1;
const srcW = wide ? 2 * ATLAS_CELL_W : ATLAS_CELL_W;
const srcOff = ATLAS_OFFSETS[rank]!;
for (let gy = 0; gy < ATLAS_CELL_H; gy++) {
const py = y + gy;
const glyph = bitGlyph(codepoint, font);
if (!glyph) return 0;
const { atlas, rank } = glyph;
const wide = atlas.wideFlags[rank] === 1;
const srcW = wide ? 2 * atlas.cellW : atlas.cellW;
const srcOff = atlas.offsets[rank]!;
const yOffset = atlasSet(font).bit.ascent - atlas.ascent;
for (let gy = 0; gy < atlas.cellH; gy++) {
const py = y + yOffset + gy;
if (py >= fbH) break;
const bitRowStart = srcOff + gy * srcW;
for (let gx = 0; gx < srcW; gx++) {
const bitIdx = bitRowStart + gx;
const byte = ATLAS_PIXELS[bitIdx >>> 3]!;
const byte = atlas.pixels[bitIdx >>> 3]!;
if (((byte >>> (7 - (bitIdx & 7))) & 1) === 0) continue;
for (let sx = 0; sx < scaleX; sx++) {
const px = x + gx * scaleX + sx;
@@ -497,18 +659,19 @@ export async function renderChunkToPng(
slotText?: string,
): Promise<RenderedImage> {
const useAA = style.aa === true;
const atlasH = useAA ? ATLAS_GRAY_CELL_H : ATLAS_CELL_H;
const atlasW = useAA ? ATLAS_GRAY_CELL_W : ATLAS_CELL_W;
const selected = atlasSet(style.font);
const atlasH = useAA ? selected.gray.cellH : selected.bit.cellH;
const atlasW = useAA ? selected.gray.cellW : selected.bit.cellW;
const markerScale = Math.max(1, Math.floor(style.markerScale ?? 1));
const cellH = atlasH + Math.max(0, Math.floor(style.cellHBonus ?? DEFAULT_CELL_H_BONUS));
const cellW = Math.max(1, atlasW + Math.floor(style.cellWBonus ?? DEFAULT_CELL_W_BONUS));
const lines = wrapLines(text, cols, markerScale);
const cellH = renderCellHeight(style);
const cellW = renderCellWidth(style);
const lines = wrapLines(text, cols, markerScale, style.font);
// Slot string carries role attribution by position. It is width-identical to
// `text`, so wrapLines splits it into the exact same rows — fitSlotLines[r] aligns
// codepoint-for-codepoint with fitLines[r]. Only built when coloring is on.
const slotLines: string[] | null =
style.colorByRole === true && slotText !== undefined
? wrapLines(slotText, cols, markerScale)
? wrapLines(slotText, cols, markerScale, style.font)
: null;
const maxLines = Math.max(1, Math.floor((maxHeightPx - 2 * PAD_Y) / cellH));
@@ -567,7 +730,7 @@ export async function renderChunkToPng(
: (glyphIndex % GLYPH_PALETTE.length) + 1; // 0 reserved for background in colorMask
let advance: number;
if (isMarker && markerScale > 1) {
advance = blitGlyphScaled(fb, markerMask, width, height, baseX, baseY, codepoint, markerScale);
advance = blitGlyphScaled(fb, markerMask, width, height, baseX, baseY, codepoint, markerScale, style.font);
if (colorMask) {
for (let gy = 0; gy < atlasH; gy++) {
const py = baseY + gy;
@@ -581,7 +744,7 @@ export async function renderChunkToPng(
}
}
} else if (useAA) {
advance = blitGlyphGray(fb, width, baseX, baseY, codepoint);
advance = blitGlyphGray(fb, width, baseX, baseY, codepoint, style.font);
if (colorMask && advance > 0) {
const srcW = advance * atlasW;
for (let gy = 0; gy < atlasH; gy++) {
@@ -596,7 +759,7 @@ export async function renderChunkToPng(
}
}
} else {
advance = blitGlyph(fb, width, baseX, baseY, codepoint, isMarker ? markerMask : null);
advance = blitGlyph(fb, width, baseX, baseY, codepoint, style.font, isMarker ? markerMask : null);
if (colorMask && advance > 0) {
const srcW = advance * atlasW;
for (let gy = 0; gy < atlasH; gy++) {
@@ -694,13 +857,13 @@ export async function renderTextToPngsWithCharLimit(
slotText?: string,
): Promise<RenderedImage[]> {
const markerScale = Math.max(1, Math.floor(style.markerScale ?? 1));
const cellH = ATLAS_CELL_H + Math.max(0, Math.floor(style.cellHBonus ?? DEFAULT_CELL_H_BONUS));
const lines = wrapLines(text, cols, markerScale);
const cellH = renderCellHeight(style);
const lines = wrapLines(text, cols, markerScale, style.font);
// Width-identical slot rows align 1:1 with `lines`; pages are contiguous slices,
// so the same index range gives each page its slot half. Only built when coloring.
const slotLines: string[] | null =
style.colorByRole === true && slotText !== undefined
? wrapLines(slotText, cols, markerScale)
? wrapLines(slotText, cols, markerScale, style.font)
: null;
const hardLinesPerImg = Math.max(1, Math.floor((maxHeightPx - 2 * PAD_Y) / cellH));
// Dense pages (DENSE_CONTENT_CHARS_PER_IMAGE) fill the full 1932 px height;
src/core/transform.ts
+6 -3
View File
@@ -32,6 +32,7 @@ import {
DENSE_CONTENT_CHARS_PER_IMAGE,
DENSE_CONTENT_COLS,
DENSE_RENDER_STYLE,
ANTHROPIC_SLAB_COLS,
renderTextToPngsWithCharLimit,
} from './render.js';
import { factSheetText } from './factsheet.js';
@@ -134,8 +135,8 @@ const DEFAULTS: Required<TransformOptions> = {
minToolResultChars: 6000,
// system field rejects images (400 system.N.type: Input should be 'text') —
// images always go into the first user message.
// 313 cols × 5 px + 8 px pad = 1573 px slab width (under 2000 px ceiling).
cols: 313,
// 312 cols × 5 px + 8 px pad = 1568 px (Anthropic no-resize edge).
cols: ANTHROPIC_SLAB_COLS,
maxImagesPerToolResult: 10,
charsPerToken: 4,
historyAmortizationHorizon: 1,
@@ -1097,10 +1098,12 @@ export function estimateImageCount(
cols: number,
numCols: number = 1,
maxCharsPerImage: number = READABLE_CHARS_PER_IMAGE,
maxLinesPerColumn: number = LINES_PER_IMAGE,
): number {
const n = Math.max(1, numCols | 0);
const readableLinesPerCol = Math.max(1, Math.floor(maxCharsPerImage / Math.max(1, cols)));
const linesPerImage = Math.min(LINES_PER_IMAGE, readableLinesPerCol) * n;
const hardLinesPerCol = Math.max(1, Math.floor(maxLinesPerColumn));
const linesPerImage = Math.min(hardLinesPerCol, readableLinesPerCol) * n;
const charBudget = Math.max(1, maxCharsPerImage * n);
if (typeof textOrLen === 'number') {
// Back-compat shim — numeric arg gets the looser chars-based estimate.
src/dashboard/fragments.ts
+18 -6
View File
@@ -85,7 +85,7 @@ const MODEL_CATALOG: ReadonlyArray<{ id: string; label: string }> = [
];
const GPT_MODEL_CATALOG: ReadonlyArray<{ id: string; label: string }> = [
{ id: 'gpt-5.6', label: 'GPT 5.6' },
{ id: 'gpt-5.6-sol', label: 'GPT 5.6 Sol' },
{ id: 'gpt-5.5', label: 'GPT 5.5' },
];
@@ -147,7 +147,7 @@ export function renderModelsFragment(
otherChips +
`<span class="hint">opt-in only · OpenAI Responses path · set PXPIPE_MODELS to persist</span>${moot}` +
`</div>` +
`<div class="models" style="display:none">` +
`<div class="models">` +
`<span class="models-label">Image GPT models</span>` +
gptChips +
`<span class="hint">imaging only, no Anthropic cache_control · one scope for all families · set PXPIPE_MODELS (CSV of bases, or off) to persist</span>${moot}` +
@@ -227,10 +227,11 @@ function statTile(
cls = '',
tip = '',
): string {
const t = tip ? ` title="${escapeHtml(tip)}"` : '';
const q = tip ? `<span class="q">?</span>` : '';
const q = tip
? `<span class="q" tabindex="0" aria-label="${escapeHtml(tip)}" data-tip="${escapeHtml(tip)}">?</span>`
: '';
return (
`<div class="tile"${t}>` +
`<div class="tile">` +
`<div class="tile-label">${label}${q}</div>` +
`<div class="tile-value ${cls}">${value}</div>` +
`<div class="tile-sub">${sub}</div>` +
@@ -830,7 +831,18 @@ const CSS = `
.tile-sub { font-size: 11.5px; color: var(--muted); margin-top: 6px; }
.q { display: inline-flex; align-items: center; justify-content: center; width: 14px; height: 14px;
border-radius: 50%; background: var(--surface-2); border: 1px solid var(--border-strong);
color: var(--muted); font-size: 9px; font-weight: 700; cursor: help; }
color: var(--muted); font-size: 9px; font-weight: 700; cursor: help; position: relative; outline: none; }
.q:hover, .q:focus-visible { color: var(--flame-ink); border-color: var(--flame); }
.q::after { content: attr(data-tip); position: absolute; z-index: 50; left: 50%; bottom: calc(100% + 8px);
width: min(280px, 75vw); transform: translate(-50%, 4px); padding: 8px 10px; border-radius: 7px;
background: var(--ink); color: var(--surface); box-shadow: var(--shadow); font-size: 11px; font-weight: 500;
line-height: 1.4; text-align: left; pointer-events: none; opacity: 0; visibility: hidden;
transition: opacity .12s, transform .12s, visibility .12s; }
.q::before { content: ''; position: absolute; z-index: 51; left: 50%; bottom: calc(100% + 3px);
transform: translateX(-50%); border: 5px solid transparent; border-top-color: var(--ink);
pointer-events: none; opacity: 0; visibility: hidden; transition: opacity .12s, visibility .12s; }
.q:hover::after, .q:focus-visible::after { opacity: 1; visibility: visible; transform: translate(-50%, 0); }
.q:hover::before, .q:focus-visible::before { opacity: 1; visibility: visible; }
/* drawer */
.drawer { margin: 0 0 14px; background: var(--surface); border: 1px solid var(--border);
src/node.ts
+1 -1
View File
@@ -161,7 +161,7 @@ Environment:
PXPIPE_GATEWAY_BASE_URL gateway base URL (required with PXPIPE_PROVIDER)
PXPIPE_GATEWAY_HEADERS extra upstream headers: JSON object or k=v;k2=v2
PXPIPE_MODELS comma-separated model bases to image (Claude/GPT/Grok);
default claude-fable-5,gpt-5.6 (Opus/GPT-5.5/Grok opt-in);
default claude-fable-5 (Sol/Opus/GPT-5.5/Grok opt-in);
off disables
PXPIPE_CONFIG JSON config path (default ~/.config/pxpipe/config.json)
supports {"models": [...]} or {"models": "off"}
src/worker.ts
+3 -1
View File
@@ -108,7 +108,9 @@ export default {
// (e.g. MIN_TOOL_RESULT_CHARS=200 to skip absurdly small dumps).
minReminderChars: env.MIN_REMINDER_CHARS ? Number(env.MIN_REMINDER_CHARS) : 0,
minToolResultChars: env.MIN_TOOL_RESULT_CHARS ? Number(env.MIN_TOOL_RESULT_CHARS) : 0,
cols: env.COLS ? Number(env.COLS) : 100,
// Omit by default so OpenAI-shaped requests use their exact model profile;
// COLS remains an explicit operator override for every family.
...(env.COLS ? { cols: Number(env.COLS) } : {}),
// R2 multi-column ON (2 cols) — single-col drops below break-even on
// real tool-doc slabs. Override via MULTI_COL=1 if OCR misreads layout.
multiCol: env.MULTI_COL ? Math.max(1, Number(env.MULTI_COL) | 0) : 2,
tests/cache-stability-e2e.test.ts
+16 -3
View File
@@ -17,11 +17,24 @@
*
* Run just this file: pnpm vitest run tests/cache-stability-e2e.test.ts
*/
import { describe, expect, it } from 'vitest';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { createProxy } from '../src/core/proxy.js';
import { countCacheControlMarkers } from '../src/core/measurement.js';
import { HISTORY_SYNTHETIC_INTRO } from '../src/core/history.js';
// These proxy-contract tests deliberately exercise the opt-in Sol transform.
// Snapshot the developer shell so the suite is deterministic now that Sol is
// intentionally absent from the built-in default scope.
let ambientPxpipeModels: string | undefined;
beforeAll(() => {
ambientPxpipeModels = process.env.PXPIPE_MODELS;
process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6-sol';
});
afterAll(() => {
if (ambientPxpipeModels === undefined) delete process.env.PXPIPE_MODELS;
else process.env.PXPIPE_MODELS = ambientPxpipeModels;
});
// ---------------------------------------------------------------------------
// Fake upstream — records every outbound MAIN request body and answers with a
// canned, well-formed response so the proxy completes. The /count_tokens probe
@@ -487,7 +500,7 @@ describe('e2e cache alignment — GPT (OpenAI) through the real proxy', () => {
turns: { role: 'user' | 'assistant'; text: string }[];
}): string {
return JSON.stringify({
model: opts.model ?? 'gpt-5.6',
model: opts.model ?? 'gpt-5.6-sol',
messages: [
{ role: 'system', content: slab(opts.systemChars) },
...opts.turns.map((t) => ({ role: t.role, content: t.text })),
@@ -500,7 +513,7 @@ describe('e2e cache alignment — GPT (OpenAI) through the real proxy', () => {
turns: { role: 'user' | 'assistant'; text: string }[];
}): string {
return JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: slab(opts.systemChars),
input: opts.turns.map((t) => ({ role: t.role, content: t.text })),
});
tests/dashboard-api.test.ts
+29 -12
View File
@@ -15,6 +15,7 @@ import { getAllowedModelBases, setAllowedModelBases } from '../src/core/applicab
import type { SessionsPaths } from '../src/sessions.js';
import type { TrackEvent } from '../src/core/tracker.js';
import type { StatsPayload, RecentPayload } from '../src/dashboard/types.js';
import { renderPage } from '../src/dashboard/fragments.js';
function makeTmp(): SessionsPaths {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'pxpipe-dashapi-'));
@@ -176,27 +177,28 @@ describe('serveFragment', () => {
dash.handleCompressionToggle({ enabled: true });
});
it('renders and mutates GPT 5.5/5.6 chips via the single model scope', async () => {
it('renders opt-in GPT 5.5/5.6 chips and mutates the single model scope', async () => {
const prev = process.env.PXPIPE_MODELS;
try {
delete process.env.PXPIPE_MODELS;
setAllowedModelBases(null); // reset to built-in default (Fable 5 + GPT 5.6)
const on = await (await dash.serveFragment('models', url, 1234)).text();
expect(on).toContain('Image GPT models');
// GPT 5.6 is on by default; GPT 5.5 is opt-in (off until toggled).
expect(on).toContain('GPT 5.6 ✓');
expect(on).toContain('GPT 5.5</button>');
// GPT 5.6 renders to the left of GPT 5.5.
expect(on.indexOf('GPT 5.6')).toBeLessThan(on.indexOf('GPT 5.5'));
expect(getAllowedModelBases()).toContain('gpt-5.6');
setAllowedModelBases(null); // reset to built-in Fable-only default
const off = await (await dash.serveFragment('models', url, 1234)).text();
expect(off).toContain('Image GPT models');
expect(off).not.toContain('<div class="models" style="display:none">');
expect(off).toContain('GPT 5.6 Sol</button>');
expect(off).toContain('GPT 5.5</button>');
// Sol remains available and ordered first, but neither GPT is silently on.
expect(off.indexOf('GPT 5.6 Sol')).toBeLessThan(off.indexOf('GPT 5.5'));
expect(getAllowedModelBases()).not.toContain('gpt-5.6-sol');
expect(getAllowedModelBases()).not.toContain('gpt-5.5');
dash.handleModelsToggle('gpt-5.6-sol', true);
dash.handleModelsToggle('gpt-5.5', true);
const onBoth = await (await dash.serveFragment('models', url, 1234)).text();
expect(onBoth).toContain('GPT 5.5 ✓');
expect(onBoth).toContain('GPT 5.6 ✓');
expect(onBoth).toContain('GPT 5.6 Sol ✓');
expect(getAllowedModelBases()).toContain('gpt-5.5');
expect(getAllowedModelBases()).toContain('gpt-5.6');
expect(getAllowedModelBases()).toContain('gpt-5.6-sol');
} finally {
setAllowedModelBases(null);
if (prev === undefined) delete process.env.PXPIPE_MODELS;
@@ -218,6 +220,13 @@ describe('serveFragment', () => {
expect(stats).toContain('requests');
});
it('renders keyboard-accessible hover help for stat question marks', async () => {
const header = await (await dash.serveFragment('header', url, 4711)).text();
expect(header).toContain('class="q" tabindex="0"');
expect(header).toContain('data-tip=');
expect(header).toContain('aria-label=');
});
it('escapes HTML in latest source text', async () => {
dash.captureImage({
imagePngs: [new Uint8Array([137, 80, 78, 71])],
@@ -236,6 +245,14 @@ describe('serveFragment', () => {
});
});
describe('dashboard page help UI', () => {
it('ships visible hover/focus tooltip CSS for question-mark controls', () => {
const html = renderPage(47821);
expect(html).toContain('.q:hover::after, .q:focus-visible::after');
expect(html).toContain('content: attr(data-tip)');
});
});
// ---- GPT (OpenAI) savings split ------------------------------------------
// The dashboard math was built entirely around the Anthropic cache-aware
// baseline, so GPT rows used to surface all-zero columns. These lock the
tests/design-behavior-e2e.test.ts
+15 -2
View File
@@ -18,9 +18,22 @@
*
* Run just this file: pnpm vitest run tests/design-behavior-e2e.test.ts
*/
import { describe, expect, it } from 'vitest';
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { createProxy } from '../src/core/proxy.js';
// These proxy-contract tests deliberately exercise the opt-in Sol transform.
// Snapshot the developer shell so the suite is deterministic now that Sol is
// intentionally absent from the built-in default scope.
let ambientPxpipeModels: string | undefined;
beforeAll(() => {
ambientPxpipeModels = process.env.PXPIPE_MODELS;
process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6-sol';
});
afterAll(() => {
if (ambientPxpipeModels === undefined) delete process.env.PXPIPE_MODELS;
else process.env.PXPIPE_MODELS = ambientPxpipeModels;
});
function fakeUpstream() {
const main: string[] = [];
const real = globalThis.fetch;
@@ -190,7 +203,7 @@ describe('design: RECENT REQUEST stays legible (GPT)', () => {
const out = await drive(
'/v1/chat/completions',
JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [{ role: 'system', content: 'SYS ' + big(60_000) }, ...turns],
}),
);
tests/export.test.ts
+7
View File
@@ -531,6 +531,13 @@ describe('exportImageTokens model routing', () => {
expect(gpTokens).toBeGreaterThan(0);
});
it('uses measured Grok pixel pricing instead of the GPT fallback', () => {
expect(exportImageTokens('grok-4.5', 768, 360)).toBe(Math.ceil((768 * 360) / 1000));
expect(exportImageTokens('grok-4.5', 768, 360)).not.toBe(
exportImageTokens('gpt-4o', 768, 360),
);
});
it('Claude image tokens are substantially higher than GPT for the same full-page image', () => {
// The issue was a ~7x underestimate when using GPT formula for Claude.
// Verify the ratio is at least 5x so the fix is clearly meaningful.
tests/openai-gpt5.test.ts
+150 -47
View File
@@ -2,7 +2,7 @@
* Tests for GPT-5 applicability gate, OpenAI vision-token cost model,
* Chat Completions transformer, and Responses API transformer.
*/
import { describe, expect, it } from 'vitest';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { isPxpipeSupportedGptModel } from '../src/core/applicability.js';
import { openAIVisionTokens, visionTokensForModel, isClaudeModel, resolveVisionCost, transformOpenAIChatCompletions, transformOpenAIResponses } from '../src/core/openai.js';
import { resolveGptProfile } from '../src/core/gpt-model-profiles.js';
@@ -10,16 +10,39 @@ import { resolveGptProfile } from '../src/core/gpt-model-profiles.js';
const enc = new TextEncoder();
const dec = new TextDecoder();
let ambientPxpipeModels: string | undefined;
beforeEach(() => {
ambientPxpipeModels = process.env.PXPIPE_MODELS;
delete process.env.PXPIPE_MODELS;
});
afterEach(() => {
if (ambientPxpipeModels === undefined) delete process.env.PXPIPE_MODELS;
else process.env.PXPIPE_MODELS = ambientPxpipeModels;
});
// ── Task 1: applicability gate ──────────────────────────────────────────────
describe('isPxpipeSupportedGptModel', () => {
it('matches GPT 5.6 by default; GPT 5.5 is opt-in only', () => {
it('keeps GPT 5.6 Sol, sibling variants, and GPT 5.5 off by default', () => {
expect(isPxpipeSupportedGptModel('gpt-5')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.5')).toBe(false); // off by default — degrades on imaged context
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.5')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-terra')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5-mini')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-nano')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6[1m]')).toBe(true); // variant tag stripped
expect(isPxpipeSupportedGptModel('gpt-5.6-nano')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol[1m]')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol-codex[1m]')).toBe(false);
});
it('enables only exact Sol ids and suffix aliases when explicitly opted in', () => {
process.env.PXPIPE_MODELS = 'gpt-5.6-sol';
expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol[1m]')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol-codex')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol-codex[1m]')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-terra')).toBe(false);
});
it('rejects non-GPT-5 models', () => {
@@ -59,7 +82,7 @@ describe('openAIVisionTokens', () => {
it('resolveVisionCost returns correct regimes', () => {
expect(resolveVisionCost('gpt-5').regime).toBe('tile');
expect(resolveVisionCost('gpt-5.6').regime).toBe('patch');
expect(resolveVisionCost('gpt-5.6-sol').regime).toBe('patch');
expect(resolveVisionCost('gpt-5-mini').regime).toBe('patch');
expect(resolveVisionCost('gpt-5.6-nano').regime).toBe('patch');
expect(resolveVisionCost('gpt-4o').regime).toBe('tile');
@@ -72,7 +95,7 @@ describe('visionTokensForModel (Claude on the Responses path)', () => {
expect(isClaudeModel('claude-opus-4-8')).toBe(true);
expect(isClaudeModel('claude-sonnet-5')).toBe(true);
expect(isClaudeModel('anthropic/claude-3-5')).toBe(true);
expect(isClaudeModel('gpt-5.6')).toBe(false);
expect(isClaudeModel('gpt-5.6-sol')).toBe(false);
expect(isClaudeModel(undefined)).toBe(false);
});
@@ -110,10 +133,10 @@ const TASK_LIKE_PARAMS = {
additionalProperties: false,
};
describe('transformOpenAIChatCompletions (gpt-5.6)', () => {
describe('transformOpenAIChatCompletions (gpt-5.6-sol)', () => {
it('compresses GPT system + tool docs while preserving native tool selection metadata', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [
{ role: 'system', content: BIG_SYSTEM },
{ role: 'user', content: 'hello' },
@@ -145,8 +168,9 @@ describe('transformOpenAIChatCompletions (gpt-5.6)', () => {
expect(parts[0]!.type).toBe('image_url');
expect(parts[0]!.image_url!.url).toMatch(/^data:image\/png;base64,/);
// Image width should be 768px (152 cols * 5px + 8px pad).
expect(result.info.firstImageWidth).toBe(768);
// GPT 5.6 Sol has its own 6x11 JetBrains Mono profile. 126 cols keeps the
// physical strip below OpenAI's 768px short-side resize floor.
expect(result.info.firstImageWidth).toBe(764);
// System message replaced with pointer.
const sysMsg = messages.find((m) => m.role === 'system')!;
@@ -163,7 +187,7 @@ describe('transformOpenAIChatCompletions (gpt-5.6)', () => {
it('images GPT tool definitions even when there is no instruction context', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [{ role: 'user', content: 'hello' }],
tools: [{
type: 'function',
@@ -186,7 +210,7 @@ describe('transformOpenAIChatCompletions (gpt-5.6)', () => {
it('keeps a parameter literally named "description" (task-tool regression)', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [{ role: 'user', content: 'hello' }],
tools: [{
type: 'function',
@@ -211,7 +235,7 @@ describe('transformOpenAIChatCompletions (gpt-5.6)', () => {
it('returns compressed=false with not_profitable reason for small input', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [
{ role: 'system', content: 'short' },
{ role: 'user', content: 'hi' },
@@ -231,10 +255,10 @@ const BIG_FLAT_TOOL_DESC = 'Flat tool description with lots of context. '.repeat
const RESPONSES_TOOL_PARAMS = { type: 'object', description: 'Param root.', properties: { x: { type: 'string', description: 'x param' } } };
const RESPONSES_TOOL_DOC = `## Tool: do_thing\n${BIG_FLAT_TOOL_DESC}\n\`\`\`json\n${JSON.stringify(RESPONSES_TOOL_PARAMS)}\n\`\`\``;
describe('transformOpenAIResponses (gpt-5.6)', () => {
describe('transformOpenAIResponses (gpt-5.6-sol)', () => {
it('compresses GPT Responses instructions + tool docs while preserving native tool selection metadata', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_INSTRUCTIONS,
input: [
{ role: 'user', content: 'Please do the thing.' },
@@ -281,7 +305,7 @@ describe('transformOpenAIResponses (gpt-5.6)', () => {
// array form for a developer/system item used to be dropped: not imaged and
// not stubbed, so the verbose text rode uncompressed as native input.
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
input: [
{ role: 'developer', content: [{ type: 'input_text', text: BIG_INSTRUCTIONS }] },
{ role: 'user', content: 'Please do the thing.' },
@@ -307,7 +331,7 @@ describe('transformOpenAIResponses (gpt-5.6)', () => {
it('images GPT Responses tool definitions even when there is no instruction context', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
input: [{ role: 'user', content: 'Please do the thing.' }],
tools: [{
type: 'function',
@@ -328,7 +352,7 @@ describe('transformOpenAIResponses (gpt-5.6)', () => {
it('keeps a parameter literally named "description" (task-tool regression)', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
input: [{ role: 'user', content: 'Please do the thing.' }],
tools: [{ type: 'function', name: 'task', description: BIG_FLAT_TOOL_DESC, parameters: TASK_LIKE_PARAMS }],
}));
@@ -346,7 +370,7 @@ describe('transformOpenAIResponses (gpt-5.6)', () => {
it('handles bare string input (wraps into user item with images)', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_INSTRUCTIONS,
input: 'Do the thing please.',
}));
@@ -369,7 +393,7 @@ describe('transformOpenAIResponses (gpt-5.6)', () => {
it('records outgoingTextChars for compressed Responses requests, counting text but not image base64', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_INSTRUCTIONS,
input: [{ role: 'user', content: 'Please do the thing.' }],
tools: [{ type: 'function', name: 'do_thing', description: 'pick a thing', parameters: { type: 'object', properties: {} } }],
@@ -424,7 +448,7 @@ describe('transformOpenAIResponses (gpt-5.6)', () => {
it('returns compressed=false with not_profitable/below_min reason for small input', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: 'Short.',
input: [{ role: 'user', content: 'hi' }],
}));
@@ -488,7 +512,7 @@ function buildChatMessages(turns: number): Array<Record<string, unknown>> {
describe('transformOpenAIResponses — history collapse', () => {
it('collapses the OLD transcript prefix into history images, keeps the tail as text', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_SLAB,
input: buildResponsesInput(20),
}));
@@ -531,7 +555,7 @@ describe('transformOpenAIResponses — history collapse', () => {
it('produces a byte-stable history image sha across identical requests', async () => {
const make = () => enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_SLAB,
input: buildResponsesInput(20),
}));
@@ -543,7 +567,7 @@ describe('transformOpenAIResponses — history collapse', () => {
it('does not collapse when collapseHistory is off', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_SLAB,
input: buildResponsesInput(20),
}));
@@ -559,7 +583,7 @@ describe('transformOpenAIResponses — history collapse', () => {
it('partially collapses GPT history up to the image cap and leaves the rest as text', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_SLAB,
input: buildResponsesInput(30),
}));
@@ -586,7 +610,7 @@ describe('transformOpenAIResponses — history collapse', () => {
describe('transformOpenAIChatCompletions — history collapse', () => {
it('collapses the OLD transcript into a synthetic user message with image_url parts', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: buildChatMessages(20),
}));
const result = await transformOpenAIChatCompletions(body, { charsPerToken: 1, minCompressChars: 1 });
@@ -654,7 +678,7 @@ function buildAutonomousChat(turns: number): Array<Record<string, unknown>> {
describe('GPT history collapse — pins the live request as text (autonomous shape)', () => {
it('Responses: lone request kept as legible text + echoed in the guard, work imaged', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_SLAB,
input: buildAutonomousResponses(24),
}));
@@ -685,7 +709,7 @@ describe('GPT history collapse — pins the live request as text (autonomous sha
it('Chat: lone request kept as legible text + echoed in the guard, work imaged', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: buildAutonomousChat(24),
}));
const result = await transformOpenAIChatCompletions(body, { charsPerToken: 1, minCompressChars: 1 });
@@ -712,7 +736,7 @@ describe('GPT history collapse — pins the live request as text (autonomous sha
it('Responses: byte-stable history image sha across identical autonomous requests', async () => {
const make = () => enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_SLAB,
input: buildAutonomousResponses(24),
}));
@@ -732,18 +756,18 @@ describe('GPT history collapse — pins the live request as text (autonomous sha
describe('openAIVisionTokens — gpt-5.x flagship patch model', () => {
it('flagship multiplier is 1.0, not the mini 1.62', () => {
// 768x1932 → patches = ceil(768/32)*ceil(1932/32) = 24*61 = 1464; ×1.0 = 1464.
expect(openAIVisionTokens('gpt-5.6', 768, 1932)).toBe(1464);
expect(openAIVisionTokens('gpt-5.6-sol', 768, 1932)).toBe(1464);
expect(openAIVisionTokens('gpt-5.5', 768, 1932)).toBe(1464);
});
it('flagship patch budget is 10,000 (original detail), not 2,500', () => {
// 4000x4000 → patches = 125*125 = 15625, capped at the budget.
// Pre-fix (cap 2500, ×1.62) this returned 4050; correct is min(15625,10000)=10000.
expect(openAIVisionTokens('gpt-5.6', 4000, 4000)).toBe(10000);
expect(openAIVisionTokens('gpt-5.6-sol', 4000, 4000)).toBe(10000);
});
it('resolveVisionCost flagship = patch, multiplier 1, cap 10000; mini stays 1.62/1536', () => {
expect(resolveVisionCost('gpt-5.6')).toMatchObject({ regime: 'patch', multiplier: 1, patchCap: 10000 });
expect(resolveVisionCost('gpt-5.6-sol')).toMatchObject({ regime: 'patch', multiplier: 1, patchCap: 10000 });
expect(resolveVisionCost('gpt-5.5')).toMatchObject({ regime: 'patch', multiplier: 1, patchCap: 10000 });
expect(resolveVisionCost('gpt-5.6-mini')).toMatchObject({ regime: 'patch', multiplier: 1.62, patchCap: 1536 });
expect(resolveVisionCost('gpt-5.6-nano')).toMatchObject({ regime: 'patch', multiplier: 2.46, patchCap: 1536 });
@@ -755,7 +779,7 @@ describe('openAIVisionTokens — gpt-5.x flagship patch model', () => {
describe('image parts request detail = "original" (avoid downscale of dense text)', () => {
it('Chat Completions image_url parts use detail:"original"', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [
{ role: 'system', content: BIG_SYSTEM },
{ role: 'user', content: 'hello' },
@@ -773,7 +797,7 @@ describe('image parts request detail = "original" (avoid downscale of dense text
it('Responses input_image parts use detail:"original"', async () => {
const body = enc.encode(JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: BIG_INSTRUCTIONS,
input: [{ role: 'user', content: [{ type: 'input_text', text: 'hello' }] }],
}));
@@ -799,21 +823,101 @@ describe('resolveGptProfile (Claude on Responses)', () => {
expect(p.stripCols).toBe(312);
expect(resolveGptProfile('claude-fable-5').maxHeightPx).toBe(728);
expect(resolveGptProfile('claude-fable-5').stripCols).toBe(312);
expect(resolveGptProfile('gpt-5.6').maxHeightPx).toBe(1932);
expect(resolveGptProfile('gpt-5.6').stripCols).toBe(152);
for (const model of [
'gpt-5.6-sol',
'gpt-5.6-sol[1m]',
'gpt-5.6-sol-codex',
'gpt-5.6-sol-codex[1m]',
'gpt-5.6-sol-2026-07-09',
]) {
const sol = resolveGptProfile(model);
expect(sol.maxHeightPx, model).toBe(1932);
expect(sol.stripCols, model).toBe(126);
expect(sol.style.font, model).toBe('jetbrains-mono-10');
}
for (const model of ['gpt-5.6', 'gpt-5.6-terra', 'gpt-5.6-terra[1m]']) {
const notSol = resolveGptProfile(model);
expect(notSol.stripCols, model).toBe(152);
expect(notSol.style.font, model).toBe('spleen-5x8');
}
expect(resolveGptProfile('claude-fable-5').style.font).toBe('spleen-5x8');
});
it('renders Claude Responses at Claude width instead of the GPT default cap', async () => {
const body = enc.encode(JSON.stringify({
model: 'claude-fable-5',
instructions: BIG_INSTRUCTIONS,
input: [{ role: 'user', content: 'hello' }],
}));
const result = await transformOpenAIResponses(body, { charsPerToken: 1, minCompressChars: 1 });
expect(result.info.compressed).toBe(true);
expect(result.info.firstImageWidth).toBe(1568);
});
});
describe('resolveGptProfile (Grok)', () => {
it('uses production 5x8 packing; exact IDs ride the fact-sheet, not cell padding', () => {
// Dense packing maximizes measured Grok savings (~1000 tok/MPix). Exact
// tokens (paths/hex/ports/camelCase) are kept as text beside the image.
it('uses the measured opt-in 9x12 profile plus the fact-sheet', () => {
// Live climb 2026-07-09: 5x8 was 0/4 exact with four confabulations;
// effective 9x12 was the densest arm at 4/4 exact and zero confabulation.
const p = resolveGptProfile('grok-4.5');
expect(p.stripCols).toBe(152);
expect(p.style?.cellWBonus).toBe(0);
expect(p.style?.cellHBonus).toBe(0);
expect(p.style?.aa).toBe(true);
expect(resolveGptProfile('grok-4').stripCols).toBe(152);
expect(p.stripCols).toBe(84);
expect(p.style.font).toBe('spleen-5x8');
expect(p.style.cellWBonus).toBe(4);
expect(p.style.cellHBonus).toBe(4);
expect(p.style.aa).toBe(true);
expect(resolveGptProfile('grok-4').stripCols).toBe(84);
});
it('renders the opt-in profile at 764px wide', async () => {
const body = enc.encode(JSON.stringify({
model: 'grok-4.5',
instructions: BIG_INSTRUCTIONS,
input: [{ role: 'user', content: 'hello' }],
}));
const result = await transformOpenAIResponses(body, { charsPerToken: 1, minCompressChars: 1 });
expect(result.info.compressed).toBe(true);
expect(result.info.firstImageWidth).toBe(764);
});
});
describe('resolveGptProfile style overrides', () => {
it('merges every render knob into the selected model profile', () => {
const prev = process.env.PXPIPE_GPT_PROFILES;
try {
process.env.PXPIPE_GPT_PROFILES = JSON.stringify({
'gpt-5.6-sol': {
stripCols: 100,
style: {
font: 'spleen-5x8',
cellWBonus: 2,
cellHBonus: 3,
aa: false,
grid: true,
gridCols: 4,
colorCycle: true,
markerScale: 2,
markerRed: true,
},
},
});
expect(resolveGptProfile('gpt-5.6-sol-codex')).toMatchObject({
stripCols: 100,
style: {
font: 'spleen-5x8',
cellWBonus: 2,
cellHBonus: 3,
aa: false,
grid: true,
gridCols: 4,
colorCycle: true,
markerScale: 2,
markerRed: true,
},
});
} finally {
if (prev === undefined) delete process.env.PXPIPE_GPT_PROFILES;
else process.env.PXPIPE_GPT_PROFILES = prev;
}
});
});
@@ -828,4 +932,3 @@ describe('visionTokensForModel (Grok)', () => {
);
});
});
tests/proxy-usage.test.ts
+19 -6
View File
@@ -1,6 +1,19 @@
import { describe, it, expect } from 'vitest';
import { afterAll, beforeAll, describe, it, expect } from 'vitest';
import { createProxy, type ProxyEvent } from '../src/core/proxy.js';
// These proxy-contract tests deliberately exercise the opt-in Sol transform.
// Snapshot the developer shell so the suite is deterministic now that Sol is
// intentionally absent from the built-in default scope.
let ambientPxpipeModels: string | undefined;
beforeAll(() => {
ambientPxpipeModels = process.env.PXPIPE_MODELS;
process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6-sol';
});
afterAll(() => {
if (ambientPxpipeModels === undefined) delete process.env.PXPIPE_MODELS;
else process.env.PXPIPE_MODELS = ambientPxpipeModels;
});
/** Tiny in-process mock upstream — accepts any request and returns whatever
* the test fixture configured. Lets us assert that the proxy correctly
* extracts Anthropic's usage block from both SSE and JSON responses without
@@ -133,7 +146,7 @@ describe('proxy usage extraction', () => {
).toBe(true);
});
it('routes GPT 5.6 chat completions to OpenAI, transforms once, and normalizes usage', async () => {
it('routes GPT 5.6 Sol chat completions to OpenAI, transforms once, and normalizes usage', async () => {
const upstreamRequests: Request[] = [];
const restore = mockUpstream(async (req) => {
upstreamRequests.push(req.clone());
@@ -159,7 +172,7 @@ describe('proxy usage extraction', () => {
});
const reqBody = JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [
{ role: 'system', content: 'System instruction. '.repeat(900) },
{ role: 'user', content: 'hi' },
@@ -220,7 +233,7 @@ describe('proxy usage extraction', () => {
});
const reqBody = JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [
{ role: 'system', content: 'System instruction. '.repeat(900) },
{ role: 'user', content: 'hi' },
@@ -271,7 +284,7 @@ describe('proxy usage extraction', () => {
});
const reqBody = JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
instructions: 'System instruction. '.repeat(900),
input: [{ role: 'user', content: 'hi' }],
});
@@ -293,7 +306,7 @@ describe('proxy usage extraction', () => {
const sent = JSON.parse(await upstreamRequests[0]!.text()) as any;
const firstUser = sent.input.find((m: any) => m.role === 'user');
expect(firstUser.content[0].type).toBe('input_image');
expect(captured?.model).toBe('gpt-5.6');
expect(captured?.model).toBe('gpt-5.6-sol');
expect(captured?.info?.compressed).toBe(true);
expect(captured?.info?.firstUserSha8).toMatch(/^[0-9a-f]{8}$/);
});
tests/public-api.test.ts
+20 -17
View File
@@ -89,24 +89,27 @@ describe('public library API', () => {
}
});
it('recognizes GPT 5.6 as the default OpenAI imaging scope (5.5 opt-in)', () => {
it('keeps GPT 5.6 Sol off by default but preserves exact opt-in aliases', () => {
expect(isPxpipeSupportedGptModel('gpt-5')).toBe(false);
// gpt-5.5 degrades on imaged context, so it is off by default now.
expect(isPxpipeSupportedGptModel('gpt-5.5')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.5-codex')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.5-2026-06-01')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol-codex')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-terra')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5-mini')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-nano')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6[1m]')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-4o')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-50')).toBe(false);
expect(isPxpipeSupportedGptModel('')).toBe(false);
expect(isPxpipeSupportedGptModel('claude-opus-4-8')).toBe(false);
expect(isPxpipeSupportedGptModel(null)).toBe(false);
process.env.PXPIPE_MODELS = 'gpt-5.6-sol';
expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol-codex')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol[1m]')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol-codex[1m]')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-terra')).toBe(false);
});
it('keeps Grok opt-in only (off by default, like Opus)', () => {
it('keeps Grok and Sol opt-in only (off by default, like Opus)', () => {
// Pure-image exact OCR fails at production 5×8; do not image Grok unless
// the operator opts in via PXPIPE_MODELS or the dashboard chip.
const prev = process.env.PXPIPE_MODELS;
@@ -116,9 +119,9 @@ describe('public library API', () => {
expect(isPxpipeSupportedGptModel('grok-4')).toBe(false);
expect(isPxpipeSupportedGptModel('grok-4.20')).toBe(false);
expect(getAllowedModelBases()).not.toContain('grok-4.5');
expect(getAllowedModelBases()).toEqual(['claude-fable-5', 'gpt-5.6']);
expect(getAllowedModelBases()).toEqual(['claude-fable-5']);
process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6,grok-4.5';
process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6-sol,grok-4.5';
expect(isPxpipeSupportedGptModel('grok-4.5')).toBe(true);
expect(isPxpipeSupportedGptModel('grok-4.5-fast')).toBe(true); // -suffix alias
} finally {
@@ -133,17 +136,17 @@ describe('public library API', () => {
// Explicit Claude-only scope disables GPT imaging.
process.env.PXPIPE_MODELS = 'claude-fable-5';
expect(isPxpipeSupportedGptModel('gpt-5.5')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(false);
// Mixed CSV selects exactly those bases across families.
process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6';
process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6-sol';
expect(isPxpipeSupportedGptModel('gpt-5.5')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(true);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(true);
expect(isPxpipeSupportedModel('claude-fable-5')).toBe(true);
// `off` disables everything.
process.env.PXPIPE_MODELS = 'off';
expect(isPxpipeSupportedGptModel('gpt-5.6')).toBe(false);
expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(false);
expect(isPxpipeSupportedModel('claude-fable-5')).toBe(false);
} finally {
if (prev === undefined) delete process.env.PXPIPE_MODELS;
tests/render.test.ts
+27
View File
@@ -12,6 +12,8 @@ import {
SLOT_MARK_USER,
SLOT_MARK_ASSISTANT,
ROLE_PALETTE,
renderCellHeight,
renderCellWidth,
CELL_H,
CELL_W,
} from '../src/core/render.js';
@@ -42,6 +44,31 @@ import {
synthesizeText,
} from './fixtures/real-shapes.js';
describe('model-selectable font atlases', () => {
it('uses the JetBrains Mono 10 cell geometry without changing the default atlas', async () => {
expect(renderCellWidth({ aa: true })).toBe(5);
expect(renderCellHeight({ aa: true })).toBe(8);
expect(renderCellWidth({ font: 'jetbrains-mono-10', aa: true })).toBe(6);
expect(renderCellHeight({ font: 'jetbrains-mono-10', aa: true })).toBe(11);
const text = 'tokenLedgerShard a3f9c1e0b7d2';
const defaultImg = await renderChunkToPng(text, 40, { aa: true });
const solImg = await renderChunkToPng(text, 40, { font: 'jetbrains-mono-10', aa: true });
expect(defaultImg.width).toBe(208);
expect(solImg.width).toBe(248);
expect(solImg.height).toBeGreaterThan(defaultImg.height);
expect(Buffer.from(solImg.png)).not.toEqual(Buffer.from(defaultImg.png));
});
it('falls back to the full Spleen/Unifont atlas for Unicode outside the compact Sol atlas', async () => {
const img = await renderChunkToPng('한글 test', 20, {
font: 'jetbrains-mono-10',
aa: true,
});
expect(img.droppedChars).toBe(0);
});
});
describe('compactSlabWhitespace', () => {
it('returns empty string unchanged', () => {
expect(compactSlabWhitespace('')).toBe('');
tests/savings-honesty.test.ts
+3 -3
View File
@@ -27,7 +27,7 @@ import {
openAICacheReadRate,
} from '../src/core/openai-savings.js';
const GPT = 'gpt-5.6';
const GPT = 'gpt-5.6-sol';
// ===========================================================================
describe('GPT savings honesty (vs the real o200k cached-rate model)', () => {
@@ -160,7 +160,7 @@ describe('per-model pricing is applied correctly (Fable vs Opus vs GPT)', () =>
// Both bill cache reads at 0.1x. Pricing an unrelated GPT row at the
// aggressive 0.1x would overstate its cache savings, so the fallback stays
// 0.5x. The gate keeps families from bleeding each other's rates.
expect(openAICacheReadRate('gpt-5.6')).toBe(0.1);
expect(openAICacheReadRate('gpt-5.6-sol')).toBe(0.1);
expect(openAICacheReadRate('gpt-5.5')).toBe(0.1);
// claude models arrive here through the bridge and must use Anthropic's rate.
expect(openAICacheReadRate('claude-opus-4-8')).toBe(0.1);
@@ -174,6 +174,6 @@ describe('per-model pricing is applied correctly (Fable vs Opus vs GPT)', () =>
// Guard against a refactor that unifies them: they are the same number today
// for different reasons (GPT cached-input vs Anthropic cache_read). If one
// provider changes, only its own constant should move.
expect(openAICacheReadRate('gpt-5.6')).toBe(CACHE_READ_RATE);
expect(openAICacheReadRate('gpt-5.6-sol')).toBe(CACHE_READ_RATE);
});
});
tests/savings-math-e2e.test.ts
+1 -1
View File
@@ -96,7 +96,7 @@ const slab = (n: number) =>
const gptBody = (sysChars: number) =>
JSON.stringify({
model: 'gpt-5.6',
model: 'gpt-5.6-sol',
messages: [
{ role: 'system', content: slab(sysChars) },
{ role: 'user', content: 'hello' },