eval: complete Sol and Grok quality runs

FINDINGS.md

@@ -8,17 +8,16 @@
 
 ---
 
-## Update (2026-07-11) — Sol 5×8 evaluated, remains opt-in
+## Update (2026-07-17) — Sol 5×8 evaluated, remains opt-in
 
 The exact `gpt-5.6-sol` profile now uses Spleen 5×8 at 152 columns. It remains
 an explicit opt-in. Fresh novel arithmetic scored 96/100 pure-image and 98/100
 with the production factsheet. Matched input usage was 5,300 text tokens versus
 7,000 production-image tokens, 32% more on this short workload.
 
-On the portable gist corpus, Sol scored 79/93 completed answerable probes,
-18/18 state probes, and 4/15 completed never-stated confabulations. One
-six-probe session failed at the gateway and is reported as a transport error,
-not six model misses. Dense 12-character hex recall scored 0/15.
+On the portable gist corpus, Sol scored 83/98 answerable probes, 17/18 state
+probes, and 4/16 never-stated confabulations. All sessions completed without
+transport errors. Dense 12-character hex recall scored 0/15.
 
 These results keep Sol below the Fable default bar. Native recent/open state and
 the verbatim factsheet remain required guards. Sibling `gpt-5.6-*` ids do not

README.md

@@ -141,8 +141,8 @@ without running the proxy.
 - **Per-model rendering:** opt-in `gpt-5.6-sol` uses a 152-column,
   5×8 Spleen profile; Claude keeps its 312-column 5×8 Spleen profile. These
   are selected by exact model id, including history pages and profitability
-  math. **Sol quality:** production 5×8 scored 98/100 arithmetic and 79/93
-  completed gist, 18/18 state, 4/15 completed never-stated confabulations,
+  math. **Sol quality:** production 5×8 scored 98/100 arithmetic and 83/98
+  gist, 17/18 state, 4/16 never-stated confabulations,
   and 0/15 dense hex. Exact IDs therefore use the verbatim factsheet, and recent/open tool state stays native.
   [Sol receipts](eval/sol-profile/QUALITY_RESULTS.md) and
   [profile evidence](docs/MODEL_RENDER_PROFILES.md).
@@ -175,18 +175,18 @@ used for these novel-arithmetic rows.
 | novel arithmetic | `grok-4.5` | 100 | **82%** |
 | novel arithmetic | `moonshotai/kimi-k3` | 100 | **79%** |
 | gist recall A/B (decisions, values, paths, names, negations; distractors; 15k–45k char sessions) | Fable 5 | 98/arm | **98/98** |
-| same gist corpus, production images + factsheet | `gpt-5.6-sol` | 98 | **79/93 completed; 1 session error** |
+| same gist corpus, production images + factsheet | `gpt-5.6-sol` | 98 | **83/98** |
 | same gist corpus, production images + factsheet | `grok-4.5` | 98 | **83/98** |
 | state tracking (value mutated 3×, final/first/count) | Fable 5 | 18/arm | **18/18** |
-| same state-tracking corpus | `gpt-5.6-sol` | 18 | **18/18 latest** |
+| same state-tracking corpus | `gpt-5.6-sol` | 18 | **17/18** |
 | same state-tracking corpus | `grok-4.5` | 18 | **13/18** |
 | confabulation on never-stated facts (lower is better) | Fable 5 | 16/arm | **0/16** |
-| same never-stated probes (lower is better) | `gpt-5.6-sol` | 16 | **4/15 completed; 1 session error** |
+| same never-stated probes (lower is better) | `gpt-5.6-sol` | 16 | **4/16** |
 | same never-stated probes (lower is better) | `grok-4.5` | 16 | **0/16** |
 | verbatim 12-char hex, dense render | Opus | 15 | **0/15** |
 | verbatim 12-char hex, dense render | Fable 5 | 15 | **13/15** |
 | verbatim 12-char hex, same dense pages | `gpt-5.6-sol` | 15 | **0/15** |
-| verbatim 12-char hex, same dense pages | `grok-4.5` | 15 | **0/6 completed; 9 transport errors** |
+| verbatim 12-char hex, same dense pages | `grok-4.5` | 15 | **0/15** |
 
 **Harness split:** Fable/Opus quality and SWE-bench rows use **Claude**; Sol and Grok quality use
 **Codex’s Responses provider** (`OPENAI_BASE_URL`). Kimi K3
@@ -304,7 +304,7 @@ Everything else passes through byte-identical: your messages, recent turns,
 the model's output (it is the response, the proxy never touches it), sparse
 prose, and anything too small to win. Fable 5 is the only built-in default. Sol,
 Opus, GPT 5.5, and Grok remain explicit opt-ins. Sol's production 5×8 run
-scored 98/100 arithmetic, 79/93 completed gist, 18/18 state, 4/15 completed
+scored 98/100 arithmetic, 83/98 gist, 17/18 state, 4/16
 never-stated confabulations, and 0/15 dense hex. Grok scored 82/100
 arithmetic, 83/98 gist, and 13/18 state.
 

docs/MODEL_RENDER_PROFILES.md

@@ -7,7 +7,7 @@ from the model id.
 | model rule | default | cell | columns | max height | evidence |
 |---|:---:|---|---:|---:|---|
 | `claude-fable-5*` | yes | Spleen 5×8 | 312 | 728 px | established Claude suites |
-| `gpt-5.6-sol*` | opt-in | Spleen 5×8 | 152 | 1932 px | 98/100 production arithmetic; 79/93 completed gist |
+| `gpt-5.6-sol*` | opt-in | Spleen 5×8 | 152 | 1932 px | 98/100 production arithmetic; 83/98 gist |
 | `grok-*` | opt-in | Spleen 5×8 | 152 | 512 px | 82/100 arithmetic; 83/98 gist |
 | other GPT/o-series | opt-in | Spleen 5×8 | 152 | 1932 px | conservative fallback |
 

eval/grok-density/QUALITY_RESULTS.md

@@ -9,7 +9,7 @@ pxpipe and used the production 5×8 profile.
 | gist recall | 98/98 | 83/98 | no transport errors |
 | state tracking | 18/18 | 13/18 | subset of the gist corpus |
 | never-stated guards | 0/16 confabulated | 0/16 confabulated | lower is better |
-| dense 12-char hex | 15/15 | 0/6 completed | 9 image calls failed at the gateway or timed out |
+| dense 12-char hex | 15/15 | 0/15 | all calls completed after retrying transient upstream failures |
 
 Novel-arithmetic input usage was recovered from the provider log for the same
 N=100 run: 25,400 text tokens and 27,100 production-image tokens, **+6.7%**.
@@ -23,6 +23,6 @@ Receipts:
 - `../sol-profile/verbatim-hex-grok-4.5-results.json`
 - `../sol-profile/verbatim-hex-grok-4.5-text-results.json`
 
-Grok remains opt-in because arithmetic, gist, and state tracking are below the
-Fable bar. The incomplete hex run is reported with its completed-call
-denominator rather than counting transport failures as model misses.
+Grok remains opt-in because arithmetic, gist, state tracking, and dense exact
+recall are below the Fable bar. The completed hex rerun used an 8,192-token
+output cap so mandatory reasoning could not consume the final answer budget.

eval/sol-profile/QUALITY_RESULTS.md

@@ -11,9 +11,9 @@ adjacent text factsheet.
 | test | text | production image | notes |
 |---|---:|---:|---|
 | novel arithmetic, N=100 | 100/100 | 98/100 | pure image 96/100 |
-| gist recall | not measured | 79/93 completed | one six-probe session failed at the gateway |
-| state tracking | not measured | 18/18 | no transport errors |
-| never-stated guards | not measured | 4/15 completed confabulated | one guard shared the failed session |
+| gist recall | not measured | 83/98 | no transport errors |
+| state tracking | not measured | 17/18 | no transport errors |
+| never-stated guards | not measured | 4/16 confabulated | no transport errors |
 | dense 12-char hex | not run in this harness | 0/15 | all calls completed |
 
 Matched arithmetic usage was 5,300 text input tokens and 7,000 production-image

eval/sol-profile/responses-client.mjs

@@ -152,6 +152,9 @@ export async function callResponses({ model, content, maxOutputTokens, timeoutMs
     if (!response.ok) {
       throw new Error(`HTTP ${response.status}: ${json?.error?.message || raw.slice(0, 160)}`);
     }
+    if (json.status === 'incomplete') {
+      throw new Error(`incomplete response: ${json.incomplete_details?.reason || 'unknown reason'}`);
+    }
     let text = typeof json.output_text === 'string' ? json.output_text : '';
     if (!text && Array.isArray(json.output)) {
       for (const item of json.output) {

eval/sol-profile/verbatim-hex-grok-4.5-results.json

@@ -1,8 +1,10 @@
 {
-  "generatedAt": "2026-07-11T20:47:49.180Z",
+  "generatedAt": "2026-07-17T22:28:51.062Z",
   "model": "grok-4.5",
   "live": true,
   "correct": 0,
+  "completed": 15,
+  "errors": 0,
   "n": 15,
   "rows": [
     {
@@ -11,29 +13,29 @@
       "gold": "c9c947f680ec",
       "got": "",
       "ok": false,
-      "raw": "c9c4176684",
-      "ms": 21274,
+      "raw": "c9c417668e",
+      "ms": 40361,
       "error": null
     },
     {
       "page": 0,
       "dur": 812,
       "gold": "851eb3af1bd1",
-      "got": "36efd6a5e76e",
+      "got": "",
       "ok": false,
-      "raw": "36efd6a5e76e",
-      "ms": 2477,
+      "raw": "8e1eba4f1d",
+      "ms": 62406,
       "error": null
     },
     {
       "page": 0,
       "dur": 6150,
       "gold": "ade34f70fd73",
-      "got": "",
+      "got": "9e0e7d1a59bc",
       "ok": false,
-      "raw": "non-json HTTP 503: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endi",
-      "ms": null,
-      "error": "non-json HTTP 503: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endi"
+      "raw": "9e0e7d1a59bc",
+      "ms": 86829,
+      "error": null
     },
     {
       "page": 1,
@@ -41,19 +43,19 @@
       "gold": "c5d68855f46d",
       "got": "",
       "ok": false,
-      "raw": "This operation was aborted",
-      "ms": null,
-      "error": "This operation was aborted"
+      "raw": "e26595f6d",
+      "ms": 104810,
+      "error": null
     },
     {
       "page": 1,
       "dur": 8071,
       "gold": "92abade01aad",
-      "got": "",
+      "got": "a1c179b9e9e1",
       "ok": false,
-      "raw": "This operation was aborted",
-      "ms": null,
-      "error": "This operation was aborted"
+      "raw": "a1c179b9e9e1",
+      "ms": 2686,
+      "error": null
     },
     {
       "page": 1,
@@ -61,9 +63,9 @@
       "gold": "ffe21785b09d",
       "got": "",
       "ok": false,
-      "raw": "This operation was aborted",
-      "ms": null,
-      "error": "This operation was aborted"
+      "raw": "319562bac5",
+      "ms": 178195,
+      "error": null
     },
     {
       "page": 2,
@@ -71,9 +73,9 @@
       "gold": "87cb51eb0e99",
       "got": "",
       "ok": false,
-      "raw": "This operation was aborted",
-      "ms": null,
-      "error": "This operation was aborted"
+      "raw": "8c311f1b107",
+      "ms": 39440,
+      "error": null
     },
     {
       "page": 2,
@@ -81,9 +83,9 @@
       "gold": "93c3ced96dac",
       "got": "",
       "ok": false,
-      "raw": "non-json HTTP 503: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endi",
-      "ms": null,
-      "error": "non-json HTTP 503: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endi"
+      "raw": "e0c4ed0da",
+      "ms": 95645,
+      "error": null
     },
     {
       "page": 2,
@@ -91,9 +93,9 @@
       "gold": "f152ae9bfb8f",
       "got": "",
       "ok": false,
-      "raw": "This operation was aborted",
-      "ms": null,
-      "error": "This operation was aborted"
+      "raw": "aecade0cb",
+      "ms": 89792,
+      "error": null
     },
     {
       "page": 3,
@@ -101,9 +103,9 @@
       "gold": "5a7373d4187f",
       "got": "",
       "ok": false,
-      "raw": "non-json HTTP 503: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endi",
-      "ms": null,
-      "error": "non-json HTTP 503: <!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ie7 oldie\" lang=\"en-US\"> <![endi"
+      "raw": "55773d1981",
+      "ms": 85904,
+      "error": null
     },
     {
       "page": 3,
@@ -111,8 +113,8 @@
       "gold": "44ea8c7aeedd",
       "got": "",
       "ok": false,
-      "raw": "4eac37aedd",
-      "ms": 45790,
+      "raw": "e4e0c7aeed",
+      "ms": 48068,
       "error": null
     },
     {
@@ -121,18 +123,18 @@
       "gold": "8145b5a0fd46",
       "got": "",
       "ok": false,
-      "raw": "This operation was aborted",
-      "ms": null,
-      "error": "This operation was aborted"
+      "raw": "a5d1d98f0e",
+      "ms": 221591,
+      "error": null
     },
     {
       "page": 4,
       "dur": 2921,
       "gold": "b8fce698f971",
-      "got": "",
+      "got": "b0fce68f97f1",
       "ok": false,
-      "raw": "b0fce68f971",
-      "ms": 22070,
+      "raw": "b0fce68f97f1",
+      "ms": 3421,
       "error": null
     },
     {
@@ -141,8 +143,8 @@
       "gold": "4a8164556b99",
       "got": "",
       "ok": false,
-      "raw": "a816655b0a",
-      "ms": 29812,
+      "raw": "a3d16a55b0a",
+      "ms": 148181,
       "error": null
     },
     {
@@ -152,10 +154,8 @@
       "got": "",
       "ok": false,
       "raw": "a316a558ba",
-      "ms": 30782,
+      "ms": 22717,
       "error": null
     }
-  ],
-  "completed": 6,
-  "errors": 9
-}
+  ]
+}

eval/sol-profile/verbatim-hex.mjs

@@ -8,16 +8,25 @@ const ROOT=join(HERE,'../verbatim-15');
 const MODEL=process.env.SOL_QUALITY_MODEL||process.env.MODEL||'gpt-5.6-sol';
 const LIVE=process.env.SOL_QUALITY_LIVE==='1';
 const TIMEOUT=Number(process.env.SOL_QUALITY_TIMEOUT_MS||90000);
+const MAX_OUTPUT_TOKENS=Number(process.env.SOL_QUALITY_MAX_OUTPUT_TOKENS||80);
+const RETRIES=Math.max(0,Number(process.env.SOL_QUALITY_RETRIES||0));
 const trials=JSON.parse(readFileSync(join(ROOT,'golds.json'),'utf8'));
 const RESULT=join(HERE, MODEL==='gpt-5.6-sol' ? 'verbatim-hex-results.json' : 'verbatim-hex-'+MODEL.replace(/[^a-zA-Z0-9._-]+/g,'_')+'-results.json');
 
+function writeResult(rows){
+  const completed=rows.filter(r=>!r.error);
+  const result={generatedAt:new Date().toISOString(),model:MODEL,live:LIVE,correct:completed.filter(r=>r.ok).length,completed:completed.length,errors:rows.length-completed.length,n:trials.length,rows};
+  writeFileSync(RESULT, JSON.stringify(result,null,2));
+  return result;
+}
+
 async function callImage(trial){
   const png=readFileSync(join(ROOT,`page${trial.page}.png`));
   const content=[
     {type:'input_image',image_url:`data:image/png;base64,${png.toString('base64')}`,detail:'original'},
     {type:'input_text',text:`Read the image visually. Find the JSON line whose dur_ms is exactly ${trial.dur}. Return only its id field, exactly 12 lowercase hex characters.`},
   ];
-  return callResponses({model:MODEL,content,maxOutputTokens:80,timeoutMs:TIMEOUT});
+  return callResponses({model:MODEL,content,maxOutputTokens:MAX_OUTPUT_TOKENS,timeoutMs:TIMEOUT});
 }
 
 const rows=[];
@@ -25,21 +34,26 @@ for(let i=0;i<trials.length;i++){
   const t=trials[i];
   let out='', ms=null, err=null;
   process.stdout.write(`trial ${i+1}/${trials.length} page${t.page} dur=${t.dur} ... `);
-  try{
-    if(LIVE){
-      const r=await callImage(t);
-      out=r.text; ms=r.ms;
+  for(let attempt=0;attempt<=RETRIES;attempt++){
+    try{
+      if(LIVE){
+        const r=await callImage(t);
+        out=r.text; ms=r.ms;
+      }
+      err=null;
+      break;
+    }catch(e){
+      err=String(e?.message||e);
+      out=err;
+      if(attempt<RETRIES)console.log(`retry ${attempt+1}/${RETRIES}: ${err.slice(0,100)}`);
     }
-  }catch(e){
-    err=String(e?.message||e);
-    out=err;
   }
   const got=out.match(/[0-9a-f]{12}/i)?.[0]?.toLowerCase()||'';
   const ok=got===t.gold;
   rows.push({...t,got,ok,raw:out,ms,error:err});
+  if(LIVE)writeResult(rows);
   console.log(`${ok?'HIT':'MISS'} gold=${t.gold} got=${got||'-'}${ms!=null?` ${ms}ms`:''}${err?` ERR ${err.slice(0,100)}`:''}`);
 }
 if(!LIVE){console.log('dry run only; no receipt written');process.exit(0)}
-const completed=rows.filter(r=>!r.error),result={generatedAt:new Date().toISOString(),model:MODEL,live:LIVE,correct:completed.filter(r=>r.ok).length,completed:completed.length,errors:rows.length-completed.length,n:rows.length,rows};
-writeFileSync(RESULT, JSON.stringify(result,null,2));
+const result=writeResult(rows);
 console.log(`SUMMARY ${result.correct}/${result.completed} completed (${result.errors} errors) -> ${RESULT}`);