diff --git a/FINDINGS.md b/FINDINGS.md index 8880373..b2e4d02 100644 --- a/FINDINGS.md +++ b/FINDINGS.md @@ -8,17 +8,16 @@ --- -## Update (2026-07-11) — Sol 5×8 evaluated, remains opt-in +## Update (2026-07-17) — Sol 5×8 evaluated, remains opt-in The exact `gpt-5.6-sol` profile now uses Spleen 5×8 at 152 columns. It remains an explicit opt-in. Fresh novel arithmetic scored 96/100 pure-image and 98/100 with the production factsheet. Matched input usage was 5,300 text tokens versus 7,000 production-image tokens, 32% more on this short workload. -On the portable gist corpus, Sol scored 79/93 completed answerable probes, -18/18 state probes, and 4/15 completed never-stated confabulations. One -six-probe session failed at the gateway and is reported as a transport error, -not six model misses. Dense 12-character hex recall scored 0/15. +On the portable gist corpus, Sol scored 83/98 answerable probes, 17/18 state +probes, and 4/16 never-stated confabulations. All sessions completed without +transport errors. Dense 12-character hex recall scored 0/15. These results keep Sol below the Fable default bar. Native recent/open state and the verbatim factsheet remain required guards. Sibling `gpt-5.6-*` ids do not diff --git a/README.md b/README.md index e5a378e..2d1db48 100644 --- a/README.md +++ b/README.md @@ -141,8 +141,8 @@ without running the proxy. - **Per-model rendering:** opt-in `gpt-5.6-sol` uses a 152-column, 5×8 Spleen profile; Claude keeps its 312-column 5×8 Spleen profile. These are selected by exact model id, including history pages and profitability - math. **Sol quality:** production 5×8 scored 98/100 arithmetic and 79/93 - completed gist, 18/18 state, 4/15 completed never-stated confabulations, + math. **Sol quality:** production 5×8 scored 98/100 arithmetic and 83/98 + gist, 17/18 state, 4/16 never-stated confabulations, and 0/15 dense hex. Exact IDs therefore use the verbatim factsheet, and recent/open tool state stays native. [Sol receipts](eval/sol-profile/QUALITY_RESULTS.md) and [profile evidence](docs/MODEL_RENDER_PROFILES.md). @@ -175,18 +175,18 @@ used for these novel-arithmetic rows. | novel arithmetic | `grok-4.5` | 100 | **82%** | | novel arithmetic | `moonshotai/kimi-k3` | 100 | **79%** | | gist recall A/B (decisions, values, paths, names, negations; distractors; 15k–45k char sessions) | Fable 5 | 98/arm | **98/98** | -| same gist corpus, production images + factsheet | `gpt-5.6-sol` | 98 | **79/93 completed; 1 session error** | +| same gist corpus, production images + factsheet | `gpt-5.6-sol` | 98 | **83/98** | | same gist corpus, production images + factsheet | `grok-4.5` | 98 | **83/98** | | state tracking (value mutated 3×, final/first/count) | Fable 5 | 18/arm | **18/18** | -| same state-tracking corpus | `gpt-5.6-sol` | 18 | **18/18 latest** | +| same state-tracking corpus | `gpt-5.6-sol` | 18 | **17/18** | | same state-tracking corpus | `grok-4.5` | 18 | **13/18** | | confabulation on never-stated facts (lower is better) | Fable 5 | 16/arm | **0/16** | -| same never-stated probes (lower is better) | `gpt-5.6-sol` | 16 | **4/15 completed; 1 session error** | +| same never-stated probes (lower is better) | `gpt-5.6-sol` | 16 | **4/16** | | same never-stated probes (lower is better) | `grok-4.5` | 16 | **0/16** | | verbatim 12-char hex, dense render | Opus | 15 | **0/15** | | verbatim 12-char hex, dense render | Fable 5 | 15 | **13/15** | | verbatim 12-char hex, same dense pages | `gpt-5.6-sol` | 15 | **0/15** | -| verbatim 12-char hex, same dense pages | `grok-4.5` | 15 | **0/6 completed; 9 transport errors** | +| verbatim 12-char hex, same dense pages | `grok-4.5` | 15 | **0/15** | **Harness split:** Fable/Opus quality and SWE-bench rows use **Claude**; Sol and Grok quality use **Codex’s Responses provider** (`OPENAI_BASE_URL`). Kimi K3 @@ -304,7 +304,7 @@ Everything else passes through byte-identical: your messages, recent turns, the model's output (it is the response, the proxy never touches it), sparse prose, and anything too small to win. Fable 5 is the only built-in default. Sol, Opus, GPT 5.5, and Grok remain explicit opt-ins. Sol's production 5×8 run -scored 98/100 arithmetic, 79/93 completed gist, 18/18 state, 4/15 completed +scored 98/100 arithmetic, 83/98 gist, 17/18 state, 4/16 never-stated confabulations, and 0/15 dense hex. Grok scored 82/100 arithmetic, 83/98 gist, and 13/18 state. diff --git a/docs/MODEL_RENDER_PROFILES.md b/docs/MODEL_RENDER_PROFILES.md index 02052bc..3d7c14c 100644 --- a/docs/MODEL_RENDER_PROFILES.md +++ b/docs/MODEL_RENDER_PROFILES.md @@ -7,7 +7,7 @@ from the model id. | model rule | default | cell | columns | max height | evidence | |---|:---:|---|---:|---:|---| | `claude-fable-5*` | yes | Spleen 5×8 | 312 | 728 px | established Claude suites | -| `gpt-5.6-sol*` | opt-in | Spleen 5×8 | 152 | 1932 px | 98/100 production arithmetic; 79/93 completed gist | +| `gpt-5.6-sol*` | opt-in | Spleen 5×8 | 152 | 1932 px | 98/100 production arithmetic; 83/98 gist | | `grok-*` | opt-in | Spleen 5×8 | 152 | 512 px | 82/100 arithmetic; 83/98 gist | | other GPT/o-series | opt-in | Spleen 5×8 | 152 | 1932 px | conservative fallback | diff --git a/eval/grok-density/QUALITY_RESULTS.md b/eval/grok-density/QUALITY_RESULTS.md index 9fd389b..a87dcb1 100644 --- a/eval/grok-density/QUALITY_RESULTS.md +++ b/eval/grok-density/QUALITY_RESULTS.md @@ -9,7 +9,7 @@ pxpipe and used the production 5×8 profile. | gist recall | 98/98 | 83/98 | no transport errors | | state tracking | 18/18 | 13/18 | subset of the gist corpus | | never-stated guards | 0/16 confabulated | 0/16 confabulated | lower is better | -| dense 12-char hex | 15/15 | 0/6 completed | 9 image calls failed at the gateway or timed out | +| dense 12-char hex | 15/15 | 0/15 | all calls completed after retrying transient upstream failures | Novel-arithmetic input usage was recovered from the provider log for the same N=100 run: 25,400 text tokens and 27,100 production-image tokens, **+6.7%**. @@ -23,6 +23,6 @@ Receipts: - `../sol-profile/verbatim-hex-grok-4.5-results.json` - `../sol-profile/verbatim-hex-grok-4.5-text-results.json` -Grok remains opt-in because arithmetic, gist, and state tracking are below the -Fable bar. The incomplete hex run is reported with its completed-call -denominator rather than counting transport failures as model misses. +Grok remains opt-in because arithmetic, gist, state tracking, and dense exact +recall are below the Fable bar. The completed hex rerun used an 8,192-token +output cap so mandatory reasoning could not consume the final answer budget. diff --git a/eval/sol-profile/QUALITY_RESULTS.md b/eval/sol-profile/QUALITY_RESULTS.md index 6372f03..1a0ebc7 100644 --- a/eval/sol-profile/QUALITY_RESULTS.md +++ b/eval/sol-profile/QUALITY_RESULTS.md @@ -11,9 +11,9 @@ adjacent text factsheet. | test | text | production image | notes | |---|---:|---:|---| | novel arithmetic, N=100 | 100/100 | 98/100 | pure image 96/100 | -| gist recall | not measured | 79/93 completed | one six-probe session failed at the gateway | -| state tracking | not measured | 18/18 | no transport errors | -| never-stated guards | not measured | 4/15 completed confabulated | one guard shared the failed session | +| gist recall | not measured | 83/98 | no transport errors | +| state tracking | not measured | 17/18 | no transport errors | +| never-stated guards | not measured | 4/16 confabulated | no transport errors | | dense 12-char hex | not run in this harness | 0/15 | all calls completed | Matched arithmetic usage was 5,300 text input tokens and 7,000 production-image diff --git a/eval/sol-profile/gist-recall-results.json b/eval/sol-profile/gist-recall-results.json index b467a1b..42af20f 100644 --- a/eval/sol-profile/gist-recall-results.json +++ b/eval/sol-profile/gist-recall-results.json @@ -1,5 +1,5 @@ { - "generatedAt": "2026-07-11T16:21:56.523Z", + "generatedAt": "2026-07-17T17:05:45.802Z", "model": "gpt-5.6-sol", "live": true, "recipe": { @@ -20,18 +20,18 @@ "factsheet": true }, "answerable": { - "correct": 79, - "completed": 93, + "correct": 83, + "completed": 98, "n": 98 }, "state": { - "correct": 18, + "correct": 17, "completed": 18, "n": 18 }, "unanswerable": { "confabulated": 4, - "completed": 15, + "completed": 16, "n": 16 }, "rows": [ @@ -44,7 +44,19 @@ "answer": "zustand", "ok": false, "raw": "[\"zustand\",\"7800ms\",\"src/batcher/core.ts\",\"Tobias Okafor\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3456, + "input_tokens_details": { + "cache_write_tokens": 3453, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3485 + } }, { "tier": "work", @@ -55,7 +67,19 @@ "answer": "7800ms", "ok": false, "raw": "[\"zustand\",\"7800ms\",\"src/batcher/core.ts\",\"Tobias Okafor\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3456, + "input_tokens_details": { + "cache_write_tokens": 3453, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3485 + } }, { "tier": "work", @@ -66,7 +90,19 @@ "answer": "src/batcher/core.ts", "ok": true, "raw": "[\"zustand\",\"7800ms\",\"src/batcher/core.ts\",\"Tobias Okafor\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3456, + "input_tokens_details": { + "cache_write_tokens": 3453, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3485 + } }, { "tier": "work", @@ -77,7 +113,19 @@ "answer": "Tobias Okafor", "ok": true, "raw": "[\"zustand\",\"7800ms\",\"src/batcher/core.ts\",\"Tobias Okafor\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3456, + "input_tokens_details": { + "cache_write_tokens": 3453, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3485 + } }, { "tier": "work", @@ -88,7 +136,19 @@ "answer": "OFF", "ok": true, "raw": "[\"zustand\",\"7800ms\",\"src/batcher/core.ts\",\"Tobias Okafor\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3456, + "input_tokens_details": { + "cache_write_tokens": 3453, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3485 + } }, { "tier": "work", @@ -99,7 +159,19 @@ "answer": "UNKNOWN", "ok": true, "raw": "[\"zustand\",\"7800ms\",\"src/batcher/core.ts\",\"Tobias Okafor\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3456, + "input_tokens_details": { + "cache_write_tokens": 3453, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3485 + } }, { "tier": "work", @@ -110,7 +182,19 @@ "answer": "xstate", "ok": true, "raw": "[\"xstate\",\"3480ms\",\"src/cursor/sync.ts\",\"Farid Moreau\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3496, + "input_tokens_details": { + "cache_write_tokens": 3493, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3524 + } }, { "tier": "work", @@ -121,7 +205,19 @@ "answer": "3480ms", "ok": true, "raw": "[\"xstate\",\"3480ms\",\"src/cursor/sync.ts\",\"Farid Moreau\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3496, + "input_tokens_details": { + "cache_write_tokens": 3493, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3524 + } }, { "tier": "work", @@ -132,7 +228,19 @@ "answer": "src/cursor/sync.ts", "ok": true, "raw": "[\"xstate\",\"3480ms\",\"src/cursor/sync.ts\",\"Farid Moreau\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3496, + "input_tokens_details": { + "cache_write_tokens": 3493, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3524 + } }, { "tier": "work", @@ -143,7 +251,19 @@ "answer": "Farid Moreau", "ok": true, "raw": "[\"xstate\",\"3480ms\",\"src/cursor/sync.ts\",\"Farid Moreau\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3496, + "input_tokens_details": { + "cache_write_tokens": 3493, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3524 + } }, { "tier": "work", @@ -154,7 +274,19 @@ "answer": "OFF", "ok": true, "raw": "[\"xstate\",\"3480ms\",\"src/cursor/sync.ts\",\"Farid Moreau\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3496, + "input_tokens_details": { + "cache_write_tokens": 3493, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3524 + } }, { "tier": "work", @@ -165,7 +297,19 @@ "answer": "UNKNOWN", "ok": true, "raw": "[\"xstate\",\"3480ms\",\"src/cursor/sync.ts\",\"Farid Moreau\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3496, + "input_tokens_details": { + "cache_write_tokens": 3493, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3524 + } }, { "tier": "work", @@ -173,10 +317,22 @@ "type": "decision", "q": "Which package was chosen for the store layer?", "gold": "immer", - "answer": "lru-cache", + "answer": "msgpackr", "ok": false, - "raw": "[\"lru-cache\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"msgpackr\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"render-v2.1.7\"]", + "error": null, + "usage": { + "input_tokens": 3513, + "input_tokens_details": { + "cache_write_tokens": 3510, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3546 + } }, { "tier": "work", @@ -186,8 +342,20 @@ "gold": "9110", "answer": "9110ms", "ok": true, - "raw": "[\"lru-cache\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"msgpackr\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"render-v2.1.7\"]", + "error": null, + "usage": { + "input_tokens": 3513, + "input_tokens_details": { + "cache_write_tokens": 3510, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3546 + } }, { "tier": "work", @@ -197,8 +365,20 @@ "gold": "src/cursor/io.ts", "answer": "src/cursor/io.ts", "ok": true, - "raw": "[\"lru-cache\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"msgpackr\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"render-v2.1.7\"]", + "error": null, + "usage": { + "input_tokens": 3513, + "input_tokens_details": { + "cache_write_tokens": 3510, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3546 + } }, { "tier": "work", @@ -208,8 +388,20 @@ "gold": "Nadia Costa", "answer": "Nadia Costa", "ok": true, - "raw": "[\"lru-cache\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"msgpackr\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"render-v2.1.7\"]", + "error": null, + "usage": { + "input_tokens": 3513, + "input_tokens_details": { + "cache_write_tokens": 3510, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3546 + } }, { "tier": "work", @@ -219,8 +411,20 @@ "gold": "OFF", "answer": "OFF", "ok": true, - "raw": "[\"lru-cache\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"msgpackr\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"render-v2.1.7\"]", + "error": null, + "usage": { + "input_tokens": 3513, + "input_tokens_details": { + "cache_write_tokens": 3510, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3546 + } }, { "tier": "work", @@ -228,10 +432,22 @@ "type": "unanswerable", "q": "What git tag was the hotfix released under?", "gold": "UNKNOWN", - "answer": "UNKNOWN", - "ok": true, - "raw": "[\"lru-cache\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "answer": "render-v2.1.7", + "ok": false, + "raw": "[\"msgpackr\",\"9110ms\",\"src/cursor/io.ts\",\"Nadia Costa\",\"OFF\",\"render-v2.1.7\"]", + "error": null, + "usage": { + "input_tokens": 3513, + "input_tokens_details": { + "cache_write_tokens": 3510, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3546 + } }, { "tier": "work", @@ -239,10 +455,22 @@ "type": "decision", "q": "Which package was chosen for the store layer?", "gold": "immer", - "answer": "valtio", + "answer": "better-sqlite3", "ok": false, - "raw": "[\"valtio\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"better-sqlite3\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", + "error": null, + "usage": { + "input_tokens": 2973, + "input_tokens_details": { + "cache_write_tokens": 2970, + "cached_tokens": 0 + }, + "output_tokens": 30, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3003 + } }, { "tier": "work", @@ -252,8 +480,20 @@ "gold": "3470", "answer": "3470ms", "ok": true, - "raw": "[\"valtio\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"better-sqlite3\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", + "error": null, + "usage": { + "input_tokens": 2973, + "input_tokens_details": { + "cache_write_tokens": 2970, + "cached_tokens": 0 + }, + "output_tokens": 30, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3003 + } }, { "tier": "work", @@ -263,8 +503,20 @@ "gold": "src/scheduler/core.ts", "answer": "src/scheduler/core.ts", "ok": true, - "raw": "[\"valtio\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"better-sqlite3\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", + "error": null, + "usage": { + "input_tokens": 2973, + "input_tokens_details": { + "cache_write_tokens": 2970, + "cached_tokens": 0 + }, + "output_tokens": 30, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3003 + } }, { "tier": "work", @@ -274,8 +526,20 @@ "gold": "Dmitri Berg", "answer": "Dmitri Berg", "ok": true, - "raw": "[\"valtio\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"better-sqlite3\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", + "error": null, + "usage": { + "input_tokens": 2973, + "input_tokens_details": { + "cache_write_tokens": 2970, + "cached_tokens": 0 + }, + "output_tokens": 30, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3003 + } }, { "tier": "work", @@ -285,8 +549,20 @@ "gold": "OFF", "answer": "OFF", "ok": true, - "raw": "[\"valtio\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"better-sqlite3\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", + "error": null, + "usage": { + "input_tokens": 2973, + "input_tokens_details": { + "cache_write_tokens": 2970, + "cached_tokens": 0 + }, + "output_tokens": 30, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3003 + } }, { "tier": "work", @@ -296,8 +572,20 @@ "gold": "UNKNOWN", "answer": "UNKNOWN", "ok": true, - "raw": "[\"valtio\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"better-sqlite3\",\"3470ms\",\"src/scheduler/core.ts\",\"Dmitri Berg\",\"OFF\",\"UNKNOWN\"]", + "error": null, + "usage": { + "input_tokens": 2973, + "input_tokens_details": { + "cache_write_tokens": 2970, + "cached_tokens": 0 + }, + "output_tokens": 30, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3003 + } }, { "tier": "work", @@ -305,10 +593,22 @@ "type": "decision", "q": "Which package was chosen for the store layer?", "gold": "redux-toolkit", - "answer": "level", + "answer": "moxb", "ok": false, - "raw": "[\"level\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"us-east-2\"]", - "error": null + "raw": "[\"moxb\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"eu-central-1\"]", + "error": null, + "usage": { + "input_tokens": 2989, + "input_tokens_details": { + "cache_write_tokens": 2986, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3022 + } }, { "tier": "work", @@ -318,8 +618,20 @@ "gold": "6210", "answer": "6210ms", "ok": true, - "raw": "[\"level\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"us-east-2\"]", - "error": null + "raw": "[\"moxb\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"eu-central-1\"]", + "error": null, + "usage": { + "input_tokens": 2989, + "input_tokens_details": { + "cache_write_tokens": 2986, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3022 + } }, { "tier": "work", @@ -329,8 +641,20 @@ "gold": "src/cursor/io.ts", "answer": "src/cursor/io.ts", "ok": true, - "raw": "[\"level\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"us-east-2\"]", - "error": null + "raw": "[\"moxb\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"eu-central-1\"]", + "error": null, + "usage": { + "input_tokens": 2989, + "input_tokens_details": { + "cache_write_tokens": 2986, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3022 + } }, { "tier": "work", @@ -340,8 +664,20 @@ "gold": "Tobias Khoury", "answer": "Tobias Khoury", "ok": true, - "raw": "[\"level\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"us-east-2\"]", - "error": null + "raw": "[\"moxb\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"eu-central-1\"]", + "error": null, + "usage": { + "input_tokens": 2989, + "input_tokens_details": { + "cache_write_tokens": 2986, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3022 + } }, { "tier": "work", @@ -351,8 +687,20 @@ "gold": "OFF", "answer": "OFF", "ok": true, - "raw": "[\"level\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"us-east-2\"]", - "error": null + "raw": "[\"moxb\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"eu-central-1\"]", + "error": null, + "usage": { + "input_tokens": 2989, + "input_tokens_details": { + "cache_write_tokens": 2986, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3022 + } }, { "tier": "work", @@ -360,10 +708,22 @@ "type": "unanswerable", "q": "Which AWS region was the failover assigned to?", "gold": "UNKNOWN", - "answer": "us-east-2", + "answer": "eu-central-1", "ok": false, - "raw": "[\"level\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"us-east-2\"]", - "error": null + "raw": "[\"moxb\",\"6210ms\",\"src/cursor/io.ts\",\"Tobias Khoury\",\"OFF\",\"eu-central-1\"]", + "error": null, + "usage": { + "input_tokens": 2989, + "input_tokens_details": { + "cache_write_tokens": 2986, + "cached_tokens": 0 + }, + "output_tokens": 33, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3022 + } }, { "tier": "work", @@ -374,7 +734,19 @@ "answer": "redux-toolkit", "ok": true, "raw": "[\"redux-toolkit\",\"1250ms\",\"src/cursor/io.ts\",\"Lucia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3208, + "input_tokens_details": { + "cache_write_tokens": 3205, + "cached_tokens": 0 + }, + "output_tokens": 27, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3235 + } }, { "tier": "work", @@ -385,7 +757,19 @@ "answer": "1250ms", "ok": true, "raw": "[\"redux-toolkit\",\"1250ms\",\"src/cursor/io.ts\",\"Lucia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3208, + "input_tokens_details": { + "cache_write_tokens": 3205, + "cached_tokens": 0 + }, + "output_tokens": 27, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3235 + } }, { "tier": "work", @@ -396,7 +780,19 @@ "answer": "src/cursor/io.ts", "ok": true, "raw": "[\"redux-toolkit\",\"1250ms\",\"src/cursor/io.ts\",\"Lucia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3208, + "input_tokens_details": { + "cache_write_tokens": 3205, + "cached_tokens": 0 + }, + "output_tokens": 27, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3235 + } }, { "tier": "work", @@ -407,7 +803,19 @@ "answer": "Lucia Costa", "ok": true, "raw": "[\"redux-toolkit\",\"1250ms\",\"src/cursor/io.ts\",\"Lucia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3208, + "input_tokens_details": { + "cache_write_tokens": 3205, + "cached_tokens": 0 + }, + "output_tokens": 27, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3235 + } }, { "tier": "work", @@ -418,7 +826,19 @@ "answer": "OFF", "ok": true, "raw": "[\"redux-toolkit\",\"1250ms\",\"src/cursor/io.ts\",\"Lucia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3208, + "input_tokens_details": { + "cache_write_tokens": 3205, + "cached_tokens": 0 + }, + "output_tokens": 27, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3235 + } }, { "tier": "work", @@ -429,7 +849,19 @@ "answer": "UNKNOWN", "ok": true, "raw": "[\"redux-toolkit\",\"1250ms\",\"src/cursor/io.ts\",\"Lucia Costa\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3208, + "input_tokens_details": { + "cache_write_tokens": 3205, + "cached_tokens": 0 + }, + "output_tokens": 27, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3235 + } }, { "tier": "work", @@ -440,7 +872,19 @@ "answer": "nanostores", "ok": false, "raw": "[\"nanostores\",\"6290ms\",\"src/scheduler/sync.ts\",\"Tobias\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3470, + "input_tokens_details": { + "cache_write_tokens": 3467, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3498 + } }, { "tier": "work", @@ -451,7 +895,19 @@ "answer": "6290ms", "ok": true, "raw": "[\"nanostores\",\"6290ms\",\"src/scheduler/sync.ts\",\"Tobias\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3470, + "input_tokens_details": { + "cache_write_tokens": 3467, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3498 + } }, { "tier": "work", @@ -462,7 +918,19 @@ "answer": "src/scheduler/sync.ts", "ok": true, "raw": "[\"nanostores\",\"6290ms\",\"src/scheduler/sync.ts\",\"Tobias\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3470, + "input_tokens_details": { + "cache_write_tokens": 3467, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3498 + } }, { "tier": "work", @@ -473,7 +941,19 @@ "answer": "Tobias", "ok": false, "raw": "[\"nanostores\",\"6290ms\",\"src/scheduler/sync.ts\",\"Tobias\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3470, + "input_tokens_details": { + "cache_write_tokens": 3467, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3498 + } }, { "tier": "work", @@ -484,7 +964,19 @@ "answer": "OFF", "ok": true, "raw": "[\"nanostores\",\"6290ms\",\"src/scheduler/sync.ts\",\"Tobias\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3470, + "input_tokens_details": { + "cache_write_tokens": 3467, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3498 + } }, { "tier": "work", @@ -495,7 +987,19 @@ "answer": "UNKNOWN", "ok": true, "raw": "[\"nanostores\",\"6290ms\",\"src/scheduler/sync.ts\",\"Tobias\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3470, + "input_tokens_details": { + "cache_write_tokens": 3467, + "cached_tokens": 0 + }, + "output_tokens": 28, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3498 + } }, { "tier": "work", @@ -506,7 +1010,19 @@ "answer": "nanostores", "ok": false, "raw": "[\"nanostores\",\"7870ms\",\"src/batcher/sync.ts\",\"Priya Tanaka\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3651, + "input_tokens_details": { + "cache_write_tokens": 3648, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3680 + } }, { "tier": "work", @@ -517,7 +1033,19 @@ "answer": "7870ms", "ok": true, "raw": "[\"nanostores\",\"7870ms\",\"src/batcher/sync.ts\",\"Priya Tanaka\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3651, + "input_tokens_details": { + "cache_write_tokens": 3648, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3680 + } }, { "tier": "work", @@ -528,7 +1056,19 @@ "answer": "src/batcher/sync.ts", "ok": true, "raw": "[\"nanostores\",\"7870ms\",\"src/batcher/sync.ts\",\"Priya Tanaka\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3651, + "input_tokens_details": { + "cache_write_tokens": 3648, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3680 + } }, { "tier": "work", @@ -539,7 +1079,19 @@ "answer": "Priya Tanaka", "ok": true, "raw": "[\"nanostores\",\"7870ms\",\"src/batcher/sync.ts\",\"Priya Tanaka\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3651, + "input_tokens_details": { + "cache_write_tokens": 3648, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3680 + } }, { "tier": "work", @@ -550,7 +1102,19 @@ "answer": "OFF", "ok": true, "raw": "[\"nanostores\",\"7870ms\",\"src/batcher/sync.ts\",\"Priya Tanaka\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3651, + "input_tokens_details": { + "cache_write_tokens": 3648, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3680 + } }, { "tier": "work", @@ -561,7 +1125,19 @@ "answer": "UNKNOWN", "ok": true, "raw": "[\"nanostores\",\"7870ms\",\"src/batcher/sync.ts\",\"Priya Tanaka\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3651, + "input_tokens_details": { + "cache_write_tokens": 3648, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3680 + } }, { "tier": "work", @@ -571,8 +1147,20 @@ "gold": "jotai", "answer": "jotai", "ok": true, - "raw": "[\"jotai\",\"7850\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"jotai\",\"7050ms\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"8255\"]", + "error": null, + "usage": { + "input_tokens": 3534, + "input_tokens_details": { + "cache_write_tokens": 3531, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3563 + } }, { "tier": "work", @@ -580,10 +1168,22 @@ "type": "numeric", "q": "What exact value in ms was the retry budget set to?", "gold": "7850", - "answer": "7850", - "ok": true, - "raw": "[\"jotai\",\"7850\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"UNKNOWN\"]", - "error": null + "answer": "7050ms", + "ok": false, + "raw": "[\"jotai\",\"7050ms\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"8255\"]", + "error": null, + "usage": { + "input_tokens": 3534, + "input_tokens_details": { + "cache_write_tokens": 3531, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3563 + } }, { "tier": "work", @@ -593,8 +1193,20 @@ "gold": "src/retry/io.ts", "answer": "src/retry/io.ts", "ok": true, - "raw": "[\"jotai\",\"7850\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"jotai\",\"7050ms\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"8255\"]", + "error": null, + "usage": { + "input_tokens": 3534, + "input_tokens_details": { + "cache_write_tokens": 3531, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3563 + } }, { "tier": "work", @@ -604,8 +1216,20 @@ "gold": "Soren Petrov", "answer": "Soren Petrov", "ok": true, - "raw": "[\"jotai\",\"7850\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"jotai\",\"7050ms\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"8255\"]", + "error": null, + "usage": { + "input_tokens": 3534, + "input_tokens_details": { + "cache_write_tokens": 3531, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3563 + } }, { "tier": "work", @@ -615,8 +1239,20 @@ "gold": "OFF", "answer": "OFF", "ok": true, - "raw": "[\"jotai\",\"7850\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"UNKNOWN\"]", - "error": null + "raw": "[\"jotai\",\"7050ms\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"8255\"]", + "error": null, + "usage": { + "input_tokens": 3534, + "input_tokens_details": { + "cache_write_tokens": 3531, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3563 + } }, { "tier": "work", @@ -624,10 +1260,22 @@ "type": "unanswerable", "q": "What port number was the staging proxy moved to?", "gold": "UNKNOWN", - "answer": "UNKNOWN", - "ok": true, - "raw": "[\"jotai\",\"7850\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"UNKNOWN\"]", - "error": null + "answer": "8255", + "ok": false, + "raw": "[\"jotai\",\"7050ms\",\"src/retry/io.ts\",\"Soren Petrov\",\"OFF\",\"8255\"]", + "error": null, + "usage": { + "input_tokens": 3534, + "input_tokens_details": { + "cache_write_tokens": 3531, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3563 + } }, { "tier": "work", @@ -638,7 +1286,19 @@ "answer": "redux-toolkit", "ok": true, "raw": "[\"redux-toolkit\",\"4540ms\",\"src/cursor/sync.ts\",\"Ingrid Nakamura\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3422, + "input_tokens_details": { + "cache_write_tokens": 3419, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3451 + } }, { "tier": "work", @@ -649,7 +1309,19 @@ "answer": "4540ms", "ok": true, "raw": "[\"redux-toolkit\",\"4540ms\",\"src/cursor/sync.ts\",\"Ingrid Nakamura\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3422, + "input_tokens_details": { + "cache_write_tokens": 3419, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3451 + } }, { "tier": "work", @@ -660,7 +1332,19 @@ "answer": "src/cursor/sync.ts", "ok": true, "raw": "[\"redux-toolkit\",\"4540ms\",\"src/cursor/sync.ts\",\"Ingrid Nakamura\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3422, + "input_tokens_details": { + "cache_write_tokens": 3419, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3451 + } }, { "tier": "work", @@ -671,7 +1355,19 @@ "answer": "Ingrid Nakamura", "ok": true, "raw": "[\"redux-toolkit\",\"4540ms\",\"src/cursor/sync.ts\",\"Ingrid Nakamura\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3422, + "input_tokens_details": { + "cache_write_tokens": 3419, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3451 + } }, { "tier": "work", @@ -682,7 +1378,19 @@ "answer": "OFF", "ok": true, "raw": "[\"redux-toolkit\",\"4540ms\",\"src/cursor/sync.ts\",\"Ingrid Nakamura\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3422, + "input_tokens_details": { + "cache_write_tokens": 3419, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3451 + } }, { "tier": "work", @@ -693,7 +1401,19 @@ "answer": "UNKNOWN", "ok": true, "raw": "[\"redux-toolkit\",\"4540ms\",\"src/cursor/sync.ts\",\"Ingrid Nakamura\",\"OFF\",\"UNKNOWN\"]", - "error": null + "error": null, + "usage": { + "input_tokens": 3422, + "input_tokens_details": { + "cache_write_tokens": 3419, + "cached_tokens": 0 + }, + "output_tokens": 29, + "output_tokens_details": { + "reasoning_tokens": 0 + }, + "total_tokens": 3451 + } }, { "tier": "work2", @@ -701,10 +1421,22 @@ "type": "decision", "q": "What was the FINAL package chosen for the store layer?", "gold": "nanostores", - "answer": "", - "ok": false, - "raw": "", - "error": "SyntaxError: Unexpected token '<', \"\n\n\n\n\n\n\n