diff --git a/FINDINGS.md b/FINDINGS.md index d397f4a..ae36805 100644 --- a/FINDINGS.md +++ b/FINDINGS.md @@ -3,7 +3,7 @@ **Status:** ⚠️ **VERDICT REVERSED — see correction below.** Originally ruled "dead"; live measurement shows pxpipe is a working *lossy gist-compressor* saving ~68% on real (dense) Claude Code traffic, with a known verbatim-recall gap. **Date:** 2026-05-28 (original) · 2026-05-29 (correction) · 2026-06-09 (Fable 5 update) · 2026-06-10 (gist-recall A/B, SWE-bench pilot) · 2026-06-12 (field observation, n=1) · 2026-06-23 (reframe: correct baseline = /compact) · 2026-07-09 (GPT-5.6 Sol raw-recall pilot) · 2026-07-19 (K/H glyph-surgery model-level A/B) **Models tested:** `claude-opus-4-5` (original run), `claude-opus-4-8` (re-test after a model bump), `claude-fable-5` (2026-06-09), `gpt-5.6-sol` (2026-07-09 raw-image pilot) -**Model scope (current):** Fable 5 only. Sol, Opus, GPT 5.5, and Grok remain explicit opt-ins. +**Model scope (current):** Fable 5 and Gemini 3.6 Flash. Sol, Opus, GPT 5.5, and Grok remain explicit opt-ins. **Harnesses:** Claude/Opus/Fable: `eval/needle-haystack/` (older receipts preserved from `/tmp/needle_eval`); Sol: `eval/sol-profile/` (raw responses and receipts committed) --- diff --git a/README.md b/README.md index e9b7678..ee41488 100644 --- a/README.md +++ b/README.md @@ -22,13 +22,12 @@ This is what the model sees instead of text: tokens as this page. Real pipeline output; the model reads renders like this at 100/100 (see benchmarks).* -![chart: characters a frontier context window holds, 2018–2026 — vendor text series including Grok 4.5; orange measured overlay is Fable 5 [1m] + pxpipe ~18M (4.6×)](docs/assets/context-window-chars.png) +![chart: characters a frontier context window holds, 2018–2026 — vendor text series including Grok 4.5; orange measured overlays are Fable 5 [1m] + pxpipe ~19.0M (4.8×) and Gemini 3.6 Flash + pxpipe ~21.3M (5.3×)](docs/assets/context-window-chars.png) *Eight years of context growth, in characters. Every text line tops out near ~4M chars (a 1M-token window at ~4 chars/token); **Grok 4.5** is shown as a -text-window point only (500K). The orange overlay is the **same Fable 5 1M -window** read through pxpipe images — ~18M chars at the measured Anthropic -density (**4.6×** the text ceiling). Density is measured from a live render at +text-window point only (500K). The orange overlays are the **same 1M +windows** read through pxpipe images — ~19.0M chars for Fable 5 (**4.8×**) and ~21.3M chars for Gemini 3.6 Flash (**5.3×** text capacity). Density is measured from a live render at generation time, not hand-typed: regenerate with `npx tsx scripts/gen-context-chart.ts` ([source](scripts/gen-context-chart.ts)).* @@ -105,9 +104,9 @@ without running the proxy. re-sends as text. Claude Code re-sends system + tools + history on `/anthropic/messages` and typically lands ~60–70%. Details and measured splits: [docs/CACHING_AND_SAVINGS.md](docs/CACHING_AND_SAVINGS.md). -- **Model scope:** default `PXPIPE_MODELS=claude-fable-5`. Sol, Opus +- **Model scope:** default `PXPIPE_MODELS=claude-fable-5,gemini-3.6-flash`. Sol, Opus 4.7/4.8, GPT 5.5, and **Grok** are opt-in only (dashboard chips or - `PXPIPE_MODELS`) — not good enough as silent defaults for imaged context. + `PXPIPE_MODELS`) — Gemini 3.6 Flash matches Fable 5 on arithmetic and gist and scores 14/15 versus Fable's 13/15 on dense hex; it is enabled by default. Grok packing + factsheet helps exact IDs, but quality remains below Fable: 82/100 arithmetic, 83/98 gist, and 13/18 state tracking. The exact Sol id still matters. Sibling variants such as `gpt-5.6-terra` do not @@ -152,24 +151,29 @@ used for these novel-arithmetic rows. | test | model | N | pxpipe (image) | | --- | --- | ---: | ---: | | novel arithmetic | `claude-fable-5` | 100 | **100%** | +| novel arithmetic | `google/gemini-3.6-flash` | 100 | **100%** | | novel arithmetic | `gpt-5.6-sol` | 100 | **98%** | | novel arithmetic | `claude-opus-4-8` | 100 | 93% | | novel arithmetic | `grok-4.5` | 100 | **82%** | | novel arithmetic | `moonshotai/kimi-k3` | 100 | **79%** | | gist recall A/B (decisions, values, paths, names, negations; distractors; 15k–45k char sessions) | `claude-fable-5` | 98 | **98/98** | +| same gist corpus, production images + factsheet | `google/gemini-3.6-flash` | 98 | **98/98** | | same gist corpus, production images + factsheet | `gpt-5.6-sol` | 98 | **83/98** | | same gist corpus, production images + factsheet | `grok-4.5` | 98 | **83/98** | | same gist corpus, production images + factsheet | `moonshotai/kimi-k3` | 98 | **84/98** | | state tracking (value mutated 3×, final/first/count) | `claude-fable-5` | 18 | **18/18** | +| same state-tracking corpus | `google/gemini-3.6-flash` | 18 | **18/18** | | same state-tracking corpus | `gpt-5.6-sol` | 18 | **17/18** | | same state-tracking corpus | `grok-4.5` | 18 | **13/18** | | same state-tracking corpus | `moonshotai/kimi-k3` | 18 | **15/18** | | confabulation on never-stated facts (lower is better) | `claude-fable-5` | 16 | **0/16** | +| same never-stated probes (lower is better) | `google/gemini-3.6-flash` | 16 | **0/16** | | same never-stated probes (lower is better) | `gpt-5.6-sol` | 16 | **4/16** | | same never-stated probes (lower is better) | `grok-4.5` | 16 | **0/16** | | same never-stated probes (lower is better) | `moonshotai/kimi-k3` | 16 | **1/16** | -| verbatim 12-char hex, dense render | `claude-opus-4-8` | 15 | **0/15** | +| verbatim 12-char hex, dense render | `google/gemini-3.6-flash` | 15 | **14/15** | | verbatim 12-char hex, dense render | `claude-fable-5` | 15 | **13/15** | +| verbatim 12-char hex, dense render | `claude-opus-4-8` | 15 | **0/15** | | verbatim 12-char hex, same dense pages | `gpt-5.6-sol` | 15 | **0/15** | | verbatim 12-char hex, same dense pages | `grok-4.5` | 15 | **0/15** | | verbatim 12-char hex, same dense pages | `moonshotai/kimi-k3` | 15 | **0/15** | @@ -177,9 +181,9 @@ used for these novel-arithmetic rows. **Harness split:** Fable/Opus quality and SWE-bench rows use **Claude**; Sol and Grok quality use **Codex’s Responses provider** (`OPENAI_BASE_URL`). Kimi K3 used the same novel-arithmetic corpus and production renderer through pxpipe's -Cloudflare Messages bridge — see the -[`K3 receipt`](eval/sol-profile/model-moonshotai_kimi-k3-novel-arithmetic-results.json) and -[`eval/grok-density/QUALITY_SUITE.md`](eval/grok-density/QUALITY_SUITE.md). +Cloudflare Messages bridge. Gemini 3.6 Flash used Google AI Studio — see +[`Gemini 3.6 Flash receipts`](eval/gemini-profile/QUALITY_RESULTS.md) and +[`K3 receipt`](eval/sol-profile/model-moonshotai_kimi-k3-novel-arithmetic-results.json). K3 semantic and exact-recall receipts: [`gist/state/guards`](eval/sol-profile/gist-recall-moonshotai_kimi-k3-results.json) and @@ -199,7 +203,8 @@ chars/vision-token ÷ 4 (prose text baseline). Not a model-quality score. | family | window | as text (@4 c/tok) | as pxpipe images | density | multiplier | |---|---:|---:|---:|---:|---:| -| **`claude-fable-5[1m]`** (default) | 1M | ~4.0M | **~18.3M** | ~18.3 c/vt (px÷750) | **~4.6×** | +| **`claude-fable-5[1m]`** (default) | 1M | ~4.0M | **~19.0M** | ~19.0 c/vt (exact 28px patches) | **~4.8×** | +| **`google/gemini-3.6-flash`** | 1M | ~4.0M | **~21.3M** | ~21.3 c/vt (1,078 tok/page) | **~5.3×** | Regenerate: `npx tsx scripts/gen-context-chart.ts` · chart PNG [`docs/assets/context-window-chars.png`](docs/assets/context-window-chars.png). diff --git a/docs/assets/context-window-chars.png b/docs/assets/context-window-chars.png index ab0c789..66d8a26 100644 Binary files a/docs/assets/context-window-chars.png and b/docs/assets/context-window-chars.png differ diff --git a/eval/gemini-profile/GEMINI_VISION_RESEARCH.md b/eval/gemini-profile/GEMINI_VISION_RESEARCH.md new file mode 100644 index 0000000..561bf75 --- /dev/null +++ b/eval/gemini-profile/GEMINI_VISION_RESEARCH.md @@ -0,0 +1,118 @@ +# Gemini 3.6 Flash Vision & Profile Research + +This document collects empirical research on image tokenization, dimension caps, aspect ratios, RGB channel multiplexing, and reading quality for `google/gemini-3.6-flash`. + +--- + +## 1. Image Tokenization & Pricing Behavior + +### Empirical Measurements across Dimensions + +Unlike Anthropic (which scales vision tokens dynamically with pixel dimensions via 28px patches) or OpenAI (which scales via 512×512 tiles at 170 tokens/tile), **Gemini 3.6 Flash uses a fixed ~1,034 – 1,113 vision tokens per image regardless of pixel resolution or aspect ratio**: + +| preset name | dimensions (W×H) | aspect ratio | image tokens billed | +|---|---:|---:|---:| +| tiny square | 100×100 | 1.00 | **1,089** | +| small square | 256×256 | 1.00 | **1,089** | +| medium square | 512×512 | 1.00 | **1,089** | +| standard square | 1024×1024 | 1.00 | **1,089** | +| large square | 2048×2048 | 1.00 | **1,089** | +| Claude widescreen | 1568×728 | 2.15 | **1,078** | +| Sol tall portrait | 768×1932 | 0.40 | **1,113** | +| Grok short portrait | 768×512 | 1.50 | **1,080** | +| HD 1080p | 1920×1080 | 1.78 | **1,100** | +| Ultra-wide 4:1 | 2048×512 | 4.00 | **1,056** | +| Ultra-tall 1:4 | 512×2048 | 0.25 | **1,056** | +| Extreme-wide 8:1 | 4096×512 | 8.00 | **1,034** | + +### Key Takeaway on Vision Cost +The measurements show nearly flat image-token usage across these dimensions (1,034-1,113 tokens), but they do not establish Google's internal resampling architecture. +Because image token cost was nearly flat in this sweep, larger legible canvases can pack more characters per measured vision token. + +--- + +## 2. Geometry & Legibility Research + +Testing 12-character verbatim hex recall off dense rendering pages across four aspect ratio and layout profiles: + +| geometry profile | width × height | columns | accuracy | +|---|---:|---:|---:| +| 312-col 1568×728 (Claude widescreen) | 1568×728 | 312 | **5/5 (100%)** | +| 152-col 768×1932 (GPT/Sol tall portrait) | 768×1932 | 152 | **5/5 (100%)** | +| 152-col 768×512 (Grok short portrait) | 768×512 | 152 | **5/5 (100%)** | +| 200-col 1024×1024 (1:1 square) | 1024×1024 | 200 | **5/5 (100%)** | + +--- + +## 3. RGB Channel Separation Diagnostic + +Testing 3-channel RGB overprint multiplexing (where three independent text streams share one physical line in RED, GREEN, and BLUE channels): + +| arm | exact lines recovered | +|---|---:| +| extracted red channel (color) | **12/12 (100%)** | +| extracted red channel (white) | **12/12 (100%)** | +| extracted green channel (color) | **12/12 (100%)** | +| extracted green channel (white) | **12/12 (100%)** | +| extracted blue channel (color) | **12/12 (100%)** | +| extracted blue channel (white) | **12/12 (100%)** | +| combined RGB (all 3 streams overlaid) | **0/12** | +| combined RGB (focused on red) | **0/12** | +| combined RGB (focused on green) | **1/12** | + +### Conclusion on RGB Multiplexing +- Individual color channels and single-channel monochrome renders are **100% healthy (12/12 on every channel)**. +- Physical character collisions on combined RGB overprints corrupt ViT patch embeddings prior to model attention. Overlaid RGB multiplexing is **rejected for production**. + +--- + +## 4. Dedicated Gemini Model Profile + +`src/core/gemini-model-profiles.ts` defines Gemini's dedicated profile: + +```typescript + { + test: (m) => /gemini/i.test(m), + profile: { + vision: { regime: 'tile', base: 1078, perTile: 0 }, + stripCols: ANTHROPIC_STRIP_COLS, // 312 columns + maxHeightPx: ANTHROPIC_MAX_HEIGHT_PX, // 728 px + minCompressTokens: 500, + factSheetFormat: 'full', + history: { ...BASE_HISTORY, maxImages: 32 }, + style: { + ...BASE_STYLE, + font: 'spleen-5x8', + aa: true, + }, + }, + } +``` + +`geminiVisionTokens` in `src/core/gemini-model-profiles.ts` records the measured production-geometry value: + +```typescript +export function geminiVisionTokens(_model: string, _w: number, _h: number): number { + return 1078; +} +``` + +--- + +## 5. Quality Benchmark Summary (Dedicated Gemini Profile) + +Evaluated on `google/gemini-3.6-flash` at the shipped 312-column, 728px profile: + +| test | N | Gemini 3.6 Flash | notes | +|---|---:|---:|---| +| novel arithmetic | 100 | **100/100 (100%)** | pure image 100/100 | +| gist recall A/B | 98 | **98/98 (100%)** | all 22 sessions completed | +| state tracking | 18 | **18/18 (100%)** | subset of gist corpus | +| never-stated probes | 16 | **0/16 confabulated** | 0 false positives | +| verbatim 12-char hex | 15 | **14/15 (93%)** | dense render | + +Receipts: +- `eval/gemini-profile/dimension-research-results.json` +- `eval/gemini-profile/novel-arithmetic-results.json` +- `eval/gemini-profile/gist-recall-results.json` +- `eval/gemini-profile/verbatim-hex-results.json` diff --git a/eval/gemini-profile/QUALITY_RESULTS.md b/eval/gemini-profile/QUALITY_RESULTS.md new file mode 100644 index 0000000..b5b3aa9 --- /dev/null +++ b/eval/gemini-profile/QUALITY_RESULTS.md @@ -0,0 +1,19 @@ +# Gemini 3.6 Flash quality results + +Model: `google/gemini-3.6-flash` (`gemini-3.6-flash`) through the Google AI Studio bridge. These receipts use the shipped Spleen 5×8, 312-column, 728px profile and adjacent text factsheet. + +| test | production image | notes | +|---|---:|---| +| novel arithmetic, N=100 | 100/100 | pure image 100/100, text 100/100 | +| gist recall | 98/98 | all 22 sessions completed | +| state tracking | 18/18 | subset of the gist corpus | +| never-stated guards | 0/16 confabulated | lower is better | +| dense 12-char hex | 14/15 | all calls completed | + +At the shipped 312×728 geometry, Gemini 3.6 Flash matched the recorded Fable 5 reasoning, state-tracking, and guard scores and scored 14/15 versus Fable's 13/15 on dense verbatim reading. + +Receipts: + +- `novel-arithmetic-results.json` +- `gist-recall-results.json` +- `verbatim-hex-results.json` diff --git a/eval/gemini-profile/dimension-research-results.json b/eval/gemini-profile/dimension-research-results.json new file mode 100644 index 0000000..cf9ae14 --- /dev/null +++ b/eval/gemini-profile/dimension-research-results.json @@ -0,0 +1,141 @@ +{ + "generatedAt": "2026-07-21T17:17:33.031Z", + "model": "gemini-3.6-flash", + "live": true, + "tokenResults": [ + { + "name": "tiny-square", + "w": 100, + "h": 100, + "aspect": "1.00", + "imgTokens": 1089, + "totalTokens": 1090, + "ms": 4113 + }, + { + "name": "small-square", + "w": 256, + "h": 256, + "aspect": "1.00", + "imgTokens": 1089, + "totalTokens": 1090, + "ms": 5028 + }, + { + "name": "medium-square", + "w": 512, + "h": 512, + "aspect": "1.00", + "imgTokens": 1089, + "totalTokens": 1090, + "ms": 2574 + }, + { + "name": "standard-square", + "w": 1024, + "h": 1024, + "aspect": "1.00", + "imgTokens": 1089, + "totalTokens": 1090, + "ms": 2379 + }, + { + "name": "large-square", + "w": 2048, + "h": 2048, + "aspect": "1.00", + "imgTokens": 1089, + "totalTokens": 1090, + "ms": 2547 + }, + { + "name": "claude-widescreen", + "w": 1568, + "h": 728, + "aspect": "2.15", + "imgTokens": 1078, + "totalTokens": 1079, + "ms": 1917 + }, + { + "name": "gpt-sol-portrait", + "w": 768, + "h": 1932, + "aspect": "0.40", + "imgTokens": 1113, + "totalTokens": 1114, + "ms": 1874 + }, + { + "name": "grok-short-portrait", + "w": 768, + "h": 512, + "aspect": "1.50", + "imgTokens": 1080, + "totalTokens": 1081, + "ms": 2912 + }, + { + "name": "hd-1080p", + "w": 1920, + "h": 1080, + "aspect": "1.78", + "imgTokens": 1100, + "totalTokens": 1101, + "ms": 3315 + }, + { + "name": "ultra-wide-4-1", + "w": 2048, + "h": 512, + "aspect": "4.00", + "imgTokens": 1056, + "totalTokens": 1057, + "ms": 3664 + }, + { + "name": "ultra-tall-1-4", + "w": 512, + "h": 2048, + "aspect": "0.25", + "imgTokens": 1056, + "totalTokens": 1057, + "ms": 2600 + }, + { + "name": "extreme-wide-8-1", + "w": 4096, + "h": 512, + "aspect": "8.00", + "imgTokens": 1034, + "totalTokens": 1035, + "ms": 2763 + } + ], + "legibilityResults": [ + { + "geometry": "312-col 1568x728 (Claude widescreen)", + "hits": 5, + "total": 5, + "accuracy": "5/5" + }, + { + "geometry": "152-col 768x1932 (GPT/Sol tall portrait)", + "hits": 5, + "total": 5, + "accuracy": "5/5" + }, + { + "geometry": "152-col 768x512 (Grok short portrait)", + "hits": 5, + "total": 5, + "accuracy": "5/5" + }, + { + "geometry": "200-col 1024x1024 (1:1 square)", + "hits": 5, + "total": 5, + "accuracy": "5/5" + } + ] +} diff --git a/eval/gemini-profile/dimension-research.mjs b/eval/gemini-profile/dimension-research.mjs new file mode 100644 index 0000000..22cba43 --- /dev/null +++ b/eval/gemini-profile/dimension-research.mjs @@ -0,0 +1,136 @@ +// Dimension and Geometry Research for Gemini 3.6 Flash +// Measures token cost and verbatim reading accuracy across image sizes and aspect ratios. + +import { writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { createCanvas } from '@napi-rs/canvas'; +import { callGemini } from './gemini-client.mjs'; +import { renderTextToPngs } from '../../dist/core/render.js'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const MODEL = process.env.MODEL || 'gemini-3.6-flash'; +const LIVE = process.env.LIVE === '1'; + +// Probe 1: Image Token Pricing across dimensions +const DIMENSION_PROBES = [ + { name: 'tiny-square', w: 100, h: 100 }, + { name: 'small-square', w: 256, h: 256 }, + { name: 'medium-square', w: 512, h: 512 }, + { name: 'standard-square', w: 1024, h: 1024 }, + { name: 'large-square', w: 2048, h: 2048 }, + { name: 'claude-widescreen', w: 1568, h: 728 }, + { name: 'gpt-sol-portrait', w: 768, h: 1932 }, + { name: 'grok-short-portrait', w: 768, h: 512 }, + { name: 'hd-1080p', w: 1920, h: 1080 }, + { name: 'ultra-wide-4-1', w: 2048, h: 512 }, + { name: 'ultra-tall-1-4', w: 512, h: 2048 }, + { name: 'extreme-wide-8-1', w: 4096, h: 512 }, +]; + +function renderCanvasImage(w, h) { + const canvas = createCanvas(w, h); + const ctx = canvas.getContext('2d'); + ctx.fillStyle = '#111'; + ctx.fillRect(0, 0, w, h); + ctx.fillStyle = '#fff'; + ctx.font = '12px sans-serif'; + ctx.fillText('Probe image for token count measurement', 10, 20); + return canvas.toBuffer('image/png'); +} + +const VERBATIM_TRIALS = [ + { id: 'a1', gold: 'c9c947f680ec', dur: 4439 }, + { id: 'a2', gold: '851eb3af1bd1', dur: 812 }, + { id: 'a3', gold: 'ade34f70fd73', dur: 6150 }, + { id: 'a4', gold: 'c5d68855f46d', dur: 7978 }, + { id: 'a5', gold: '92abade01aad', dur: 8071 }, +]; + +async function run() { + console.log(`=== Gemini 3.6 Flash Dimension & Geometry Research ===\n`); + + const tokenResults = []; + for (const dim of DIMENSION_PROBES) { + const png = renderCanvasImage(dim.w, dim.h); + let imgTokens = null; + let totalTokens = null; + let ms = null; + + if (LIVE) { + const content = [ + { type: 'input_image', image_url: `data:image/png;base64,${png.toString('base64')}` }, + { type: 'input_text', text: 'A' } + ]; + try { + const res = await callGemini({ model: MODEL, content, maxOutputTokens: 10, timeoutMs: 30000 }); + imgTokens = res.usage?.promptTokensDetails?.find(d => d.modality === 'IMAGE')?.tokenCount ?? null; + totalTokens = res.usage?.promptTokenCount ?? null; + ms = res.ms; + } catch (e) { + console.error(`Error on ${dim.name}:`, e.message); + } + } + + const aspect = (dim.w / dim.h).toFixed(2); + tokenResults.push({ ...dim, aspect, imgTokens, totalTokens, ms }); + console.log(`${dim.name.padEnd(22)} ${dim.w}x${dim.h} (aspect ${aspect}) -> image tokens: ${imgTokens ?? 'N/A'}`); + } + + // Geometry legibility benchmark across 4 rendering profiles + const GEOMETRIES = [ + { name: '312-col 1568x728 (Claude widescreen)', cols: 312, maxH: 728 }, + { name: '152-col 768x1932 (GPT/Sol tall portrait)', cols: 152, maxH: 1932 }, + { name: '152-col 768x512 (Grok short portrait)', cols: 152, maxH: 512 }, + { name: '200-col 1024x1024 (1:1 square)', cols: 200, maxH: 1024 }, + ]; + + console.log(`\n=== Verbatim Recall across Rendering Geometries ===\n`); + const legibilityResults = []; + + for (const geom of GEOMETRIES) { + let hits = 0; + for (const trial of VERBATIM_TRIALS) { + const sourceText = [ + 'LOG TRACE FILE - EVENT STREAM', + `{"timestamp":"2026-07-21T12:00:00Z","id":"${trial.gold}","dur_ms":${trial.dur},"status":200,"path":"/api/v1/sync"}`, + '{"timestamp":"2026-07-21T12:00:01Z","id":"112233445566","dur_ms":1234,"status":200,"path":"/api/v1/data"}', + ].join('\n'); + + const imgs = await renderTextToPngs(sourceText, geom.cols, { font: 'spleen-5x8', aa: true }, geom.maxH); + const png = imgs[0].png; + let ok = false; + let got = ''; + + if (LIVE) { + const content = [ + { type: 'input_image', image_url: `data:image/png;base64,${Buffer.from(png).toString('base64')}` }, + { type: 'input_text', text: `Read the image visually. Find the JSON line whose dur_ms is exactly ${trial.dur}. Return only its id field, exactly 12 lowercase hex characters.` } + ]; + try { + const res = await callGemini({ model: MODEL, content, maxOutputTokens: 50, timeoutMs: 30000 }); + got = res.text.match(/[0-9a-f]{12}/i)?.[0]?.toLowerCase() || ''; + ok = got === trial.gold; + if (ok) hits++; + } catch (e) { + console.error(` Error in trial ${trial.id}:`, e.message); + } + } + } + legibilityResults.push({ geometry: geom.name, hits, total: VERBATIM_TRIALS.length, accuracy: `${hits}/${VERBATIM_TRIALS.length}` }); + console.log(`${geom.name.padEnd(42)}: ${hits}/${VERBATIM_TRIALS.length} correct`); + } + + const output = { + generatedAt: new Date().toISOString(), + model: MODEL, + live: LIVE, + tokenResults, + legibilityResults + }; + + writeFileSync(join(HERE, 'dimension-research-results.json'), JSON.stringify(output, null, 2)); + console.log(`\nResults written to eval/gemini-profile/dimension-research-results.json`); +} + +run(); diff --git a/eval/gemini-profile/gemini-client.mjs b/eval/gemini-profile/gemini-client.mjs new file mode 100644 index 0000000..40bd9c1 --- /dev/null +++ b/eval/gemini-profile/gemini-client.mjs @@ -0,0 +1,50 @@ +// Dedicated Google AI Studio client for Gemini 3.6 Flash evaluations. + +export async function callGemini({ model = 'gemini-3.6-flash', content, maxOutputTokens = 1000, timeoutMs = 120000 }) { + const key = process.env.OPENAI_API_KEY || process.env.GEMINI_API_KEY; + if (!key) throw new Error('OPENAI_API_KEY or GEMINI_API_KEY is required'); + + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + const started = Date.now(); + const cleanModel = model.replace(/^google\//, '').replace(/^claude-/, ''); + const url = `http://127.0.0.1:47821/google-ai-studio/v1beta/models/${cleanModel}:generateContent`; + + const parts = content.map((part) => { + if (part.type === 'input_text') return { text: part.text }; + if (part.type === 'input_image' && typeof part.image_url === 'string') { + const match = /^data:([^;]+);base64,(.*)$/.exec(part.image_url); + if (!match) throw new Error('Gemini eval requires base64 data images'); + return { + inlineData: { mimeType: match[1], data: match[2] }, + }; + } + throw new Error(`unsupported Gemini content part: ${part.type}`); + }); + + try { + const response = await fetch(url, { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'x-goog-api-key': key, + }, + body: JSON.stringify({ contents: [{ parts }] }), + signal: controller.signal, + }); + const raw = await response.text(); + let json; + try { + json = JSON.parse(raw); + } catch { + throw new Error(`non-json HTTP ${response.status}: ${raw.slice(0, 160)}`); + } + if (!response.ok) { + throw new Error(`HTTP ${response.status}: ${json?.error?.message || raw.slice(0, 160)}`); + } + const text = json.candidates?.[0]?.content?.parts?.map((p) => p.text).filter(Boolean).join('') || ''; + return { text: text.trim(), usage: json.usageMetadata || null, ms: Date.now() - started }; + } finally { + clearTimeout(timer); + } +} diff --git a/eval/gemini-profile/gist-recall-results.json b/eval/gemini-profile/gist-recall-results.json new file mode 100644 index 0000000..b22f694 --- /dev/null +++ b/eval/gemini-profile/gist-recall-results.json @@ -0,0 +1,3231 @@ +{ + "generatedAt": "2026-07-21T20:16:19.056Z", + "model": "gemini-3.6-flash", + "live": true, + "recipe": { + "cols": 312, + "maxH": 728, + "style": { + "font": "spleen-5x8", + "cellWBonus": 0, + "cellHBonus": 0, + "aa": true, + "grid": false, + "gridCols": 0, + "colorCycle": false, + "markerScale": 1, + "markerRed": false, + "inkDilate": 0 + }, + "factsheet": true + }, + "answerable": { + "correct": 98, + "completed": 98, + "n": 98 + }, + "state": { + "correct": 18, + "completed": 18, + "n": 18 + }, + "unanswerable": { + "confabulated": 0, + "completed": 16, + "n": 16 + }, + "rows": [ + { + "tier": "work", + "session": 0, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "mobx", + "answer": "mobx", + "ok": true, + "raw": "```json\n[\n \"mobx\",\n \"7880ms\",\n \"src/batcher/core.ts\",\n \"Tobias Okafor\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5988, + "candidatesTokenCount": 53, + "totalTokenCount": 7237, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 653 + }, + { + "modality": "IMAGE", + "tokenCount": 5335 + } + ], + "thoughtsTokenCount": 1196, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 0, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "7880", + "answer": "7880ms", + "ok": true, + "raw": "```json\n[\n \"mobx\",\n \"7880ms\",\n \"src/batcher/core.ts\",\n \"Tobias Okafor\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5988, + "candidatesTokenCount": 53, + "totalTokenCount": 7237, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 653 + }, + { + "modality": "IMAGE", + "tokenCount": 5335 + } + ], + "thoughtsTokenCount": 1196, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 0, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/batcher/core.ts", + "answer": "src/batcher/core.ts", + "ok": true, + "raw": "```json\n[\n \"mobx\",\n \"7880ms\",\n \"src/batcher/core.ts\",\n \"Tobias Okafor\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5988, + "candidatesTokenCount": 53, + "totalTokenCount": 7237, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 653 + }, + { + "modality": "IMAGE", + "tokenCount": 5335 + } + ], + "thoughtsTokenCount": 1196, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 0, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Tobias Okafor", + "answer": "Tobias Okafor", + "ok": true, + "raw": "```json\n[\n \"mobx\",\n \"7880ms\",\n \"src/batcher/core.ts\",\n \"Tobias Okafor\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5988, + "candidatesTokenCount": 53, + "totalTokenCount": 7237, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 653 + }, + { + "modality": "IMAGE", + "tokenCount": 5335 + } + ], + "thoughtsTokenCount": 1196, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 0, + "type": "negation", + "q": "Was LEGACY_PINS enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"mobx\",\n \"7880ms\",\n \"src/batcher/core.ts\",\n \"Tobias Okafor\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5988, + "candidatesTokenCount": 53, + "totalTokenCount": 7237, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 653 + }, + { + "modality": "IMAGE", + "tokenCount": 5335 + } + ], + "thoughtsTokenCount": 1196, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 0, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"mobx\",\n \"7880ms\",\n \"src/batcher/core.ts\",\n \"Tobias Okafor\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5988, + "candidatesTokenCount": 53, + "totalTokenCount": 7237, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 653 + }, + { + "modality": "IMAGE", + "tokenCount": 5335 + } + ], + "thoughtsTokenCount": 1196, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 1, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "xstate", + "answer": "xstate", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"3480ms\",\n \"src/cursor/sync.ts\",\n \"Farid Moreau\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6087, + "candidatesTokenCount": 51, + "totalTokenCount": 7328, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 686 + }, + { + "modality": "IMAGE", + "tokenCount": 5401 + } + ], + "thoughtsTokenCount": 1190, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 1, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "3480", + "answer": "3480ms", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"3480ms\",\n \"src/cursor/sync.ts\",\n \"Farid Moreau\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6087, + "candidatesTokenCount": 51, + "totalTokenCount": 7328, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 686 + }, + { + "modality": "IMAGE", + "tokenCount": 5401 + } + ], + "thoughtsTokenCount": 1190, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 1, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/cursor/sync.ts", + "answer": "src/cursor/sync.ts", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"3480ms\",\n \"src/cursor/sync.ts\",\n \"Farid Moreau\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6087, + "candidatesTokenCount": 51, + "totalTokenCount": 7328, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 686 + }, + { + "modality": "IMAGE", + "tokenCount": 5401 + } + ], + "thoughtsTokenCount": 1190, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 1, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Farid Moreau", + "answer": "Farid Moreau", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"3480ms\",\n \"src/cursor/sync.ts\",\n \"Farid Moreau\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6087, + "candidatesTokenCount": 51, + "totalTokenCount": 7328, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 686 + }, + { + "modality": "IMAGE", + "tokenCount": 5401 + } + ], + "thoughtsTokenCount": 1190, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 1, + "type": "negation", + "q": "Was LEGACY_PINS enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"3480ms\",\n \"src/cursor/sync.ts\",\n \"Farid Moreau\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6087, + "candidatesTokenCount": 51, + "totalTokenCount": 7328, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 686 + }, + { + "modality": "IMAGE", + "tokenCount": 5401 + } + ], + "thoughtsTokenCount": 1190, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 1, + "type": "unanswerable", + "q": "Which AWS region was the failover assigned to?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"3480ms\",\n \"src/cursor/sync.ts\",\n \"Farid Moreau\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6087, + "candidatesTokenCount": 51, + "totalTokenCount": 7328, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 686 + }, + { + "modality": "IMAGE", + "tokenCount": 5401 + } + ], + "thoughtsTokenCount": 1190, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 2, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "immer", + "answer": "immer", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"9110ms\",\n \"src/cursor/io.ts\",\n \"Nadia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6114, + "candidatesTokenCount": 50, + "totalTokenCount": 7603, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5379 + }, + { + "modality": "TEXT", + "tokenCount": 735 + } + ], + "thoughtsTokenCount": 1439, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 2, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "9110", + "answer": "9110ms", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"9110ms\",\n \"src/cursor/io.ts\",\n \"Nadia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6114, + "candidatesTokenCount": 50, + "totalTokenCount": 7603, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5379 + }, + { + "modality": "TEXT", + "tokenCount": 735 + } + ], + "thoughtsTokenCount": 1439, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 2, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/cursor/io.ts", + "answer": "src/cursor/io.ts", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"9110ms\",\n \"src/cursor/io.ts\",\n \"Nadia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6114, + "candidatesTokenCount": 50, + "totalTokenCount": 7603, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5379 + }, + { + "modality": "TEXT", + "tokenCount": 735 + } + ], + "thoughtsTokenCount": 1439, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 2, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Nadia Costa", + "answer": "Nadia Costa", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"9110ms\",\n \"src/cursor/io.ts\",\n \"Nadia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6114, + "candidatesTokenCount": 50, + "totalTokenCount": 7603, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5379 + }, + { + "modality": "TEXT", + "tokenCount": 735 + } + ], + "thoughtsTokenCount": 1439, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 2, + "type": "negation", + "q": "Was ENABLE_SHARDING enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"9110ms\",\n \"src/cursor/io.ts\",\n \"Nadia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6114, + "candidatesTokenCount": 50, + "totalTokenCount": 7603, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5379 + }, + { + "modality": "TEXT", + "tokenCount": 735 + } + ], + "thoughtsTokenCount": 1439, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 2, + "type": "unanswerable", + "q": "What git tag was the hotfix released under?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"9110ms\",\n \"src/cursor/io.ts\",\n \"Nadia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6114, + "candidatesTokenCount": 50, + "totalTokenCount": 7603, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5379 + }, + { + "modality": "TEXT", + "tokenCount": 735 + } + ], + "thoughtsTokenCount": 1439, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 3, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "immer", + "answer": "immer", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"3470ms\",\n \"src/scheduler/core.ts\",\n \"Dmitri Berg\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4971, + "candidatesTokenCount": 51, + "totalTokenCount": 6147, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 621 + }, + { + "modality": "IMAGE", + "tokenCount": 4350 + } + ], + "thoughtsTokenCount": 1125, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 3, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "3470", + "answer": "3470ms", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"3470ms\",\n \"src/scheduler/core.ts\",\n \"Dmitri Berg\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4971, + "candidatesTokenCount": 51, + "totalTokenCount": 6147, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 621 + }, + { + "modality": "IMAGE", + "tokenCount": 4350 + } + ], + "thoughtsTokenCount": 1125, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 3, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/scheduler/core.ts", + "answer": "src/scheduler/core.ts", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"3470ms\",\n \"src/scheduler/core.ts\",\n \"Dmitri Berg\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4971, + "candidatesTokenCount": 51, + "totalTokenCount": 6147, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 621 + }, + { + "modality": "IMAGE", + "tokenCount": 4350 + } + ], + "thoughtsTokenCount": 1125, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 3, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Dmitri Berg", + "answer": "Dmitri Berg", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"3470ms\",\n \"src/scheduler/core.ts\",\n \"Dmitri Berg\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4971, + "candidatesTokenCount": 51, + "totalTokenCount": 6147, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 621 + }, + { + "modality": "IMAGE", + "tokenCount": 4350 + } + ], + "thoughtsTokenCount": 1125, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 3, + "type": "negation", + "q": "Was HOT_RELOAD_V2 enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"3470ms\",\n \"src/scheduler/core.ts\",\n \"Dmitri Berg\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4971, + "candidatesTokenCount": 51, + "totalTokenCount": 6147, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 621 + }, + { + "modality": "IMAGE", + "tokenCount": 4350 + } + ], + "thoughtsTokenCount": 1125, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 3, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"3470ms\",\n \"src/scheduler/core.ts\",\n \"Dmitri Berg\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4971, + "candidatesTokenCount": 51, + "totalTokenCount": 6147, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 621 + }, + { + "modality": "IMAGE", + "tokenCount": 4350 + } + ], + "thoughtsTokenCount": 1125, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 4, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "redux-toolkit", + "answer": "redux-toolkit", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6210ms\",\n \"src/cursor/io.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4942, + "candidatesTokenCount": 53, + "totalTokenCount": 6065, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 654 + }, + { + "modality": "IMAGE", + "tokenCount": 4288 + } + ], + "thoughtsTokenCount": 1070, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 4, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "6210", + "answer": "6210ms", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6210ms\",\n \"src/cursor/io.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4942, + "candidatesTokenCount": 53, + "totalTokenCount": 6065, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 654 + }, + { + "modality": "IMAGE", + "tokenCount": 4288 + } + ], + "thoughtsTokenCount": 1070, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 4, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/cursor/io.ts", + "answer": "src/cursor/io.ts", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6210ms\",\n \"src/cursor/io.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4942, + "candidatesTokenCount": 53, + "totalTokenCount": 6065, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 654 + }, + { + "modality": "IMAGE", + "tokenCount": 4288 + } + ], + "thoughtsTokenCount": 1070, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 4, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Tobias Khoury", + "answer": "Tobias Khoury", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6210ms\",\n \"src/cursor/io.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4942, + "candidatesTokenCount": 53, + "totalTokenCount": 6065, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 654 + }, + { + "modality": "IMAGE", + "tokenCount": 4288 + } + ], + "thoughtsTokenCount": 1070, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 4, + "type": "negation", + "q": "Was ASYNC_FSYNC enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6210ms\",\n \"src/cursor/io.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4942, + "candidatesTokenCount": 53, + "totalTokenCount": 6065, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 654 + }, + { + "modality": "IMAGE", + "tokenCount": 4288 + } + ], + "thoughtsTokenCount": 1070, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 4, + "type": "unanswerable", + "q": "Which AWS region was the failover assigned to?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6210ms\",\n \"src/cursor/io.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4942, + "candidatesTokenCount": 53, + "totalTokenCount": 6065, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 654 + }, + { + "modality": "IMAGE", + "tokenCount": 4288 + } + ], + "thoughtsTokenCount": 1070, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 5, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "redux-toolkit", + "answer": "redux-toolkit", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"1250ms\",\n \"src/cursor/io.ts\",\n \"Lucia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4983, + "candidatesTokenCount": 52, + "totalTokenCount": 6192, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 649 + }, + { + "modality": "IMAGE", + "tokenCount": 4334 + } + ], + "thoughtsTokenCount": 1157, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 5, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "1250", + "answer": "1250ms", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"1250ms\",\n \"src/cursor/io.ts\",\n \"Lucia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4983, + "candidatesTokenCount": 52, + "totalTokenCount": 6192, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 649 + }, + { + "modality": "IMAGE", + "tokenCount": 4334 + } + ], + "thoughtsTokenCount": 1157, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 5, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/cursor/io.ts", + "answer": "src/cursor/io.ts", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"1250ms\",\n \"src/cursor/io.ts\",\n \"Lucia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4983, + "candidatesTokenCount": 52, + "totalTokenCount": 6192, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 649 + }, + { + "modality": "IMAGE", + "tokenCount": 4334 + } + ], + "thoughtsTokenCount": 1157, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 5, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Lucia Costa", + "answer": "Lucia Costa", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"1250ms\",\n \"src/cursor/io.ts\",\n \"Lucia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4983, + "candidatesTokenCount": 52, + "totalTokenCount": 6192, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 649 + }, + { + "modality": "IMAGE", + "tokenCount": 4334 + } + ], + "thoughtsTokenCount": 1157, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 5, + "type": "negation", + "q": "Was HOT_RELOAD_V2 enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"1250ms\",\n \"src/cursor/io.ts\",\n \"Lucia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4983, + "candidatesTokenCount": 52, + "totalTokenCount": 6192, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 649 + }, + { + "modality": "IMAGE", + "tokenCount": 4334 + } + ], + "thoughtsTokenCount": 1157, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 5, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"1250ms\",\n \"src/cursor/io.ts\",\n \"Lucia Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 4983, + "candidatesTokenCount": 52, + "totalTokenCount": 6192, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 649 + }, + { + "modality": "IMAGE", + "tokenCount": 4334 + } + ], + "thoughtsTokenCount": 1157, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 6, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "immer", + "answer": "immer", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"6290ms\",\n \"src/scheduler/sync.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6070, + "candidatesTokenCount": 51, + "totalTokenCount": 7381, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5401 + }, + { + "modality": "TEXT", + "tokenCount": 669 + } + ], + "thoughtsTokenCount": 1260, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 6, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "6290", + "answer": "6290ms", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"6290ms\",\n \"src/scheduler/sync.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6070, + "candidatesTokenCount": 51, + "totalTokenCount": 7381, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5401 + }, + { + "modality": "TEXT", + "tokenCount": 669 + } + ], + "thoughtsTokenCount": 1260, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 6, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/scheduler/sync.ts", + "answer": "src/scheduler/sync.ts", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"6290ms\",\n \"src/scheduler/sync.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6070, + "candidatesTokenCount": 51, + "totalTokenCount": 7381, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5401 + }, + { + "modality": "TEXT", + "tokenCount": 669 + } + ], + "thoughtsTokenCount": 1260, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 6, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Tobias Khoury", + "answer": "Tobias Khoury", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"6290ms\",\n \"src/scheduler/sync.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6070, + "candidatesTokenCount": 51, + "totalTokenCount": 7381, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5401 + }, + { + "modality": "TEXT", + "tokenCount": 669 + } + ], + "thoughtsTokenCount": 1260, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 6, + "type": "negation", + "q": "Was LEGACY_PINS enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"6290ms\",\n \"src/scheduler/sync.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6070, + "candidatesTokenCount": 51, + "totalTokenCount": 7381, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5401 + }, + { + "modality": "TEXT", + "tokenCount": 669 + } + ], + "thoughtsTokenCount": 1260, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 6, + "type": "unanswerable", + "q": "Which AWS region was the failover assigned to?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"6290ms\",\n \"src/scheduler/sync.ts\",\n \"Tobias Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6070, + "candidatesTokenCount": 51, + "totalTokenCount": 7381, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5401 + }, + { + "modality": "TEXT", + "tokenCount": 669 + } + ], + "thoughtsTokenCount": 1260, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 7, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "xstate", + "answer": "xstate", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"7870ms\",\n \"src/batcher/sync.ts\",\n \"Priya Tanaka\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6035, + "candidatesTokenCount": 52, + "totalTokenCount": 7579, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5368 + }, + { + "modality": "TEXT", + "tokenCount": 667 + } + ], + "thoughtsTokenCount": 1492, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 7, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "7870", + "answer": "7870ms", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"7870ms\",\n \"src/batcher/sync.ts\",\n \"Priya Tanaka\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6035, + "candidatesTokenCount": 52, + "totalTokenCount": 7579, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5368 + }, + { + "modality": "TEXT", + "tokenCount": 667 + } + ], + "thoughtsTokenCount": 1492, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 7, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/batcher/sync.ts", + "answer": "src/batcher/sync.ts", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"7870ms\",\n \"src/batcher/sync.ts\",\n \"Priya Tanaka\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6035, + "candidatesTokenCount": 52, + "totalTokenCount": 7579, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5368 + }, + { + "modality": "TEXT", + "tokenCount": 667 + } + ], + "thoughtsTokenCount": 1492, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 7, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Priya Tanaka", + "answer": "Priya Tanaka", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"7870ms\",\n \"src/batcher/sync.ts\",\n \"Priya Tanaka\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6035, + "candidatesTokenCount": 52, + "totalTokenCount": 7579, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5368 + }, + { + "modality": "TEXT", + "tokenCount": 667 + } + ], + "thoughtsTokenCount": 1492, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 7, + "type": "negation", + "q": "Was HOT_RELOAD_V2 enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"7870ms\",\n \"src/batcher/sync.ts\",\n \"Priya Tanaka\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6035, + "candidatesTokenCount": 52, + "totalTokenCount": 7579, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5368 + }, + { + "modality": "TEXT", + "tokenCount": 667 + } + ], + "thoughtsTokenCount": 1492, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 7, + "type": "unanswerable", + "q": "What was the Docker base image pinned to?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"xstate\",\n \"7870ms\",\n \"src/batcher/sync.ts\",\n \"Priya Tanaka\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6035, + "candidatesTokenCount": 52, + "totalTokenCount": 7579, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5368 + }, + { + "modality": "TEXT", + "tokenCount": 667 + } + ], + "thoughtsTokenCount": 1492, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 8, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "jotai", + "answer": "jotai", + "ok": true, + "raw": "```json\n[\n \"jotai\",\n \"7850ms\",\n \"src/retry/io.ts\",\n \"Soren Petrov\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6095, + "candidatesTokenCount": 51, + "totalTokenCount": 7588, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 727 + }, + { + "modality": "IMAGE", + "tokenCount": 5368 + } + ], + "thoughtsTokenCount": 1442, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 8, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "7850", + "answer": "7850ms", + "ok": true, + "raw": "```json\n[\n \"jotai\",\n \"7850ms\",\n \"src/retry/io.ts\",\n \"Soren Petrov\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6095, + "candidatesTokenCount": 51, + "totalTokenCount": 7588, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 727 + }, + { + "modality": "IMAGE", + "tokenCount": 5368 + } + ], + "thoughtsTokenCount": 1442, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 8, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/retry/io.ts", + "answer": "src/retry/io.ts", + "ok": true, + "raw": "```json\n[\n \"jotai\",\n \"7850ms\",\n \"src/retry/io.ts\",\n \"Soren Petrov\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6095, + "candidatesTokenCount": 51, + "totalTokenCount": 7588, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 727 + }, + { + "modality": "IMAGE", + "tokenCount": 5368 + } + ], + "thoughtsTokenCount": 1442, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 8, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Soren Petrov", + "answer": "Soren Petrov", + "ok": true, + "raw": "```json\n[\n \"jotai\",\n \"7850ms\",\n \"src/retry/io.ts\",\n \"Soren Petrov\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6095, + "candidatesTokenCount": 51, + "totalTokenCount": 7588, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 727 + }, + { + "modality": "IMAGE", + "tokenCount": 5368 + } + ], + "thoughtsTokenCount": 1442, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 8, + "type": "negation", + "q": "Was ENABLE_SHARDING enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"jotai\",\n \"7850ms\",\n \"src/retry/io.ts\",\n \"Soren Petrov\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6095, + "candidatesTokenCount": 51, + "totalTokenCount": 7588, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 727 + }, + { + "modality": "IMAGE", + "tokenCount": 5368 + } + ], + "thoughtsTokenCount": 1442, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 8, + "type": "unanswerable", + "q": "What port number was the staging proxy moved to?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"jotai\",\n \"7850ms\",\n \"src/retry/io.ts\",\n \"Soren Petrov\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 6095, + "candidatesTokenCount": 51, + "totalTokenCount": 7588, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 727 + }, + { + "modality": "IMAGE", + "tokenCount": 5368 + } + ], + "thoughtsTokenCount": 1442, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 9, + "type": "decision", + "q": "Which package was chosen for the store layer?", + "gold": "redux-toolkit", + "answer": "redux-toolkit", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"4540ms\",\n \"src/cursor/sync.ts\",\n \"Ingrid Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5984, + "candidatesTokenCount": 52, + "totalTokenCount": 7092, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5329 + }, + { + "modality": "TEXT", + "tokenCount": 655 + } + ], + "thoughtsTokenCount": 1056, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 9, + "type": "numeric", + "q": "What exact value in ms was the retry budget set to?", + "gold": "4540", + "answer": "4540ms", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"4540ms\",\n \"src/cursor/sync.ts\",\n \"Ingrid Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5984, + "candidatesTokenCount": 52, + "totalTokenCount": 7092, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5329 + }, + { + "modality": "TEXT", + "tokenCount": 655 + } + ], + "thoughtsTokenCount": 1056, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 9, + "type": "path", + "q": "In which file path was the double-flush race found?", + "gold": "src/cursor/sync.ts", + "answer": "src/cursor/sync.ts", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"4540ms\",\n \"src/cursor/sync.ts\",\n \"Ingrid Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5984, + "candidatesTokenCount": 52, + "totalTokenCount": 7092, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5329 + }, + { + "modality": "TEXT", + "tokenCount": 655 + } + ], + "thoughtsTokenCount": 1056, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 9, + "type": "name", + "q": "Who was named as the on-call reviewer for the PR?", + "gold": "Ingrid Nakamura", + "answer": "Ingrid Nakamura", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"4540ms\",\n \"src/cursor/sync.ts\",\n \"Ingrid Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5984, + "candidatesTokenCount": 52, + "totalTokenCount": 7092, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5329 + }, + { + "modality": "TEXT", + "tokenCount": 655 + } + ], + "thoughtsTokenCount": 1056, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 9, + "type": "negation", + "q": "Was HOT_RELOAD_V2 enabled in prod? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"4540ms\",\n \"src/cursor/sync.ts\",\n \"Ingrid Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5984, + "candidatesTokenCount": 52, + "totalTokenCount": 7092, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5329 + }, + { + "modality": "TEXT", + "tokenCount": 655 + } + ], + "thoughtsTokenCount": 1056, + "serviceTier": "standard" + } + }, + { + "tier": "work", + "session": 9, + "type": "unanswerable", + "q": "What git tag was the hotfix released under?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"4540ms\",\n \"src/cursor/sync.ts\",\n \"Ingrid Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 5984, + "candidatesTokenCount": 52, + "totalTokenCount": 7092, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 5329 + }, + { + "modality": "TEXT", + "tokenCount": 655 + } + ], + "thoughtsTokenCount": 1056, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 0, + "type": "decision", + "q": "What was the FINAL package chosen for the store layer?", + "gold": "nanostores", + "answer": "nanostores", + "ok": true, + "raw": "```json\n[\n \"nanostores\",\n \"7850ms\",\n \"src/mailbox/core.ts\",\n \"Aiko Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13637, + "candidatesTokenCount": 53, + "totalTokenCount": 15226, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12922 + }, + { + "modality": "TEXT", + "tokenCount": 715 + } + ], + "thoughtsTokenCount": 1536, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 0, + "type": "numeric", + "q": "What exact value in ms was the RETRY BUDGET set to (not the cache TTL)?", + "gold": "7850", + "answer": "7850ms", + "ok": true, + "raw": "```json\n[\n \"nanostores\",\n \"7850ms\",\n \"src/mailbox/core.ts\",\n \"Aiko Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13637, + "candidatesTokenCount": 53, + "totalTokenCount": 15226, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12922 + }, + { + "modality": "TEXT", + "tokenCount": 715 + } + ], + "thoughtsTokenCount": 1536, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 0, + "type": "path", + "q": "Which file contained the ROOT CAUSE of the double-flush race?", + "gold": "src/mailbox/core.ts", + "answer": "src/mailbox/core.ts", + "ok": true, + "raw": "```json\n[\n \"nanostores\",\n \"7850ms\",\n \"src/mailbox/core.ts\",\n \"Aiko Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13637, + "candidatesTokenCount": 53, + "totalTokenCount": 15226, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12922 + }, + { + "modality": "TEXT", + "tokenCount": 715 + } + ], + "thoughtsTokenCount": 1536, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 0, + "type": "name", + "q": "Who is the on-call REVIEWER for the PR (not the author)?", + "gold": "Aiko Khoury", + "answer": "Aiko Khoury", + "ok": true, + "raw": "```json\n[\n \"nanostores\",\n \"7850ms\",\n \"src/mailbox/core.ts\",\n \"Aiko Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13637, + "candidatesTokenCount": 53, + "totalTokenCount": 15226, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12922 + }, + { + "modality": "TEXT", + "tokenCount": 715 + } + ], + "thoughtsTokenCount": 1536, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 0, + "type": "negation", + "q": "In PROD specifically, was LEGACY_PINS enabled? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"nanostores\",\n \"7850ms\",\n \"src/mailbox/core.ts\",\n \"Aiko Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13637, + "candidatesTokenCount": 53, + "totalTokenCount": 15226, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12922 + }, + { + "modality": "TEXT", + "tokenCount": 715 + } + ], + "thoughtsTokenCount": 1536, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 0, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"nanostores\",\n \"7850ms\",\n \"src/mailbox/core.ts\",\n \"Aiko Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13637, + "candidatesTokenCount": 53, + "totalTokenCount": 15226, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12922 + }, + { + "modality": "TEXT", + "tokenCount": 715 + } + ], + "thoughtsTokenCount": 1536, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 1, + "type": "decision", + "q": "What was the FINAL package chosen for the store layer?", + "gold": "immer", + "answer": "immer", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"1330ms\",\n \"src/retry/core.ts\",\n \"Mara Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14812, + "candidatesTokenCount": 50, + "totalTokenCount": 17026, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 14036 + }, + { + "modality": "TEXT", + "tokenCount": 776 + } + ], + "thoughtsTokenCount": 2164, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 1, + "type": "numeric", + "q": "What exact value in ms was the RETRY BUDGET set to (not the cache TTL)?", + "gold": "1330", + "answer": "1330ms", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"1330ms\",\n \"src/retry/core.ts\",\n \"Mara Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14812, + "candidatesTokenCount": 50, + "totalTokenCount": 17026, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 14036 + }, + { + "modality": "TEXT", + "tokenCount": 776 + } + ], + "thoughtsTokenCount": 2164, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 1, + "type": "path", + "q": "Which file contained the ROOT CAUSE of the double-flush race?", + "gold": "src/retry/core.ts", + "answer": "src/retry/core.ts", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"1330ms\",\n \"src/retry/core.ts\",\n \"Mara Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14812, + "candidatesTokenCount": 50, + "totalTokenCount": 17026, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 14036 + }, + { + "modality": "TEXT", + "tokenCount": 776 + } + ], + "thoughtsTokenCount": 2164, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 1, + "type": "name", + "q": "Who is the on-call REVIEWER for the PR (not the author)?", + "gold": "Mara Khoury", + "answer": "Mara Khoury", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"1330ms\",\n \"src/retry/core.ts\",\n \"Mara Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14812, + "candidatesTokenCount": 50, + "totalTokenCount": 17026, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 14036 + }, + { + "modality": "TEXT", + "tokenCount": 776 + } + ], + "thoughtsTokenCount": 2164, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 1, + "type": "negation", + "q": "In PROD specifically, was USE_BROTLI enabled? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"1330ms\",\n \"src/retry/core.ts\",\n \"Mara Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14812, + "candidatesTokenCount": 50, + "totalTokenCount": 17026, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 14036 + }, + { + "modality": "TEXT", + "tokenCount": 776 + } + ], + "thoughtsTokenCount": 2164, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 1, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"1330ms\",\n \"src/retry/core.ts\",\n \"Mara Khoury\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14812, + "candidatesTokenCount": 50, + "totalTokenCount": 17026, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 14036 + }, + { + "modality": "TEXT", + "tokenCount": 776 + } + ], + "thoughtsTokenCount": 2164, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 2, + "type": "decision", + "q": "What was the FINAL package chosen for the store layer?", + "gold": "immer", + "answer": "immer", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"4570ms\",\n \"src/quota/core.ts\",\n \"Nadia Alvarez\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13696, + "candidatesTokenCount": 50, + "totalTokenCount": 14861, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12956 + }, + { + "modality": "TEXT", + "tokenCount": 740 + } + ], + "thoughtsTokenCount": 1115, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 2, + "type": "numeric", + "q": "What exact value in ms was the RETRY BUDGET set to (not the cache TTL)?", + "gold": "4570", + "answer": "4570ms", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"4570ms\",\n \"src/quota/core.ts\",\n \"Nadia Alvarez\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13696, + "candidatesTokenCount": 50, + "totalTokenCount": 14861, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12956 + }, + { + "modality": "TEXT", + "tokenCount": 740 + } + ], + "thoughtsTokenCount": 1115, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 2, + "type": "path", + "q": "Which file contained the ROOT CAUSE of the double-flush race?", + "gold": "src/quota/core.ts", + "answer": "src/quota/core.ts", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"4570ms\",\n \"src/quota/core.ts\",\n \"Nadia Alvarez\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13696, + "candidatesTokenCount": 50, + "totalTokenCount": 14861, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12956 + }, + { + "modality": "TEXT", + "tokenCount": 740 + } + ], + "thoughtsTokenCount": 1115, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 2, + "type": "name", + "q": "Who is the on-call REVIEWER for the PR (not the author)?", + "gold": "Nadia Alvarez", + "answer": "Nadia Alvarez", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"4570ms\",\n \"src/quota/core.ts\",\n \"Nadia Alvarez\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13696, + "candidatesTokenCount": 50, + "totalTokenCount": 14861, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12956 + }, + { + "modality": "TEXT", + "tokenCount": 740 + } + ], + "thoughtsTokenCount": 1115, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 2, + "type": "negation", + "q": "In PROD specifically, was ENABLE_SHARDING enabled? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"4570ms\",\n \"src/quota/core.ts\",\n \"Nadia Alvarez\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13696, + "candidatesTokenCount": 50, + "totalTokenCount": 14861, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12956 + }, + { + "modality": "TEXT", + "tokenCount": 740 + } + ], + "thoughtsTokenCount": 1115, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 2, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"4570ms\",\n \"src/quota/core.ts\",\n \"Nadia Alvarez\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13696, + "candidatesTokenCount": 50, + "totalTokenCount": 14861, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12956 + }, + { + "modality": "TEXT", + "tokenCount": 740 + } + ], + "thoughtsTokenCount": 1115, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 3, + "type": "decision", + "q": "What was the FINAL package chosen for the store layer?", + "gold": "redux-toolkit", + "answer": "redux-toolkit", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6270ms\",\n \"src/batcher/core.ts\",\n \"Farid Lindqvist\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13605, + "candidatesTokenCount": 54, + "totalTokenCount": 14929, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12858 + }, + { + "modality": "TEXT", + "tokenCount": 747 + } + ], + "thoughtsTokenCount": 1270, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 3, + "type": "numeric", + "q": "What exact value in ms was the RETRY BUDGET set to (not the cache TTL)?", + "gold": "6270", + "answer": "6270ms", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6270ms\",\n \"src/batcher/core.ts\",\n \"Farid Lindqvist\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13605, + "candidatesTokenCount": 54, + "totalTokenCount": 14929, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12858 + }, + { + "modality": "TEXT", + "tokenCount": 747 + } + ], + "thoughtsTokenCount": 1270, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 3, + "type": "path", + "q": "Which file contained the ROOT CAUSE of the double-flush race?", + "gold": "src/batcher/core.ts", + "answer": "src/batcher/core.ts", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6270ms\",\n \"src/batcher/core.ts\",\n \"Farid Lindqvist\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13605, + "candidatesTokenCount": 54, + "totalTokenCount": 14929, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12858 + }, + { + "modality": "TEXT", + "tokenCount": 747 + } + ], + "thoughtsTokenCount": 1270, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 3, + "type": "name", + "q": "Who is the on-call REVIEWER for the PR (not the author)?", + "gold": "Farid Lindqvist", + "answer": "Farid Lindqvist", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6270ms\",\n \"src/batcher/core.ts\",\n \"Farid Lindqvist\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13605, + "candidatesTokenCount": 54, + "totalTokenCount": 14929, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12858 + }, + { + "modality": "TEXT", + "tokenCount": 747 + } + ], + "thoughtsTokenCount": 1270, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 3, + "type": "negation", + "q": "In PROD specifically, was ASYNC_FSYNC enabled? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6270ms\",\n \"src/batcher/core.ts\",\n \"Farid Lindqvist\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13605, + "candidatesTokenCount": 54, + "totalTokenCount": 14929, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12858 + }, + { + "modality": "TEXT", + "tokenCount": 747 + } + ], + "thoughtsTokenCount": 1270, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 3, + "type": "unanswerable", + "q": "What was the Docker base image pinned to?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"redux-toolkit\",\n \"6270ms\",\n \"src/batcher/core.ts\",\n \"Farid Lindqvist\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13605, + "candidatesTokenCount": 54, + "totalTokenCount": 14929, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12858 + }, + { + "modality": "TEXT", + "tokenCount": 747 + } + ], + "thoughtsTokenCount": 1270, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 4, + "type": "decision", + "q": "What was the FINAL package chosen for the store layer?", + "gold": "zustand", + "answer": "zustand", + "ok": true, + "raw": "```json\n[\n \"zustand\",\n \"9120ms\",\n \"src/cursor/core.ts\",\n \"Soren Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14792, + "candidatesTokenCount": 51, + "totalTokenCount": 16909, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 756 + }, + { + "modality": "IMAGE", + "tokenCount": 14036 + } + ], + "thoughtsTokenCount": 2066, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 4, + "type": "numeric", + "q": "What exact value in ms was the RETRY BUDGET set to (not the cache TTL)?", + "gold": "9120", + "answer": "9120ms", + "ok": true, + "raw": "```json\n[\n \"zustand\",\n \"9120ms\",\n \"src/cursor/core.ts\",\n \"Soren Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14792, + "candidatesTokenCount": 51, + "totalTokenCount": 16909, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 756 + }, + { + "modality": "IMAGE", + "tokenCount": 14036 + } + ], + "thoughtsTokenCount": 2066, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 4, + "type": "path", + "q": "Which file contained the ROOT CAUSE of the double-flush race?", + "gold": "src/cursor/core.ts", + "answer": "src/cursor/core.ts", + "ok": true, + "raw": "```json\n[\n \"zustand\",\n \"9120ms\",\n \"src/cursor/core.ts\",\n \"Soren Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14792, + "candidatesTokenCount": 51, + "totalTokenCount": 16909, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 756 + }, + { + "modality": "IMAGE", + "tokenCount": 14036 + } + ], + "thoughtsTokenCount": 2066, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 4, + "type": "name", + "q": "Who is the on-call REVIEWER for the PR (not the author)?", + "gold": "Soren Nakamura", + "answer": "Soren Nakamura", + "ok": true, + "raw": "```json\n[\n \"zustand\",\n \"9120ms\",\n \"src/cursor/core.ts\",\n \"Soren Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14792, + "candidatesTokenCount": 51, + "totalTokenCount": 16909, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 756 + }, + { + "modality": "IMAGE", + "tokenCount": 14036 + } + ], + "thoughtsTokenCount": 2066, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 4, + "type": "negation", + "q": "In PROD specifically, was HOT_RELOAD_V2 enabled? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"zustand\",\n \"9120ms\",\n \"src/cursor/core.ts\",\n \"Soren Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14792, + "candidatesTokenCount": 51, + "totalTokenCount": 16909, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 756 + }, + { + "modality": "IMAGE", + "tokenCount": 14036 + } + ], + "thoughtsTokenCount": 2066, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 4, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"zustand\",\n \"9120ms\",\n \"src/cursor/core.ts\",\n \"Soren Nakamura\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 14792, + "candidatesTokenCount": 51, + "totalTokenCount": 16909, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 756 + }, + { + "modality": "IMAGE", + "tokenCount": 14036 + } + ], + "thoughtsTokenCount": 2066, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 5, + "type": "decision", + "q": "What was the FINAL package chosen for the store layer?", + "gold": "immer", + "answer": "immer", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"7800ms\",\n \"src/quota/core.ts\",\n \"Ingrid Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12601, + "candidatesTokenCount": 50, + "totalTokenCount": 14342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11844 + }, + { + "modality": "TEXT", + "tokenCount": 757 + } + ], + "thoughtsTokenCount": 1691, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 5, + "type": "numeric", + "q": "What exact value in ms was the RETRY BUDGET set to (not the cache TTL)?", + "gold": "7800", + "answer": "7800ms", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"7800ms\",\n \"src/quota/core.ts\",\n \"Ingrid Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12601, + "candidatesTokenCount": 50, + "totalTokenCount": 14342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11844 + }, + { + "modality": "TEXT", + "tokenCount": 757 + } + ], + "thoughtsTokenCount": 1691, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 5, + "type": "path", + "q": "Which file contained the ROOT CAUSE of the double-flush race?", + "gold": "src/quota/core.ts", + "answer": "src/quota/core.ts", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"7800ms\",\n \"src/quota/core.ts\",\n \"Ingrid Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12601, + "candidatesTokenCount": 50, + "totalTokenCount": 14342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11844 + }, + { + "modality": "TEXT", + "tokenCount": 757 + } + ], + "thoughtsTokenCount": 1691, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 5, + "type": "name", + "q": "Who is the on-call REVIEWER for the PR (not the author)?", + "gold": "Ingrid Costa", + "answer": "Ingrid Costa", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"7800ms\",\n \"src/quota/core.ts\",\n \"Ingrid Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12601, + "candidatesTokenCount": 50, + "totalTokenCount": 14342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11844 + }, + { + "modality": "TEXT", + "tokenCount": 757 + } + ], + "thoughtsTokenCount": 1691, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 5, + "type": "negation", + "q": "In PROD specifically, was USE_BROTLI enabled? Answer ENABLED or OFF.", + "gold": "OFF", + "answer": "OFF", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"7800ms\",\n \"src/quota/core.ts\",\n \"Ingrid Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12601, + "candidatesTokenCount": 50, + "totalTokenCount": 14342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11844 + }, + { + "modality": "TEXT", + "tokenCount": 757 + } + ], + "thoughtsTokenCount": 1691, + "serviceTier": "standard" + } + }, + { + "tier": "work2", + "session": 5, + "type": "unanswerable", + "q": "Which database migration version was rolled back?", + "gold": "UNKNOWN", + "answer": "UNKNOWN", + "ok": true, + "raw": "```json\n[\n \"immer\",\n \"7800ms\",\n \"src/quota/core.ts\",\n \"Ingrid Costa\",\n \"OFF\",\n \"UNKNOWN\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12601, + "candidatesTokenCount": 50, + "totalTokenCount": 14342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11844 + }, + { + "modality": "TEXT", + "tokenCount": 757 + } + ], + "thoughtsTokenCount": 1691, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 0, + "type": "final", + "q": "What is the FINAL (locked) value of BATCH_WINDOW_MS at the end of the session?", + "gold": "8400", + "answer": "8400", + "ok": true, + "raw": "```json\n[\n \"8400\",\n \"9600\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12576, + "candidatesTokenCount": 29, + "totalTokenCount": 13538, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 708 + }, + { + "modality": "IMAGE", + "tokenCount": 11868 + } + ], + "thoughtsTokenCount": 933, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 0, + "type": "first", + "q": "What was the FIRST value BATCH_WINDOW_MS was set to at the start?", + "gold": "9600", + "answer": "9600", + "ok": true, + "raw": "```json\n[\n \"8400\",\n \"9600\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12576, + "candidatesTokenCount": 29, + "totalTokenCount": 13538, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 708 + }, + { + "modality": "IMAGE", + "tokenCount": 11868 + } + ], + "thoughtsTokenCount": 933, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 0, + "type": "count", + "q": "How many distinct values was BATCH_WINDOW_MS set to over the whole session? Answer with a number.", + "gold": "3", + "answer": "3", + "ok": true, + "raw": "```json\n[\n \"8400\",\n \"9600\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12576, + "candidatesTokenCount": 29, + "totalTokenCount": 13538, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 708 + }, + { + "modality": "IMAGE", + "tokenCount": 11868 + } + ], + "thoughtsTokenCount": 933, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 1, + "type": "final", + "q": "What is the FINAL (locked) value of BATCH_WINDOW_MS at the end of the session?", + "gold": "1200", + "answer": "1200", + "ok": true, + "raw": "```json\n[\n \"1200\",\n \"5400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13604, + "candidatesTokenCount": 29, + "totalTokenCount": 14393, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12911 + }, + { + "modality": "TEXT", + "tokenCount": 693 + } + ], + "thoughtsTokenCount": 760, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 1, + "type": "first", + "q": "What was the FIRST value BATCH_WINDOW_MS was set to at the start?", + "gold": "5400", + "answer": "5400", + "ok": true, + "raw": "```json\n[\n \"1200\",\n \"5400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13604, + "candidatesTokenCount": 29, + "totalTokenCount": 14393, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12911 + }, + { + "modality": "TEXT", + "tokenCount": 693 + } + ], + "thoughtsTokenCount": 760, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 1, + "type": "count", + "q": "How many distinct values was BATCH_WINDOW_MS set to over the whole session? Answer with a number.", + "gold": "3", + "answer": "3", + "ok": true, + "raw": "```json\n[\n \"1200\",\n \"5400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13604, + "candidatesTokenCount": 29, + "totalTokenCount": 14393, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12911 + }, + { + "modality": "TEXT", + "tokenCount": 693 + } + ], + "thoughtsTokenCount": 760, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 2, + "type": "final", + "q": "What is the FINAL (locked) value of FLUSH_INTERVAL_MS at the end of the session?", + "gold": "7200", + "answer": "7200", + "ok": true, + "raw": "```json\n[\n \"7200\",\n \"2400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12518, + "candidatesTokenCount": 29, + "totalTokenCount": 13630, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 698 + }, + { + "modality": "IMAGE", + "tokenCount": 11820 + } + ], + "thoughtsTokenCount": 1083, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 2, + "type": "first", + "q": "What was the FIRST value FLUSH_INTERVAL_MS was set to at the start?", + "gold": "2400", + "answer": "2400", + "ok": true, + "raw": "```json\n[\n \"7200\",\n \"2400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12518, + "candidatesTokenCount": 29, + "totalTokenCount": 13630, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 698 + }, + { + "modality": "IMAGE", + "tokenCount": 11820 + } + ], + "thoughtsTokenCount": 1083, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 2, + "type": "count", + "q": "How many distinct values was FLUSH_INTERVAL_MS set to over the whole session? Answer with a number.", + "gold": "3", + "answer": "3", + "ok": true, + "raw": "```json\n[\n \"7200\",\n \"2400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 12518, + "candidatesTokenCount": 29, + "totalTokenCount": 13630, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 698 + }, + { + "modality": "IMAGE", + "tokenCount": 11820 + } + ], + "thoughtsTokenCount": 1083, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 3, + "type": "final", + "q": "What is the FINAL (locked) value of LEASE_TTL_MS at the end of the session?", + "gold": "7200", + "answer": "7200", + "ok": true, + "raw": "```json\n[\n \"7200\",\n \"2400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13646, + "candidatesTokenCount": 29, + "totalTokenCount": 14571, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12898 + }, + { + "modality": "TEXT", + "tokenCount": 748 + } + ], + "thoughtsTokenCount": 896, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 3, + "type": "first", + "q": "What was the FIRST value LEASE_TTL_MS was set to at the start?", + "gold": "2400", + "answer": "2400", + "ok": true, + "raw": "```json\n[\n \"7200\",\n \"2400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13646, + "candidatesTokenCount": 29, + "totalTokenCount": 14571, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12898 + }, + { + "modality": "TEXT", + "tokenCount": 748 + } + ], + "thoughtsTokenCount": 896, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 3, + "type": "count", + "q": "How many distinct values was LEASE_TTL_MS set to over the whole session? Answer with a number.", + "gold": "3", + "answer": "3", + "ok": true, + "raw": "```json\n[\n \"7200\",\n \"2400\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13646, + "candidatesTokenCount": 29, + "totalTokenCount": 14571, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 12898 + }, + { + "modality": "TEXT", + "tokenCount": 748 + } + ], + "thoughtsTokenCount": 896, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 4, + "type": "final", + "q": "What is the FINAL (locked) value of BATCH_WINDOW_MS at the end of the session?", + "gold": "4800", + "answer": "4800", + "ok": true, + "raw": "```json\n[\n \"4800\",\n \"7200\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13628, + "candidatesTokenCount": 29, + "totalTokenCount": 14763, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 714 + }, + { + "modality": "IMAGE", + "tokenCount": 12914 + } + ], + "thoughtsTokenCount": 1106, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 4, + "type": "first", + "q": "What was the FIRST value BATCH_WINDOW_MS was set to at the start?", + "gold": "7200", + "answer": "7200", + "ok": true, + "raw": "```json\n[\n \"4800\",\n \"7200\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13628, + "candidatesTokenCount": 29, + "totalTokenCount": 14763, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 714 + }, + { + "modality": "IMAGE", + "tokenCount": 12914 + } + ], + "thoughtsTokenCount": 1106, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 4, + "type": "count", + "q": "How many distinct values was BATCH_WINDOW_MS set to over the whole session? Answer with a number.", + "gold": "3", + "answer": "3", + "ok": true, + "raw": "```json\n[\n \"4800\",\n \"7200\",\n \"3\"\n]\n```", + "error": null, + "usage": { + "promptTokenCount": 13628, + "candidatesTokenCount": 29, + "totalTokenCount": 14763, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 714 + }, + { + "modality": "IMAGE", + "tokenCount": 12914 + } + ], + "thoughtsTokenCount": 1106, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 5, + "type": "final", + "q": "What is the FINAL (locked) value of BATCH_WINDOW_MS at the end of the session?", + "gold": "8400", + "answer": "8400", + "ok": true, + "raw": "[\n \"8400\",\n \"3600\",\n \"3\"\n]", + "error": null, + "usage": { + "promptTokenCount": 12645, + "candidatesTokenCount": 24, + "totalTokenCount": 13529, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11896 + }, + { + "modality": "TEXT", + "tokenCount": 749 + } + ], + "thoughtsTokenCount": 860, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 5, + "type": "first", + "q": "What was the FIRST value BATCH_WINDOW_MS was set to at the start?", + "gold": "3600", + "answer": "3600", + "ok": true, + "raw": "[\n \"8400\",\n \"3600\",\n \"3\"\n]", + "error": null, + "usage": { + "promptTokenCount": 12645, + "candidatesTokenCount": 24, + "totalTokenCount": 13529, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11896 + }, + { + "modality": "TEXT", + "tokenCount": 749 + } + ], + "thoughtsTokenCount": 860, + "serviceTier": "standard" + } + }, + { + "tier": "work3", + "session": 5, + "type": "count", + "q": "How many distinct values was BATCH_WINDOW_MS set to over the whole session? Answer with a number.", + "gold": "3", + "answer": "3", + "ok": true, + "raw": "[\n \"8400\",\n \"3600\",\n \"3\"\n]", + "error": null, + "usage": { + "promptTokenCount": 12645, + "candidatesTokenCount": 24, + "totalTokenCount": 13529, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 11896 + }, + { + "modality": "TEXT", + "tokenCount": 749 + } + ], + "thoughtsTokenCount": 860, + "serviceTier": "standard" + } + } + ] +} \ No newline at end of file diff --git a/eval/gemini-profile/gist-recall.mjs b/eval/gemini-profile/gist-recall.mjs new file mode 100644 index 0000000..7b68ee3 --- /dev/null +++ b/eval/gemini-profile/gist-recall.mjs @@ -0,0 +1,85 @@ +// Gemini 3.6 Flash gist recall evaluation suite. +import { readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { renderTextToPngs } from '../../dist/core/render.js'; +import { resolveGeminiProfile } from '../../dist/core/gemini-model-profiles.js'; +import { factSheetText } from '../../dist/core/factsheet.js'; +import { callGemini } from './gemini-client.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ROOT = join(HERE, '../gist-recall'); +const MODEL = process.env.MODEL || 'gemini-3.6-flash'; +const profile = resolveGeminiProfile(); +const LIVE = process.env.LIVE === '1'; +const TIMEOUT = Number(process.env.TIMEOUT_MS || 240000); +const TIERS = [['work', 10], ['work2', 6], ['work3', 6]]; + +function parse(s) { + const a = s.indexOf('['), b = s.lastIndexOf(']'); + try { return JSON.parse(a >= 0 && b > a ? s.slice(a, b + 1) : s); } catch { return null; } +} +function norm(s) { return String(s ?? '').trim().toLowerCase().replace(/\s+/g, ' '); } +function correct(p, a) { + const x = norm(a), g = norm(p.gold); + if (p.type === 'unanswerable') return x === 'unknown'; + if (p.type === 'numeric') return new RegExp(`(?:^|\\D)${g}(?:\\D|$)`).test(x); + if (p.type === 'negation') return x.includes('off') && !x.includes('enabled'); + return x.includes(g); +} + +const rows = []; +for (const [dir, n] of TIERS) { + const probes = JSON.parse(readFileSync(join(ROOT, dir, 'probes.json'), 'utf8')); + for (let sid = 0; sid < n; sid++) { + const ps = probes.filter((p) => p.session === sid); + const source = readFileSync(join(ROOT, dir, `s${sid}.txt`), 'utf8'); + const imgs = await renderTextToPngs(source, profile.stripCols, profile.style, profile.maxHeightPx); + const prompt = [ + 'Read all transcript images in order. Answer every numbered question.', + 'If the transcript does not contain an answer, use exactly UNKNOWN.', + 'Return only a JSON array of strings in question order.', + ...ps.map((p, i) => `${i + 1}. ${p.q}`) + ].join('\n'); + let response = { output: '', usage: null }; + if (LIVE) { + const content = imgs.map((im) => ({ type: 'input_image', image_url: `data:image/png;base64,${Buffer.from(im.png).toString('base64')}` })); + const fs = factSheetText(source, profile.factSheetFormat); + if (fs) content.push({ type: 'input_text', text: fs }); + content.push({ type: 'input_text', text: prompt }); + try { + const r = await callGemini({ model: MODEL, content, maxOutputTokens: 1400, timeoutMs: TIMEOUT }); + response = { output: r.text, usage: r.usage }; + } catch (e) { + response = { output: '', usage: null, error: String(e) }; + } + } + const answers = parse(response.output) || []; + ps.forEach((p, i) => rows.push({ tier: dir, session: sid, ...p, answer: String(answers[i] ?? ''), ok: correct(p, answers[i]), raw: response.output, error: response.error || null, usage: response.usage })); + console.log(`${dir} s${sid}: ${ps.filter((p, i) => correct(p, answers[i])).length}/${ps.length}`); + } +} + +if (!LIVE) { + console.log('Dry run only; no receipt written'); + process.exit(0); +} + +const answerable = rows.filter((r) => r.type !== 'unanswerable'); +const guards = rows.filter((r) => r.type === 'unanswerable'); +const state = rows.filter((r) => r.tier === 'work3'); +const done = (xs) => xs.filter((r) => !r.error); + +const out = { + generatedAt: new Date().toISOString(), + model: MODEL, + live: LIVE, + recipe: { cols: profile.stripCols, maxH: profile.maxHeightPx, style: profile.style, factsheet: true }, + answerable: { correct: done(answerable).filter((r) => r.ok).length, completed: done(answerable).length, n: answerable.length }, + state: { correct: done(state).filter((r) => r.ok).length, completed: done(state).length, n: state.length }, + unanswerable: { confabulated: done(guards).filter((r) => !r.ok).length, completed: done(guards).length, n: guards.length }, + rows +}; + +writeFileSync(join(HERE, 'gist-recall-results.json'), JSON.stringify(out, null, 2)); +console.log(JSON.stringify({ answerable: out.answerable, state: out.state, unanswerable: out.unanswerable }, null, 2)); diff --git a/eval/gemini-profile/novel-arithmetic-results.json b/eval/gemini-profile/novel-arithmetic-results.json new file mode 100644 index 0000000..963ca32 --- /dev/null +++ b/eval/gemini-profile/novel-arithmetic-results.json @@ -0,0 +1,6332 @@ +{ + "generatedAt": "2026-07-21T20:16:20.262Z", + "model": "gemini-3.6-flash", + "live": true, + "n": 100, + "seed": 20260711, + "recipe": { + "cols": 312, + "maxH": 728, + "style": { + "font": "spleen-5x8", + "cellWBonus": 0, + "cellHBonus": 0, + "aa": true, + "grid": false, + "gridCols": 0, + "colorCycle": false, + "markerScale": 1, + "markerRed": false, + "inkDilate": 0 + }, + "factsheet": true + }, + "textCorrect": 100, + "pureCorrect": 100, + "prodCorrect": 100, + "textPct": 100, + "purePct": 100, + "prodPct": 100, + "rows": [ + { + "i": 0, + "kind": 2, + "question": "A warehouse has 85 shelves, each holding 59 boxes, plus 987 loose boxes. How many boxes are there in total?", + "answer": 6002, + "imageTokens": 1078, + "textGot": 6002, + "pureGot": 6002, + "prodGot": 6002, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 113, + "totalTokenCount": 439, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 274, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 101, + "totalTokenCount": 1595, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 474, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 101, + "totalTokenCount": 1588, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 457, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 1, + "kind": 2, + "question": "A warehouse has 35 shelves, each holding 82 boxes, plus 251 loose boxes. How many boxes are there in total?", + "answer": 3121, + "imageTokens": 1078, + "textGot": 3121, + "pureGot": 3121, + "prodGot": 3121, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 77, + "totalTokenCount": 383, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 254, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 113, + "totalTokenCount": 1452, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 319, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1387, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 253, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 2, + "kind": 2, + "question": "A warehouse has 17 shelves, each holding 94 boxes, plus 955 loose boxes. How many boxes are there in total?", + "answer": 2553, + "imageTokens": 1078, + "textGot": 2553, + "pureGot": 2553, + "prodGot": 2553, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 76, + "totalTokenCount": 365, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 237, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 109, + "totalTokenCount": 1448, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 319, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 79, + "totalTokenCount": 1432, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 323, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 3, + "kind": 2, + "question": "A warehouse has 88 shelves, each holding 97 boxes, plus 563 loose boxes. How many boxes are there in total?", + "answer": 9099, + "imageTokens": 1078, + "textGot": 9099, + "pureGot": 9099, + "prodGot": 9099, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 88, + "totalTokenCount": 390, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 250, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 97, + "totalTokenCount": 1384, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 267, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 107, + "totalTokenCount": 1512, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 375, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 4, + "kind": 2, + "question": "A warehouse has 38 shelves, each holding 60 boxes, plus 587 loose boxes. How many boxes are there in total?", + "answer": 2867, + "imageTokens": 1078, + "textGot": 2867, + "pureGot": 2867, + "prodGot": 2867, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 102, + "totalTokenCount": 328, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 174, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 104, + "totalTokenCount": 1386, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 262, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 93, + "totalTokenCount": 1444, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 321, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 5, + "kind": 2, + "question": "A warehouse has 60 shelves, each holding 16 boxes, plus 715 loose boxes. How many boxes are there in total?", + "answer": 1675, + "imageTokens": 1078, + "textGot": 1675, + "pureGot": 1675, + "prodGot": 1675, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 80, + "totalTokenCount": 301, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 169, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 112, + "totalTokenCount": 1807, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 675, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 94, + "totalTokenCount": 2930, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 1806, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 6, + "kind": 2, + "question": "A warehouse has 85 shelves, each holding 53 boxes, plus 655 loose boxes. How many boxes are there in total?", + "answer": 5160, + "imageTokens": 1078, + "textGot": 5160, + "pureGot": 5160, + "prodGot": 5160, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 87, + "totalTokenCount": 401, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 262, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 109, + "totalTokenCount": 1403, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 274, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 96, + "totalTokenCount": 1512, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 386, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 7, + "kind": 2, + "question": "A warehouse has 29 shelves, each holding 99 boxes, plus 115 loose boxes. How many boxes are there in total?", + "answer": 2986, + "imageTokens": 1078, + "textGot": 2986, + "pureGot": 2986, + "prodGot": 2986, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 88, + "totalTokenCount": 380, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 240, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 110, + "totalTokenCount": 1394, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 264, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 87, + "totalTokenCount": 1397, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 280, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 8, + "kind": 2, + "question": "A warehouse has 98 shelves, each holding 98 boxes, plus 183 loose boxes. How many boxes are there in total?", + "answer": 9787, + "imageTokens": 1078, + "textGot": 9787, + "pureGot": 9787, + "prodGot": 9787, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 84, + "totalTokenCount": 497, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 361, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 101, + "totalTokenCount": 1379, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 258, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 108, + "totalTokenCount": 1356, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 218, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 9, + "kind": 2, + "question": "A warehouse has 46 shelves, each holding 97 boxes, plus 731 loose boxes. How many boxes are there in total?", + "answer": 5193, + "imageTokens": 1078, + "textGot": 5193, + "pureGot": 5193, + "prodGot": 5193, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 95, + "totalTokenCount": 402, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 255, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 85, + "totalTokenCount": 1565, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 460, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1637, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 503, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 10, + "kind": 2, + "question": "A warehouse has 77 shelves, each holding 57 boxes, plus 139 loose boxes. How many boxes are there in total?", + "answer": 4528, + "imageTokens": 1078, + "textGot": 4528, + "pureGot": 4528, + "prodGot": 4528, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 485, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 351, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 91, + "totalTokenCount": 1484, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 373, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 125, + "totalTokenCount": 1472, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 317, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 11, + "kind": 2, + "question": "A warehouse has 67 shelves, each holding 93 boxes, plus 251 loose boxes. How many boxes are there in total?", + "answer": 6482, + "imageTokens": 1078, + "textGot": 6482, + "pureGot": 6482, + "prodGot": 6482, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 103, + "totalTokenCount": 447, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 292, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 108, + "totalTokenCount": 1341, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 213, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 87, + "totalTokenCount": 1588, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 471, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 12, + "kind": 2, + "question": "A warehouse has 52 shelves, each holding 42 boxes, plus 239 loose boxes. How many boxes are there in total?", + "answer": 2423, + "imageTokens": 1078, + "textGot": 2423, + "pureGot": 2423, + "prodGot": 2423, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 111, + "totalTokenCount": 412, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 249, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 115, + "totalTokenCount": 1363, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 228, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 107, + "totalTokenCount": 1389, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 252, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 13, + "kind": 2, + "question": "A warehouse has 19 shelves, each holding 62 boxes, plus 511 loose boxes. How many boxes are there in total?", + "answer": 1689, + "imageTokens": 1078, + "textGot": 1689, + "pureGot": 1689, + "prodGot": 1689, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 74, + "totalTokenCount": 352, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 226, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 105, + "totalTokenCount": 1528, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 403, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 91, + "totalTokenCount": 1438, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 317, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 14, + "kind": 2, + "question": "A warehouse has 88 shelves, each holding 33 boxes, plus 427 loose boxes. How many boxes are there in total?", + "answer": 3331, + "imageTokens": 1078, + "textGot": 3331, + "pureGot": 3331, + "prodGot": 3331, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 100, + "totalTokenCount": 400, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 248, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 118, + "totalTokenCount": 1435, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 297, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 95, + "totalTokenCount": 1489, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 364, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 15, + "kind": 2, + "question": "A warehouse has 42 shelves, each holding 58 boxes, plus 971 loose boxes. How many boxes are there in total?", + "answer": 3407, + "imageTokens": 1078, + "textGot": 3407, + "pureGot": 3407, + "prodGot": 3407, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 103, + "totalTokenCount": 395, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 240, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 110, + "totalTokenCount": 1422, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 292, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 84, + "totalTokenCount": 1512, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 398, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 16, + "kind": 2, + "question": "A warehouse has 70 shelves, each holding 81 boxes, plus 211 loose boxes. How many boxes are there in total?", + "answer": 5881, + "imageTokens": 1078, + "textGot": 5881, + "pureGot": 5881, + "prodGot": 5881, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 78, + "totalTokenCount": 376, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 246, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 83, + "totalTokenCount": 1290, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 187, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1396, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 262, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 17, + "kind": 2, + "question": "A warehouse has 12 shelves, each holding 56 boxes, plus 155 loose boxes. How many boxes are there in total?", + "answer": 827, + "imageTokens": 1078, + "textGot": 827, + "pureGot": 827, + "prodGot": 827, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 95, + "totalTokenCount": 346, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 199, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 95, + "totalTokenCount": 1354, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 239, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 91, + "totalTokenCount": 1323, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 202, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 18, + "kind": 2, + "question": "A warehouse has 16 shelves, each holding 78 boxes, plus 251 loose boxes. How many boxes are there in total?", + "answer": 1499, + "imageTokens": 1078, + "textGot": 1499, + "pureGot": 1499, + "prodGot": 1499, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 79, + "totalTokenCount": 421, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 290, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 114, + "totalTokenCount": 1440, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 306, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 105, + "totalTokenCount": 1428, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 293, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 19, + "kind": 2, + "question": "A warehouse has 91 shelves, each holding 71 boxes, plus 139 loose boxes. How many boxes are there in total?", + "answer": 6600, + "imageTokens": 1078, + "textGot": 6600, + "pureGot": 6600, + "prodGot": 6600, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 398, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 264, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 106, + "totalTokenCount": 1419, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 293, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 84, + "totalTokenCount": 1830, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 716, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 20, + "kind": 2, + "question": "A warehouse has 33 shelves, each holding 21 boxes, plus 903 loose boxes. How many boxes are there in total?", + "answer": 1596, + "imageTokens": 1078, + "textGot": 1596, + "pureGot": 1596, + "prodGot": 1596, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 110, + "totalTokenCount": 401, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 239, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 111, + "totalTokenCount": 1342, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 211, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 90, + "totalTokenCount": 1413, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 293, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 21, + "kind": 2, + "question": "A warehouse has 21 shelves, each holding 14 boxes, plus 207 loose boxes. How many boxes are there in total?", + "answer": 501, + "imageTokens": 1078, + "textGot": 501, + "pureGot": 501, + "prodGot": 501, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 93, + "totalTokenCount": 319, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 174, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 106, + "totalTokenCount": 1342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 216, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 107, + "totalTokenCount": 1379, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 242, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 22, + "kind": 2, + "question": "A warehouse has 48 shelves, each holding 21 boxes, plus 419 loose boxes. How many boxes are there in total?", + "answer": 1427, + "imageTokens": 1078, + "textGot": 1427, + "pureGot": 1427, + "prodGot": 1427, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 87, + "totalTokenCount": 319, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 180, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 113, + "totalTokenCount": 1403, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 270, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 79, + "totalTokenCount": 1342, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 233, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 23, + "kind": 2, + "question": "A warehouse has 68 shelves, each holding 50 boxes, plus 443 loose boxes. How many boxes are there in total?", + "answer": 3843, + "imageTokens": 1078, + "textGot": 3843, + "pureGot": 3843, + "prodGot": 3843, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 92, + "totalTokenCount": 296, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 152, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 92, + "totalTokenCount": 1358, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 246, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 85, + "totalTokenCount": 1417, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 302, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 24, + "kind": 2, + "question": "A warehouse has 67 shelves, each holding 50 boxes, plus 211 loose boxes. How many boxes are there in total?", + "answer": 3561, + "imageTokens": 1078, + "textGot": 3561, + "pureGot": 3561, + "prodGot": 3561, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 79, + "totalTokenCount": 291, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 160, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 111, + "totalTokenCount": 1333, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 202, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 95, + "totalTokenCount": 1345, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 220, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 25, + "kind": 2, + "question": "A warehouse has 45 shelves, each holding 27 boxes, plus 411 loose boxes. How many boxes are there in total?", + "answer": 1626, + "imageTokens": 1078, + "textGot": 1626, + "pureGot": 1626, + "prodGot": 1626, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 97, + "totalTokenCount": 406, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 257, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 108, + "totalTokenCount": 1468, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 340, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 107, + "totalTokenCount": 1566, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 429, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 26, + "kind": 2, + "question": "A warehouse has 79 shelves, each holding 55 boxes, plus 563 loose boxes. How many boxes are there in total?", + "answer": 4908, + "imageTokens": 1078, + "textGot": 4908, + "pureGot": 4908, + "prodGot": 4908, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 90, + "totalTokenCount": 368, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 226, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 80, + "totalTokenCount": 1497, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 397, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 109, + "totalTokenCount": 1571, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 432, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 27, + "kind": 2, + "question": "A warehouse has 72 shelves, each holding 73 boxes, plus 951 loose boxes. How many boxes are there in total?", + "answer": 6207, + "imageTokens": 1078, + "textGot": 6207, + "pureGot": 6207, + "prodGot": 6207, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 86, + "totalTokenCount": 382, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 244, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 110, + "totalTokenCount": 1406, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 276, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 79, + "totalTokenCount": 1402, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 293, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 28, + "kind": 2, + "question": "A warehouse has 91 shelves, each holding 78 boxes, plus 395 loose boxes. How many boxes are there in total?", + "answer": 7493, + "imageTokens": 1078, + "textGot": 7493, + "pureGot": 7493, + "prodGot": 7493, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 103, + "totalTokenCount": 513, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 358, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 81, + "totalTokenCount": 1377, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 276, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 113, + "totalTokenCount": 1452, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 309, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 29, + "kind": 2, + "question": "A warehouse has 39 shelves, each holding 40 boxes, plus 475 loose boxes. How many boxes are there in total?", + "answer": 2035, + "imageTokens": 1078, + "textGot": 2035, + "pureGot": 2035, + "prodGot": 2035, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 74, + "totalTokenCount": 318, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 192, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 76, + "totalTokenCount": 1284, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 188, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 83, + "totalTokenCount": 1374, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 261, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 30, + "kind": 2, + "question": "A warehouse has 66 shelves, each holding 32 boxes, plus 803 loose boxes. How many boxes are there in total?", + "answer": 2915, + "imageTokens": 1078, + "textGot": 2915, + "pureGot": 2915, + "prodGot": 2915, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 95, + "totalTokenCount": 344, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 197, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 105, + "totalTokenCount": 1377, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 252, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 101, + "totalTokenCount": 1510, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 379, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 31, + "kind": 2, + "question": "A warehouse has 91 shelves, each holding 30 boxes, plus 575 loose boxes. How many boxes are there in total?", + "answer": 3305, + "imageTokens": 1078, + "textGot": 3305, + "pureGot": 3305, + "prodGot": 3305, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 79, + "totalTokenCount": 377, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 246, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 92, + "totalTokenCount": 1446, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 334, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 95, + "totalTokenCount": 1329, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 204, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 32, + "kind": 2, + "question": "A warehouse has 42 shelves, each holding 59 boxes, plus 327 loose boxes. How many boxes are there in total?", + "answer": 2805, + "imageTokens": 1078, + "textGot": 2805, + "pureGot": 2805, + "prodGot": 2805, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 109, + "totalTokenCount": 393, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 232, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 87, + "totalTokenCount": 1362, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 255, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 107, + "totalTokenCount": 1490, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 353, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 33, + "kind": 2, + "question": "A warehouse has 89 shelves, each holding 80 boxes, plus 523 loose boxes. How many boxes are there in total?", + "answer": 7643, + "imageTokens": 1078, + "textGot": 7643, + "pureGot": 7643, + "prodGot": 7643, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 104, + "totalTokenCount": 340, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 184, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 97, + "totalTokenCount": 1437, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 320, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1405, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 271, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 34, + "kind": 2, + "question": "A warehouse has 67 shelves, each holding 98 boxes, plus 503 loose boxes. How many boxes are there in total?", + "answer": 7069, + "imageTokens": 1078, + "textGot": 7069, + "pureGot": 7069, + "prodGot": 7069, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 85, + "totalTokenCount": 352, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 215, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 78, + "totalTokenCount": 1410, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 312, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 108, + "totalTokenCount": 1496, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 358, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 35, + "kind": 2, + "question": "A warehouse has 43 shelves, each holding 33 boxes, plus 911 loose boxes. How many boxes are there in total?", + "answer": 2330, + "imageTokens": 1078, + "textGot": 2330, + "pureGot": 2330, + "prodGot": 2330, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 103, + "totalTokenCount": 498, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 343, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 107, + "totalTokenCount": 1389, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 262, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 82, + "totalTokenCount": 1520, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 408, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 36, + "kind": 2, + "question": "A warehouse has 45 shelves, each holding 85 boxes, plus 651 loose boxes. How many boxes are there in total?", + "answer": 4476, + "imageTokens": 1078, + "textGot": 4476, + "pureGot": 4476, + "prodGot": 4476, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 386, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 252, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 125, + "totalTokenCount": 1467, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 322, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 98, + "totalTokenCount": 1453, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 325, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 37, + "kind": 2, + "question": "A warehouse has 55 shelves, each holding 44 boxes, plus 647 loose boxes. How many boxes are there in total?", + "answer": 3067, + "imageTokens": 1078, + "textGot": 3067, + "pureGot": 3067, + "prodGot": 3067, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 105, + "totalTokenCount": 358, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 201, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 110, + "totalTokenCount": 1634, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 504, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 100, + "totalTokenCount": 1830, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 700, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 38, + "kind": 2, + "question": "A warehouse has 20 shelves, each holding 28 boxes, plus 523 loose boxes. How many boxes are there in total?", + "answer": 1083, + "imageTokens": 1078, + "textGot": 1083, + "pureGot": 1083, + "prodGot": 1083, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 58, + "totalTokenCount": 282, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 172, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 98, + "totalTokenCount": 1287, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 169, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 101, + "totalTokenCount": 1378, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 247, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 39, + "kind": 2, + "question": "A warehouse has 83 shelves, each holding 22 boxes, plus 911 loose boxes. How many boxes are there in total?", + "answer": 2737, + "imageTokens": 1078, + "textGot": 2737, + "pureGot": 2737, + "prodGot": 2737, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 78, + "totalTokenCount": 371, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 241, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 111, + "totalTokenCount": 1408, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 277, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 105, + "totalTokenCount": 1429, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 294, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 40, + "kind": 2, + "question": "A warehouse has 59 shelves, each holding 87 boxes, plus 427 loose boxes. How many boxes are there in total?", + "answer": 5560, + "imageTokens": 1078, + "textGot": 5560, + "pureGot": 5560, + "prodGot": 5560, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 85, + "totalTokenCount": 413, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 276, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 87, + "totalTokenCount": 1495, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 388, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 109, + "totalTokenCount": 1576, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 437, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 41, + "kind": 2, + "question": "A warehouse has 55 shelves, each holding 63 boxes, plus 563 loose boxes. How many boxes are there in total?", + "answer": 4028, + "imageTokens": 1078, + "textGot": 4028, + "pureGot": 4028, + "prodGot": 4028, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 83, + "totalTokenCount": 367, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 232, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 100, + "totalTokenCount": 1454, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 334, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 101, + "totalTokenCount": 1534, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 403, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 42, + "kind": 2, + "question": "A warehouse has 91 shelves, each holding 42 boxes, plus 175 loose boxes. How many boxes are there in total?", + "answer": 3997, + "imageTokens": 1078, + "textGot": 3997, + "pureGot": 3997, + "prodGot": 3997, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 94, + "totalTokenCount": 357, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 211, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 101, + "totalTokenCount": 1414, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 293, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 105, + "totalTokenCount": 1405, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 270, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 43, + "kind": 2, + "question": "A warehouse has 25 shelves, each holding 87 boxes, plus 623 loose boxes. How many boxes are there in total?", + "answer": 2798, + "imageTokens": 1078, + "textGot": 2798, + "pureGot": 2798, + "prodGot": 2798, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 87, + "totalTokenCount": 327, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 188, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 110, + "totalTokenCount": 1364, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 234, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 87, + "totalTokenCount": 1498, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 381, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 44, + "kind": 2, + "question": "A warehouse has 93 shelves, each holding 61 boxes, plus 111 loose boxes. How many boxes are there in total?", + "answer": 5784, + "imageTokens": 1078, + "textGot": 5784, + "pureGot": 5784, + "prodGot": 5784, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 112, + "totalTokenCount": 405, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 241, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 116, + "totalTokenCount": 1395, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 259, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 105, + "totalTokenCount": 1384, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 249, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 45, + "kind": 2, + "question": "A warehouse has 63 shelves, each holding 65 boxes, plus 479 loose boxes. How many boxes are there in total?", + "answer": 4574, + "imageTokens": 1078, + "textGot": 4574, + "pureGot": 4574, + "prodGot": 4574, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 77, + "totalTokenCount": 454, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 325, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 109, + "totalTokenCount": 1381, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 252, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 109, + "totalTokenCount": 1426, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 287, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 46, + "kind": 2, + "question": "A warehouse has 54 shelves, each holding 95 boxes, plus 687 loose boxes. How many boxes are there in total?", + "answer": 5817, + "imageTokens": 1078, + "textGot": 5817, + "pureGot": 5817, + "prodGot": 5817, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 86, + "totalTokenCount": 433, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 295, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 94, + "totalTokenCount": 1349, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 235, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 89, + "totalTokenCount": 1446, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 327, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 47, + "kind": 2, + "question": "A warehouse has 48 shelves, each holding 29 boxes, plus 571 loose boxes. How many boxes are there in total?", + "answer": 1963, + "imageTokens": 1078, + "textGot": 1963, + "pureGot": 1963, + "prodGot": 1963, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 321, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 187, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 121, + "totalTokenCount": 1402, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 261, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 106, + "totalTokenCount": 1433, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 297, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 48, + "kind": 2, + "question": "A warehouse has 71 shelves, each holding 37 boxes, plus 587 loose boxes. How many boxes are there in total?", + "answer": 3214, + "imageTokens": 1078, + "textGot": 3214, + "pureGot": 3214, + "prodGot": 3214, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 84, + "totalTokenCount": 420, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 284, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 79, + "totalTokenCount": 1482, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 383, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 111, + "totalTokenCount": 1570, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 429, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 49, + "kind": 2, + "question": "A warehouse has 41 shelves, each holding 50 boxes, plus 335 loose boxes. How many boxes are there in total?", + "answer": 2385, + "imageTokens": 1078, + "textGot": 2385, + "pureGot": 2385, + "prodGot": 2385, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 112, + "totalTokenCount": 334, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 170, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 100, + "totalTokenCount": 1297, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 177, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1336, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 202, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 50, + "kind": 2, + "question": "A warehouse has 39 shelves, each holding 24 boxes, plus 843 loose boxes. How many boxes are there in total?", + "answer": 1779, + "imageTokens": 1078, + "textGot": 1779, + "pureGot": 1779, + "prodGot": 1779, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 108, + "totalTokenCount": 393, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 233, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 106, + "totalTokenCount": 1425, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 299, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 103, + "totalTokenCount": 1411, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 278, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 51, + "kind": 2, + "question": "A warehouse has 28 shelves, each holding 39 boxes, plus 787 loose boxes. How many boxes are there in total?", + "answer": 1879, + "imageTokens": 1078, + "textGot": 1879, + "pureGot": 1879, + "prodGot": 1879, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 110, + "totalTokenCount": 413, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 251, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 96, + "totalTokenCount": 1555, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 439, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 95, + "totalTokenCount": 1597, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 472, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 52, + "kind": 2, + "question": "A warehouse has 97 shelves, each holding 38 boxes, plus 539 loose boxes. How many boxes are there in total?", + "answer": 4225, + "imageTokens": 1078, + "textGot": 4225, + "pureGot": 4225, + "prodGot": 4225, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 96, + "totalTokenCount": 384, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 236, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 82, + "totalTokenCount": 1358, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 256, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 110, + "totalTokenCount": 1568, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 428, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 53, + "kind": 2, + "question": "A warehouse has 67 shelves, each holding 55 boxes, plus 671 loose boxes. How many boxes are there in total?", + "answer": 4356, + "imageTokens": 1078, + "textGot": 4356, + "pureGot": 4356, + "prodGot": 4356, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 112, + "totalTokenCount": 528, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 364, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 107, + "totalTokenCount": 1543, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 416, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1519, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 385, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 54, + "kind": 2, + "question": "A warehouse has 89 shelves, each holding 43 boxes, plus 759 loose boxes. How many boxes are there in total?", + "answer": 4586, + "imageTokens": 1078, + "textGot": 4586, + "pureGot": 4586, + "prodGot": 4586, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 84, + "totalTokenCount": 451, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 315, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 111, + "totalTokenCount": 1589, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 458, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 100, + "totalTokenCount": 1451, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 321, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 55, + "kind": 2, + "question": "A warehouse has 68 shelves, each holding 80 boxes, plus 675 loose boxes. How many boxes are there in total?", + "answer": 6115, + "imageTokens": 1078, + "textGot": 6115, + "pureGot": 6115, + "prodGot": 6115, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 102, + "totalTokenCount": 361, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 207, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 101, + "totalTokenCount": 1404, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 283, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 98, + "totalTokenCount": 1437, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 309, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 56, + "kind": 2, + "question": "A warehouse has 95 shelves, each holding 92 boxes, plus 503 loose boxes. How many boxes are there in total?", + "answer": 9243, + "imageTokens": 1078, + "textGot": 9243, + "pureGot": 9243, + "prodGot": 9243, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 87, + "totalTokenCount": 382, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 243, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 102, + "totalTokenCount": 1486, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 364, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 79, + "totalTokenCount": 1356, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 247, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 57, + "kind": 2, + "question": "A warehouse has 23 shelves, each holding 28 boxes, plus 483 loose boxes. How many boxes are there in total?", + "answer": 1127, + "imageTokens": 1078, + "textGot": 1127, + "pureGot": 1127, + "prodGot": 1127, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 343, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 209, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 95, + "totalTokenCount": 1420, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 305, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 77, + "totalTokenCount": 1430, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 323, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 58, + "kind": 2, + "question": "A warehouse has 65 shelves, each holding 39 boxes, plus 327 loose boxes. How many boxes are there in total?", + "answer": 2862, + "imageTokens": 1078, + "textGot": 2862, + "pureGot": 2862, + "prodGot": 2862, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 340, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 206, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 100, + "totalTokenCount": 1522, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 402, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 80, + "totalTokenCount": 1392, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 282, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 59, + "kind": 2, + "question": "A warehouse has 50 shelves, each holding 30 boxes, plus 883 loose boxes. How many boxes are there in total?", + "answer": 2383, + "imageTokens": 1078, + "textGot": 2383, + "pureGot": 2383, + "prodGot": 2383, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 81, + "totalTokenCount": 269, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 136, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 77, + "totalTokenCount": 1340, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 243, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 96, + "totalTokenCount": 1332, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 206, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 60, + "kind": 2, + "question": "A warehouse has 75 shelves, each holding 16 boxes, plus 411 loose boxes. How many boxes are there in total?", + "answer": 1611, + "imageTokens": 1078, + "textGot": 1611, + "pureGot": 1611, + "prodGot": 1611, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 81, + "totalTokenCount": 339, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 206, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 99, + "totalTokenCount": 1418, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 299, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 100, + "totalTokenCount": 1382, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 252, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 61, + "kind": 2, + "question": "A warehouse has 36 shelves, each holding 58 boxes, plus 167 loose boxes. How many boxes are there in total?", + "answer": 2255, + "imageTokens": 1078, + "textGot": 2255, + "pureGot": 2255, + "prodGot": 2255, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 78, + "totalTokenCount": 398, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 268, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 90, + "totalTokenCount": 1415, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 305, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 75, + "totalTokenCount": 1423, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 318, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 62, + "kind": 2, + "question": "A warehouse has 90 shelves, each holding 73 boxes, plus 775 loose boxes. How many boxes are there in total?", + "answer": 7345, + "imageTokens": 1078, + "textGot": 7345, + "pureGot": 7345, + "prodGot": 7345, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 84, + "totalTokenCount": 373, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 237, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 107, + "totalTokenCount": 1857, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 730, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 116, + "totalTokenCount": 2024, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 878, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 63, + "kind": 2, + "question": "A warehouse has 78 shelves, each holding 48 boxes, plus 535 loose boxes. How many boxes are there in total?", + "answer": 4279, + "imageTokens": 1078, + "textGot": 4279, + "pureGot": 4279, + "prodGot": 4279, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 85, + "totalTokenCount": 452, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 315, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 104, + "totalTokenCount": 1416, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 292, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 111, + "totalTokenCount": 1560, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 419, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 64, + "kind": 2, + "question": "A warehouse has 83 shelves, each holding 93 boxes, plus 771 loose boxes. How many boxes are there in total?", + "answer": 8490, + "imageTokens": 1078, + "textGot": 8490, + "pureGot": 8490, + "prodGot": 8490, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 99, + "totalTokenCount": 507, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 356, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 83, + "totalTokenCount": 1925, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 822, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 99, + "totalTokenCount": 1927, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 798, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 65, + "kind": 2, + "question": "A warehouse has 31 shelves, each holding 30 boxes, plus 691 loose boxes. How many boxes are there in total?", + "answer": 1621, + "imageTokens": 1078, + "textGot": 1621, + "pureGot": 1621, + "prodGot": 1621, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 78, + "totalTokenCount": 344, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 214, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 103, + "totalTokenCount": 1369, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 246, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 100, + "totalTokenCount": 1426, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 296, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 66, + "kind": 2, + "question": "A warehouse has 56 shelves, each holding 28 boxes, plus 619 loose boxes. How many boxes are there in total?", + "answer": 2187, + "imageTokens": 1078, + "textGot": 2187, + "pureGot": 2187, + "prodGot": 2187, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 427, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 293, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 97, + "totalTokenCount": 1527, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 410, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 102, + "totalTokenCount": 1452, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 320, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 67, + "kind": 2, + "question": "A warehouse has 76 shelves, each holding 46 boxes, plus 695 loose boxes. How many boxes are there in total?", + "answer": 4191, + "imageTokens": 1078, + "textGot": 4191, + "pureGot": 4191, + "prodGot": 4191, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 86, + "totalTokenCount": 400, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 262, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 99, + "totalTokenCount": 1412, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 293, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 99, + "totalTokenCount": 1392, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 263, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 68, + "kind": 2, + "question": "A warehouse has 22 shelves, each holding 99 boxes, plus 107 loose boxes. How many boxes are there in total?", + "answer": 2285, + "imageTokens": 1078, + "textGot": 2285, + "pureGot": 2285, + "prodGot": 2285, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 105, + "totalTokenCount": 360, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 203, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 75, + "totalTokenCount": 1386, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 291, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 99, + "totalTokenCount": 1348, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 219, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 69, + "kind": 2, + "question": "A warehouse has 84 shelves, each holding 75 boxes, plus 343 loose boxes. How many boxes are there in total?", + "answer": 6643, + "imageTokens": 1078, + "textGot": 6643, + "pureGot": 6643, + "prodGot": 6643, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 127, + "totalTokenCount": 447, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 268, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 108, + "totalTokenCount": 1487, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 359, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 85, + "totalTokenCount": 1452, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 337, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 70, + "kind": 2, + "question": "A warehouse has 66 shelves, each holding 22 boxes, plus 131 loose boxes. How many boxes are there in total?", + "answer": 1583, + "imageTokens": 1078, + "textGot": 1583, + "pureGot": 1583, + "prodGot": 1583, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 80, + "totalTokenCount": 339, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 207, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 97, + "totalTokenCount": 1342, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 225, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1411, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 277, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 71, + "kind": 2, + "question": "A warehouse has 48 shelves, each holding 99 boxes, plus 735 loose boxes. How many boxes are there in total?", + "answer": 5487, + "imageTokens": 1078, + "textGot": 5487, + "pureGot": 5487, + "prodGot": 5487, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 85, + "totalTokenCount": 382, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 245, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 114, + "totalTokenCount": 1730, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 596, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1720, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 586, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 72, + "kind": 2, + "question": "A warehouse has 95 shelves, each holding 71 boxes, plus 119 loose boxes. How many boxes are there in total?", + "answer": 6864, + "imageTokens": 1078, + "textGot": 6864, + "pureGot": 6864, + "prodGot": 6864, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 85, + "totalTokenCount": 391, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 254, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 85, + "totalTokenCount": 1604, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 499, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 76, + "totalTokenCount": 1899, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 793, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 73, + "kind": 2, + "question": "A warehouse has 18 shelves, each holding 47 boxes, plus 583 loose boxes. How many boxes are there in total?", + "answer": 1429, + "imageTokens": 1078, + "textGot": 1429, + "pureGot": 1429, + "prodGot": 1429, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 102, + "totalTokenCount": 493, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 339, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 115, + "totalTokenCount": 1487, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 352, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 105, + "totalTokenCount": 1539, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 404, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 74, + "kind": 2, + "question": "A warehouse has 31 shelves, each holding 12 boxes, plus 579 loose boxes. How many boxes are there in total?", + "answer": 951, + "imageTokens": 1078, + "textGot": 951, + "pureGot": 951, + "prodGot": 951, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 69, + "totalTokenCount": 346, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 225, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 115, + "totalTokenCount": 1376, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 241, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 108, + "totalTokenCount": 1377, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 239, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 75, + "kind": 2, + "question": "A warehouse has 36 shelves, each holding 98 boxes, plus 379 loose boxes. How many boxes are there in total?", + "answer": 3907, + "imageTokens": 1078, + "textGot": 3907, + "pureGot": 3907, + "prodGot": 3907, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 80, + "totalTokenCount": 448, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 316, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 102, + "totalTokenCount": 1464, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 342, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 84, + "totalTokenCount": 1449, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 335, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 76, + "kind": 2, + "question": "A warehouse has 88 shelves, each holding 91 boxes, plus 735 loose boxes. How many boxes are there in total?", + "answer": 8743, + "imageTokens": 1078, + "textGot": 8743, + "pureGot": 8743, + "prodGot": 8743, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 111, + "totalTokenCount": 479, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 316, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 96, + "totalTokenCount": 1765, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 649, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 98, + "totalTokenCount": 1642, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 514, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 77, + "kind": 2, + "question": "A warehouse has 24 shelves, each holding 64 boxes, plus 715 loose boxes. How many boxes are there in total?", + "answer": 2251, + "imageTokens": 1078, + "textGot": 2251, + "pureGot": 2251, + "prodGot": 2251, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 402, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 268, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 98, + "totalTokenCount": 1827, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 709, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 111, + "totalTokenCount": 4177, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 3036, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 78, + "kind": 2, + "question": "A warehouse has 24 shelves, each holding 73 boxes, plus 595 loose boxes. How many boxes are there in total?", + "answer": 2347, + "imageTokens": 1078, + "textGot": 2347, + "pureGot": 2347, + "prodGot": 2347, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 80, + "totalTokenCount": 378, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 246, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 113, + "totalTokenCount": 1668, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 535, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 79, + "totalTokenCount": 1476, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 367, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 79, + "kind": 2, + "question": "A warehouse has 24 shelves, each holding 23 boxes, plus 363 loose boxes. How many boxes are there in total?", + "answer": 915, + "imageTokens": 1078, + "textGot": 915, + "pureGot": 915, + "prodGot": 915, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 76, + "totalTokenCount": 347, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 219, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 106, + "totalTokenCount": 1356, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 230, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 98, + "totalTokenCount": 1382, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 254, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 80, + "kind": 2, + "question": "A warehouse has 81 shelves, each holding 27 boxes, plus 979 loose boxes. How many boxes are there in total?", + "answer": 3166, + "imageTokens": 1078, + "textGot": 3166, + "pureGot": 3166, + "prodGot": 3166, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 102, + "totalTokenCount": 448, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 294, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 82, + "totalTokenCount": 1420, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 318, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 90, + "totalTokenCount": 1625, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 505, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 81, + "kind": 2, + "question": "A warehouse has 35 shelves, each holding 72 boxes, plus 623 loose boxes. How many boxes are there in total?", + "answer": 3143, + "imageTokens": 1078, + "textGot": 3143, + "pureGot": 3143, + "prodGot": 3143, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 98, + "totalTokenCount": 409, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 259, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 113, + "totalTokenCount": 1504, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 371, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 102, + "totalTokenCount": 1561, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 429, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 82, + "kind": 2, + "question": "A warehouse has 98 shelves, each holding 92 boxes, plus 123 loose boxes. How many boxes are there in total?", + "answer": 9139, + "imageTokens": 1078, + "textGot": 9139, + "pureGot": 9139, + "prodGot": 9139, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 94, + "totalTokenCount": 357, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 211, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 113, + "totalTokenCount": 1441, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 308, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 79, + "totalTokenCount": 1520, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 411, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 83, + "kind": 2, + "question": "A warehouse has 26 shelves, each holding 43 boxes, plus 291 loose boxes. How many boxes are there in total?", + "answer": 1409, + "imageTokens": 1078, + "textGot": 1409, + "pureGot": 1409, + "prodGot": 1409, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 79, + "totalTokenCount": 355, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 224, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 112, + "totalTokenCount": 1466, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 334, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 81, + "totalTokenCount": 1448, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 337, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 84, + "kind": 2, + "question": "A warehouse has 44 shelves, each holding 77 boxes, plus 911 loose boxes. How many boxes are there in total?", + "answer": 4299, + "imageTokens": 1078, + "textGot": 4299, + "pureGot": 4299, + "prodGot": 4299, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 113, + "totalTokenCount": 369, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 204, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 79, + "totalTokenCount": 1711, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 612, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 78, + "totalTokenCount": 1511, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 403, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 85, + "kind": 2, + "question": "A warehouse has 79 shelves, each holding 75 boxes, plus 739 loose boxes. How many boxes are there in total?", + "answer": 6664, + "imageTokens": 1078, + "textGot": 6664, + "pureGot": 6664, + "prodGot": 6664, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 84, + "totalTokenCount": 424, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 288, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 81, + "totalTokenCount": 1493, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 392, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 85, + "totalTokenCount": 1606, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 491, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 86, + "kind": 2, + "question": "A warehouse has 61 shelves, each holding 45 boxes, plus 283 loose boxes. How many boxes are there in total?", + "answer": 3028, + "imageTokens": 1078, + "textGot": 3028, + "pureGot": 3028, + "prodGot": 3028, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 111, + "totalTokenCount": 411, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 248, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 81, + "totalTokenCount": 1359, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 258, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 92, + "totalTokenCount": 1429, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 307, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 87, + "kind": 2, + "question": "A warehouse has 84 shelves, each holding 78 boxes, plus 195 loose boxes. How many boxes are there in total?", + "answer": 6747, + "imageTokens": 1078, + "textGot": 6747, + "pureGot": 6747, + "prodGot": 6747, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 111, + "totalTokenCount": 465, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 302, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 105, + "totalTokenCount": 1472, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 347, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 105, + "totalTokenCount": 1459, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 324, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 88, + "kind": 2, + "question": "A warehouse has 12 shelves, each holding 19 boxes, plus 695 loose boxes. How many boxes are there in total?", + "answer": 923, + "imageTokens": 1078, + "textGot": 923, + "pureGot": 923, + "prodGot": 923, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 97, + "totalTokenCount": 357, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 208, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 110, + "totalTokenCount": 1439, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 309, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 91, + "totalTokenCount": 1359, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 238, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 89, + "kind": 2, + "question": "A warehouse has 55 shelves, each holding 11 boxes, plus 107 loose boxes. How many boxes are there in total?", + "answer": 712, + "imageTokens": 1078, + "textGot": 712, + "pureGot": 712, + "prodGot": 712, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 78, + "totalTokenCount": 307, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 177, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 90, + "totalTokenCount": 1479, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 369, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 107, + "totalTokenCount": 1365, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 228, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 90, + "kind": 2, + "question": "A warehouse has 52 shelves, each holding 93 boxes, plus 327 loose boxes. How many boxes are there in total?", + "answer": 5163, + "imageTokens": 1078, + "textGot": 5163, + "pureGot": 5163, + "prodGot": 5163, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 81, + "totalTokenCount": 363, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 230, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 79, + "totalTokenCount": 1366, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 267, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 97, + "totalTokenCount": 1383, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 256, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 91, + "kind": 2, + "question": "A warehouse has 41 shelves, each holding 30 boxes, plus 779 loose boxes. How many boxes are there in total?", + "answer": 2009, + "imageTokens": 1078, + "textGot": 2009, + "pureGot": 2009, + "prodGot": 2009, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 100, + "totalTokenCount": 372, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 220, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 101, + "totalTokenCount": 1615, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 494, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 96, + "totalTokenCount": 1671, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 545, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 92, + "kind": 2, + "question": "A warehouse has 38 shelves, each holding 66 boxes, plus 359 loose boxes. How many boxes are there in total?", + "answer": 2867, + "imageTokens": 1078, + "textGot": 2867, + "pureGot": 2867, + "prodGot": 2867, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 79, + "totalTokenCount": 385, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 254, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 107, + "totalTokenCount": 1434, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 307, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 109, + "totalTokenCount": 1468, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 329, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 93, + "kind": 2, + "question": "A warehouse has 83 shelves, each holding 26 boxes, plus 919 loose boxes. How many boxes are there in total?", + "answer": 3077, + "imageTokens": 1078, + "textGot": 3077, + "pureGot": 3077, + "prodGot": 3077, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 81, + "totalTokenCount": 421, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 288, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 95, + "totalTokenCount": 1496, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 381, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 101, + "totalTokenCount": 1514, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 383, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 94, + "kind": 2, + "question": "A warehouse has 49 shelves, each holding 58 boxes, plus 791 loose boxes. How many boxes are there in total?", + "answer": 3633, + "imageTokens": 1078, + "textGot": 3633, + "pureGot": 3633, + "prodGot": 3633, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 102, + "totalTokenCount": 459, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 305, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 116, + "totalTokenCount": 1646, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 510, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 99, + "totalTokenCount": 1686, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 557, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 95, + "kind": 2, + "question": "A warehouse has 73 shelves, each holding 58 boxes, plus 687 loose boxes. How many boxes are there in total?", + "answer": 4921, + "imageTokens": 1078, + "textGot": 4921, + "pureGot": 4921, + "prodGot": 4921, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 379, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 245, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 67, + "totalTokenCount": 1396, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 309, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 104, + "totalTokenCount": 1462, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 328, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 96, + "kind": 2, + "question": "A warehouse has 58 shelves, each holding 73 boxes, plus 835 loose boxes. How many boxes are there in total?", + "answer": 5069, + "imageTokens": 1078, + "textGot": 5069, + "pureGot": 5069, + "prodGot": 5069, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 76, + "totalTokenCount": 380, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 252, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 107, + "totalTokenCount": 1504, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 377, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 77, + "totalTokenCount": 1510, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 403, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 97, + "kind": 2, + "question": "A warehouse has 61 shelves, each holding 39 boxes, plus 383 loose boxes. How many boxes are there in total?", + "answer": 2762, + "imageTokens": 1078, + "textGot": 2762, + "pureGot": 2762, + "prodGot": 2762, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 112, + "totalTokenCount": 469, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 305, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 115, + "totalTokenCount": 1517, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 382, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 114, + "totalTokenCount": 1527, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 383, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 98, + "kind": 2, + "question": "A warehouse has 44 shelves, each holding 98 boxes, plus 347 loose boxes. How many boxes are there in total?", + "answer": 4659, + "imageTokens": 1078, + "textGot": 4659, + "pureGot": 4659, + "prodGot": 4659, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 82, + "totalTokenCount": 364, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 230, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 86, + "totalTokenCount": 1459, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 27 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 353, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 89, + "totalTokenCount": 1401, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 37 + } + ], + "thoughtsTokenCount": 282, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + }, + { + "i": 99, + "kind": 2, + "question": "A warehouse has 64 shelves, each holding 98 boxes, plus 867 loose boxes. How many boxes are there in total?", + "answer": 7139, + "imageTokens": 1078, + "textGot": 7139, + "pureGot": 7139, + "prodGot": 7139, + "textOk": true, + "pureOk": true, + "prodOk": true, + "textUsage": { + "promptTokenCount": 52, + "candidatesTokenCount": 87, + "totalTokenCount": 473, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 52 + } + ], + "thoughtsTokenCount": 334, + "serviceTier": "standard" + }, + "pureUsage": { + "promptTokenCount": 1020, + "candidatesTokenCount": 84, + "totalTokenCount": 1504, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 993 + }, + { + "modality": "TEXT", + "tokenCount": 27 + } + ], + "thoughtsTokenCount": 400, + "serviceTier": "standard" + }, + "prodUsage": { + "promptTokenCount": 1030, + "candidatesTokenCount": 87, + "totalTokenCount": 1543, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 37 + }, + { + "modality": "IMAGE", + "tokenCount": 993 + } + ], + "thoughtsTokenCount": 426, + "serviceTier": "standard" + }, + "textError": null, + "pureError": null, + "prodError": null + } + ] +} \ No newline at end of file diff --git a/eval/gemini-profile/novel-arithmetic.mjs b/eval/gemini-profile/novel-arithmetic.mjs new file mode 100644 index 0000000..f2d1a5f --- /dev/null +++ b/eval/gemini-profile/novel-arithmetic.mjs @@ -0,0 +1,110 @@ +// Gemini 3.6 Flash novel arithmetic benchmark suite. +import { existsSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { renderTextToPngs } from '../../dist/core/render.js'; +import { resolveGeminiProfile } from '../../dist/core/gemini-model-profiles.js'; +import { factSheetText } from '../../dist/core/factsheet.js'; +import { visionTokensForModel } from '../../dist/core/openai.js'; +import { callGemini } from './gemini-client.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const MODEL = process.env.MODEL || 'gemini-3.6-flash'; +const LIVE = process.env.LIVE === '1'; +const N = Math.max(1, Number(process.env.N || 100)); +const SEED = Number(process.env.SEED || 20260711); +const CONCURRENCY = Math.max(1, Number(process.env.CONCURRENCY || 5)); +const TIMEOUT = Number(process.env.TIMEOUT_MS || 180000); +const profile = resolveGeminiProfile(); +const RESULT = join(HERE, `novel-arithmetic-results.json`); + +function lcg(seed) { let s = seed >>> 0; return () => s = (Math.imul(s, 1664525) + 1013904223) >>> 0; } +function ri(r, a, b) { return a + (r() % (b - a + 1)); } +function num(text) { const m = String(text ?? '').match(/ANSWER:\s*(-?\d+)/i); return m ? Number(m[1]) : NaN; } + +function problems(n, seed) { + const r = lcg(seed), out = []; + for (let i = 0; i < n; i++) { + const k = r() % 4; + let question, answer; + if (k === 0) { + const a = ri(r, 1000, 9999), b = ri(r, 1000, 9999), c = ri(r, 1000, 9999); + question = `A factory produced ${a} units on Monday, ${b} units on Tuesday, and ${c} units on Wednesday. How many units did it produce in total over the three days?`; + answer = a + b + c; + } else if (k === 1) { + const a = ri(r, 3000, 9999), b = ri(r, 100, 999), c = ri(r, 100, 999); + question = `A reservoir contains ${a} gallons of water. ${b} gallons are pumped out, and later ${c} gallons flow in. How many gallons are in the reservoir now?`; + answer = a - b + c; + } else if (k === 2) { + const a = ri(r, 11, 99), b = ri(r, 11, 99), c = ri(r, 100, 999); + question = `A warehouse has ${a} shelves, each holding ${b} boxes, plus ${c} loose boxes. How many boxes are there in total?`; + answer = a * b + c; + } else { + const a = ri(r, 5000, 9999), b = ri(r, 1000, 4999); + question = `A stadium has ${a} seats. ${b} are already sold. How many seats remain unsold?`; + answer = a - b; + } + out.push({ i, kind: k, question, answer }); + } + return out; +} + +async function pool(items, limit, fn) { + const out = new Array(items.length); + let next = 0; + async function w() { + while (next < items.length) { + const i = next++; + out[i] = await fn(items[i]); + } + } + await Promise.all(Array.from({ length: Math.min(limit, items.length) }, w)); + return out; +} + +const ps = problems(N, SEED); +console.log(`novel arithmetic · model=${MODEL} · live=${LIVE} · N=${N}`); + +if (!LIVE) { + console.log('Dry run only. Exiting.'); + process.exit(0); +} + +const rows = await pool(ps, CONCURRENCY, async (p) => { + const ask = "Solve the math word problem. Show brief reasoning and end with exactly 'ANSWER: '."; + const imgs = await renderTextToPngs(p.question, profile.stripCols, profile.style, profile.maxHeightPx); + const urls = imgs.map((im) => ({ type: 'input_image', image_url: `data:image/png;base64,${Buffer.from(im.png).toString('base64')}` })); + const imageTokens = imgs.reduce((n, im) => n + visionTokensForModel(MODEL, im.width, im.height), 0); + + let text, pure, prod; + try { text = await callGemini({ model: MODEL, content: [{ type: 'input_text', text: `${ask}\n\n${p.question}` }], maxOutputTokens: 256, timeoutMs: TIMEOUT }); } catch (e) { text = { text: '', error: String(e.message || e) }; } + try { pure = await callGemini({ model: MODEL, content: [...urls, { type: 'input_text', text: `The problem is in the image. ${ask}` }], maxOutputTokens: 256, timeoutMs: TIMEOUT }); } catch (e) { pure = { text: '', error: String(e.message || e) }; } + try { + const fs = factSheetText(p.question, profile.factSheetFormat); + prod = await callGemini({ model: MODEL, content: [...urls, ...(fs ? [{ type: 'input_text', text: fs }] : []), { type: 'input_text', text: `The problem is in the image; use the exact-number factsheet if present. ${ask}` }], maxOutputTokens: 256, timeoutMs: TIMEOUT }); + } catch (e) { prod = { text: '', error: String(e.message || e) }; } + + const textGot = num(text.text), pureGot = num(pure.text), prodGot = num(prod.text); + const row = { ...p, imageTokens, textGot, pureGot, prodGot, textOk: textGot === p.answer, pureOk: pureGot === p.answer, prodOk: prodGot === p.answer, textUsage: text.usage || null, pureUsage: pure.usage || null, prodUsage: prod.usage || null, textError: text.error || null, pureError: pure.error || null, prodError: prod.error || null }; + console.log(`q${p.i} text=${row.textOk ? 'Y' : 'N'}(${textGot}) pure=${row.pureOk ? 'Y' : 'N'}(${pureGot}) prod=${row.prodOk ? 'Y' : 'N'}(${prodGot}) gold=${p.answer}`); + return row; +}); + +const count = (k) => rows.filter((r) => r[k]).length; +const summary = { + generatedAt: new Date().toISOString(), + model: MODEL, + live: true, + n: N, + seed: SEED, + recipe: { cols: profile.stripCols, maxH: profile.maxHeightPx, style: profile.style, factsheet: true }, + textCorrect: count('textOk'), + pureCorrect: count('pureOk'), + prodCorrect: count('prodOk'), + textPct: 100 * count('textOk') / N, + purePct: 100 * count('pureOk') / N, + prodPct: 100 * count('prodOk') / N, + rows +}; +writeFileSync(RESULT, JSON.stringify(summary, null, 2)); +console.log(`\nSUMMARY text ${summary.textCorrect}/${N} (${summary.textPct}%) · pure ${summary.pureCorrect}/${N} (${summary.purePct}%) · prod ${summary.prodCorrect}/${N} (${summary.prodPct}%)`); diff --git a/eval/gemini-profile/resolution-sweep-results.json b/eval/gemini-profile/resolution-sweep-results.json new file mode 100644 index 0000000..ec4a261 --- /dev/null +++ b/eval/gemini-profile/resolution-sweep-results.json @@ -0,0 +1,98 @@ +{ + "generatedAt": "2026-07-21T18:03:44.997Z", + "model": "gemini-3.6-flash", + "live": true, + "results": [ + { + "name": "152-col x 728px (768x728)", + "cols": 152, + "maxH": 728, + "dimensions": { + "width": 768, + "height": 648, + "pages": 1 + }, + "hits": 5, + "total": 5, + "accuracy": "5/5" + }, + { + "name": "152-col x 1932px (768x1932 Sol tall)", + "cols": 152, + "maxH": 1932, + "dimensions": { + "width": 768, + "height": 648, + "pages": 1 + }, + "hits": 5, + "total": 5, + "accuracy": "5/5" + }, + { + "name": "312-col x 728px (1568x728 Claude wide)", + "cols": 312, + "maxH": 728, + "dimensions": { + "width": 1568, + "height": 648, + "pages": 1 + }, + "hits": 4, + "total": 5, + "accuracy": "4/5" + }, + { + "name": "312-col x 1568px (1568x1568 Square 1.5M)", + "cols": 312, + "maxH": 1568, + "dimensions": { + "width": 1568, + "height": 648, + "pages": 1 + }, + "hits": 5, + "total": 5, + "accuracy": "5/5" + }, + { + "name": "312-col x 2048px (1568x2048 Tall wide)", + "cols": 312, + "maxH": 2048, + "dimensions": { + "width": 1568, + "height": 648, + "pages": 1 + }, + "hits": 4, + "total": 5, + "accuracy": "4/5" + }, + { + "name": "400-col x 1024px (2000x1024 Ultra wide)", + "cols": 400, + "maxH": 1024, + "dimensions": { + "width": 2008, + "height": 568, + "pages": 2 + }, + "hits": 5, + "total": 5, + "accuracy": "5/5" + }, + { + "name": "500-col x 1200px (2500x1200 Extreme wide)", + "cols": 500, + "maxH": 1200, + "dimensions": { + "width": 2508, + "height": 456, + "pages": 2 + }, + "hits": 5, + "total": 5, + "accuracy": "5/5" + } + ] +} diff --git a/eval/gemini-profile/resolution-sweep.mjs b/eval/gemini-profile/resolution-sweep.mjs new file mode 100644 index 0000000..a109acf --- /dev/null +++ b/eval/gemini-profile/resolution-sweep.mjs @@ -0,0 +1,97 @@ +// Empirical Long Side & Resolution Sweep for Gemini 3.6 Flash +// Sweeps width (cols), height (long side px), and font scaling to find exact breakdown threshold. + +import { writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { callGemini } from './gemini-client.mjs'; +import { renderTextToPngs } from '../../dist/core/render.js'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const MODEL = process.env.MODEL || 'gemini-3.6-flash'; +const LIVE = process.env.LIVE === '1'; + +// Dense test fixtures containing exact hex IDs and dur_ms targets +const SWEEP_TRIALS = [ + { id: 't1', gold: 'c9c947f680ec', dur: 4439 }, + { id: 't2', gold: '851eb3af1bd1', dur: 812 }, + { id: 't3', gold: 'ade34f70fd73', dur: 6150 }, + { id: 't4', gold: 'c5d68855f46d', dur: 7978 }, + { id: 't5', gold: '92abade01aad', dur: 8071 }, +]; + +function generateDenseLog(trial, totalLines = 60) { + const lines = [ + `BEGIN EVENT LOG TRACE - SYSTEM SESSION ${trial.id}`, + `{"timestamp":"2026-07-21T12:00:00Z","id":"${trial.gold}","dur_ms":${trial.dur},"status":200,"path":"/api/v1/sync","msg":"target line"}`, + ]; + for (let i = 0; i < totalLines - 2; i++) { + const fakeHex = (i * 12345678911 + 987654321).toString(16).padEnd(12, '0').slice(0, 12); + lines.push(`{"timestamp":"2026-07-21T12:01:${(i % 60).toString().padStart(2, '0')}Z","id":"${fakeHex}","dur_ms":${1000 + i * 17},"status":200,"path":"/api/v1/filler_${i}"}`); + } + return lines.join('\n'); +} + +// Sweep configurations +const CONFIGS = [ + { name: '152-col x 728px (768x728)', cols: 152, maxH: 728 }, + { name: '152-col x 1932px (768x1932 Sol tall)', cols: 152, maxH: 1932 }, + { name: '312-col x 728px (1568x728 Claude wide)', cols: 312, maxH: 728 }, + { name: '312-col x 1568px (1568x1568 Square 1.5M)', cols: 312, maxH: 1568 }, + { name: '312-col x 2048px (1568x2048 Tall wide)', cols: 312, maxH: 2048 }, + { name: '400-col x 1024px (2000x1024 Ultra wide)', cols: 400, maxH: 1024 }, + { name: '500-col x 1200px (2500x1200 Extreme wide)', cols: 500, maxH: 1200 }, +]; + +async function run() { + console.log(`=== Gemini 3.6 Flash Resolution & Long Side Sweep ===\n`); + + const results = []; + for (const cfg of CONFIGS) { + let hits = 0; + let sampleImgDetails = null; + + console.log(`Testing config: ${cfg.name}...`); + for (const trial of SWEEP_TRIALS) { + const sourceText = generateDenseLog(trial, 80); + const imgs = await renderTextToPngs(sourceText, cfg.cols, { font: 'spleen-5x8', aa: true }, cfg.maxH); + const png = imgs[0].png; + if (!sampleImgDetails) { + sampleImgDetails = { width: imgs[0].width, height: imgs[0].height, pages: imgs.length }; + } + + let got = ''; + let ok = false; + if (LIVE) { + const content = [ + { type: 'input_image', image_url: `data:image/png;base64,${Buffer.from(png).toString('base64')}` }, + { type: 'input_text', text: `Read the image visually. Find the JSON line whose dur_ms is exactly ${trial.dur}. Return only its id field, exactly 12 lowercase hex characters.` } + ]; + try { + const res = await callGemini({ model: MODEL, content, maxOutputTokens: 50, timeoutMs: 30000 }); + got = res.text.match(/[0-9a-f]{12}/i)?.[0]?.toLowerCase() || ''; + ok = got === trial.gold; + if (ok) hits++; + } catch (e) { + console.error(` Error on ${trial.id}:`, e.message); + } + } + } + + const accuracy = `${hits}/${SWEEP_TRIALS.length}`; + results.push({ name: cfg.name, cols: cfg.cols, maxH: cfg.maxH, dimensions: sampleImgDetails, hits, total: SWEEP_TRIALS.length, accuracy }); + console.log(` -> Dimensions: ${sampleImgDetails?.width}x${sampleImgDetails?.height} | Accuracy: ${accuracy}\n`); + } + + const output = { + generatedAt: new Date().toISOString(), + model: MODEL, + live: LIVE, + results + }; + + writeFileSync(join(HERE, 'resolution-sweep-results.json'), JSON.stringify(output, null, 2)); + console.log(`Results saved to eval/gemini-profile/resolution-sweep-results.json`); +} + +run(); diff --git a/eval/gemini-profile/rgb-separation-diagnostic.mjs b/eval/gemini-profile/rgb-separation-diagnostic.mjs new file mode 100644 index 0000000..8dc28c0 --- /dev/null +++ b/eval/gemini-profile/rgb-separation-diagnostic.mjs @@ -0,0 +1,193 @@ +// Controlled diagnostic for Gemini 3.6 Flash RGB separation. +// Compares one candidate RGB-overprint image with exact channel extractions from that same PNG. + +import { mkdirSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { createCanvas, loadImage } from '@napi-rs/canvas'; +import { callGemini } from './gemini-client.mjs'; +import { renderRgbMultiplex } from '../sol-profile/rgb-multiplex-renderer.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const OUT = join(HERE, '.work', 'rgb-separation-diagnostic'); +const RESULT = join(HERE, 'rgb-separation-diagnostic-results.json'); +const MODEL = process.env.MODEL || 'gemini-3.6-flash'; +const LIVE = process.env.LIVE === '1'; +const TIMEOUT = Number(process.env.TIMEOUT_MS || 120000); +mkdirSync(OUT, { recursive: true }); + +const words = [ + 'amber', 'birch', 'cedar', 'delta', 'ember', 'fjord', 'grove', 'harbor', + 'ivory', 'jungle', 'karma', 'linen', 'maple', 'north', 'olive', 'pearl', + 'quartz', 'river', 'solar', 'tulip', 'umber', 'violet', 'willow', 'xenon', + 'yellow', 'zephyr', 'acorn', 'bloom', 'coral', 'dune', 'elm', 'flint', + 'glade', 'hazel', 'iris', 'jade', +]; +const logical = Array.from({ length: 36 }, (_, i) => { + const code = String((i * 7919 + 104729) % 100000).padStart(5, '0'); + return `L${String(i + 1).padStart(2, '0')} word=${words[i]} code=${code}`; +}); +const expected = { + red: logical.filter((_, i) => i % 3 === 0), + green: logical.filter((_, i) => i % 3 === 1), + blue: logical.filter((_, i) => i % 3 === 2), +}; + +const images = renderRgbMultiplex(logical.join('\n'), { cols: 95, maxHeightPx: 1932 }); +if (images.length !== 1) throw new Error(`expected one diagnostic image, got ${images.length}`); +const combined = images[0]; +writeFileSync(join(OUT, 'combined.png'), combined.png); + +const decoded = await loadImage(Buffer.from(combined.png)); +const sourceCanvas = createCanvas(decoded.width, decoded.height); +const sourceCtx = sourceCanvas.getContext('2d'); +sourceCtx.drawImage(decoded, 0, 0); +const source = sourceCtx.getImageData(0, 0, decoded.width, decoded.height).data; +const channelNames = ['red', 'green', 'blue']; +const artifacts = { combined: combined.png }; + +for (let channel = 0; channel < 3; channel++) { + for (const grayscale of [false, true]) { + const canvas = createCanvas(decoded.width, decoded.height); + const ctx = canvas.getContext('2d'); + const image = ctx.createImageData(decoded.width, decoded.height); + for (let i = 0; i < decoded.width * decoded.height; i++) { + const value = source[i * 4 + channel]; + image.data[i * 4] = grayscale ? value : channel === 0 ? value : 0; + image.data[i * 4 + 1] = grayscale ? value : channel === 1 ? value : 0; + image.data[i * 4 + 2] = grayscale ? value : channel === 2 ? value : 0; + image.data[i * 4 + 3] = 255; + } + ctx.putImageData(image, 0, 0); + const name = `${channelNames[channel]}-${grayscale ? 'white' : 'color'}`; + artifacts[name] = canvas.toBuffer('image/png'); + writeFileSync(join(OUT, `${name}.png`), artifacts[name]); + } +} + +const imagePart = (png) => ({ + type: 'input_image', + image_url: `data:image/png;base64,${Buffer.from(png).toString('base64')}`, +}); + +function parseJson(text) { + const start = Math.min(...['{', '['].map((ch) => { + const i = text.indexOf(ch); return i < 0 ? Infinity : i; + })); + const end = Math.max(text.lastIndexOf('}'), text.lastIndexOf(']')); + if (!Number.isFinite(start) || end < start) return null; + try { return JSON.parse(text.slice(start, end + 1)); } catch { return null; } +} + +function score(expectedLines, answer) { + const got = Array.isArray(answer) ? answer.map(String) : []; + const exact = expectedLines.filter((line) => got.includes(line)).length; + return { exact, total: expectedLines.length, got }; +} + +async function run() { + console.log(`=== Gemini 3.6 Flash RGB Channel Separation Diagnostic ===\n`); + const rows = []; + + if (LIVE) { + // 1. Combined RGB arm + console.log('Testing combined RGB arm...'); + const combinedPrompt = [ + 'This image overlays three independent text streams in RGB color channels.', + 'Within each physical row, read RED first, then GREEN, then BLUE.', + 'Return only JSON with keys red, green, blue. Each value must be an array of the 12 exact lines in reading order.', + ].join(' '); + try { + const response = await callGemini({ + model: MODEL, + content: [imagePart(artifacts.combined), { type: 'input_text', text: combinedPrompt }], + maxOutputTokens: 1600, + timeoutMs: TIMEOUT, + }); + const parsed = parseJson(response.text) || {}; + const row = { + arm: 'combined', + red: score(expected.red, parsed.red), + green: score(expected.green, parsed.green), + blue: score(expected.blue, parsed.blue), + raw: response.text, + usage: response.usage, + ms: response.ms, + }; + rows.push(row); + console.log(`Combined RGB -> Red: ${row.red.exact}/12, Green: ${row.green.exact}/12, Blue: ${row.blue.exact}/12`); + } catch (error) { + console.error('Error on combined RGB:', error.message); + rows.push({ arm: 'combined', error: String(error?.message || error) }); + } + + // 2. Extracted single channels + for (const channel of channelNames) { + for (const mode of ['color', 'white']) { + const arm = `${channel}-${mode}`; + console.log(`Testing extracted arm ${arm}...`); + const prompt = `Read the single visible text stream. Return only a JSON array containing all 12 exact lines in top-to-bottom order.`; + try { + const response = await callGemini({ + model: MODEL, + content: [imagePart(artifacts[arm]), { type: 'input_text', text: prompt }], + maxOutputTokens: 700, + timeoutMs: TIMEOUT, + }); + const sc = score(expected[channel], parseJson(response.text)); + rows.push({ + arm, + score: sc, + raw: response.text, + usage: response.usage, + ms: response.ms, + }); + console.log(`Extracted ${arm} -> ${sc.exact}/12`); + } catch (error) { + console.error(`Error on ${arm}:`, error.message); + rows.push({ arm, error: String(error?.message || error) }); + } + } + } + + // 3. Combined focused arms + for (const channel of channelNames) { + const arm = `combined-focus-${channel}`; + console.log(`Testing focused arm ${arm}...`); + const prompt = `Ignore every other color. Read only the ${channel.toUpperCase()} text stream from this RGB-overlaid image. Return only a JSON array containing its 12 exact lines in top-to-bottom order.`; + try { + const response = await callGemini({ + model: MODEL, + content: [imagePart(artifacts.combined), { type: 'input_text', text: prompt }], + maxOutputTokens: 700, + timeoutMs: TIMEOUT, + }); + const sc = score(expected[channel], parseJson(response.text)); + rows.push({ + arm, + score: sc, + raw: response.text, + usage: response.usage, + ms: response.ms, + }); + console.log(`Focused ${arm} -> ${sc.exact}/12`); + } catch (error) { + console.error(`Error on ${arm}:`, error.message); + rows.push({ arm, error: String(error?.message || error) }); + } + } + } + + const result = { + generatedAt: new Date().toISOString(), + model: MODEL, + live: LIVE, + image: { width: combined.width, height: combined.height, bytes: combined.png.length }, + expected, + rows, + }; + writeFileSync(RESULT, JSON.stringify(result, null, 2) + '\n'); + console.log(`\nRGB diagnostic results saved to ${RESULT}`); +} + +run(); diff --git a/eval/gemini-profile/verbatim-hex-results.json b/eval/gemini-profile/verbatim-hex-results.json new file mode 100644 index 0000000..c5050b9 --- /dev/null +++ b/eval/gemini-profile/verbatim-hex-results.json @@ -0,0 +1,267 @@ +{ + "generatedAt": "2026-07-21T20:18:28.695Z", + "model": "gemini-3.6-flash", + "live": true, + "correct": 14, + "completed": 15, + "errors": 0, + "n": 15, + "recipe": { + "cols": 312, + "maxH": 728, + "style": { + "font": "spleen-5x8", + "cellWBonus": 0, + "cellHBonus": 0, + "aa": true, + "grid": false, + "gridCols": 0, + "colorCycle": false, + "markerScale": 1, + "markerRed": false, + "inkDilate": 0 + } + }, + "rows": [ + { + "page": 0, + "dur": 4439, + "gold": "c9c947f680ec", + "got": "c9c947f680ec", + "ok": true, + "raw": "c9c947f680ec", + "ms": 3617, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 0, + "dur": 812, + "gold": "851eb3af1bd1", + "got": "851eb3af1bd1", + "ok": true, + "raw": "851eb3af1bd1", + "ms": 3797, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 0, + "dur": 6150, + "gold": "ade34f70fd73", + "got": "ade34f70fd73", + "ok": true, + "raw": "ade34f70fd73", + "ms": 3768, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 1, + "dur": 7978, + "gold": "c5d68855f46d", + "got": "c5d68855f46d", + "ok": true, + "raw": "c5d68855f46d", + "ms": 4464, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 1, + "dur": 8071, + "gold": "92abade01aad", + "got": "92abade01aad", + "ok": true, + "raw": "92abade01aad", + "ms": 4215, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 1, + "dur": 3309, + "gold": "ffe21785b09d", + "got": "ffe21785b09d", + "ok": true, + "raw": "ffe21785b09d", + "ms": 4399, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 2, + "dur": 7215, + "gold": "87cb51eb0e99", + "got": "87cb51eb0e99", + "ok": true, + "raw": "87cb51eb0e99", + "ms": 4315, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 2, + "dur": 4397, + "gold": "93c3ced96dac", + "got": "93c3ced96dac", + "ok": true, + "raw": "93c3ced96dac", + "ms": 3469, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 2, + "dur": 4495, + "gold": "f152ae9bfb8f", + "got": "f152ae9bfb8f", + "ok": true, + "raw": "f152ae9bfb8f", + "ms": 4938, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 3, + "dur": 1622, + "gold": "5a7373d4187f", + "got": "5a7373d4187f", + "ok": true, + "raw": "5a7373d4187f", + "ms": 3935, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 3, + "dur": 2025, + "gold": "44ea8c7aeedd", + "got": "44ea8c7aeedd", + "ok": true, + "raw": "44ea8c7aeedd", + "ms": 3961, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 3, + "dur": 6533, + "gold": "8145b5a0fd46", + "got": "8145b5a0fd46", + "ok": true, + "raw": "8145b5a0fd46", + "ms": 2918, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 4, + "dur": 2921, + "gold": "b8fce698f971", + "got": "b3fce698f971", + "ok": false, + "raw": "b3fce698f971", + "ms": 24902, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 4, + "dur": 8475, + "gold": "4a8164556b99", + "got": "4a8164556b99", + "ok": true, + "raw": "4a8164556b99", + "ms": 4330, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + }, + { + "page": 4, + "dur": 8799, + "gold": "e53112c4b5a4", + "got": "e53112c4b5a4", + "ok": true, + "raw": "e53112c4b5a4", + "ms": 4311, + "dimensions": [ + { + "width": 1568, + "height": 648 + } + ], + "error": null + } + ] +} \ No newline at end of file diff --git a/eval/gemini-profile/verbatim-hex.mjs b/eval/gemini-profile/verbatim-hex.mjs new file mode 100644 index 0000000..260a60c --- /dev/null +++ b/eval/gemini-profile/verbatim-hex.mjs @@ -0,0 +1,98 @@ +// Gemini 3.6 Flash verbatim hex evaluation suite. +import { readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { callGemini } from './gemini-client.mjs'; +import { renderTextToPngs } from '../../dist/core/render.js'; +import { resolveGeminiProfile } from '../../dist/core/gemini-model-profiles.js'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ROOT = join(HERE, '../verbatim-15'); +const MODEL = process.env.MODEL || 'gemini-3.6-flash'; +const LIVE = process.env.LIVE === '1'; +const TIMEOUT = Number(process.env.TIMEOUT_MS || 90000); +const MAX_OUTPUT_TOKENS = Number(process.env.MAX_OUTPUT_TOKENS || 80); +const trials = JSON.parse(readFileSync(join(ROOT, 'golds.json'), 'utf8')); +const RESULT = join(HERE, 'verbatim-hex-results.json'); +const profile = resolveGeminiProfile(); + +function denseLog(trial, totalLines = 80) { + const lines = [ + `BEGIN EVENT LOG TRACE - PAGE ${trial.page}`, + `{"timestamp":"2026-07-21T12:00:00Z","id":"${trial.gold}","dur_ms":${trial.dur},"status":200,"path":"/api/v1/sync","msg":"target line"}`, + ]; + for (let i = 0; i < totalLines - 2; i++) { + const fakeHex = (i * 12345678911 + 987654321).toString(16).padEnd(12, '0').slice(0, 12); + lines.push(`{"timestamp":"2026-07-21T12:01:${String(i % 60).padStart(2, '0')}Z","id":"${fakeHex}","dur_ms":${1000 + i * 17},"status":200,"path":"/api/v1/filler_${i}"}`); + } + return lines.join('\n'); +} + +function writeResult(rows) { + const completed = rows.filter((r) => !r.error); + const result = { + generatedAt: new Date().toISOString(), + model: MODEL, + live: LIVE, + correct: completed.filter((r) => r.ok).length, + completed: completed.length, + errors: rows.length - completed.length, + n: trials.length, + recipe: { cols: profile.stripCols, maxH: profile.maxHeightPx, style: profile.style }, + rows + }; + writeFileSync(RESULT, JSON.stringify(result, null, 2)); + return result; +} + +async function callImage(trial) { + const images = await renderTextToPngs( + denseLog(trial), + profile.stripCols, + profile.style, + profile.maxHeightPx, + ); + const content = [ + ...images.map((image) => ({ + type: 'input_image', + image_url: `data:image/png;base64,${Buffer.from(image.png).toString('base64')}`, + })), + { type: 'input_text', text: `Read the image visually. Find the JSON line whose dur_ms is exactly ${trial.dur}. Return only its id field, exactly 12 lowercase hex characters.` }, + ]; + const result = await callGemini({ model: MODEL, content, maxOutputTokens: MAX_OUTPUT_TOKENS, timeoutMs: TIMEOUT }); + return { + ...result, + dimensions: images.map((image) => ({ width: image.width, height: image.height })), + }; +} + +const rows = []; +for (let i = 0; i < trials.length; i++) { + const t = trials[i]; + let out = '', ms = null, err = null, dimensions = []; + process.stdout.write(`trial ${i + 1}/${trials.length} page${t.page} dur=${t.dur} ... `); + try { + if (LIVE) { + const r = await callImage(t); + out = r.text; + ms = r.ms; + dimensions = r.dimensions; + } + } catch (e) { + err = String(e?.message || e); + out = err; + } + const got = out.match(/[0-9a-f]{12}/i)?.[0]?.toLowerCase() || ''; + const ok = got === t.gold; + rows.push({ ...t, got, ok, raw: out, ms, dimensions, error: err }); + if (LIVE) writeResult(rows); + console.log(`${ok ? 'HIT' : 'MISS'} gold=${t.gold} got=${got || '-'}${ms != null ? ` ${ms}ms` : ''}${err ? ` ERR ${err.slice(0, 100)}` : ''}`); +} + +if (!LIVE) { + console.log('Dry run only; no receipt written'); + process.exit(0); +} + +const result = writeResult(rows); +console.log(`SUMMARY ${result.correct}/${result.completed} completed (${result.errors} errors) -> ${RESULT}`); diff --git a/scripts/gen-context-chart.ts b/scripts/gen-context-chart.ts index b6f87cb..5499115 100644 --- a/scripts/gen-context-chart.ts +++ b/scripts/gen-context-chart.ts @@ -13,8 +13,8 @@ * - Text points: window tokens × 4 chars/token (the standard English-prose * rule of thumb; token-dense content like code/JSON tokenizes *worse*, * ~2-2.5, so 4 is the generous assumption for the text series). - * - Fable pxpipe point: window tokens × measured chars-per-vision-token via - * Claude dense geometry + Anthropic 750 px/token pricing. + * - pxpipe points: window tokens × measured chars-per-vision-token using the + * shipped 312-column geometry and each provider's image-token accounting. * * Window sizes (announcement-era frontier defaults), with release dates used * for x-axis placement: @@ -36,7 +36,7 @@ const OUT = join(ROOT, 'docs/assets/context-window-chars.png'); const README_EXAMPLE = join(ROOT, 'docs/assets/example-render.png'); const TEXT_CPT = 4; // chars per text token, prose rule of thumb -const PX_PER_VISION_TOKEN = 750; // Anthropic image pricing: tokens = w*h/750 +const GEMINI_TOKENS_PER_FULL_PAGE = 1078; // measured at 1568x728 interface Density { /** chars per vision-token for this family's render+billing profile */ @@ -70,7 +70,10 @@ async function measureFableDensity(fixture: string): Promise { if (r.droppedChars > 0) { throw new Error(`fable fixture dropped ${r.droppedChars} chars — atlas gap, fix before charting`); } - const visionTokens = Math.ceil(r.pixels / PX_PER_VISION_TOKEN); + const visionTokens = r.pages.reduce( + (total, page) => total + Math.ceil(page.width / 28) * Math.ceil(page.height / 28), + 0, + ); const cpt = fixture.length / visionTokens; console.log( `Fable density: ${fixture.length} chars → ${r.pages.length} pages, ` + @@ -79,6 +82,21 @@ async function measureFableDensity(fixture: string): Promise { return { cpt, pages: r.pages.length, pixels: r.pixels, visionTokens }; } +async function measureGeminiDensity(fixture: string): Promise { + // Widescreen geometry (312-col Spleen): measured 1078 tokens per page. + const r = await renderTextToImages(fixture, { reflow: true }); + if (r.droppedChars > 0) { + throw new Error(`gemini fixture dropped ${r.droppedChars} chars — atlas gap, fix before charting`); + } + const visionTokens = r.pages.length * GEMINI_TOKENS_PER_FULL_PAGE; + const cpt = fixture.length / visionTokens; + console.log( + `Gemini density: ${fixture.length} chars → ${r.pages.length} pages, ` + + `${visionTokens} vision tokens → ${cpt.toFixed(2)} chars/vision-token`, + ); + return { cpt, pages: r.pages.length, pixels: r.pixels, visionTokens }; +} + // --------------------------------------------------------------------------- // 2. Data @@ -100,7 +118,7 @@ interface Point { dy?: number; } -function points(fableCpt: number): Point[] { +function points(fableCpt: number, geminiCpt: number): Point[] { const t = ( kind: Point['kind'], name: string, @@ -133,6 +151,8 @@ function points(fableCpt: number): Point[] { t('openai', 'GPT-5.6', 2026.52, 1_050_000, 'right'), // the model powering this session: claude-fable-5[1m], 1M-token window t('claude', 'Fable 5 [1m]', 2026.05, 1_000_000, 'below'), + // Gemini 3.6 Flash 1M window + t('gemini', 'Gemini 3.6 Flash', 2026.50, 1_000_000, 'below'), // Grok 4.5: text-window only (opt-in; no pxpipe overlay on this chart). // Window = xAI docs maxPromptLength for grok-4.5 (500000). t('grok', 'Grok 4.5', 2026.42, 500_000, 'below', 10), @@ -144,6 +164,14 @@ function points(fableCpt: number): Point[] { kind: 'pxpipe', label: 'above', }, + { + name: 'Gemini 3.6 Flash + pxpipe', + x: 2026.50, + tokens: 1_000_000, + chars: Math.round(1_000_000 * geminiCpt), + kind: 'pxpipe', + label: 'above', + }, ]; } @@ -175,7 +203,7 @@ function pngWidth(path: string): number { return png.readUInt32BE(16); } -function draw(data: Point[], fableCpt: number): Buffer { +function draw(data: Point[], fableCpt: number, geminiCpt: number): Buffer { const logicalWidth = 1180; const logicalHeight = 1000; const outputWidth = pngWidth(README_EXAMPLE); @@ -208,7 +236,7 @@ function draw(data: Point[], fableCpt: number): Buffer { ctx.font = '400 14px sans-serif'; ctx.fillText( `each point: model · context window (tokens) → characters it holds · text at ~${TEXT_CPT} chars/token · ` + - `Fable pxpipe measured at ${fableCpt.toFixed(1)} chars/vision-token (px ÷ ${PX_PER_VISION_TOKEN})`, + `pxpipe uses shipped 312-column pages · Fable ${fableCpt.toFixed(1)} · Gemini ${geminiCpt.toFixed(1)} chars/vision-token`, 36, 68, ); @@ -220,7 +248,7 @@ function draw(data: Point[], fableCpt: number): Buffer { const bottom = 1000 - 72; // Linear scale — log gave every decade equal height, which flattened the whole // point of the chart: 18M must physically tower ~4.5× over the 4M pack. - const Y_MAX = 20_000_000; + const Y_MAX = 24_000_000; const X_MIN = 2018; const X_MAX = 2027.25; const y = (v: number) => bottom - (v / Y_MAX) * (bottom - top); @@ -277,10 +305,11 @@ function draw(data: Point[], fableCpt: number): Buffer { ctx.stroke(); } - // Dashed vertical connector: Fable text window → same window imaged (pxpipe). + // Dashed vertical connector: Fable / Gemini text window → same window imaged (pxpipe). // Grok is plotted as a text series only (no pxpipe overlay). const overlays: Array<{ textName: string; pxName: string }> = [ { textName: 'Fable 5 [1m]', pxName: 'Fable 5 [1m] + pxpipe' }, + { textName: 'Gemini 3.6 Flash', pxName: 'Gemini 3.6 Flash + pxpipe' }, ]; for (const o of overlays) { const base = data.find((p) => p.name === o.textName)!; @@ -384,7 +413,7 @@ function draw(data: Point[], fableCpt: number): Buffer { [colors.gemini, 'Google · Gemini'], [colors.claude, 'Anthropic · Claude → Fable'], [colors.grok, 'xAI · Grok 4.5'], - [colors.pxpipe, 'Fable 5 [1m] · same 1M window · pxpipe images (measured)'], + [colors.pxpipe, 'pxpipe images (measured overlays)'], ]; ctx.font = '400 13px sans-serif'; let ly = top + 20; @@ -413,7 +442,8 @@ function draw(data: Point[], fableCpt: number): Buffer { // --------------------------------------------------------------------------- const fixture = loadFixture(); const fable = await measureFableDensity(fixture); -const data = points(fable.cpt); +const gemini = await measureGeminiDensity(fixture); +const data = points(fable.cpt, gemini.cpt); console.log('\n model released window (tokens) chars in window'); for (const p of data) { @@ -423,10 +453,16 @@ for (const p of data) { } const fableText = data.find((p) => p.name === 'Fable 5 [1m]')!; const fablePx = data.find((p) => p.name === 'Fable 5 [1m] + pxpipe')!; +const geminiText = data.find((p) => p.name === 'Gemini 3.6 Flash')!; +const geminiPx = data.find((p) => p.name === 'Gemini 3.6 Flash + pxpipe')!; + console.log( `\n Fable pxpipe multiplier on the same window: ${(fablePx.chars / fableText.chars).toFixed(2)}×`, ); +console.log( + ` Gemini 3.6 Flash pxpipe multiplier on the same window: ${(geminiPx.chars / geminiText.chars).toFixed(2)}×`, +); mkdirSync(dirname(OUT), { recursive: true }); -writeFileSync(OUT, draw(data, fable.cpt)); +writeFileSync(OUT, draw(data, fable.cpt, gemini.cpt)); console.log(`\nwrote ${OUT}`); diff --git a/src/core/applicability.ts b/src/core/applicability.ts index 4b72b0b..1882daf 100644 --- a/src/core/applicability.ts +++ b/src/core/applicability.ts @@ -24,7 +24,7 @@ function baseModelId(model: string): string { /** Dashboard runtime override; null = fall back to PXPIPE_MODELS env / built-in default. In-memory only. */ let runtimeModelBases: readonly string[] | null = null; -/** Built-in default scope when PXPIPE_MODELS is unset: Fable 5 only. +/** Built-in default scope when PXPIPE_MODELS is unset: Fable 5 and Gemini 3.6 Flash. * Everything else is opt-in via dashboard chips or PXPIPE_MODELS: * - Opus 4.7/4.8 — worse at reading imaged content (FINDINGS.md 2026-06-16: * Opus 4.8 ~2pp arithmetic, 6/15 dense-hex vs Fable 100/100). @@ -34,7 +34,7 @@ let runtimeModelBases: readonly string[] | null = null; * - Grok 4.5 — 82/100 arithmetic, 83/98 gist, and 13/18 state tracking. * Both profiles remain available for explicit opt-in. * Silently imaging weak or unvalidated readers is the wrong default. */ -const DEFAULT_MODEL_BASES = ['claude-fable-5']; +const DEFAULT_MODEL_BASES = ['claude-fable-5', 'gemini-3.6-flash']; function falsey(v: string): boolean { return /^(0|false|no|off|none)$/i.test(v.trim()); @@ -80,8 +80,11 @@ export function setAllowedModelBases(list: readonly string[] | null): void { * alias; [variant] tags stripped first. */ function isAllowed(model: string | null | undefined): boolean { if (typeof model !== 'string') return false; - const base = baseModelId(model); - return allowedModelBases().some((b) => base === b || base.startsWith(`${b}-`)); + const base = baseModelId(model).toLowerCase(); + return allowedModelBases().some((b) => { + const target = b.toLowerCase(); + return base === target || base.startsWith(`${target}-`) || base === `google/${target}`; + }); } /** True when pxpipe may transform this Anthropic model. */ diff --git a/src/core/gemini-model-profiles.ts b/src/core/gemini-model-profiles.ts new file mode 100644 index 0000000..fcc106b --- /dev/null +++ b/src/core/gemini-model-profiles.ts @@ -0,0 +1,49 @@ +import { + MAX_HEIGHT_PX as ANTHROPIC_MAX_HEIGHT_PX, + ANTHROPIC_SLAB_COLS as ANTHROPIC_STRIP_COLS, +} from './render.js'; +import type { GptModelProfile } from './gpt-model-profiles.js'; + +/** Dedicated profile for the validated Gemini 3.6 Flash model. The production + * 1568×728 geometry measured 1,078 image tokens. */ +export const GEMINI_3_6_FLASH_PROFILE: GptModelProfile = { + vision: { regime: 'tile', base: 1078, perTile: 0 }, + stripCols: ANTHROPIC_STRIP_COLS, + maxHeightPx: ANTHROPIC_MAX_HEIGHT_PX, + minCompressTokens: 500, + factSheetFormat: 'full', + history: { + maxImages: 32, + keepTail: 6, + keepRecentPairs: 6, + minCollapseTokens: 2000, + responsesMode: 'pairs', + framing: 'full', + factSheetScope: 'per-segment', + }, + style: { + font: 'spleen-5x8', + cellWBonus: 0, + cellHBonus: 0, + aa: true, + grid: false, + gridCols: 0, + colorCycle: false, + markerScale: 1, + markerRed: false, + inkDilate: 0, + }, +}; + +export function isGeminiModel(model: string | null | undefined): boolean { + const id = (model ?? '').toLowerCase(); + return id === 'gemini-3.6-flash' || id === 'google/gemini-3.6-flash'; +} + +export function resolveGeminiProfile(): GptModelProfile { + return GEMINI_3_6_FLASH_PROFILE; +} + +export function geminiVisionTokens(_model: string, _w: number, _h: number): number { + return 1078; +} diff --git a/src/core/google.ts b/src/core/google.ts new file mode 100644 index 0000000..b0c03d3 --- /dev/null +++ b/src/core/google.ts @@ -0,0 +1,210 @@ +/** + * Google AI Studio / Gemini API request transformer and usage extractor. + * Intercepts /google-ai-studio/v1beta/models/*:generateContent and :streamGenerateContent + * requests, extracts static system instructions, renders them to PNG image parts, and + * passes transformed payloads to Google AI Studio. + */ + +import { renderTextToPngs, shrinkColsToContent } from './render.js'; +import { geminiVisionTokens, resolveGeminiProfile } from './gemini-model-profiles.js'; +import { bytesToBase64 } from './png.js'; +import { compactSlabWhitespace, type TransformInfo } from './transform.js'; +import { prepareImagedRenderText, CHAT_HEADER } from './openai.js'; +import { factSheetText } from './factsheet.js'; + +export interface GooglePart { + text?: string; + inlineData?: { + mimeType: string; + data: string; + }; + [key: string]: unknown; +} + +export interface GoogleContent { + role?: 'user' | 'model' | string; + parts?: GooglePart[]; + [key: string]: unknown; +} + +export interface GoogleGenerateContentRequest { + contents?: GoogleContent[]; + systemInstruction?: { + role?: string; + parts?: GooglePart[]; + [key: string]: unknown; + }; + tools?: unknown[]; + [key: string]: unknown; +} + +const GOOGLE_ROUTE = /^\/google-ai-studio\/(?:v1|v1beta)\/models\/([^/:]+):(generateContent|streamGenerateContent)$/; + +export function parseGoogleModelFromPath(pathname: string): string | null { + const match = GOOGLE_ROUTE.exec(pathname); + return match && match[1] ? match[1] : null; +} + +const SYSTEM_POINTER = + 'The original system instruction is rendered in the image at the start of the first user turn. ' + + 'Treat that image as this system instruction, with the same authority and priority.'; + +function record(value: unknown): Record | null { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? value as Record + : null; +} + +export async function transformGoogleGenerateContent( + bodyBytes: Uint8Array, + modelName: string, + options: { + compress?: boolean; + cols?: number; + reflow?: boolean; + } = {}, +): Promise<{ body: Uint8Array; info: TransformInfo }> { + const text = new TextDecoder().decode(bodyBytes); + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch { + return { body: bodyBytes, info: createDefaultInfo(modelName) }; + } + const reqRecord = record(parsed); + if (!reqRecord) return { body: bodyBytes, info: createDefaultInfo(modelName) }; + const req = reqRecord as GoogleGenerateContentRequest; + + const info = createDefaultInfo(modelName); + if (options.compress === false) { + info.reason = 'compression_disabled'; + return { body: bodyBytes, info }; + } + + // Extract system instructions + const systemTexts: string[] = []; + const systemInstruction = record(req.systemInstruction); + const systemParts = systemInstruction?.parts; + if (systemParts !== undefined && !Array.isArray(systemParts)) { + return { body: bodyBytes, info }; + } + if (Array.isArray(systemParts)) { + for (const rawPart of systemParts) { + const part = record(rawPart); + if (!part) return { body: bodyBytes, info }; + if (typeof part.text === 'string' && part.text.trim()) { + systemTexts.push(part.text); + info.staticChars += part.text.length; + } + } + } + + const combinedRaw = systemTexts.join('\n\n'); + info.origChars = combinedRaw.length; + if (!combinedRaw) { + info.reason = 'no_static_context'; + return { body: bodyBytes, info }; + } + + const profile = resolveGeminiProfile(); + const combined = compactSlabWhitespace(combinedRaw).trimEnd(); + const reflowNote = options.reflow !== false + ? ' The glyph ↵ (U+21B5) marks an original hard line break in content; treat it as a real newline.' + : ''; + const header = CHAT_HEADER.replace('\n====', reflowNote + '\n===='); + const renderedText = prepareImagedRenderText(header + combined, options.reflow !== false); + + const maxCols = options.cols ?? profile.stripCols; + const cols = Math.min( + shrinkColsToContent(renderedText, maxCols, profile.style.markerScale, profile.style.font), + profile.stripCols, + ); + + const images = await renderTextToPngs(renderedText, cols, profile.style, profile.maxHeightPx); + const imageTokens = images.reduce( + (total, image) => total + geminiVisionTokens(modelName, image.width, image.height), + 0, + ); + const textTokens = Math.max(1, Math.ceil(combinedRaw.length / 3.5)); + const fsText = factSheetText(combinedRaw, profile.factSheetFormat); + const nativeText = SYSTEM_POINTER + (fsText ?? ''); + const nativeInjectedTokens = Math.ceil(nativeText.length / 3.5); + + info.gateEval = { + site: 'slab', + imageTokens, + textTokens, + burnImageSide: nativeInjectedTokens, + burnTextSide: 0, + profitable: true, + }; + + // Build image parts + const imageParts: GooglePart[] = images.map((img) => ({ + inlineData: { + mimeType: 'image/png', + data: bytesToBase64(img.png), + }, + })); + + if (fsText) { + imageParts.push({ text: fsText }); + } + + // Prepare transformed request + if (req.contents !== undefined && !Array.isArray(req.contents)) { + return { body: bodyBytes, info: createDefaultInfo(modelName) }; + } + const contents = Array.isArray(req.contents) ? [...req.contents] : []; + if (contents.length > 0 && contents[0] && contents[0].role === 'user') { + const firstTurn = contents[0]; + if (firstTurn.parts !== undefined && !Array.isArray(firstTurn.parts)) { + return { body: bodyBytes, info: createDefaultInfo(modelName) }; + } + contents[0] = { + ...firstTurn, + parts: [...imageParts, ...(firstTurn.parts || [])], + }; + } else { + contents.unshift({ + role: 'user', + parts: imageParts, + }); + } + + // Keep a native system-level pointer so the imaged instruction retains its + // original authority instead of being demoted to ordinary user content. + const transformedReq: GoogleGenerateContentRequest = { + ...req, + contents, + systemInstruction: { + ...req.systemInstruction, + parts: [{ text: SYSTEM_POINTER }], + }, + }; + + info.compressed = true; + info.imageCount = images.length; + info.imageBytes = images.reduce((acc, img) => acc + img.png.byteLength, 0); + info.imageTokens = imageTokens; + info.baselineImagedTokens = textTokens; + info.nativeInjectedTokens = nativeInjectedTokens; + + const transformedBytes = new TextEncoder().encode(JSON.stringify(transformedReq)); + return { body: transformedBytes, info }; +} + +function createDefaultInfo(_model: string): TransformInfo { + return { + compressed: false, + reason: undefined, + origChars: 0, + compressedChars: 0, + staticChars: 0, + dynamicChars: 0, + dynamicBlockCount: 0, + imageCount: 0, + imageBytes: 0, + droppedCodepointsTop: {}, + }; +} diff --git a/src/core/gpt-model-profiles.ts b/src/core/gpt-model-profiles.ts index 275c456..d334bc0 100644 --- a/src/core/gpt-model-profiles.ts +++ b/src/core/gpt-model-profiles.ts @@ -29,6 +29,7 @@ import { DEFAULT_RENDER_FONT, type RenderFont, } from './render.js'; +import { isGeminiModel, resolveGeminiProfile } from './gemini-model-profiles.js'; export const GPT_MAX_HEIGHT_PX = 1932; @@ -375,6 +376,7 @@ export function resolveGptProfile(model: string | null | undefined): GptModelPro // Match applicability.ts: bracketed transport variants (for example [1m]) // do not define a different visual reader profile. const m = (model ?? '').toLowerCase().replace(/\[[^\]]*\]/g, ''); + if (isGeminiModel(m)) return resolveGeminiProfile(); const env = envProfiles(); if (env.size > 0) { let best: GptModelProfile | undefined; diff --git a/src/core/openai.ts b/src/core/openai.ts index ca461bb..bc65f31 100644 --- a/src/core/openai.ts +++ b/src/core/openai.ts @@ -22,6 +22,7 @@ import { resolveGptProfile, type GptVisionCost, } from './gpt-model-profiles.js'; +import { isGeminiModel, geminiVisionTokens } from './gemini-model-profiles.js'; import { bytesToBase64 } from './png.js'; import { compactSlabWhitespace, @@ -127,9 +128,12 @@ export function isGrokModel(model: string | null | undefined): boolean { export const GROK_TOKENS_PER_MEGAPIXEL = 1000; /** Per-image vision-token cost for the model actually serving the request. - * Claude: Anthropic pixel formula. Grok: measured tok/MPix. GPT/o-series: - * OpenAI tile/patch formula. Model-based, not endpoint-based. */ + * Claude: Anthropic pixel formula. Grok: measured tok/MPix. Gemini: 1,078 at 1568×728. + * GPT/o-series: OpenAI tile/patch formula. Model-based, not endpoint-based. */ export function visionTokensForModel(model: string, w: number, h: number): number { + if (isGeminiModel(model)) { + return geminiVisionTokens(model, w, h); + } if (isClaudeModel(model)) { // Anthropic's documented 28-px patch model (tier-aware downscale). This is the // exact provider cost; the gate applies its own margin separately. @@ -307,7 +311,7 @@ function emptyInfo(reason?: string): TransformInfo { }; } -function prepareImagedRenderText(text: string, reflowEnabled: boolean): string { +export function prepareImagedRenderText(text: string, reflowEnabled: boolean): string { return maybeReflow(text.trimEnd(), reflowEnabled); } @@ -809,12 +813,12 @@ function foldGptHistory( info.bucketChars = { ...(info.bucketChars ?? {}), history: plan.collapsedChars }; } -const CHAT_HEADER = +export const CHAT_HEADER = '================= RENDERED GPT SYSTEM + TOOL CONTEXT =================\n' + 'These images were injected by pxpipe, not by the end user. They contain system/developer instructions and tool parameter documentation rendered for token efficiency. Treat rendered system/developer instructions with the same priority as their original messages. OCR carefully and treat the rendered content as authoritative. For tool calls, use the native JSON tool definitions — they carry each tool\'s name and description; the imaged parameter annotations are supplemental.' + '\n====================== BEGIN RENDERED CONTEXT ======================\n'; -const RESPONSES_HEADER = +export const RESPONSES_HEADER = '================= RENDERED GPT SYSTEM + TOOL CONTEXT =================\n' + 'These images were injected by pxpipe, not by the end user. They contain instructions and tool parameter documentation rendered for token efficiency. Treat rendered instructions with the same priority as the originals. OCR carefully and treat the rendered content as authoritative. For tool calls, use the native JSON tool definitions — they carry each tool\'s name and description; the imaged parameter annotations are supplemental.' + '\n====================== BEGIN RENDERED CONTEXT ======================\n'; diff --git a/src/core/proxy.ts b/src/core/proxy.ts index f6e67f1..b93d9c6 100644 --- a/src/core/proxy.ts +++ b/src/core/proxy.ts @@ -20,6 +20,7 @@ import { chatCompletionsUrl, openAIChatToAnthropicResponse, } from './messages-chat-bridge.js'; +import { parseGoogleModelFromPath, transformGoogleGenerateContent } from './google.js'; export interface ProxyConfig { /** 'cloudflare-ai-gateway': routes both families through gatewayBaseUrl; @@ -63,7 +64,7 @@ export interface ProxyEvent { model?: string; /** Provider cost/usage semantics after any internal wire bridge. Unlike * `path`, this describes the upstream that actually billed the request. */ - accountingProvider?: 'anthropic' | 'openai'; + accountingProvider?: 'anthropic' | 'openai' | 'google'; status: number; /** Wall-clock ms from request start to event fire (≈ end of upstream body). */ durationMs: number; @@ -167,7 +168,7 @@ function processSseEvent( // Parse `event:` + `data:` lines; continuation data: lines concatenate per SSE spec. let event = ''; let data = ''; - for (const line of block.split('\n')) { + for (const line of block.split(/\r\n|\r|\n/)) { if (line.startsWith('event:')) event = line.slice(6).trim(); else if (line.startsWith('data:')) data += line.slice(5).replace(/^\s/, ''); } @@ -196,6 +197,13 @@ function processSseEvent( state.stopReason = typeof reason === 'string' ? reason : event === 'response.incomplete' ? 'incomplete' : 'stop'; } + // Google AI Studio streaming chunks: usageMetadata object. + if (obj.usageMetadata && typeof obj.usageMetadata === 'object') { + const gUsage = normalizeUsage(obj.usageMetadata); + if (gUsage) state.usage = gUsage; + } + measureGoogleCandidates(obj, m, state); + measureOpenAIChoices(obj, m); // OpenAI chat chunks: the final chunk carries choices[].finish_reason (earlier chunks ship null). const choices = obj.choices; @@ -247,6 +255,33 @@ function processSseEvent( } } +function objectRecord(value: unknown): Record | null { + return value !== null && typeof value === 'object' && !Array.isArray(value) + ? value as Record + : null; +} + +function parseJsonObjects(text: string): Record[] | null { + const trimmed = text.trim(); + try { + const parsed: unknown = JSON.parse(trimmed); + if (Array.isArray(parsed)) { + return parsed.map(objectRecord).filter((item): item is Record => item !== null); + } + const item = objectRecord(parsed); + return item ? [item] : null; + } catch { + const match = /"usageMetadata"\s*:\s*(\{[^}]+\})/.exec(trimmed); + if (match && match[1]) { + try { + const usageMetadata: unknown = JSON.parse(match[1]); + return [{ usageMetadata }]; + } catch {} + } + } + return null; +} + function normalizeUsage(raw: unknown): Usage | undefined { if (!raw || typeof raw !== 'object') return undefined; const u = raw as Record; @@ -270,6 +305,15 @@ function normalizeUsage(raw: unknown): Usage | undefined { // OpenAI field aliases. if (typeof u.prompt_tokens === 'number') out.input_tokens = u.prompt_tokens; if (typeof u.completion_tokens === 'number') out.output_tokens = u.completion_tokens; + + // Google AI Studio field aliases. + if (typeof u.promptTokenCount === 'number') out.input_tokens = u.promptTokenCount; + if (typeof u.candidatesTokenCount === 'number' || typeof u.thoughtsTokenCount === 'number') { + out.output_tokens = + (typeof u.candidatesTokenCount === 'number' ? u.candidatesTokenCount : 0) + + (typeof u.thoughtsTokenCount === 'number' ? u.thoughtsTokenCount : 0); + } + if (typeof u.cachedContentTokenCount === 'number') out.cached_tokens = u.cachedContentTokenCount; // OpenAI prompt-cache hits live in a details sub-object: Responses uses // `input_tokens_details.cached_tokens`, Chat uses `prompt_tokens_details`. const details = @@ -302,6 +346,35 @@ function measureOpenAIChoices(obj: Record, m: OutputMeasurement } } +function measureGoogleCandidates( + obj: Record, + m: OutputMeasurement, + state?: { stopReason: string | undefined }, +): boolean { + if (!Array.isArray(obj.candidates)) return false; + for (const rawCandidate of obj.candidates) { + const candidate = objectRecord(rawCandidate); + if (!candidate) continue; + if (state && typeof candidate.finishReason === 'string') state.stopReason = candidate.finishReason; + const content = objectRecord(candidate.content); + if (!Array.isArray(content?.parts)) continue; + for (const rawPart of content.parts) { + const part = objectRecord(rawPart); + if (!part) continue; + if (typeof part.text === 'string') { + if (part.thought === true) m.thinkingChars += part.text.length; + else m.textChars += part.text.length; + } + if (part.functionCall !== undefined) { + try { + m.toolUseChars += JSON.stringify(part.functionCall).length; + } catch {} + } + } + } + return true; +} + /** Measure non-streaming messages.content[] — same OutputMeasurement shape as the SSE accumulator. */ function measureFromMessageJson(j: unknown): OutputMeasurement { const m: OutputMeasurement = { textChars: 0, thinkingChars: 0, toolUseChars: 0, redactedBlockCount: 0 }; @@ -449,11 +522,11 @@ function teeForUsage(res: Response): { const { done, value } = await reader.read(); if (done) break; buf += decoder.decode(value, { stream: true }); - // SSE events are terminated by a blank line. - let evEnd: number; - while ((evEnd = buf.indexOf('\n\n')) >= 0) { - const block = buf.slice(0, evEnd); - buf = buf.slice(evEnd + 2); + // SSE events are terminated by a blank line; support LF and CRLF. + let boundary: RegExpExecArray | null; + while ((boundary = /\r\n\r\n|\n\n|\r\r/.exec(buf)) !== null) { + const block = buf.slice(0, boundary.index); + buf = buf.slice(boundary.index + boundary[0].length); processSseEvent(block, m, state); } } @@ -471,11 +544,24 @@ function teeForUsage(res: Response): { buf += decoder.decode(value, { stream: true }); } try { - const j = JSON.parse(buf); + const objects = parseJsonObjects(buf); + if (!objects) return { usage: undefined, measurement: undefined, stopReason: undefined }; + let usage: Usage | undefined; + let recognizedGoogle = false; + const measurement: OutputMeasurement = { + textChars: 0, thinkingChars: 0, toolUseChars: 0, redactedBlockCount: 0, + }; + const state: { stopReason: string | undefined } = { stopReason: undefined }; + for (const object of objects) { + const nextUsage = normalizeUsage(object.usage ?? object.usageMetadata); + if (nextUsage) usage = nextUsage; + recognizedGoogle = measureGoogleCandidates(object, measurement, state) || recognizedGoogle; + } + const last = objects[objects.length - 1]; return { - usage: normalizeUsage(j?.usage), - measurement: measureFromMessageJson(j), - stopReason: readStopReasonFromJson(j), + usage, + measurement: recognizedGoogle ? measurement : measureFromMessageJson(last), + stopReason: state.stopReason ?? readStopReasonFromJson(last), }; } catch { return { usage: undefined, measurement: undefined, stopReason: undefined }; @@ -523,6 +609,7 @@ const STRIP_REQ_HEADERS = new Set([ 'content-length', // we recompute 'expect', 'accept-encoding', // let upstream choose + 'x-pxpipe-bypass', // pxpipe-only opt-out signal; never forwarded upstream ]); const STRIP_RES_HEADERS = new Set([ @@ -603,6 +690,30 @@ async function countTokensUpstream( } } +async function countGoogleTokensUpstream( + countTokensUrl: string, + body: Uint8Array, + headers: Headers, + model: string, +): Promise { + try { + const request = JSON.parse(new TextDecoder().decode(body)) as Record; + const countBody = JSON.stringify({ + generateContentRequest: { ...request, model: `models/${model}` }, + }); + const res = await fetch(countTokensUrl, { + method: 'POST', + headers, + body: countBody, + }); + if (!res.ok) return null; + const json = await res.json() as { totalTokens?: unknown }; + return typeof json.totalTokens === 'number' ? json.totalTokens : null; + } catch { + return null; + } +} + /** Resolve upstream URLs from config. Pure — unit-testable. */ export function resolveUpstreams(config: ProxyConfig): { anthropic: string; @@ -774,8 +885,9 @@ export function createProxy(config: ProxyConfig = {}) { // Responses-shaped even though they are not canonical `/v1/*` paths. // Classify by the parsed wire route, otherwise the dashboard ignores // GPT image/baseline telemetry and renders As text / Saved as dashes. - accountingProvider: - isOpenAIChat || isOpenAIResponses || bridgedGptMessages || bridgedChatMessages + accountingProvider: isGoogleRoute + ? 'google' + : isOpenAIChat || isOpenAIResponses || bridgedGptMessages || bridgedChatMessages ? 'openai' : 'anthropic', status, @@ -795,20 +907,34 @@ export function createProxy(config: ProxyConfig = {}) { }; // Transform only known shapes; everything else passes through. + // Explicit per-request opt-out (#111): a subprocess that merely inherited + // ANTHROPIC_BASE_URL (e.g. a plugin's internal `claude` probe) can send + // `x-pxpipe-bypass: 1` — via Claude Code's ANTHROPIC_CUSTOM_HEADERS — to + // have its traffic forwarded byte-for-byte untouched. Routing and auth + // still apply; the header itself is stripped before forwarding. + const bypassHeader = req.headers.get('x-pxpipe-bypass'); + const bypass = bypassHeader !== null && !/^(?:0|false|off|no)$/i.test(bypassHeader.trim()); const providerPrefixed = isProviderPrefixedPath(url.pathname); - const isMessages = req.method === 'POST' && isAnthropicMessagesPath(url.pathname); - const isOpenAIChat = req.method === 'POST' && isOpenAIChatPath(url.pathname); - const isOpenAIResponses = req.method === 'POST' && isOpenAIResponsesPath(url.pathname); + const isMessages = !bypass && req.method === 'POST' && isAnthropicMessagesPath(url.pathname); + const isOpenAIChat = !bypass && req.method === 'POST' && isOpenAIChatPath(url.pathname); + const isOpenAIResponses = !bypass && req.method === 'POST' && isOpenAIResponsesPath(url.pathname); + const googleModel = req.method === 'POST' + ? parseGoogleModelFromPath(url.pathname) + : null; + const isGoogleRoute = googleModel !== null; + const isGoogle = isGoogleRoute && !bypass; const isOpenAIPath = isCanonicalOpenAIPath( url.pathname, req.headers, config.openAIApiKey !== undefined, ); - const upstreamBase = providerPrefixed ? passthroughUpstream : isOpenAIPath ? openAIUpstream : upstream; + const upstreamBase = isGoogleRoute || providerPrefixed + ? passthroughUpstream + : isOpenAIPath ? openAIUpstream : upstream; let bodyOut: BodyInit | null = null; let info: TransformInfo | undefined; - let requestModel: string | undefined; + let requestModel: string | undefined = googleModel ?? undefined; let bridgedGptMessages = false; let bridgedChatMessages = false; let modelRouteForRequest: 'openai' | 'cloudflare' | undefined; @@ -821,13 +947,13 @@ export function createProxy(config: ProxyConfig = {}) { let baselineCacheablePromise: Promise | undefined; let baselineStatusApplies = false; - if (isMessages || isOpenAIChat || isOpenAIResponses) { + if (isMessages || isOpenAIChat || isOpenAIResponses || isGoogle) { const bodyIn = new Uint8Array(await req.arrayBuffer()); try { const transformOpts = typeof config.transform === 'function' ? config.transform() : config.transform; // Fail-closed: unreadable model → no compression, not a risky guess. - const model = readModelField(bodyIn); + const model = googleModel ?? readModelField(bodyIn); requestModel = model ?? undefined; // /v1/messages is only a wire schema: Claude Code can target a non- // Anthropic model (for example GPT-5.6 Sol). Do not apply Claude's @@ -853,11 +979,13 @@ export function createProxy(config: ProxyConfig = {}) { bridgedChatMessages = forceChat; const chatStamp = bridgedChatMessages ? routedModel : undefined; const effectiveModel = chatStamp ?? model; - const modelOk = isMessages - ? (messagesAnthropic && isPxpipeSupportedModel(model)) - || bridgedGptMessages - || (bridgedChatMessages && isPxpipeSupportedGptModel(effectiveModel)) - : isPxpipeSupportedGptModel(model); + const modelOk = isGoogle + ? isPxpipeSupportedModel(model) + : isMessages + ? (messagesAnthropic && isPxpipeSupportedModel(model)) + || bridgedGptMessages + || (bridgedChatMessages && isPxpipeSupportedGptModel(effectiveModel)) + : isPxpipeSupportedGptModel(model); // Compression eligibility and telemetry follow the model that actually // receives the request, not Claude Code's local gateway alias. if (bridgedChatMessages && effectiveModel) requestModel = effectiveModel; @@ -869,15 +997,64 @@ export function createProxy(config: ProxyConfig = {}) { : bridgedChatMessages ? anthropicMessagesToOpenAIChat(bodyIn, chatStamp ?? undefined) : bodyIn; - const r = isMessages - ? bridgedGptMessages - ? await transformOpenAIResponses(bridgeBody, effectiveOpts) - : bridgedChatMessages - ? await transformOpenAIChatCompletions(bridgeBody, effectiveOpts) - : await transformRequest(bodyIn, effectiveOpts) - : isOpenAIChat - ? await transformOpenAIChatCompletions(bodyIn, effectiveOpts) - : await transformOpenAIResponses(bodyIn, effectiveOpts); + let r = isGoogle + ? await transformGoogleGenerateContent(bodyIn, model!, effectiveOpts) + : isMessages + ? bridgedGptMessages + ? await transformOpenAIResponses(bridgeBody, effectiveOpts) + : bridgedChatMessages + ? await transformOpenAIChatCompletions(bridgeBody, effectiveOpts) + : await transformRequest(bodyIn, effectiveOpts) + : isOpenAIChat + ? await transformOpenAIChatCompletions(bodyIn, effectiveOpts) + : await transformOpenAIResponses(bodyIn, effectiveOpts); + if (isGoogle && r.info.compressed) { + const countHeaders = applyGatewayHeaders(filterHeaders(req.headers, STRIP_REQ_HEADERS)); + countHeaders.set('content-type', 'application/json'); + const countUrl = new URL( + passthroughUpstream + url.pathname.replace( + /:(?:generateContent|streamGenerateContent)$/, + ':countTokens', + ), + ); + for (const [key, value] of url.searchParams) countUrl.searchParams.append(key, value); + countUrl.searchParams.delete('alt'); + const [baseline, transformed] = await Promise.all([ + countGoogleTokensUpstream(countUrl.toString(), bodyIn, countHeaders, model!), + countGoogleTokensUpstream(countUrl.toString(), r.body, countHeaders, model!), + ]); + if (baseline === null || transformed === null || transformed >= baseline) { + r = { + body: bodyIn, + info: { + ...r.info, + compressed: false, + reason: baseline === null || transformed === null + ? 'baseline_probe_failed' + : `not_profitable (${transformed} >= ${baseline} tokens)`, + imageCount: 0, + imageBytes: 0, + imageTokens: undefined, + baselineImagedTokens: undefined, + nativeInjectedTokens: undefined, + baselineProbeStatus: baseline === null || transformed === null ? 'failed' : 'ok', + }, + }; + } else { + r.info.baselineTokens = baseline; + r.info.baselineImagedTokens = baseline; + r.info.nativeInjectedTokens = undefined; + r.info.baselineProbeStatus = 'ok'; + r.info.gateEval = { + site: 'slab', + imageTokens: transformed, + textTokens: baseline, + burnImageSide: 0, + burnTextSide: 0, + profitable: true, + }; + } + } if (!modelOk) r.info.reason = 'unsupported_model'; bodyOut = r.body as unknown as BodyInit; // TS narrows Uint8Array away from BodyInit info = r.info; diff --git a/src/core/tracker.ts b/src/core/tracker.ts index e3f047c..2eefacc 100644 --- a/src/core/tracker.ts +++ b/src/core/tracker.ts @@ -14,7 +14,7 @@ export interface TrackEvent { path: string; /** Top-level request model when present. */ model?: string; - accounting_provider?: 'anthropic' | 'openai'; + accounting_provider?: 'anthropic' | 'openai' | 'google'; status: number; duration_ms: number; first_byte_ms?: number; diff --git a/src/dashboard.ts b/src/dashboard.ts index d64aab8..b0303f9 100644 --- a/src/dashboard.ts +++ b/src/dashboard.ts @@ -378,7 +378,7 @@ export const ASSUMED_INPUT_USD_PER_MTOK = 10.0; * Anthropic paths are `/v1/messages[/count_tokens]`; neither word appears. */ function isOpenAIEvent( path: string | undefined, - accountingProvider?: 'anthropic' | 'openai', + accountingProvider?: 'anthropic' | 'openai' | 'google', ): boolean { if (accountingProvider) return accountingProvider === 'openai'; if (!path) return false; @@ -518,12 +518,20 @@ export class DashboardState { * the developer's actual Claude Code session files. */ private readonly ccMapFn: () => Promise>; + /** Host-provided persistence hook for the runtime model scope. The core + * override stays in-memory (Edge-safe); a Node host passes a saver that + * writes the `models` key of the config file so chip toggles survive a + * restart. Best-effort: failures are the hook's problem, never the API's. */ + private readonly persistModelBases: ((bases: readonly string[]) => void) | undefined; + constructor( paths?: SessionsPaths, ccMapFn?: () => Promise>, + persistModelBases?: (bases: readonly string[]) => void, ) { this.paths = paths; this.ccMapFn = ccMapFn ?? (() => claudeCodeMap()); + this.persistModelBases = persistModelBases; } private totalsForModel(model: string | undefined): Totals { @@ -611,6 +619,8 @@ export class DashboardState { const cc1h = u?.cache_creation?.ephemeral_1h_input_tokens ?? 0; const cr = u?.cache_read_input_tokens ?? 0; const gpt = isOpenAIEvent(ev.path, ev.accountingProvider); + const google = ev.accountingProvider === 'google' + || ev.path.includes('google-ai-studio') || ev.path.includes('generateContent'); // Unified per-row accounting, filled by the provider branch below. The // downstream totals / per-session / recent-row code reads only these — @@ -628,7 +638,22 @@ export class DashboardState { let warmForRow: boolean; // did the TEXT baseline read warm? Server-observed: // Anthropic cr>0 or GPT cached_tokens>0. Drives Context Map narration. - if (gpt) { + if (google) { + haveUsage = u !== undefined && (inp > 0 || out > 0); + const baseline = info?.baselineTokens; + haveBaseline = info?.baselineProbeStatus === 'ok' + && typeof baseline === 'number' && baseline > 0; + creditSaving = haveBaseline && haveUsage && compressed; + const measuredBaseline = haveBaseline ? baseline as number : inp; + actualInputEff = inp; + baselineInputEff = creditSaving ? measuredBaseline : inp; + outputEquiv = out; + rawActual = inp; + rawBaseline = creditSaving ? measuredBaseline : inp; + baselineForRow = creditSaving ? measuredBaseline : 0; + cacheReadForRow = u?.cached_tokens ?? 0; + warmForRow = cacheReadForRow > 0; + } else if (gpt) { // GPT cost model: no count_tokens probe, no cache-create premium, no // per-session warmth — the discount is automatic and folded into the // cached-input rate. Baseline is the measured imaged-vs-text delta. @@ -799,7 +824,8 @@ export class DashboardState { // Measured headline: only compressed rows with a usable probe. An // uncompressed row contributes zero saved (baseline === actual), so // including it here would only dilute the "saved on rows we moved" %. - if (creditSaving) { + const dollarEligible = !google; + if (creditSaving && dollarEligible) { totals.baselineInputWeighted += baselineInputEff; totals.actualInputWeighted += actualInputEff; totals.outputWeighted += outputEquiv; @@ -812,7 +838,7 @@ export class DashboardState { // can't measure the counterfactual, so actual ≈ baseline). This // keeps the ratio bounded at 100% — you can't save more than you // would have paid. - if (haveUsage) { + if (haveUsage && dollarEligible) { // baselineInputEff already folds the uncompressed/probe-failed fallback // to actualInputEff, so passthrough rows contribute zero saved here. totals.allBaselineEquivalentWeighted += baselineInputEff; @@ -874,7 +900,7 @@ export class DashboardState { // update() so the lifetime totals block (above) and the per-session // block (here) read the same values. Re-deriving them here would // duplicate the cache-aware-baseline math and invite drift. - if (creditSaving) { + if (creditSaving && dollarEligible) { s.baselineInputWeighted += baselineInputEff; s.actualInputWeighted += actualInputEff; s.baselineMeasuredCount += 1; @@ -887,7 +913,7 @@ export class DashboardState { // above (allActualInputWeighted / allOutputWeighted). Used as the // honest denominator for the session's saved-% so caching wins on // unmeasured requests still count toward "what you actually paid". - if (haveUsage) { + if (haveUsage && dollarEligible) { s.allActualInputWeighted += actualInputEff; s.allOutputWeighted += outputEquiv; } @@ -964,6 +990,8 @@ export class DashboardState { const cr = t.cache_read_tokens ?? 0; const compressed = t.compressed === true; const gpt = isOpenAIEvent(t.path, t.accounting_provider); + const google = t.accounting_provider === 'google' + || t.path.includes('google-ai-studio') || t.path.includes('generateContent'); // Same unified accounting as update(); see the branch comments there. let haveUsage: boolean; @@ -977,7 +1005,21 @@ export class DashboardState { let cacheReadForRow: number; let warmForRow: boolean; // text-baseline warmth for the Context Map narration - if (gpt) { + if (google) { + haveUsage = inp > 0 || out > 0; + const baseline = (t as { baseline_tokens?: number }).baseline_tokens; + haveBaseline = (t as { baseline_probe_status?: string }).baseline_probe_status === 'ok' + && typeof baseline === 'number' && baseline > 0; + creditSaving = haveBaseline && haveUsage && compressed; + const measuredBaseline = haveBaseline ? baseline as number : inp; + actualInputEff = inp; + baselineInputEff = creditSaving ? measuredBaseline : inp; + rawActual = inp; + rawBaseline = creditSaving ? measuredBaseline : inp; + baselineForRow = creditSaving ? measuredBaseline : 0; + cacheReadForRow = (t as { cached_tokens?: number }).cached_tokens ?? 0; + warmForRow = cacheReadForRow > 0; + } else if (gpt) { const e = gptEff({ model: t.model, inputTokens: inp, @@ -1112,7 +1154,7 @@ export class DashboardState { input_tokens: t.input_tokens, output_tokens: t.output_tokens, cache_create: t.cache_create_tokens, - cache_read: gpt ? cacheReadForRow : t.cache_read_tokens, + cache_read: gpt || google ? cacheReadForRow : t.cache_read_tokens, actual_input: haveUsage ? round1(actualInputEff) : undefined, baseline_input: creditSaving ? round1(baselineInputEff) : undefined, @@ -1508,25 +1550,36 @@ export class DashboardState { } /** POST /fragments/models — add/remove ONE model (Claude or GPT) from the - * runtime compress scope. In-memory only; restart resets to the PXPIPE_MODELS - * env / built-in default. The model checks read this live. */ + * runtime compress scope. The model checks read this live. Persisted via + * the host's `persistModelBases` hook when provided (Node writes the + * config file); otherwise in-memory only and restart resets to the + * PXPIPE_MODELS env / built-in default. */ handleModelsToggle(model: string, on: boolean): void { const next = new Set(getAllowedModelBases()); if (on) next.add(model); else next.delete(model); - setAllowedModelBases([...next]); + this.applyModelBases([...next]); } /** POST /fragments/models with {list} — replace the WHOLE runtime compress * scope from the PXPIPE_MODELS textbox. Same CSV shape as the env var; - * empty or off/false/0/no/none = compress nothing. In-memory only. */ + * empty or off/false/0/no/none = compress nothing. Persistence as above. */ handleModelsSet(csv: string): void { const trimmed = csv.trim(); const bases = !trimmed || /^(0|false|no|off|none)$/i.test(trimmed) ? [] : trimmed.split(',').map((s) => s.trim()).filter(Boolean); + this.applyModelBases(bases); + } + + private applyModelBases(bases: string[]): void { setAllowedModelBases(bases); + try { + this.persistModelBases?.(bases); + } catch { + // Persistence is best-effort; the live flip already took effect. + } } } diff --git a/src/dashboard/fragments.ts b/src/dashboard/fragments.ts index dbf1051..bf1e20e 100644 --- a/src/dashboard/fragments.ts +++ b/src/dashboard/fragments.ts @@ -93,6 +93,10 @@ const GROK_MODEL_CATALOG: ReadonlyArray<{ id: string; label: string }> = [ { id: 'grok-4.5', label: 'Grok 4.5' }, ]; +const GEMINI_MODEL_CATALOG: ReadonlyArray<{ id: string; label: string }> = [ + { id: 'gemini-3.6-flash', label: 'Gemini 3.6 Flash' }, +]; + export function renderModelsFragment( active: string[], configured: string[], @@ -100,17 +104,18 @@ export function renderModelsFragment( ): string { const on = new Set(active); const labelOf = new Map( - [...MODEL_CATALOG, ...GPT_MODEL_CATALOG, ...GROK_MODEL_CATALOG].map((m) => [m.id, m.label]), + [...MODEL_CATALOG, ...GPT_MODEL_CATALOG, ...GROK_MODEL_CATALOG, ...GEMINI_MODEL_CATALOG].map((m) => [m.id, m.label]), ); // Union the catalog with env-configured + active ids so PXPIPE_MODELS-enabled - // families always show as toggles, then split into two chip rows (Claude / - // OpenAI Responses) plus the PXPIPE_MODELS CSV textbox that mirrors the scope. + // families always show as toggles, then split into chip rows (Claude / + // OpenAI Responses / Gemini) plus the PXPIPE_MODELS CSV textbox that mirrors the scope. const ids: string[] = []; const seen = new Set(); for (const id of [ ...MODEL_CATALOG.map((m) => m.id), ...GPT_MODEL_CATALOG.map((m) => m.id), ...GROK_MODEL_CATALOG.map((m) => m.id), + ...GEMINI_MODEL_CATALOG.map((m) => m.id), ...configured, ...active, ]) { @@ -129,10 +134,11 @@ export function renderModelsFragment( ); }; const claudeChips = ids.filter((id) => id.startsWith('claude')).map(chipFor).join(''); + const geminiChips = ids.filter((id) => id.includes('gemini')).map(chipFor).join(''); const gptChips = ids.filter((id) => id.startsWith('gpt')).map(chipFor).join(''); const grokChips = ids.filter((id) => id.startsWith('grok')).map(chipFor).join(''); const otherChips = ids - .filter((id) => !id.startsWith('claude') && !id.startsWith('gpt') && !id.startsWith('grok')) + .filter((id) => !id.startsWith('claude') && !id.startsWith('gpt') && !id.startsWith('grok') && !id.includes('gemini')) .map(chipFor) .join(''); const moot = enabled @@ -146,6 +152,11 @@ export function renderModelsFragment( `unlisted models get plain text` + `` + `
` + + `Image Gemini models` + + geminiChips + + `enabled by default · 100/100 vision reader` + + `
` + + `
` + `Image OpenAI Responses models` + gptChips + grokChips + @@ -1178,8 +1189,8 @@ export function renderPage(port: number): string {
- Image model scope Fable 5 only by default · expand to experiment with other families -
⚠ Image compression is tuned for Fable 5 only — other families can use more tokens, not less. Opt in only for deliberate experiments (custom system prompt, subagent model setup, …).
+ Image model scope Fable 5 and Gemini 3.6 Flash by default · expand to experiment with other families +
⚠ Image compression is validated for Fable 5 and Gemini 3.6 Flash — other families can use more tokens, not less. Opt in only for deliberate experiments.
imaging scope ≠ provider routing — non-Anthropic IDs also need routing env on the proxy
diff --git a/src/node.ts b/src/node.ts index ff200cd..51fe2d8 100644 --- a/src/node.ts +++ b/src/node.ts @@ -94,6 +94,47 @@ function applyConfigFileDefaults(): void { } } +/** Dashboard persistence hook: write the runtime model scope back to the + * config file's `models` key so chip toggles survive a restart. Other keys + * are preserved; an invalid existing file is left untouched. + * NOTE: on the next start an explicit PXPIPE_MODELS env still wins over the + * persisted value (same precedence as every other config-file default). */ +function persistModelBasesToConfig(bases: readonly string[]): void { + const file = process.env.PXPIPE_CONFIG ?? DEFAULT_CONFIG_FILE; + let cfg: Record = {}; + let mode = 0o600; + try { + const parsed = JSON.parse(fs.readFileSync(file, 'utf8')) as unknown; + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { + console.warn(`[pxpipe] could not persist model scope: invalid config object ${file}`); + return; + } + cfg = parsed as Record; + mode = fs.statSync(file).mode & 0o777; + } catch (e) { + if ((e as NodeJS.ErrnoException).code !== 'ENOENT') { + console.warn(`[pxpipe] could not persist model scope: invalid config ${file}: ${(e as Error).message}`); + return; + } + } + // Empty array round-trips as 'off' via normalizeModelsConfig on load. + cfg.models = [...bases]; + const tmp = `${file}.tmp-${process.pid}`; + try { + fs.mkdirSync(path.dirname(file), { recursive: true }); + // Write-then-rename so a crash mid-write can't corrupt the config. + fs.writeFileSync(tmp, `${JSON.stringify(cfg, null, 2)}\n`, { mode }); + fs.renameSync(tmp, file); + } catch (e) { + try { + fs.unlinkSync(tmp); + } catch { + // The write may have failed before the temporary file was created. + } + console.warn(`[pxpipe] could not persist model scope to ${file}: ${(e as Error).message}`); + } +} + function parseCli(argv: string[]): RuntimeConfig { // Only flags accepted are --help and --version. Anything else is an // error — there is exactly ONE way to run pxpipe and the dashboard @@ -192,8 +233,8 @@ Environment: families through one gateway base URL PXPIPE_GATEWAY_BASE_URL gateway base URL (required with PXPIPE_PROVIDER) PXPIPE_GATEWAY_HEADERS extra upstream headers: JSON object or k=v;k2=v2 - PXPIPE_MODELS comma-separated model bases to image (Claude/GPT/Grok); - default claude-fable-5 (Sol/Opus/GPT-5.5/Grok opt-in); + PXPIPE_MODELS comma-separated model bases to image (Claude/Gemini/GPT/Grok); + default claude-fable-5,gemini-3.6-flash (Sol/Opus/GPT-5.5/Grok opt-in); off disables PXPIPE_CONFIG JSON config path (default ~/.config/pxpipe/config.json) supports {"models": [...]} or {"models": "off"} @@ -984,10 +1025,14 @@ async function main(): Promise { // served via the route interception in front of the proxy handler. The // SessionsPaths handle lets the dashboard surface session/disk/stats data // without reaching back into module-scope globals. - const dashboard = new DashboardState({ - eventsFile: opts.eventsFile, - sidecarDir: bodySidecarDir, - }); + const dashboard = new DashboardState( + { + eventsFile: opts.eventsFile, + sidecarDir: bodySidecarDir, + }, + undefined, + persistModelBasesToConfig, + ); // Seed the "recent requests" table from the JSONL log so a process restart // doesn't reset what you can see in the UI. Best-effort; ignored on error. await dashboard.replay(opts.eventsFile).catch(() => {}); @@ -1049,7 +1094,9 @@ async function main(): Promise { const tag = e.info?.compressed ? `compressed ${e.info.origChars}ch → ${e.info.imageCount}img/${e.info.imageBytes}B${extraTag}` : e.info?.reason - ? `savings:skip(${e.info.reason})` + ? e.info.reason === 'unsupported_model' && e.model + ? `skip(unsupported=${e.model})` + : `skip(${e.info.reason})` : ''; const cacheRead = e.usage?.cache_read_input_tokens ?? 0; const inputTokens = e.usage?.input_tokens ?? 0; diff --git a/src/worker.ts b/src/worker.ts index b9d8058..327021a 100644 --- a/src/worker.ts +++ b/src/worker.ts @@ -149,7 +149,9 @@ export default { const tag = e.info?.compressed ? `compressed ${e.info.origChars}ch → ${e.info.imageCount}img/${e.info.imageBytes}B` : e.info?.reason - ? `savings:skip(${e.info.reason})` + ? e.info.reason === 'unsupported_model' && e.model + ? `skip(unsupported=${e.model})` + : `skip(${e.info.reason})` : ''; const cacheRead = e.usage?.cache_read_input_tokens ?? 0; console.log(`${e.method} ${e.path} → ${e.status} (${e.durationMs}ms) ${tag} cache_read=${cacheRead}`); diff --git a/tests/dashboard-api.test.ts b/tests/dashboard-api.test.ts index 51de9a0..16750d6 100644 --- a/tests/dashboard-api.test.ts +++ b/tests/dashboard-api.test.ts @@ -188,7 +188,7 @@ describe('serveFragment', () => { expect(off).not.toContain('