Commit Graph

182 Commits

Author SHA1 Message Date
teamchong e98bf36387 fix(eval/swe-bench-pro): deconfound ON-arm numbers from operator session sharing port 47821 2026-06-11 05:00:01 -04:00
teamchong 78b2247a92 eval(swe-bench-pro): single-pair pilot - 1/1 both arms, -64% per-request, Rosetta gotcha documented 2026-06-11 04:56:06 -04:00
teamchong 8e8cb09223 docs: SWE-bench tables lead with per-request -65%, demote confounded run totals 2026-06-10 23:49:38 -04:00
teamchong 83764a18a0 docs: surface SWE-bench parity in honest-part section, link eval dir 2026-06-10 23:31:55 -04:00
teamchong 76542b4a3c eval(swe-bench): Lite pilot 10 paired instances — 10/10 resolved both arms, ON $27.27 vs OFF $53.61 (-49%)
- run_pilot.py harness (paired generation via proxy ON 47821 / OFF 47822)
- official swebench Docker grading via colima, reports + patches committed
- README.md results table, FINDINGS.md task-completion parity update
- repo README benchmark table gains SWE-bench row
2026-06-10 23:25:24 -04:00
teamchong 5274e74bd4 docs: lead with whole-bill savings (59%, all 13.7k requests) over touched-only 72% 2026-06-10 19:08:32 -04:00
teamchong c6a0d61f00 eval(gist-recall): A/B imaged vs text history — 98/98 both arms, 0 confabulation
- 3 tiers: facts-at-depth w/ distractors (50), hard near-miss distractors
  on 45k-char sessions (30), 3x-mutated state tracking (18); 16 unanswerable
  confabulation guards. claude-fable-5, real production renderer, exact-match
  grading, no LLM grader.
- text 98/98, image 98/98, 0 wrong, 0 confabulated UNKNOWNs in either arm
- README benchmark table + FINDINGS dated update; PNGs gitignored,
  probes/results.jsonl committed as receipts
2026-06-10 18:54:03 -04:00
teamchong 1685935759 docs: remove em dashes from README 2026-06-10 07:28:20 -04:00
teamchong ab254d84ec fix(ci): ignore-pnpmfile so lockfile checksum doesn't depend on dev machine's global pnpmfile 2026-06-10 07:22:37 -04:00
teamchong 39b77eb246 ci: let pnpm/action-setup read version from packageManager field (fixes multiple-version conflict) 2026-06-10 07:16:53 -04:00
teamchong e406ec0de0 ci: switch release to npm Trusted Publisher (OIDC, environment: release), drop NPM_TOKEN 2026-06-10 07:15:09 -04:00
teamchong bb48aa4017 ci: add CI (typecheck/test/build) and tag-triggered npm release workflows 2026-06-10 07:13:02 -04:00
teamchong e57cc07c1b chore(publish): rename npm package to pxpipe-proxy (registry typo-squat filter blocked pxpipe); CLI command stays pxpipe v0.2.0 2026-06-10 07:06:22 -04:00
teamchong 63c1eab83e chore(publish): absolute README image URL, repo/homepage/bugs fields, Fable-scoped description+keywords, drop pnpm engine 2026-06-10 06:59:02 -04:00
teamchong cb522a70fe feat(dashboard): replace Svelte bundle with AHA stack (htmx + Alpine, server-rendered fragments)
- src/dashboard/fragments.ts: server-side HTML fragment renderers reusing the
  same JSON payloads (HTML and JSON surfaces can't drift)
- node.ts/dashboard.ts: /fragments/<name> routes (header, session-summary,
  recent, latest, sessions, stats, toggle) with htmx polling + Alpine for
  local state; toggle is a POST-returning-fragment htmx swap
- vendored htmx 2.0.4 + Alpine 3.14.9 in src/dashboard/vendor.ts (offline,
  single-file npm package, no CDN)
- deleted Svelte components/stores/bundle build step; dropped svelte/vite
  devDeps; build is now just tsc + esbuild
- tests: +5 fragment contract tests (routing, toggle state, HTML escaping of
  source text, 404) - 339 total
2026-06-09 22:17:53 -04:00
teamchong 40a3eb5cdd docs: caption matches the real render measurements 2026-06-09 21:58:37 -04:00
teamchong e520749b84 docs: example render is now real transformRequest output (reflow + instruction banner) 2026-06-09 21:57:50 -04:00
teamchong e20bf015c1 docs: add example render (full-density synthetic page) 2026-06-09 21:47:52 -04:00
teamchong 7a80bd1799 docs: fix image caption to match new dense example render 2026-06-09 21:45:46 -04:00
teamchong 898be465a3 docs: README example image now matches real dashboard density (full dense page, synthetic content) 2026-06-09 21:45:12 -04:00
teamchong 02f9c91564 fix(dashboard): dollar rate to Fable 5's $10/MTok input (was $5) 2026-06-09 21:11:11 -04:00
teamchong 4ffb75e665 docs: drop GPT 5.5 route mention, Fable 5 only 2026-06-09 20:36:16 -04:00
teamchong 528034cd82 feat(proxy): cloudflare-ai-gateway provider mode
- PXPIPE_PROVIDER=cloudflare-ai-gateway + PXPIPE_GATEWAY_BASE_URL derive both
  family routes ({base}/anthropic, {base}/openai) from one configurable base
- PXPIPE_GATEWAY_HEADERS injects auth headers (JSON or k=v;k2=v2) on every
  upstream request incl. count_tokens probes
- OpenAI paths drop /v1 to match gateway shape; Anthropic paths unchanged;
  unknown paths pass through on the family route; streaming untouched
- no hardcoded hostnames; tests use fake gateway URLs/tokens (14 new)
2026-06-09 19:21:01 -04:00
teamchong c35b3ab2d4 docs: rewrite README for readability — hook first, real render example, honest-caveats section 2026-06-09 19:10:49 -04:00
teamchong 65cd8aab71 docs: hide unproven Cloudflare Workers quick start 2026-06-09 19:08:49 -04:00
teamchong 5eb7ec6156 docs: demote unmeasured OpenAI route to a footnote, fix stale test count and ocproxy mention 2026-06-09 19:08:33 -04:00
teamchong c369b931e5 chore: rename remaining pixelpipe mentions in eval scripts/docs
frozen eval outputs under eval/results*/ left as-is (historical records)
2026-06-09 19:07:13 -04:00
teamchong b5504d166e docs: drop redundant (npm: pxpipe) from README title 2026-06-09 19:05:56 -04:00
teamchong 99bb931014 chore: rename all pixelpipe references to pxpipe
- code, tests, docs, README, wrangler.toml, help text
- data dir ~/.pixelpipe -> ~/.pxpipe (PXPIPE_LOG env var)
- rebuilt dist; 323/323 tests
2026-06-09 19:04:09 -04:00
teamchong a1c51b3543 feat: publish as pxpipe on npm
- rename package to pxpipe (pixelpipe is an npm security placeholder), bin: pxpipe
- move @napi-rs/canvas to dependencies (runtime dep for npx consumers)
- drop only-allow pnpm preinstall (blocked npm/npx installs)
- add prepublishOnly guard (typecheck + test + build)
- exclude sourcemaps from tarball; README quick start: npx pxpipe
2026-06-09 19:01:22 -04:00
teamchong 7ee2847e6c feat(dashboard): show source text behind each rendered image
- ImageEntry carries sourceText (capped 64KiB upstream in transform.ts)
- new GET /api/image-source[?id=N] endpoint
- LatestPng gains a 'view source text' toggle so the operator can see
  the exact text -> image conversion side by side
2026-06-09 18:47:48 -04:00
teamchong 0e3c95a079 docs: add CLI proxy quick start, fix stale constants table and test count 2026-06-09 18:33:34 -04:00
teamchong 3aae9a6253 chore: ignore generated eval images 2026-06-09 18:33:34 -04:00
teamchong 3d0f9ef08a scope: narrow model gate to fable-5, drop opus; switch dense render to bare 5x8 cell
- gate regex: fable-5 only (opus 4.7/4.8 read tax gone on fable: 100/100 novel-arithmetic, identical image billing)
- remove dead opus-4.6 cpt fork in transform.ts
- DENSE_RENDER_STYLE: 7x10 padded -> bare 5x8 cell (~42% fewer image tokens/page, recall flat in A/B)
- dashboard: compression defaults ON
- README/FINDINGS/TRANSFORM_INFO updated with 2026-06-09 measurements
2026-06-09 18:33:34 -04:00
Steven Chong 613ca3bc71 feat(proxy): support gpt 5.5 chat completions 2026-06-05 00:35:26 -04:00
teamchong f6150ba2bd docs(readme): lead benchmark with the clean 93%, demote contaminated GSM8K to a footnote
One headline number (93% novel reading), 0/15 as the boundary, GSM8K 96%
moved to a flagged footnote so the result isn't ambiguous.
2026-05-31 18:50:31 -04:00
teamchong 05a5ac98dc docs(eval): reproducible reading-fidelity benchmark + honest README table
Does the model actually read pixelpipe's render, or coast on memorized answers?
- novel random-number problems (un-memorizable): text 100% vs image 93% (-7pp real reading tax)
- GSM8K (in training data): 97% vs 96% -- inflated ~3pp by recall
- verbatim recall from a dense render: 0/15

README 'Benchmarks' section leads with the clean 93% and reports the failure
rows too (no cherry-pick). Harness in eval/gsm8k/: gen_novel.py, render_cfg.mjs
(pixelpipe's real renderTextToPngs), bench.py -- claude -p, exact-match grading.
2026-05-31 18:48:19 -04:00
teamchong 0e2979e13a fix(dashboard): default compression OFF so a restart keeps it off
The runtime kill switch reset to ON on every restart, silently re-enabling
the lossy text->image path (which also broke extended-thinking turns).
Default compressionEnabled to false: off on startup, survives restart;
only an explicit dashboard toggle turns it on.
2026-05-31 18:48:19 -04:00
teamchong 52f6b4a0e6 docs: apply FINDINGS reframe + ref fixes (follow-up to 2c38f30)
The prior commit captured only the rename (a stale pathspec aborted the
content staging). This applies the actual edits: retitle to FINDINGS,
'## Verdict' + superseded TL;DR, and the two cross-reference updates.
2026-05-29 22:47:12 -04:00
teamchong 2c38f30f7d docs: rename POSTMORTEM.md -> FINDINGS.md, lead with corrected verdict
The project isn't dead (verdict reversed to working ~68% gist-compressor),
so 'postmortem' was a misnomer. Rename to FINDINGS, retitle, promote the
correction to '## Verdict', and mark the old TL;DR as superseded. Update the
two cross-references (eval README link, public-api test comment).
2026-05-29 22:46:19 -04:00
teamchong 378dd7f6c4 feat(applicability)!: scope to opus 4.7+ and enforce model gate at proxy
Widen the model gate to /^claude-opus-4-(?:[7-9]|[1-9]\d)(?:-|$)/ (Opus 4.7
and newer; previously 4.6/4.7). Wire isPixelpipeSupportedModel into the proxy
boundary in proxy.ts — the proxy previously gated on economics only and
ignored the model, so it compressed 4.8 traffic despite the docs claiming a
4.6/4.7 scope. Unsupported models now pass through with reason
'unsupported_model'.

Update public-api tests to the 4.7+ spec (320/320 pass).

Correct the docs to live measurement: reverse the POSTMORTEM "dead" verdict
(pixelpipe is a lossy gist-compressor saving ~68% on real dense Claude Code
traffic; the verbatim 0/15 needle finding stands as a caveat, not the
verdict), rewrite README Status/Limitations, and add a correction pointer to
the eval README. Original POSTMORTEM body preserved below the correction.

BREAKING CHANGE: Opus 4.6 and older are no longer supported.
Verbatim-risk guard (skip imaging unique IDs/hashes/exact values) still pending.
2026-05-29 22:34:55 -04:00
teamchong 175d2bbb4d fix(dashboard): parse ?id=N on /proxy-latest-png
Router was calling dashboard.servePng() with no argument, so every per-id
thumbnail URL fell back to 'latest image' and all N thumbs on a row
rendered the same PNG. Forward the parsed id from the query string.
2026-05-27 08:13:12 -04:00
Steven Chong 2f78ecaf9f fix(transform): use conservative cpt defaults for opus 4.6 2026-05-26 15:46:19 -04:00
Steven Chong d3992c6bad feat(applicability): enable pixelpipe for opus 4.6 2026-05-26 15:35:12 -04:00
Steven Chong 36ac1faa96 fix(render): reduce dense page size 2026-05-26 13:15:54 -04:00
Steven Chong 63a7810b52 fix(history): preserve git status output as text 2026-05-26 11:05:59 -04:00
Steven Chong 09fb339b39 fix(render): use readable profile for dense images 2026-05-25 15:37:14 -04:00
Steven Chong 0e95b81728 fix(render): page dense tool and history images 2026-05-25 15:33:00 -04:00
Steven Chong d089f0d32b fix(transform): preserve system cache_control on slab image 2026-05-25 14:54:59 -04:00
Steven Chong be545b7ccd feat(render): full-canvas single-column rendering, 50k chars/page
Pixelpipe now always uses the full 1568px canvas (cols=313) and packs up
to 50,000 chars per image. The multi-column code path and the
shrink-to-content path are no longer used by transform.ts - both were
sacrificing token savings on dense content to make sparse content marginally
more readable, and the readability gain wasn't real (the renderer's
output was unreadable in either layout once the content was actually dense).

Key changes:
- READABLE_CHARS_PER_IMAGE: 6_000 -> 50_000
- DEFAULT_COLS: 100 -> 313 (full 1568px / 5px cell)
- shrinkColsToContent() is now a no-op (returns cols unchanged)
- MinToolResultChars / MinReminderChars default to 50k (was 6k)
- transform.ts always renders single-column at full canvas

Practical impact: a 6 KB tool_result that previously rendered as 4 narrow
508x488 pages (1324 image tokens, 86% of text cost) now renders as 1
full-canvas 1568x488 page (~331 image tokens, 22% of text cost) - 4x
more savings on the same content with no quality change.

Rebuilt dist/. Tests: 315/315 pass.
2026-05-25 14:20:43 -04:00