191 Commits

Author SHA1 Message Date
teamchong 8bc1478406 v0.2.1: README scope corrections (docs-only) v0.2.1 2026-06-12 05:55:28 -04:00
teamchong 69eb96d17e docs: README scope wording - imaging covers system prompt + tool docs, not just old history 2026-06-12 05:40:57 -04:00
teamchong 5704c84237 docs: add real-world failure FAQ; scrub usernames/home paths from eval corpus 2026-06-11 17:17:19 -04:00
teamchong 62790e7371 docs: add FAQ - end-to-end bill math vs touched-only, measurement method, compression scope 2026-06-11 16:53:18 -04:00
teamchong d990faf646 eval(verbatim): expand to n=15 dense hex recall on Fable 5 - 13/15 vs Opus 0/15, both misses single-glyph confabulations 2026-06-11 16:48:12 -04:00
teamchong 7d62a4c715 eval(swe-bench-pro): expand to 19 pairs — ON 14/19 vs OFF 15/19, agree 18/19; navidrome 3/3 replication clears the ON loss 2026-06-11 15:47:54 -04:00
teamchong 156945977e eval(swe-bench-pro): 10-pair bench - ON 6/9 vs OFF 7/9, -61% per-request, receipts committed 2026-06-11 10:43:02 -04:00
teamchong 4eef1ec1a0 fix(eval/swe-bench-pro): verified proxy startup + atomic repo cache clone 2026-06-11 08:09:48 -04:00
teamchong b8e1e855bc feat(eval/swe-bench-pro): resumable 10-pair runner on bench-dedicated proxies
- ON 47823 / OFF 47824 with own PXPIPE_LOG files: operator session can
  never pollute bench measurement (separation by construction)
- resume = skip existing patch files; clean stop on quota errors
- Docker grading documented as proxy-free (no model calls in containers)
- README: mention Pro pilot alongside Lite
2026-06-11 08:05:06 -04:00
teamchong e98bf36387 fix(eval/swe-bench-pro): deconfound ON-arm numbers from operator session sharing port 47821 2026-06-11 05:00:01 -04:00
teamchong 78b2247a92 eval(swe-bench-pro): single-pair pilot - 1/1 both arms, -64% per-request, Rosetta gotcha documented 2026-06-11 04:56:06 -04:00
teamchong 8e8cb09223 docs: SWE-bench tables lead with per-request -65%, demote confounded run totals 2026-06-10 23:49:38 -04:00
teamchong 83764a18a0 docs: surface SWE-bench parity in honest-part section, link eval dir 2026-06-10 23:31:55 -04:00
teamchong 76542b4a3c eval(swe-bench): Lite pilot 10 paired instances — 10/10 resolved both arms, ON $27.27 vs OFF $53.61 (-49%)
- run_pilot.py harness (paired generation via proxy ON 47821 / OFF 47822)
- official swebench Docker grading via colima, reports + patches committed
- README.md results table, FINDINGS.md task-completion parity update
- repo README benchmark table gains SWE-bench row
2026-06-10 23:25:24 -04:00
teamchong 5274e74bd4 docs: lead with whole-bill savings (59%, all 13.7k requests) over touched-only 72% 2026-06-10 19:08:32 -04:00
teamchong c6a0d61f00 eval(gist-recall): A/B imaged vs text history — 98/98 both arms, 0 confabulation
- 3 tiers: facts-at-depth w/ distractors (50), hard near-miss distractors
  on 45k-char sessions (30), 3x-mutated state tracking (18); 16 unanswerable
  confabulation guards. claude-fable-5, real production renderer, exact-match
  grading, no LLM grader.
- text 98/98, image 98/98, 0 wrong, 0 confabulated UNKNOWNs in either arm
- README benchmark table + FINDINGS dated update; PNGs gitignored,
  probes/results.jsonl committed as receipts
2026-06-10 18:54:03 -04:00
teamchong 1685935759 docs: remove em dashes from README 2026-06-10 07:28:20 -04:00
teamchong ab254d84ec fix(ci): ignore-pnpmfile so lockfile checksum doesn't depend on dev machine's global pnpmfile 2026-06-10 07:22:37 -04:00
teamchong 39b77eb246 ci: let pnpm/action-setup read version from packageManager field (fixes multiple-version conflict) 2026-06-10 07:16:53 -04:00
teamchong e406ec0de0 ci: switch release to npm Trusted Publisher (OIDC, environment: release), drop NPM_TOKEN 2026-06-10 07:15:09 -04:00
teamchong bb48aa4017 ci: add CI (typecheck/test/build) and tag-triggered npm release workflows 2026-06-10 07:13:02 -04:00
teamchong e57cc07c1b chore(publish): rename npm package to pxpipe-proxy (registry typo-squat filter blocked pxpipe); CLI command stays pxpipe v0.2.0 2026-06-10 07:06:22 -04:00
teamchong 63c1eab83e chore(publish): absolute README image URL, repo/homepage/bugs fields, Fable-scoped description+keywords, drop pnpm engine 2026-06-10 06:59:02 -04:00
teamchong cb522a70fe feat(dashboard): replace Svelte bundle with AHA stack (htmx + Alpine, server-rendered fragments)
- src/dashboard/fragments.ts: server-side HTML fragment renderers reusing the
  same JSON payloads (HTML and JSON surfaces can't drift)
- node.ts/dashboard.ts: /fragments/<name> routes (header, session-summary,
  recent, latest, sessions, stats, toggle) with htmx polling + Alpine for
  local state; toggle is a POST-returning-fragment htmx swap
- vendored htmx 2.0.4 + Alpine 3.14.9 in src/dashboard/vendor.ts (offline,
  single-file npm package, no CDN)
- deleted Svelte components/stores/bundle build step; dropped svelte/vite
  devDeps; build is now just tsc + esbuild
- tests: +5 fragment contract tests (routing, toggle state, HTML escaping of
  source text, 404) - 339 total
2026-06-09 22:17:53 -04:00
teamchong 40a3eb5cdd docs: caption matches the real render measurements 2026-06-09 21:58:37 -04:00
teamchong e520749b84 docs: example render is now real transformRequest output (reflow + instruction banner) 2026-06-09 21:57:50 -04:00
teamchong e20bf015c1 docs: add example render (full-density synthetic page) 2026-06-09 21:47:52 -04:00
teamchong 7a80bd1799 docs: fix image caption to match new dense example render 2026-06-09 21:45:46 -04:00
teamchong 898be465a3 docs: README example image now matches real dashboard density (full dense page, synthetic content) 2026-06-09 21:45:12 -04:00
teamchong 02f9c91564 fix(dashboard): dollar rate to Fable 5's $10/MTok input (was $5) 2026-06-09 21:11:11 -04:00
teamchong 4ffb75e665 docs: drop GPT 5.5 route mention, Fable 5 only 2026-06-09 20:36:16 -04:00
teamchong 528034cd82 feat(proxy): cloudflare-ai-gateway provider mode
- PXPIPE_PROVIDER=cloudflare-ai-gateway + PXPIPE_GATEWAY_BASE_URL derive both
  family routes ({base}/anthropic, {base}/openai) from one configurable base
- PXPIPE_GATEWAY_HEADERS injects auth headers (JSON or k=v;k2=v2) on every
  upstream request incl. count_tokens probes
- OpenAI paths drop /v1 to match gateway shape; Anthropic paths unchanged;
  unknown paths pass through on the family route; streaming untouched
- no hardcoded hostnames; tests use fake gateway URLs/tokens (14 new)
2026-06-09 19:21:01 -04:00
teamchong c35b3ab2d4 docs: rewrite README for readability — hook first, real render example, honest-caveats section 2026-06-09 19:10:49 -04:00
teamchong 65cd8aab71 docs: hide unproven Cloudflare Workers quick start 2026-06-09 19:08:49 -04:00
teamchong 5eb7ec6156 docs: demote unmeasured OpenAI route to a footnote, fix stale test count and ocproxy mention 2026-06-09 19:08:33 -04:00
teamchong c369b931e5 chore: rename remaining pixelpipe mentions in eval scripts/docs
frozen eval outputs under eval/results*/ left as-is (historical records)
2026-06-09 19:07:13 -04:00
teamchong b5504d166e docs: drop redundant (npm: pxpipe) from README title 2026-06-09 19:05:56 -04:00
teamchong 99bb931014 chore: rename all pixelpipe references to pxpipe
- code, tests, docs, README, wrangler.toml, help text
- data dir ~/.pixelpipe -> ~/.pxpipe (PXPIPE_LOG env var)
- rebuilt dist; 323/323 tests
2026-06-09 19:04:09 -04:00
teamchong a1c51b3543 feat: publish as pxpipe on npm
- rename package to pxpipe (pixelpipe is an npm security placeholder), bin: pxpipe
- move @napi-rs/canvas to dependencies (runtime dep for npx consumers)
- drop only-allow pnpm preinstall (blocked npm/npx installs)
- add prepublishOnly guard (typecheck + test + build)
- exclude sourcemaps from tarball; README quick start: npx pxpipe
2026-06-09 19:01:22 -04:00
teamchong 7ee2847e6c feat(dashboard): show source text behind each rendered image
- ImageEntry carries sourceText (capped 64KiB upstream in transform.ts)
- new GET /api/image-source[?id=N] endpoint
- LatestPng gains a 'view source text' toggle so the operator can see
  the exact text -> image conversion side by side
2026-06-09 18:47:48 -04:00
teamchong 0e3c95a079 docs: add CLI proxy quick start, fix stale constants table and test count 2026-06-09 18:33:34 -04:00
teamchong 3aae9a6253 chore: ignore generated eval images 2026-06-09 18:33:34 -04:00
teamchong 3d0f9ef08a scope: narrow model gate to fable-5, drop opus; switch dense render to bare 5x8 cell
- gate regex: fable-5 only (opus 4.7/4.8 read tax gone on fable: 100/100 novel-arithmetic, identical image billing)
- remove dead opus-4.6 cpt fork in transform.ts
- DENSE_RENDER_STYLE: 7x10 padded -> bare 5x8 cell (~42% fewer image tokens/page, recall flat in A/B)
- dashboard: compression defaults ON
- README/FINDINGS/TRANSFORM_INFO updated with 2026-06-09 measurements
2026-06-09 18:33:34 -04:00
Steven Chong 613ca3bc71 feat(proxy): support gpt 5.5 chat completions 2026-06-05 00:35:26 -04:00
teamchong f6150ba2bd docs(readme): lead benchmark with the clean 93%, demote contaminated GSM8K to a footnote
One headline number (93% novel reading), 0/15 as the boundary, GSM8K 96%
moved to a flagged footnote so the result isn't ambiguous.
2026-05-31 18:50:31 -04:00
teamchong 05a5ac98dc docs(eval): reproducible reading-fidelity benchmark + honest README table
Does the model actually read pixelpipe's render, or coast on memorized answers?
- novel random-number problems (un-memorizable): text 100% vs image 93% (-7pp real reading tax)
- GSM8K (in training data): 97% vs 96% -- inflated ~3pp by recall
- verbatim recall from a dense render: 0/15

README 'Benchmarks' section leads with the clean 93% and reports the failure
rows too (no cherry-pick). Harness in eval/gsm8k/: gen_novel.py, render_cfg.mjs
(pixelpipe's real renderTextToPngs), bench.py -- claude -p, exact-match grading.
2026-05-31 18:48:19 -04:00
teamchong 0e2979e13a fix(dashboard): default compression OFF so a restart keeps it off
The runtime kill switch reset to ON on every restart, silently re-enabling
the lossy text->image path (which also broke extended-thinking turns).
Default compressionEnabled to false: off on startup, survives restart;
only an explicit dashboard toggle turns it on.
2026-05-31 18:48:19 -04:00
teamchong 52f6b4a0e6 docs: apply FINDINGS reframe + ref fixes (follow-up to 2c38f30)
The prior commit captured only the rename (a stale pathspec aborted the
content staging). This applies the actual edits: retitle to FINDINGS,
'## Verdict' + superseded TL;DR, and the two cross-reference updates.
2026-05-29 22:47:12 -04:00
teamchong 2c38f30f7d docs: rename POSTMORTEM.md -> FINDINGS.md, lead with corrected verdict
The project isn't dead (verdict reversed to working ~68% gist-compressor),
so 'postmortem' was a misnomer. Rename to FINDINGS, retitle, promote the
correction to '## Verdict', and mark the old TL;DR as superseded. Update the
two cross-references (eval README link, public-api test comment).
2026-05-29 22:46:19 -04:00
teamchong 378dd7f6c4 feat(applicability)!: scope to opus 4.7+ and enforce model gate at proxy
Widen the model gate to /^claude-opus-4-(?:[7-9]|[1-9]\d)(?:-|$)/ (Opus 4.7
and newer; previously 4.6/4.7). Wire isPixelpipeSupportedModel into the proxy
boundary in proxy.ts — the proxy previously gated on economics only and
ignored the model, so it compressed 4.8 traffic despite the docs claiming a
4.6/4.7 scope. Unsupported models now pass through with reason
'unsupported_model'.

Update public-api tests to the 4.7+ spec (320/320 pass).

Correct the docs to live measurement: reverse the POSTMORTEM "dead" verdict
(pixelpipe is a lossy gist-compressor saving ~68% on real dense Claude Code
traffic; the verbatim 0/15 needle finding stands as a caveat, not the
verdict), rewrite README Status/Limitations, and add a correction pointer to
the eval README. Original POSTMORTEM body preserved below the correction.

BREAKING CHANGE: Opus 4.6 and older are no longer supported.
Verbatim-risk guard (skip imaging unique IDs/hashes/exact values) still pending.
2026-05-29 22:34:55 -04:00