Commit Graph

200 Commits

Author SHA1 Message Date
teamchong 514a4f0b74 feat(gate): config-driven model scope (PXPIPE_MODELS) + Opus 4.8 re-test
- applicability: isPxpipeSupportedModel reads PXPIPE_MODELS (comma-sep,
  default claude-fable-5 only) and strips bracket/variant tags like [1m];
  Opus becomes opt-in with no code change. Default behavior unchanged.
- FINDINGS: Opus 4.8 re-test - arithmetic 98/100 (-2pp), verbatim 6/15
  (was 93/100, 0/15): improved but still taxed -> default stays Fable-only.
- FINDINGS + eval/glyph-matrix/sweep: root cause of the read tax is per-glyph
  RESOLUTION, not font shape (confusions scale with px and vanish by 10x16).
  Opus needs ~4x glyph area to read reliably; the 1568px ceiling makes that
  break-even, so there is no profitable Opus operating point.
2026-06-16 12:22:55 -04:00
teamchong 50f3f1f86b docs: add prompt-caching alignment + honest savings-math doc
New docs/CACHING_AND_SAVINGS.md explains the two things that keep coming up:

1. How pxpipe stays aligned with Anthropic's prompt cache when it images the
   static prefix: split static/dynamic blocks first (keep drifting <env> etc.
   as cheap text, never imaged), then relocate (never add) the caller's
   cache_control onto the last image so the breakpoint anchors at the end of
   the stable slab. Covers the cache-friendly splice, the symmetric
   anti-flapping burn term, and the horizon-amortized history-collapse gate.

2. How savings are measured without double-counting the cache discount: both
   the real request and a count_tokens counterfactual of the original body are
   priced with identical cache rates at the same moment, so the discount
   cancels in the difference and only the token reduction survives.

Also reconcile the priorWarmTokens JSDoc in transform.ts: the burn term is
applied undivided, matching all three gate call sites and the symmetric
priorWarmImageTokens doc. The '/ N' was stale (the per-turn gate has no
horizon to divide by; the horizon scales the lifetime cost, not the burn).
2026-06-16 08:35:48 -04:00
teamchong be85f58d90 demo: pivot to smart-zone demo (plain Claude rots at large context, pxpipe stays sharp)
- generate.mjs builds one large context (filler logs + small state-tracking needle)
- README: two-column run (plain vs pxpipe), same prompt, compare answers + /context
- drop the e-commerce /context demo; context/ is generated, gitignored
2026-06-16 08:18:21 -04:00
teamchong 0277facc5d docs: reframe savings as workload-dependent snapshots; add demo + render-sizing doc
- README: cost savings now presented as measured snapshots (~59-70% end-to-end,
  ~72-74% compressed) bound to trace + workload, not a fixed 59%
- add demo/ (synthetic side-by-side /context demo: data, prompt, expected answer)
- add docs/RENDER_SIZING.md (image-sizing rules, rationale, commit history)
2026-06-15 22:33:31 -04:00
teamchong 7ee263c596 docs: drop glyph-matrix resume date (reader is fable-5, currently unavailable) 2026-06-14 17:27:35 -04:00
teamchong a10837dcdb docs: add roadmap - glyph legibility, effective context, context-rot hypotheses (all unmeasured) 2026-06-14 17:24:45 -04:00
teamchong 0ec9efe9ca docs: FINDINGS field observation (n=1) + glyph-matrix PLAN.md
- FINDINGS.md: dated 2026-06-12 entry logging in-the-wild verbatim
  misreads (7/10 exact, ~3.3% per-char, 2/3 silent) alongside flawless
  gist-tier recall in the same session. Explicitly anecdotal, n=1,
  non-reproducible — motivation only.
- eval/glyph-matrix/PLAN.md: public plan for the paused confusion-matrix
  experiment (Task #7) so the pending work is visible outside README,
  per the measured-numbers-only README rule.
2026-06-12 13:25:51 -04:00
teamchong 958b17357b Revert "docs: troubleshooting note for minimum-release-age install failures"
This reverts commit c865520657.
2026-06-12 06:23:21 -04:00
teamchong c865520657 docs: troubleshooting note for minimum-release-age install failures 2026-06-12 06:22:20 -04:00
teamchong 8bc1478406 v0.2.1: README scope corrections (docs-only) v0.2.1 2026-06-12 05:55:28 -04:00
teamchong 69eb96d17e docs: README scope wording - imaging covers system prompt + tool docs, not just old history 2026-06-12 05:40:57 -04:00
teamchong 5704c84237 docs: add real-world failure FAQ; scrub usernames/home paths from eval corpus 2026-06-11 17:17:19 -04:00
teamchong 62790e7371 docs: add FAQ - end-to-end bill math vs touched-only, measurement method, compression scope 2026-06-11 16:53:18 -04:00
teamchong d990faf646 eval(verbatim): expand to n=15 dense hex recall on Fable 5 - 13/15 vs Opus 0/15, both misses single-glyph confabulations 2026-06-11 16:48:12 -04:00
teamchong 7d62a4c715 eval(swe-bench-pro): expand to 19 pairs — ON 14/19 vs OFF 15/19, agree 18/19; navidrome 3/3 replication clears the ON loss 2026-06-11 15:47:54 -04:00
teamchong 156945977e eval(swe-bench-pro): 10-pair bench - ON 6/9 vs OFF 7/9, -61% per-request, receipts committed 2026-06-11 10:43:02 -04:00
teamchong 4eef1ec1a0 fix(eval/swe-bench-pro): verified proxy startup + atomic repo cache clone 2026-06-11 08:09:48 -04:00
teamchong b8e1e855bc feat(eval/swe-bench-pro): resumable 10-pair runner on bench-dedicated proxies
- ON 47823 / OFF 47824 with own PXPIPE_LOG files: operator session can
  never pollute bench measurement (separation by construction)
- resume = skip existing patch files; clean stop on quota errors
- Docker grading documented as proxy-free (no model calls in containers)
- README: mention Pro pilot alongside Lite
2026-06-11 08:05:06 -04:00
teamchong e98bf36387 fix(eval/swe-bench-pro): deconfound ON-arm numbers from operator session sharing port 47821 2026-06-11 05:00:01 -04:00
teamchong 78b2247a92 eval(swe-bench-pro): single-pair pilot - 1/1 both arms, -64% per-request, Rosetta gotcha documented 2026-06-11 04:56:06 -04:00
teamchong 8e8cb09223 docs: SWE-bench tables lead with per-request -65%, demote confounded run totals 2026-06-10 23:49:38 -04:00
teamchong 83764a18a0 docs: surface SWE-bench parity in honest-part section, link eval dir 2026-06-10 23:31:55 -04:00
teamchong 76542b4a3c eval(swe-bench): Lite pilot 10 paired instances — 10/10 resolved both arms, ON $27.27 vs OFF $53.61 (-49%)
- run_pilot.py harness (paired generation via proxy ON 47821 / OFF 47822)
- official swebench Docker grading via colima, reports + patches committed
- README.md results table, FINDINGS.md task-completion parity update
- repo README benchmark table gains SWE-bench row
2026-06-10 23:25:24 -04:00
teamchong 5274e74bd4 docs: lead with whole-bill savings (59%, all 13.7k requests) over touched-only 72% 2026-06-10 19:08:32 -04:00
teamchong c6a0d61f00 eval(gist-recall): A/B imaged vs text history — 98/98 both arms, 0 confabulation
- 3 tiers: facts-at-depth w/ distractors (50), hard near-miss distractors
  on 45k-char sessions (30), 3x-mutated state tracking (18); 16 unanswerable
  confabulation guards. claude-fable-5, real production renderer, exact-match
  grading, no LLM grader.
- text 98/98, image 98/98, 0 wrong, 0 confabulated UNKNOWNs in either arm
- README benchmark table + FINDINGS dated update; PNGs gitignored,
  probes/results.jsonl committed as receipts
2026-06-10 18:54:03 -04:00
teamchong 1685935759 docs: remove em dashes from README 2026-06-10 07:28:20 -04:00
teamchong ab254d84ec fix(ci): ignore-pnpmfile so lockfile checksum doesn't depend on dev machine's global pnpmfile 2026-06-10 07:22:37 -04:00
teamchong 39b77eb246 ci: let pnpm/action-setup read version from packageManager field (fixes multiple-version conflict) 2026-06-10 07:16:53 -04:00
teamchong e406ec0de0 ci: switch release to npm Trusted Publisher (OIDC, environment: release), drop NPM_TOKEN 2026-06-10 07:15:09 -04:00
teamchong bb48aa4017 ci: add CI (typecheck/test/build) and tag-triggered npm release workflows 2026-06-10 07:13:02 -04:00
teamchong e57cc07c1b chore(publish): rename npm package to pxpipe-proxy (registry typo-squat filter blocked pxpipe); CLI command stays pxpipe v0.2.0 2026-06-10 07:06:22 -04:00
teamchong 63c1eab83e chore(publish): absolute README image URL, repo/homepage/bugs fields, Fable-scoped description+keywords, drop pnpm engine 2026-06-10 06:59:02 -04:00
teamchong cb522a70fe feat(dashboard): replace Svelte bundle with AHA stack (htmx + Alpine, server-rendered fragments)
- src/dashboard/fragments.ts: server-side HTML fragment renderers reusing the
  same JSON payloads (HTML and JSON surfaces can't drift)
- node.ts/dashboard.ts: /fragments/<name> routes (header, session-summary,
  recent, latest, sessions, stats, toggle) with htmx polling + Alpine for
  local state; toggle is a POST-returning-fragment htmx swap
- vendored htmx 2.0.4 + Alpine 3.14.9 in src/dashboard/vendor.ts (offline,
  single-file npm package, no CDN)
- deleted Svelte components/stores/bundle build step; dropped svelte/vite
  devDeps; build is now just tsc + esbuild
- tests: +5 fragment contract tests (routing, toggle state, HTML escaping of
  source text, 404) - 339 total
2026-06-09 22:17:53 -04:00
teamchong 40a3eb5cdd docs: caption matches the real render measurements 2026-06-09 21:58:37 -04:00
teamchong e520749b84 docs: example render is now real transformRequest output (reflow + instruction banner) 2026-06-09 21:57:50 -04:00
teamchong e20bf015c1 docs: add example render (full-density synthetic page) 2026-06-09 21:47:52 -04:00
teamchong 7a80bd1799 docs: fix image caption to match new dense example render 2026-06-09 21:45:46 -04:00
teamchong 898be465a3 docs: README example image now matches real dashboard density (full dense page, synthetic content) 2026-06-09 21:45:12 -04:00
teamchong 02f9c91564 fix(dashboard): dollar rate to Fable 5's $10/MTok input (was $5) 2026-06-09 21:11:11 -04:00
teamchong 4ffb75e665 docs: drop GPT 5.5 route mention, Fable 5 only 2026-06-09 20:36:16 -04:00
teamchong 528034cd82 feat(proxy): cloudflare-ai-gateway provider mode
- PXPIPE_PROVIDER=cloudflare-ai-gateway + PXPIPE_GATEWAY_BASE_URL derive both
  family routes ({base}/anthropic, {base}/openai) from one configurable base
- PXPIPE_GATEWAY_HEADERS injects auth headers (JSON or k=v;k2=v2) on every
  upstream request incl. count_tokens probes
- OpenAI paths drop /v1 to match gateway shape; Anthropic paths unchanged;
  unknown paths pass through on the family route; streaming untouched
- no hardcoded hostnames; tests use fake gateway URLs/tokens (14 new)
2026-06-09 19:21:01 -04:00
teamchong c35b3ab2d4 docs: rewrite README for readability — hook first, real render example, honest-caveats section 2026-06-09 19:10:49 -04:00
teamchong 65cd8aab71 docs: hide unproven Cloudflare Workers quick start 2026-06-09 19:08:49 -04:00
teamchong 5eb7ec6156 docs: demote unmeasured OpenAI route to a footnote, fix stale test count and ocproxy mention 2026-06-09 19:08:33 -04:00
teamchong c369b931e5 chore: rename remaining pixelpipe mentions in eval scripts/docs
frozen eval outputs under eval/results*/ left as-is (historical records)
2026-06-09 19:07:13 -04:00
teamchong b5504d166e docs: drop redundant (npm: pxpipe) from README title 2026-06-09 19:05:56 -04:00
teamchong 99bb931014 chore: rename all pixelpipe references to pxpipe
- code, tests, docs, README, wrangler.toml, help text
- data dir ~/.pixelpipe -> ~/.pxpipe (PXPIPE_LOG env var)
- rebuilt dist; 323/323 tests
2026-06-09 19:04:09 -04:00
teamchong a1c51b3543 feat: publish as pxpipe on npm
- rename package to pxpipe (pixelpipe is an npm security placeholder), bin: pxpipe
- move @napi-rs/canvas to dependencies (runtime dep for npx consumers)
- drop only-allow pnpm preinstall (blocked npm/npx installs)
- add prepublishOnly guard (typecheck + test + build)
- exclude sourcemaps from tarball; README quick start: npx pxpipe
2026-06-09 19:01:22 -04:00
teamchong 7ee2847e6c feat(dashboard): show source text behind each rendered image
- ImageEntry carries sourceText (capped 64KiB upstream in transform.ts)
- new GET /api/image-source[?id=N] endpoint
- LatestPng gains a 'view source text' toggle so the operator can see
  the exact text -> image conversion side by side
2026-06-09 18:47:48 -04:00
teamchong 0e3c95a079 docs: add CLI proxy quick start, fix stale constants table and test count 2026-06-09 18:33:34 -04:00