119 Commits

Author SHA1 Message Date
teamchong de0483cf51 fix(transform): wrap relocated env text in <system-reminder> for correct attribution 2026-07-03 01:50:08 -04:00
teamchong 06a8b70c01 fix(transform): relocate volatile env text behind ALL cache breakpoints
Volatile env/context text (git status, cwd, date) rode in req.system —
BEFORE the slab anchor in Anthropic's prefix order (tools → system →
messages) — so any git-state change cold-restarted the entire anchored
prefix. Telemetry attribution (events.jsonl 2026-06-26..07-02): 48.8%
of cold-create waste, ~2.6k tokens/session vs ~200 saved by imaging it.

Now appended as a trailing text block on the LAST user message — the
per-turn live tail that re-caches incrementally anyway — placed AFTER
history collapse (never baked into frozen chunks) and after tool_result
compression. Session-stable billingLine/sysRemainder stay in system;
fallback keeps env in system when no user message exists to carry it.
New telemetry field: envRelocatedChars.
2026-07-02 23:17:13 -04:00
teamchong e49bb11878 feat(telemetry): log stop_reason + safety-flag per request 2026-07-02 23:07:37 -04:00
teamchong beacd67b4a fix(transform): freeze imaged slab at first render — keep volatile content (skill listings, cwd caches) out of the imaged prefix so turn-2 system sha matches turn-1 2026-07-02 21:11:18 -04:00
teamchong 0ad3c68664 fix(transform): cross-session slab stability + drop compressSchemas knob
Fix #1 (cross-session cache bust): strip the volatile "# Environment"
markdown section (working dir, git status, platform, model ID) out of
the system text BEFORE the static/dynamic split. It carries no XML
wrapper so splitStaticDynamic couldn't catch it, and its git-status
lines change across sessions - baking them into the slab PNG busted
the cross-session cache (system_sha8 717f1fce -> 5efaa4bb for a
one-file edit). Parallel to stripBillingLine: the section re-enters
the system tail as plain text.

Schema handling simplified, knob removed: the Anthropic text-reference
path now renders prose-only tool docs (schemas stay untouched in
tools[] - duplicating them in the reference was pure bloat), while the
GPT imaged path always carries the compact schema (there it IS the
compression). Deletes the {type:'object'} stub + SCHEMA_STRUCTURAL_KEYS
machinery, stripSchemaDescriptions import, and the COMPRESS_SCHEMAS env.

Tests: e2e regressions for anchor relocation onto byte-frozen carry-over
history images and first-collapse anchor staying on the slab; history
relocation fixture grown past the collapseChunk=50 snap (56 messages ->
cutoff 50, frozen windows 11/21/31/41, collapsedTurns 49 - the old
35-message fixture floored to one window and never froze a chunk).

tsc clean; 615/615 tests pass.
2026-07-02 17:36:18 -04:00
teamchong baa1e1ee01 fix(transform): reword tool-docs stub/header to avoid refusal classifier
The text-based Tool Reference triggered Anthropic's model-cloning
safeguard (stop_reason: refusal -> silent Opus fallback) because the
per-tool stub said docs lived in "the system prompt" — the exact
phrasing the 169521c retrip identified as the trigger for the imaged
slab, now reintroduced in text form.

- Stub now reads: 'ⓘ Full docs: see "## Tool: <name>" in the Tool
  Reference section.' — no "system prompt"/"authoritative" framing.
- Tool Reference header is provenance-framed as first-party ("pxpipe
  (this user's local proxy) moved the full tool documentation...").
- tracker.ts now maps tool_docs_chars into events.jsonl so refusals
  are attributable to the text-reference path.
- Regression test: stub + reference header must never match
  /system prompt|authoritative/i and must carry provenance framing
  (scoped to the header; quoted third-party tool docs below it may
  legitimately contain the phrase).

Verified: 614/614 tests, tsc --noEmit clean, and a live cold-start
through the rebuilt proxy on claude-fable-5 with the text reference
active — no refusal event, no fallback.
2026-07-02 13:17:39 -04:00
teamchong ed968a7495 Reapply "fix(render): fit Anthropic page to 1568×728 (~1.15 MP) for WYSIWYG glyphs"
This reverts commit 6eaa8bfcd0.
2026-07-01 19:18:20 -04:00
teamchong 6eaa8bfcd0 Revert "fix(render): fit Anthropic page to 1568×728 (~1.15 MP) for WYSIWYG glyphs"
This reverts commit 838db8778e.
2026-07-01 18:50:25 -04:00
teamchong 838db8778e fix(render): fit Anthropic page to 1568×728 (~1.15 MP) for WYSIWYG glyphs
Measured 2026-07-01 (count_tokens sweep, claude-sonnet-4-5; see
docs/LEGIBILITY-AUDIT-2026-07-01.md): the API downscales every image to fit
BOTH long-edge ≤1568 AND ~1.15 MP (≈1,143,750 px), then bills ≈px/750
(~1525 tok/img cap). The old 1932×1932 page hit the cap but was resampled
0.555× before the vision encoder, so 5×8 glyphs arrived at ~2.8×4.4 px — the
root cause of the legibility misses this session.

New page shape 1568×728 = 1,141,504 px fits both bounds → glyphs reach the
encoder WYSIWYG (and still ≤2000 px/side for >20-image requests).

render.ts geometry:
  MAX_HEIGHT_PX          1932  -> 728
  MAX_WIDTH_PX           1932  -> 1568
  DEFAULT_COLS            313  -> 312  (1568 px exactly; 313=1573 would 0.997× blur)
  DENSE_CONTENT_COLS     384  -> 312
  READABLE/DENSE chars  50000/92160 -> 28080  (312×90, tracks real pagination)

Decouple the GPT path: OpenAI resizes differently (2048 bbox, 768 short side),
so a 768-wide strip up to 2048 tall survives un-resampled. Add
GPT_MAX_HEIGHT_PX = 1932 (gpt-model-profiles.ts) where the built-in cost
numbers were calibrated; openai-history.ts now uses it instead of the
now-Anthropic-specific MAX_HEIGHT_PX. Doc-only touch-ups in library.ts/export.ts.

tests/render + tests/paging updated to the new geometry (171 pass).
2026-07-01 17:54:51 -04:00
teamchong a346ec7ac5 feat(factsheet): carry occurrence counts, match ticket-style codes
- extractFactSheetEntries() returns {token, count}; the kept-token set
  and order are byte-identical to the old extractFactSheetTokens(),
  which now delegates. Counts render as a ×N suffix, so tally questions
  over imaged content are answerable from the text sheet instead of by
  counting 5x8 px glyph rows.
- Same-token spans matched by overlapping patterns dedupe by offset
  (token + match index) so nothing double-counts.
- New tier-0 shape: uppercase hyphenated codes with a digit
  (PROJ-1482, CVE-2024-30078); digit lookahead is bounded.
- export.ts consumes entries across all pages; 14 new/extended tests.
2026-07-01 16:41:00 -04:00
Steven Chong bf49065dce fix(savings): use observed cache state for text baseline 2026-06-30 12:24:09 -04:00
Steven Chong ff8a05e30c fix(savings): gate text-counterfactual warmth on static-prefix hash
A fresh same-session prior within the TTL is no longer sufficient to price
the text counterfactual warm: deriveBaselineWarmth now also requires the
static-prefix hash (system_sha8) to match. When opencode rotates the system
prompt / tool docs mid-session, the cacheable prefix changes, so a text-only
client would hit a new provider cache key too — pricing it warm against a
cold actual fabricated a huge phantom "loss" (the dashboard's 800%-worse
report). cr>0 still rescues a genuine warm read with no in-memory prior.

Wired through update(), replay(), and aggregateSessions via system_sha8.

Also surface losses honestly in the recent table (Saved/lost: negative
deltas in red instead of hidden as "—") and clarify headings as
billing-equivalent input tokens. Docs updated to match.
2026-06-29 15:59:10 -04:00
teamchong 966f9a144e refactor(export): single-source REPORT_CHARS_PER_TOKEN in transform.ts
Drop the duplicate CHARS_PER_TOKEN=3.7 from export.ts and import REPORT_CHARS_PER_TOKEN from transform.ts so the reporting-estimate constant has one source of truth. Clarify the dashboard baseline-warmth invariant comment. No behavior change.
2026-06-27 23:41:11 -04:00
teamchong cde21b2e73 fix(savings): base text-counterfactual warmth on wall-clock, not image-cache fate
The savings accounting priced the text baseline as cold whenever pxpipe's own
image cache missed this turn (cr==0), fabricating a 1.25x create the text path
would never have paid and inflating reported savings ~2x. Decouple them: the
text prefix reads warm whenever a fresh same-session prior exists within the
300s TTL (wall-clock), unioned with an observed read (cr>0) for the post-restart
case. Centralised in deriveBaselineWarmth; used by all three call sites
(sessions, dashboard, fragments).

Empirical replay over ~/.pxpipe/events.jsonl (13,402 message rows): honest
headline 504.6M tokens saved vs the old cr-gated 1.05B; 1,409/12,305 fresh-prior
rows (11.4%) read warm despite a busted image cache. Dashboard now narrates the
busted-image case explicitly. Adds baseline/sessions/context-map regression
tests. Measurement only — no change to proxy behavior.

typecheck + 600 tests + build all green.
2026-06-27 21:26:55 -04:00
teamchong 76a371028a feat(export): replace in-band @pxpipe directive with explicit pxpipe export CLI
The magic-word @pxpipe directive fired unreliably and added per-request body
scanning to the hot path, so drop it entirely: removed the directive module +
its test and unwired it from transform.ts / proxy.ts / node.ts / index.ts.
transformRequest now leaves message bodies byte-identical — no per-request
directive parsing.

In its place, `pxpipe export <paths>` renders code/text to dense PNG pages via
the same SDK primitive the proxy uses, so exported pages are byte-identical to
what ships to the model. Width is locked to proxy density (--cols rejected as
an unknown option, guarded by tests) to prevent drift. Added --open to preview
the rendered pages plus a result footer (artifact summary + token-savings line).

Tests: export.test.ts 64 passed (incl. --open parse + default coverage); full
suite 589 passed / 29 files; typecheck clean.
2026-06-27 18:28:28 -04:00
teamchong 6d1c46dd20 fix(cache): pin prefix-cache anchor to the byte-stable history image
Intermittent prefix-cache busts on long sessions (#11): the single cache
breakpoint relocated onto the LAST history image — the still-growing chunk
whose bytes change every window advance — so the slab+history prefix
re-created at 1.25x instead of being read.

- relocateAnchorToHistoryImage pins collapseHistory's carry-over ordinal
  (the last byte-stable chunk) instead of the last image.
- demoteProtectedHeadText keeps pxpipe's slab scaffolding (images +
  fact-sheet + the [End of rendered context.] boundary) verbatim and only
  quarantines the user's stale opening turn, so the relocator can still
  find the slab anchor (#14 demotion had clobbered the boundary it keys on).
- cachePrefixDigest joins parts with \x00 so block-boundary differences
  can't hash-collide into false prefix-stability.

520/520 green, tsc clean.
2026-06-26 00:17:10 -04:00
teamchong 2e8f89288b fix(factsheet): rank tokens by shape so short high-consequence ids survive
The 64-token budget was filled longest-first, so long low-risk doc-URLs evicted short zero-redundancy tokens (git SHAs, ports, CLI flags, CONST_IDS) — the strings a coding agent must reproduce verbatim. Allocate by priority tier (ids/flags/numbers > paths/versions > URLs, capped); preserve substring-collapse and byte-stable, cache-safe output. Adds an eviction regression test.
2026-06-25 19:58:54 -04:00
teamchong 78abd081e4 feat(factsheet): byte-stable fact-sheet of exact identifiers on Claude + OpenAI paths
Recovers precision-critical strings (paths/ids/versions) from the imaged
slab as cache-stable text so they survive OCR. Byte-identical across turns,
validated on both Claude and gpt-5.5 paths.
2026-06-25 17:23:03 -04:00
teamchong 8417380061 test: e2e guards for cache stability, dashboard honesty, and design behaviors
Test-only (no src changes). Adds fake-upstream + real-createProxy suites:

- cache-stability-e2e: byte-identical cacheable prefix across growth, marker
  conservation + relocation onto the history image, tool-pair integrity.
- savings-math-e2e: gate never images at a net loss; baselineImagedTokens
  cross-checked against the real o200k tokenizer; baselineTokens wired from
  the count_tokens probe.
- savings-honesty: categorical no-overclaim invariants (warm <= cold,
  saved==delta*weight, zero credit on passthrough) + per-model pricing guards
  (Anthropic 1.25/0.1 shared policy; GPT 0.1x model-gated).
- design-behavior-e2e: system-prompt imaging, history collapse (old imaged /
  recent legible), tools-in-recent-turns kept usable.
- dashboard-api: GPT cold-cache parity (full text delta, not 0.1x warm rate).

Each guard mutation-verified (broke source, confirmed red, reverted).
2026-06-23 12:30:16 -04:00
Steven Chong 9eb66f54f2 fix(history): anchor collapsed history recency 2026-06-23 06:37:44 -04:00
teamchong 08a2b0a118 feat(gpt): pin the live user request as text in history collapse
Autonomous GPT agents (OpenCode/gpt-5.5) send ONE human request then a long
run of tool turns. The lone request is the OLDEST turn, so history collapse
imaged it first and the model lost it — "I wonder what the user actually asked"
→ off-task drift (observed: agent edited compaction.ts instead of answering a
compare question). The live-request guard's "the request is the trailing user
message" heuristic assumes interactive chat and points at nothing here.

Fix: keep the most-recent user turn OVERALL as legible TEXT, spliced between
before-pin and after-pin history images inside the synthetic user message;
older user turns stay imaged (must not look live — snap-to-first-prompt guard).
The guard now echoes the request verbatim (capped). History is imaged on both
sides of the pin, so compression barely changes.

Cache safety (adversarially reviewed):
- Pin ONLY when the latest user turn is INSIDE the collapse range. If it's in
  the kept tail (ordinary interactive turn) it's already native text — pinning
  an older in-range turn would migrate the pin across collapse-chunk boundaries
  and re-image frozen history. Restricting to the latest turn fixes its position
  until the next prompt, so before/after sections stay byte-stable across a run.
- Undersized before-pin remainder merges into the previous before-section rather
  than emitting a sub-threshold (net-negative) image.

Both Chat Completions and Responses paths. 473 tests pass; tsc + build clean.
Not yet released — validate on the gpt-5.5 machine first.
2026-06-22 12:15:16 -04:00
teamchong 22d89ee37d fix(sessions): complete honest cache-math warmth gate
b9418bf repriced cr>0-but-no-prior turns warm in dashboard.ts (update + replay)
but left the identical gate in sessions.ts unfixed — so the Sessions panel still
fabricated inflated savings after a restart/eviction or on the first tracked turn
(a cr>0 turn priced cold bills the text counterfactual a 1.25x create on a prefix
we KNOW was cached). Mirror the dashboard fix so "honest cache math" holds on every
panel.

- sessions.ts: warm follows the observed read (cr>0) alone; with no fresh prior,
  assume the cacheable prefix was fully reused (prevCacheable = cacheable) rather
  than mispricing a known-warm read.
- sessions.test.ts: the "real prefix compression" case had cr=100 on its first
  turn (observably warm) yet asserted the cold-priced 22490 fabrication; corrected
  to the honest 1790 (warm) + 95 = 1885. Now a regression guard for this fix.
- dashboard.ts: drop the stale "no per-session state" replay comment — replay now
  mirrors update()'s per-session warmth.

Found by adversarial review of b9418bf. 462 tests pass; tsc clean.
2026-06-22 11:07:02 -04:00
Steven Chong b9418bfa6f fix(gpt): cap history images and honest cache math 2026-06-22 10:46:47 -04:00
Steven Chong f6be3d3f88 fix(openai): keep opening prompt collapsible 2026-06-21 17:37:51 -04:00
teamchong fe6b0ec2a8 feat: structure-through role attribution + turn-index recency anchor for history images
Collapsed-history images now carry who-said-what AND when, instead of leaving
the model to reconstruct it from a flattened, role-ambiguous blob (which led it
to resurface the opening turn as if it were the live request).

- Turn-index recency anchor: each collapsed turn serializes as <user t="N"> /
  <assistant t="N"> with an ABSOLUTE turn index (larger N = more recent), so the
  model can tell turn 1 from turn 60. Absolute (never "N ago"/"i of total") so
  frozen chunks stay byte-identical and keep hitting the prompt cache. Mirrored on
  the GPT path. (history.ts, openai-history.ts, openai.ts)
- colorByRole: role tags are tinted via a parallel "slot string" carried from
  serialize time (structure-through), replacing the parse-back that miscolored a
  body quoting a literal <tag>. (render.ts, history.ts)
- neutralizeSentinel: a pre-existing ↵ (U+21B5) in content is swapped to ⏎
  (U+23CE) in render-prep so reflow packs newlines instead of bailing to a raw,
  unpacked render — common when the content is about pxpipe itself. Render-only;
  originals preserved. (render.ts, transform.ts, history.ts, openai-history.ts)
- Per-model GPT vision/render profiles, retunable via PXPIPE_GPT_PROFILES,
  behavior-identical to the old hardcoded values. (gpt-model-profiles.ts)

Review fixes folded in:
- Banner intro/outro consolidated to a single source of truth; the GPT path now
  aliases the Anthropic constant instead of byte-copying it (can't drift).
- slotCopyBody neutralizes literal slot markers to a width-equivalent control
  char (U+0003), not a space the minifier would strip and misalign.

459 tests pass; tsc clean.
2026-06-21 17:17:08 -04:00
Steven Chong 47b5e412b3 feat: cache-stable history-collapse imaging for GPT (OpenAI) + Anthropic
Collapse old conversation history into rendered PNG sections so the model
reads a compact image instead of re-billed text, while preserving prompt
caching and tool-call behavior. Measures real vs compressed token/cost.

Core:
- GPT history collapse (openai-history.ts): append-only, o200k token-length
  sectioning. Sections seal only at a tool-closed boundary (open call-id set
  empty), so a function_call and its function_call_output never split across
  the collapse cut. Fixes the OpenAI 400 "No tool call found for function
  call output with call_id ..." that hit long Responses-API sessions.
- Anthropic cache contract (history.ts): append-only per-chunk rendering;
  cache_control markers are preserved/moved, never added; chunk boundaries
  align with caller marker seams for byte-stable prefix caching.
- GPT image budget (openai.ts): detail:'original' for gpt-5.x, flagship
  vision-multiplier fix, patch cap; schema-strip preserves real descriptions.
- Savings accounting (openai-savings.ts): cached_tokens + vision-token basis.

Model scope (applicability.ts):
- Default imaged scope = claude-fable-5 + gpt-5.6.
- gpt-5.5 and claude-opus-4-8 stay opt-in: same pipeline, but they degrade
  reading dense imaged history (gist drift), so silently imaging them by
  default is wrong. Promotion is gated on an OCR/recall threshold.

Dashboard: GPT + Anthropic rendering, per-family model toggles, persisted
metrics, thumbnail-expired session UI, reflow/newline handling.

Tests: cache-alignment (GPT + Anthropic), history sectioning + tool-boundary
invariants, savings, dashboard, sessions/restart-restore. 452 passing.
2026-06-20 21:11:24 -04:00
teamchong 8b15525cb9 fix(transform): relocate cache anchor onto history image (stop 1.25x re-create)
The history image sits after the slab in prefix order, but the single cache
breakpoint was anchored on the slab. So the largest block — the ~141k-token
history image — only cached when the caller's roaming downstream marker happened
to land after it. When it didn't, the whole history image re-created at the 1.25x
cache_create rate every turn instead of amortizing into 0.1x reads. On a warm
session that turned a real compression win into a net loss (the "411% more"
the dashboard correctly surfaced).

Fix is pure relocation: move pxpipe's single breakpoint from the slab onto the
LAST history image (which sits after the slab), so slab + history cache as one
stable prefix — created once, then read. Marker count is unchanged: pxpipe still
never ADDS a marker (Task #21), it only moves the caller's existing one. When no
slab imaged, the history path keeps its own relocation.

- transform.ts: relocateAnchorToHistoryImage on the collapse-success branch
- history.test.ts: two old "history carries no marker" invariants upgraded to
  assert the relocation (last history image carries it; exactly one marker total)
2026-06-19 16:39:46 -04:00
teamchong b20c976f61 fix(dashboard): restore Saved + Details on replayed rows after restart
replay() rebuilt RecentRow from the JSONL but never set
session_saved_so_far_delta, img_id, or contextHistory — so after a restart
old rows showed "-" in Saved and had no Details link, while live rows did.
Live update() does all three; replay() now matches.

- replay() sets session_saved_so_far_delta (baselineInputEff - actualInputEff,
  same cache-weighted math as live) so the Saved column survives a restart.
- replay() rebuilds the Context Map breakdown from persisted fields
  (baseline_tokens, bucket_chars, image_count, usage) with a synthetic
  nextImageId, and sets img_id/img_ids so the Details link resolves.
- The PNG ring is in-memory and not restorable; restored entries carry
  imageIds:[] and a `restored` flag, and the Details panel shows
  "thumbnails expired when the proxy restarted" instead of dead <img>s.
- +2 replay tests (restart-restore.test.ts).

376 tests pass; typecheck + build clean.

Claude-Session: https://claude.ai/code/session_01RV8uiMYNhrAdTUzJ9fkpcp
2026-06-19 16:06:59 -04:00
teamchong 48f82ffee2 fix(dashboard): hero uses cache-weighted tokens (match Details/Saved)
The session hero computed (1 - sent / raw_count_tokens_baseline) — the same
cache-blind math just removed from the Details panel. It priced the all-text
baseline at full rate, ignoring that most of it would have been cheap
cache-reads, so a warm net-loss session showed a big "fewer tokens" win up top
while Details/Saved showed a loss. Two panels, one session, opposite signs.

- renderSessionSummaryFragment now reads baselineInputWeighted /
  actualInputWeighted (the same cache-weighted pair as Details + the Saved
  column) instead of rawActualTokens / rawBaselineTokens.
- Headline is input-only; dropped the output-on-both-sides lumping (output is
  never "sent" and only dampened the %).
- Honest labels: "effective tokens … after normal cache discounts"; still no $
  assumptions (0.1x/1.25x are token multipliers, not $/Mtok).
- types.ts: corrected stale field comments (raw fields are math-drawer only).
- +4 hero tests pinning direction to the weighted pair and that output can't
  move the headline %.

374 tests pass; typecheck + build clean.
2026-06-19 15:58:22 -04:00
teamchong c899d7d836 fix(dashboard): Details headline uses cache-weighted tokens (match row)
The Details panel divided the raw count_tokens baseline by sent tokens
(cache-blind), over-claiming "X% smaller" even when the cache-aware row
showed a net loss — the two panels could flatly contradict each other.

- dashboard.ts: feed baselineInputEff/actualInputEff/haveBaseline into
  ContextMapData (same weighted pair the recent row uses); push after
  the eff-tokens are computed so the breakdown matches.
- fragments.ts: headline divides weighted baseline by weighted actual,
  says "bigger" on a net loss, with a clarifying sub-line; no savings
  claim when the baseline probe did not resolve.
- +4 tests (context-map.test.ts) pinning headline direction to the row.

370 tests pass; typecheck + build clean.
2026-06-19 14:38:17 -04:00
teamchong 97d50513aa feat(openai): GPT-5.x family + Responses API support
- Generalize isPxpipeSupportedGptModel to the GPT-5 family (gpt-5, 5.x,
  -mini/-nano), default-on; [variant] tags stripped.
- OpenAI render switches to the 768px portrait strip (GPT_STRIP_COLS=152)
  so the API's mandatory shortest-side->768 downscale never shrinks our
  5px glyphs; downscale-free in both tile and patch regimes.
- Add OpenAI vision-token cost model (resolveVisionCost/openAIVisionTokens)
  and rewire the Chat Completions profitability gate to use it instead of
  the Anthropic 750px/token model.
- Add transformOpenAIResponses for the /v1/responses (Responses API) shape
  used by Codex: instructions + input items + flat tools -> rendered
  input_image parts; wire it into the proxy (was passthrough before).
- Tests: +12 (gate family, vision-token math incl. downscale path, chat +
  responses transforms). 366 passing; Anthropic path unchanged.

README intentionally not updated (GPT OCR fidelity unverified until tested).
2026-06-19 13:47:44 -04:00
teamchong ef9ddac58a feat(library): caller fidelity + recoverable channel, edge-safe; trim comments
- keepSharp: caller predicate pins exact-match-critical blocks (IDs, hashes,
  paths) as text, overriding the chars/token heuristic. Reported via
  info.keptSharpBlocks. Never forces imaging; throwing predicate = false.
- emitRecoverable: optional recovery map (block -> original text + provenance)
  so a stateful caller can re-inject or re-fetch imaged content. The recover
  half of the fidelity contract; documented mitigation for lossy recall.
- edge/Workers-safe: process.env read is typeof-guarded; @napi-rs/canvas moved
  to devDependencies (atlas is build-time only, runtime PNG encoder is pure-JS).
- comments: trimmed ~1,640 lines of essay/derivation/changelog narration across
  src/ (transform.ts -73%), keeping one-line rationale; deep math points to docs/.
- New exports/types in index.ts + library.ts; +12 tests (keep-sharp, recoverable).

Build clean; 354/354 tests pass.
2026-06-19 13:02:30 -04:00
teamchong 716d9e0965 test: cap corpus L0 test to an 8MB per-file prefix
The real-corpus reflow check read whole transcript files; on a large ~/.claude/projects (multi-hundred-MB files) that blew the 30s timeout and aborted 'pnpm test' (hence prepublishOnly). Read only the first 8MB per file — enough for the per-file sample; a truncated trailing line just fails JSON.parse and is skipped. Corpus test now runs in ~5s.
2026-06-17 10:44:46 -04:00
teamchong 1cb62c1786 feat(dashboard): context-map transparency, flexible model chips, Opus default-off
- "How your context works" panel: per-request token flow (as-text -> real) plus
  the exact-char bucket breakdown of what became images and a gallery of every
  rendered page, opened via a "view" link on each recent-requests row.
- "compress models" chips are the union of a catalog (Fable 5, Opus 4.8/4.7,
  Sonnet 4.6, Haiku 4.5), the PXPIPE_MODELS env scope, and the active scope, so
  any env-enabled model stays toggleable (off <-> on); runtime-only override.
- Opus is OFF by default (Fable-5-only scope); opt-in via env or the chips.

Fixes from the code review:
- contextHistory cap aligned to the recent table (was 30 < 50, so older "view"
  links silently showed the latest request's breakdown); an evicted or
  unrecorded request now shows an explicit "no longer available" message.
- the session headline phrases a net loss as "N% more tokens" instead of
  rendering a garbled "-7% fewer tokens".
2026-06-17 10:01:33 -04:00
teamchong a9bda51210 feat(savings): honest negative accounting; drop the false >=0 clamp claim
Per-turn and per-session savings are the real baseline_eff - actual_eff with no
floor: a net-losing turn (cache_create-heavy image rewrite that costs more than
the text prefix it replaced) now reports the real loss instead of a fabricated
0. Updates baseline.ts and the caller comments that claimed a >=0 clamp the code
never applied -- the negative is intended and covered by tests/sessions.test.ts.
2026-06-17 10:01:33 -04:00
teamchong 8af9829872 feat(render): raise page ceiling to ~1932x1932 and align gate/paging geometry
Fable 5 / Opus 4.8 accept up to 2576px / 4784 visual tokens per image, but a
request with >20 images (pxpipe always sends many) is held to the stricter
<=2000px/side rule, so the real ceiling is ~1932x1932 (1928x1928 = 69x69 = 4761
tokens). MAX_HEIGHT_PX 1568->1932; dense tool/history pages now render at
DENSE_CONTENT_COLS=384 / DENSE_CONTENT_CHARS_PER_IMAGE=92160 (1928x1928 full
page) -- fewer image blocks at the same OCR-validated 5x8 cell. The static slab
is unchanged (313 cols / 1573x1280).

Also fixes a regression the code review caught: the break-even gate,
truncateForBudget, and estimateImageCount predicted against the slab geometry
(313 cols / 159 rows) while the dense renderer emits 384 cols / 240 rows, so
large tool_results were truncated far earlier than the 10-image cap required --
silently dropping output that would have rendered. A per-page char budget is
threaded through the prediction chain (defaults to READABLE so the slab and unit
tests are byte-identical) plus a denseGateGeometry helper, so the gate prices the
same page the renderer actually produces.
2026-06-17 10:01:33 -04:00
teamchong 514a4f0b74 feat(gate): config-driven model scope (PXPIPE_MODELS) + Opus 4.8 re-test
- applicability: isPxpipeSupportedModel reads PXPIPE_MODELS (comma-sep,
  default claude-fable-5 only) and strips bracket/variant tags like [1m];
  Opus becomes opt-in with no code change. Default behavior unchanged.
- FINDINGS: Opus 4.8 re-test - arithmetic 98/100 (-2pp), verbatim 6/15
  (was 93/100, 0/15): improved but still taxed -> default stays Fable-only.
- FINDINGS + eval/glyph-matrix/sweep: root cause of the read tax is per-glyph
  RESOLUTION, not font shape (confusions scale with px and vanish by 10x16).
  Opus needs ~4x glyph area to read reliably; the 1568px ceiling makes that
  break-even, so there is no profitable Opus operating point.
2026-06-16 12:22:55 -04:00
teamchong cb522a70fe feat(dashboard): replace Svelte bundle with AHA stack (htmx + Alpine, server-rendered fragments)
- src/dashboard/fragments.ts: server-side HTML fragment renderers reusing the
  same JSON payloads (HTML and JSON surfaces can't drift)
- node.ts/dashboard.ts: /fragments/<name> routes (header, session-summary,
  recent, latest, sessions, stats, toggle) with htmx polling + Alpine for
  local state; toggle is a POST-returning-fragment htmx swap
- vendored htmx 2.0.4 + Alpine 3.14.9 in src/dashboard/vendor.ts (offline,
  single-file npm package, no CDN)
- deleted Svelte components/stores/bundle build step; dropped svelte/vite
  devDeps; build is now just tsc + esbuild
- tests: +5 fragment contract tests (routing, toggle state, HTML escaping of
  source text, 404) - 339 total
2026-06-09 22:17:53 -04:00
teamchong 528034cd82 feat(proxy): cloudflare-ai-gateway provider mode
- PXPIPE_PROVIDER=cloudflare-ai-gateway + PXPIPE_GATEWAY_BASE_URL derive both
  family routes ({base}/anthropic, {base}/openai) from one configurable base
- PXPIPE_GATEWAY_HEADERS injects auth headers (JSON or k=v;k2=v2) on every
  upstream request incl. count_tokens probes
- OpenAI paths drop /v1 to match gateway shape; Anthropic paths unchanged;
  unknown paths pass through on the family route; streaming untouched
- no hardcoded hostnames; tests use fake gateway URLs/tokens (14 new)
2026-06-09 19:21:01 -04:00
teamchong 99bb931014 chore: rename all pixelpipe references to pxpipe
- code, tests, docs, README, wrangler.toml, help text
- data dir ~/.pixelpipe -> ~/.pxpipe (PXPIPE_LOG env var)
- rebuilt dist; 323/323 tests
2026-06-09 19:04:09 -04:00
teamchong 3d0f9ef08a scope: narrow model gate to fable-5, drop opus; switch dense render to bare 5x8 cell
- gate regex: fable-5 only (opus 4.7/4.8 read tax gone on fable: 100/100 novel-arithmetic, identical image billing)
- remove dead opus-4.6 cpt fork in transform.ts
- DENSE_RENDER_STYLE: 7x10 padded -> bare 5x8 cell (~42% fewer image tokens/page, recall flat in A/B)
- dashboard: compression defaults ON
- README/FINDINGS/TRANSFORM_INFO updated with 2026-06-09 measurements
2026-06-09 18:33:34 -04:00
Steven Chong 613ca3bc71 feat(proxy): support gpt 5.5 chat completions 2026-06-05 00:35:26 -04:00
teamchong 52f6b4a0e6 docs: apply FINDINGS reframe + ref fixes (follow-up to 2c38f30)
The prior commit captured only the rename (a stale pathspec aborted the
content staging). This applies the actual edits: retitle to FINDINGS,
'## Verdict' + superseded TL;DR, and the two cross-reference updates.
2026-05-29 22:47:12 -04:00
teamchong 378dd7f6c4 feat(applicability)!: scope to opus 4.7+ and enforce model gate at proxy
Widen the model gate to /^claude-opus-4-(?:[7-9]|[1-9]\d)(?:-|$)/ (Opus 4.7
and newer; previously 4.6/4.7). Wire isPixelpipeSupportedModel into the proxy
boundary in proxy.ts — the proxy previously gated on economics only and
ignored the model, so it compressed 4.8 traffic despite the docs claiming a
4.6/4.7 scope. Unsupported models now pass through with reason
'unsupported_model'.

Update public-api tests to the 4.7+ spec (320/320 pass).

Correct the docs to live measurement: reverse the POSTMORTEM "dead" verdict
(pixelpipe is a lossy gist-compressor saving ~68% on real dense Claude Code
traffic; the verbatim 0/15 needle finding stands as a caveat, not the
verdict), rewrite README Status/Limitations, and add a correction pointer to
the eval README. Original POSTMORTEM body preserved below the correction.

BREAKING CHANGE: Opus 4.6 and older are no longer supported.
Verbatim-risk guard (skip imaging unique IDs/hashes/exact values) still pending.
2026-05-29 22:34:55 -04:00
Steven Chong d3992c6bad feat(applicability): enable pixelpipe for opus 4.6 2026-05-26 15:35:12 -04:00
Steven Chong 36ac1faa96 fix(render): reduce dense page size 2026-05-26 13:15:54 -04:00
Steven Chong 63a7810b52 fix(history): preserve git status output as text 2026-05-26 11:05:59 -04:00
Steven Chong 09fb339b39 fix(render): use readable profile for dense images 2026-05-25 15:37:14 -04:00
Steven Chong 0e95b81728 fix(render): page dense tool and history images 2026-05-25 15:33:00 -04:00
Steven Chong d089f0d32b fix(transform): preserve system cache_control on slab image 2026-05-25 14:54:59 -04:00