From 413ce7aa62d90758e237b93108bece254a977964 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 15 Feb 2026 05:07:17 +0000 Subject: [PATCH] docs: add resource usage and efficiency optimization plan Co-authored-by: Ali Khokhar --- docs/RESOURCE_OPTIMIZATION_PLAN.md | 242 +++++++++++++++++++++++++++++ 1 file changed, 242 insertions(+) create mode 100644 docs/RESOURCE_OPTIMIZATION_PLAN.md diff --git a/docs/RESOURCE_OPTIMIZATION_PLAN.md b/docs/RESOURCE_OPTIMIZATION_PLAN.md new file mode 100644 index 00000000..488787c4 --- /dev/null +++ b/docs/RESOURCE_OPTIMIZATION_PLAN.md @@ -0,0 +1,242 @@ +# Resource Usage & Efficiency Optimization Plan + +This document outlines opportunities to improve efficiency and resource usage across the Free Claude Code proxy. Each section identifies the current behavior, optimization opportunity, and suggested approach. + +--- + +## 1. API Request Path Optimizations + +### 1.1 Skip Token Counting for Optimized Responses +**Location:** `api/routes.py` (create_message) + +**Current:** `get_token_count()` is only called when the request proceeds to streaming (after `try_optimizations` returns `None`). ✅ Already optimized. + +**Status:** No change needed. + +--- + +### 1.2 Detection Short-Circuit & Shared Text Extraction +**Location:** `api/detection.py`, `api/optimization_handlers.py` + +**Current:** Each optimization handler may call `extract_text_from_content()` independently. For a single request, detection functions can extract text from the same content multiple times (e.g., `is_quota_check_request`, `is_prefix_detection_request` both inspect the first user message). + +**Opportunity:** +- Add a **single-pass detection** that extracts text once and runs all cheap checks (quota, title, suggestion, prefix, filepath) against it. +- Reorder handlers by **cheapest-first** (already done: quota → prefix → title → suggestion → filepath). +- Consider a **unified detection** function that returns a `(optimization_type, extracted_data)` tuple to avoid redundant parsing. + +**Effort:** Low–Medium | **Impact:** Medium (reduces CPU per request on hot path) + +--- + +### 1.3 Lazy Tiktoken Encoder +**Location:** `api/request_utils.py` (line 14) + +**Current:** `ENCODER = tiktoken.get_encoding("cl100k_base")` is loaded at module import time. Tiktoken loading involves reading/parsing encoding files. + +**Opportunity:** Use lazy initialization—only load the encoder on first `get_token_count()` call. For optimization-heavy workloads (quota, prefix, title, etc.), many requests never need token counting. + +**Effort:** Low | **Impact:** Low–Medium (faster startup, less memory if optimizations dominate) + +--- + +### 1.4 Fix Syntax Error in request_utils.py ✅ DONE +**Location:** `api/request_utils.py` line 89 + +**Current:** `except TypeError, ValueError:` — invalid Python 3 syntax. + +**Fix:** `except (TypeError, ValueError):` — applied. + +**Effort:** Trivial | **Impact:** Bug fix (code may not run correctly in edge cases) + +--- + +## 2. Session Store & Persistence + +### 2.1 Async-Friendly Session Store +**Location:** `messaging/session.py` + +**Current:** Uses `threading.Lock` and synchronous file I/O (`open`, `json.load`, `json.dump`). All tree/session updates trigger debounced saves that block a thread. + +**Opportunity:** +- Use `asyncio.Lock` and `aiofiles` (or `run_in_executor`) for non-blocking I/O. +- Keeps the event loop responsive during heavy save operations. +- Consider batching multiple dirty writes into a single flush. + +**Effort:** Medium | **Impact:** Medium (better responsiveness under load) + +--- + +### 2.2 Reduce Session Store Write Frequency +**Location:** `messaging/session.py` — `_schedule_save`, `record_message_id`, `save_tree` + +**Current:** Debounce is 0.5s. Every `record_message_id` and `save_tree` can schedule a save. Under high message volume, this can still mean frequent disk writes. + +**Opportunity:** +- Increase debounce to 1–2 seconds for tree/session updates. +- Add a max write interval (e.g., cap at 1 write per 2s even if dirty). +- For `record_message_id`, consider batching or skipping persistence for high-frequency status updates (if acceptable for `/clear` semantics). + +**Effort:** Low | **Impact:** Low–Medium (fewer disk I/O operations) + +--- + +## 3. Rate Limiters + +### 3.1 Provider Rate Limiter — Sleep Outside Lock +**Location:** `providers/rate_limit.py` — `_acquire_proactive_slot` + +**Current:** Already sleeps outside the lock. ✅ Good. + +**Status:** No change needed. + +--- + +### 3.2 Messaging Rate Limiter — Config from Settings +**Location:** `messaging/limiter.py` (lines 99–100) + +**Current:** Reads `MESSAGING_RATE_LIMIT` and `MESSAGING_RATE_WINDOW` directly from `os.getenv()`, bypassing `config.settings`. + +**Opportunity:** Inject these from `Settings` for consistency and testability. Reduces env reads and aligns with the rest of the app. + +**Effort:** Low | **Impact:** Low (consistency, testability) + +--- + +## 4. Messaging & Handler + +### 4.1 Defer Message ID Recording +**Location:** `messaging/handler.py` — `handle_message` + +**Current:** `record_message_id` is called for every incoming message, including commands and status echoes, before early returns. + +**Opportunity:** Move `record_message_id` to after command/status filtering, or use a fire-and-forget pattern so the main path isn’t blocked by session store updates. The store already uses debounced saves, but the lock is still acquired synchronously. + +**Effort:** Low | **Impact:** Low (slightly faster handler path) + +--- + +### 4.2 Transcript Event Type Lookup +**Location:** `messaging/handler.py` — `TRANSCRIPT_EVENT_TYPES` + +**Current:** Uses `frozenset` for O(1) membership. ✅ Good. + +**Status:** No change needed. + +--- + +## 5. Provider & Streaming + +### 5.1 NIM Client — Reuse Parsers +**Location:** `providers/nvidia_nim/client.py` — `stream_response` + +**Current:** Creates `ThinkTagParser()` and `HeuristicToolParser()` per request. These are lightweight. + +**Opportunity:** If profiling shows allocation overhead, consider object pooling. Likely not necessary unless under extreme request rates. + +**Effort:** Low | **Impact:** Low + +--- + +### 5.2 SSE Builder Token Counting +**Location:** `providers/nvidia_nim/utils/sse_builder.py` + +**Current:** Uses `ENCODER.encode()` for usage reporting. Shares tiktoken with `request_utils`. + +**Opportunity:** Same lazy encoder as in 1.3. Ensure a single shared encoder instance. + +**Effort:** Low | **Impact:** Low + +--- + +## 6. CLI & Process Management + +### 6.1 Idle Session Cleanup +**Location:** `cli/manager.py` — `_cleanup_idle_sessions_unlocked` + +**Current:** Cleanup runs when hitting `max_sessions`. Sessions are removed when CLI exits. + +**Opportunity:** Add a background task that periodically prunes idle sessions (e.g., no activity for N minutes) to free subprocesses and file descriptors earlier. + +**Effort:** Medium | **Impact:** Medium (better resource reclaim under bursty usage) + +--- + +### 6.2 Process Registry — Best-Effort Cleanup +**Location:** `cli/process_registry.py`, `server.py` + +**Current:** `kill_all_best_effort()` runs in `finally` on shutdown. Good safety net. + +**Status:** No change needed. + +--- + +## 7. Application Lifespan + +### 7.1 Lazy Messaging Initialization +**Location:** `api/app.py` — `lifespan` + +**Current:** Messaging platform, session store, CLI manager, and tree queue are all initialized at startup, even when no Telegram messages are expected (e.g., CLI-only usage). + +**Opportunity:** Defer messaging initialization until the first request that needs it (e.g., first `/stop` or first webhook hit). Reduces startup time and memory when running in CLI-only mode. + +**Effort:** Medium | **Impact:** Medium (faster startup for CLI-only) + +--- + +### 7.2 Tree Restoration +**Location:** `api/app.py` — `lifespan` (lines 101–121) + +**Current:** Loads all trees and node mappings at startup, then runs `cleanup_stale_nodes()` and syncs back. + +**Opportunity:** Run restoration in a background task so the server can accept requests sooner. Ensure handlers tolerate `tree_queue` not being fully restored for a short window. + +**Effort:** Medium | **Impact:** Low–Medium (faster startup) + +--- + +## 8. Configuration & Imports + +### 8.1 Settings Loaded Twice at Startup +**Location:** `api/app.py` (lines 19–20, 46) + +**Current:** `get_settings()` is called at module level and again in `lifespan`. `get_settings` is `@lru_cache()` so it’s cheap, but the pattern is redundant. + +**Opportunity:** Use a single `settings = get_settings()` at module level and pass it where needed, or rely solely on `Depends(get_settings)` in lifespan. + +**Effort:** Low | **Impact:** Low (cleaner code) + +--- + +### 8.2 PTB Environment Variable +**Location:** `api/app.py` (line 7) + +**Current:** `os.environ["PTB_TIMEDELTA"] = "1"` is set at import time. Required for python-telegram-bot. + +**Status:** No change needed. + +--- + +## 9. Summary: Recommended Priorities + +| Priority | Item | Effort | Impact | +|----------|------|--------|--------| +| P0 | Fix `except TypeError, ValueError` syntax | Trivial | Bug fix | +| P1 | Lazy tiktoken encoder | Low | Startup/memory | +| P1 | Single-pass detection / shared text extraction | Low–Med | CPU per request | +| P2 | Messaging limiter config from Settings | Low | Consistency | +| P2 | Session store debounce tuning | Low | Disk I/O | +| P3 | Async session store I/O | Medium | Responsiveness | +| P3 | Lazy messaging initialization | Medium | Startup (CLI-only) | +| P3 | Idle session cleanup background task | Medium | Resource reclaim | +| P4 | Tree restoration in background | Medium | Startup | +| P4 | Defer message ID recording | Low | Handler latency | + +--- + +## 10. Implementation Order Suggestion + +1. **Immediate:** Fix syntax error in `request_utils.py`. +2. **Quick wins:** Lazy tiktoken, single-pass detection, messaging limiter from Settings. +3. **Next:** Session store debounce, defer message ID recording. +4. **Larger refactors:** Async session store, lazy messaging init, idle session cleanup, background tree restoration.