mirror of
https://github.com/giancarloerra/socraticode.git
synced 2026-07-03 14:05:21 +02:00
Tomasz Szuster
1708510c16
LiteLLM Proxy Server (https://docs.litellm.ai/docs/simple_proxy) exposes an OpenAI-compatible /v1/embeddings endpoint and fans out to 100+ underlying providers (OpenAI, Anthropic, Cohere, Voyage, HuggingFace, Bedrock, Vertex AI, Ollama, ...). Mirroring the lmstudio strategy (PR #42 + commitbb141a0) but with three meaningful differences that justify a dedicated provider rather than a flag on provider-openai: - Authentication is mandatory. LiteLLM gates /v1/models with the master key or a virtual key, unlike LM Studio (no auth by default) and OpenAI (cloud key). LITELLM_API_KEY is checked at config-load time; the provider also duck-types 401/403 in ensureReady/healthCheck via err.status to surface a distinct "auth rejected" message vs. "proxy unreachable". - Model aliases come from the proxy's config.yaml, so EMBEDDING_MODEL and EMBEDDING_DIMENSIONS have no sensible defaults. Fail-fast in loadEmbeddingConfig with provider-specific error messages pointing at litellm_params.model in the proxy config and at the underlying alias's output dim. - Whether dimensions can be forwarded depends on the underlying provider: Matryoshka-aware models (text-embedding-3-*, voyage-3) accept it, non-Matryoshka backends (BGE, nomic, Cohere v3) reject. Made opt-in via LITELLM_SEND_DIMENSIONS=true rather than hardcoded like provider-openai does for text-embedding-3-*, since LiteLLM aliases are user-defined. Encoding-format=float fix frombb141a0ports verbatim — the OpenAI SDK 6.x base64-decode path corrupts any backend that returns plain JSON float arrays (many LiteLLM aliases do, including Ollama-routed and tei-wrapped ones). Files: - src/services/provider-litellm.ts: new LiteLLMEmbeddingProvider with the same OpenAI-SDK + custom baseURL pattern. Default baseURL http://localhost:4000/v1 (LiteLLM's default port, /v1 prefix required). Batch size 256 — between OpenAI's 512 and LM Studio's 64, since the practical ceiling depends on whichever provider the alias resolves to. ensureReady distinguishes proxy-unreachable / auth-rejected / alias-not-registered. Lists up to 10 currently-registered models in the alias-missing error so the operator can sanity-check their config.yaml without leaving the log. - src/services/embedding-config.ts: extends EmbeddingProvider union with "litellm", adds litellmUrl to EmbeddingConfig, fail-fast validation for LITELLM_API_KEY + EMBEDDING_MODEL + EMBEDDING_DIMENSIONS (key first so a virtual-key user fixes the easy problem before touching the proxy config), updates Invalid EMBEDDING_PROVIDER message and hasApiKey log expression. - src/services/embedding-provider.ts: factory case for litellm with dynamic import to avoid loading the OpenAI SDK at startup for non-litellm users. - README.md: dedicated LiteLLM section, MCP host config example, env-var table entries for EMBEDDING_PROVIDER / EMBEDDING_MODEL / EMBEDDING_DIMENSIONS / EMBEDDING_CONTEXT_LENGTH (clarifying which require manual values for litellm), new LiteLLM Configuration table. - tests/unit/embedding-config.test.ts: 9 new cases (model + dim + key required, error-ordering, URL default + override, dimensions parsing, EMBEDDING_CONTEXT_LENGTH override for unknown aliases, auto-detection when alias matches a known model name) plus updated "full external config" expected object and updated invalid-provider error message. - tests/unit/embedding-provider.test.ts: factory test for litellm, plus 4 cases against a deliberately-closed port (config rejects construction without API_KEY, ensureReady unreachable error format, healthCheck short-circuits on missing key without a network call, healthCheck reaches "Not reachable" path without throwing). Backward compatible. The litellm provider is opt-in via EMBEDDING_PROVIDER=litellm. Existing ollama, openai, google, and lmstudio paths are untouched. Verified: 64/64 unit tests pass on the touched suites; biome lint clean; tsc --noEmit clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>