LiteLLM Proxy Server (https://docs.litellm.ai/docs/simple_proxy) exposes an
OpenAI-compatible /v1/embeddings endpoint and fans out to 100+ underlying
providers (OpenAI, Anthropic, Cohere, Voyage, HuggingFace, Bedrock, Vertex AI,
Ollama, ...). Mirroring the lmstudio strategy (PR #42 + commit bb141a0) but
with three meaningful differences that justify a dedicated provider rather
than a flag on provider-openai:
- Authentication is mandatory. LiteLLM gates /v1/models with the master key
or a virtual key, unlike LM Studio (no auth by default) and OpenAI (cloud
key). LITELLM_API_KEY is checked at config-load time; the provider also
duck-types 401/403 in ensureReady/healthCheck via err.status to surface a
distinct "auth rejected" message vs. "proxy unreachable".
- Model aliases come from the proxy's config.yaml, so EMBEDDING_MODEL and
EMBEDDING_DIMENSIONS have no sensible defaults. Fail-fast in
loadEmbeddingConfig with provider-specific error messages pointing at
litellm_params.model in the proxy config and at the underlying alias's
output dim.
- Whether dimensions can be forwarded depends on the underlying provider:
Matryoshka-aware models (text-embedding-3-*, voyage-3) accept it,
non-Matryoshka backends (BGE, nomic, Cohere v3) reject. Made opt-in via
LITELLM_SEND_DIMENSIONS=true rather than hardcoded like provider-openai
does for text-embedding-3-*, since LiteLLM aliases are user-defined.
Encoding-format=float fix from bb141a0 ports verbatim — the OpenAI SDK 6.x
base64-decode path corrupts any backend that returns plain JSON float arrays
(many LiteLLM aliases do, including Ollama-routed and tei-wrapped ones).
Files:
- src/services/provider-litellm.ts: new LiteLLMEmbeddingProvider with the
same OpenAI-SDK + custom baseURL pattern. Default baseURL
http://localhost:4000/v1 (LiteLLM's default port, /v1 prefix required).
Batch size 256 — between OpenAI's 512 and LM Studio's 64, since the
practical ceiling depends on whichever provider the alias resolves to.
ensureReady distinguishes proxy-unreachable / auth-rejected /
alias-not-registered. Lists up to 10 currently-registered models in the
alias-missing error so the operator can sanity-check their config.yaml
without leaving the log.
- src/services/embedding-config.ts: extends EmbeddingProvider union with
"litellm", adds litellmUrl to EmbeddingConfig, fail-fast validation for
LITELLM_API_KEY + EMBEDDING_MODEL + EMBEDDING_DIMENSIONS (key first so a
virtual-key user fixes the easy problem before touching the proxy
config), updates Invalid EMBEDDING_PROVIDER message and hasApiKey log
expression.
- src/services/embedding-provider.ts: factory case for litellm with dynamic
import to avoid loading the OpenAI SDK at startup for non-litellm users.
- README.md: dedicated LiteLLM section, MCP host config example, env-var
table entries for EMBEDDING_PROVIDER / EMBEDDING_MODEL /
EMBEDDING_DIMENSIONS / EMBEDDING_CONTEXT_LENGTH (clarifying which require
manual values for litellm), new LiteLLM Configuration table.
- tests/unit/embedding-config.test.ts: 9 new cases (model + dim + key
required, error-ordering, URL default + override, dimensions parsing,
EMBEDDING_CONTEXT_LENGTH override for unknown aliases, auto-detection
when alias matches a known model name) plus updated "full external
config" expected object and updated invalid-provider error message.
- tests/unit/embedding-provider.test.ts: factory test for litellm, plus 4
cases against a deliberately-closed port (config rejects construction
without API_KEY, ensureReady unreachable error format, healthCheck
short-circuits on missing key without a network call, healthCheck
reaches "Not reachable" path without throwing).
Backward compatible. The litellm provider is opt-in via
EMBEDDING_PROVIDER=litellm. Existing ollama, openai, google, and lmstudio
paths are untouched.
Verified: 64/64 unit tests pass on the touched suites; biome lint clean;
tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
LM Studio's Local Server speaks the OpenAI-compatible /v1/embeddings
protocol, so users running it as their model host (chat plus embedding in
one desktop app, GGUF model management) had no clean integration path.
Changes:
- src/services/provider-lmstudio.ts: new LMStudioEmbeddingProvider wrapping
the OpenAI SDK with a custom baseURL (default http://localhost:1234/v1).
Sends a placeholder API key to satisfy the OpenAI SDK while LM Studio's
Local Server runs without auth by default. Skips the dimensions parameter
because LM Studio models have no Matryoshka projection. Forces
encoding_format=float to defeat the OpenAI SDK 6.x base64 default, which
would otherwise mangle LM Studio's plain-array responses into 1024 zeros.
- src/services/embedding-config.ts: extends the EmbeddingProvider union,
reads LMSTUDIO_URL and LMSTUDIO_API_KEY, fail-fast validation when
EMBEDDING_PROVIDER=lmstudio without EMBEDDING_MODEL or EMBEDDING_DIMENSIONS.
- src/services/embedding-provider.ts: factory case for lmstudio with a
dynamic import to avoid loading the OpenAI SDK at startup for ollama users.
- ensureReady distinguishes "LM Studio unreachable" from "reachable but
embedding model not loaded" so the operator knows whether to start the
Local Server or load the configured model.
- src/services/qdrant.ts: minor refactor to extract the hybrid-search query
payload to a local const for readability.
- README.md: dedicated LM Studio section, MCP host config example, env-var
table entries.
- tests/unit/embedding-config.test.ts: 8 new cases (required-env validation,
URL default and override, optional API key, context-length override).
- tests/unit/embedding-provider.test.ts: 3 new cases (factory wiring,
ensureReady error format against a closed port, healthCheck unreachable
output).
Backward compatible. The lmstudio provider is opt-in via
EMBEDDING_PROVIDER=lmstudio. Existing ollama, openai, and google paths are
untouched.
Tomasz Szuster