Claude afaf50a972 Add queue-level concurrency limit to provider streaming ...

Adds max_concurrency cap to GlobalRateLimiter using asyncio.Semaphore.
A request now waits for a concurrency slot before the sliding window rate
limit check, so at most N streams are open to the provider simultaneously,
even when the rate window would allow more.

Changes:
- providers/rate_limit.py: max_concurrency param, _concurrency_sem, concurrency_slot() asynccontextmanager
- providers/openai_compat.py: pass max_concurrency to limiter; wrap execute_with_retry + stream iteration in concurrency_slot()
- providers/base.py: max_concurrency field on ProviderConfig
- config/settings.py: provider_max_concurrency setting (PROVIDER_MAX_CONCURRENCY env var, default None = unlimited)
- api/dependencies.py: pass provider_max_concurrency into all three provider ProviderConfig instantiations
- .env.example: document PROVIDER_MAX_CONCURRENCY (commented out)
- tests/providers/test_provider_rate_limit.py: 5 new tests covering concurrency limit enforcement, slot release on exception, noop when unconfigured
- tests/api/test_dependencies.py: add provider_max_concurrency=None to mock settings helper

https://claude.ai/code/session_014mrF1WMNgmNjtPBuoQHsbg

2026-02-19 14:23:21 +00:00

..

api

Add queue-level concurrency limit to provider streaming

2026-02-19 14:23:21 +00:00

cli

new linter rules and fixes

2026-02-18 04:13:41 -08:00

config

Remove auto support for whisper_device; only cpu and cuda allowed

2026-02-18 13:38:59 +00:00

messaging

Added fix for large replies being truncated entirely leaving no response text

2026-02-18 17:39:38 -08:00

providers

Add queue-level concurrency limit to provider streaming

2026-02-19 14:23:21 +00:00

conftest.py

Add message_thread_id support across messaging components

2026-02-18 16:10:57 -08:00