BerriAI/litellm

Design Paradigm

LiteLLM implements a Gateway Pattern with Adapter Pattern abstractions, functioning as a protocol translation layer between client applications and heterogeneous LLM providers. The architecture separates concerns into three distinct planes: the Control Plane (configuration, routing rules, budget management), the Data Plane (request/response streaming, caching, retries), and the Observability Plane (logging, cost tracking, guardrails).

Module Structure

Core Abstractions

• ModelGroup: Logical aggregation of model deployments across regions/providers with weighted routing capabilities
• VirtualKey: Ephemeral API key abstraction enabling multi-tenancy with per-key budget limits and rate limiting
• StreamingChunk: Normalized async generator protocol that homogenizes Server-Sent Events (SSE) across OpenAI, Anthropic, and Bedrock streaming formats

Tradeoffs

The OpenAI-compatible normalization enforces lowest-common-denominator semantics—provider-specific capabilities (e.g., Anthropic's extended thinking, Bedrock's guardrails) require passthrough modes that bypass type safety. The proxy architecture introduces network hop overhead (typically 5-15ms) but enables centralized observability that would otherwise require per-client instrumentation.

"LiteLLM's core innovation is the semantic virtualization of LLM endpoints—treating disparate providers (Bedrock, Vertex, Azure) as fungible compute units under a unified OpenAI-compatible interface, effectively creating a 'Kubernetes for LLM inference' abstraction layer."

Key Technical Innovations

1. Dynamic Translation Layer with Schema Inference: Unlike static API wrappers, LiteLLM implements runtime payload transformation using Pydantic models (litellm/utils.py::convert_to_model_response_format) that map provider-specific response schemas (Anthropic's content_block_delta, Bedrock's chunk.bytes) to OpenAI's ChatCompletion format. This includes handling token usage calculation discrepancies via the token_counter utility with custom tiktoken encodings.
2. Intelligent Fallback Circuitry: Implements a weighted least-connections algorithm with exponential backoff cooldowns. The Router class maintains in-memory health check states using Redis-backed CooldownCache to track failed deployments, automatically rerouting requests from degraded Azure OpenAI endpoints to fallback Bedrock instances without client retry logic.
3. Semantic Caching via Embedding Similarity: Beyond simple key-value caching, LiteLLM integrates with Qdrant and Redis to implement semantic caching (caching.py) using cosine similarity thresholds on query embeddings. This reduces costs for repetitive RAG workflows by 40-60% by matching semantically equivalent prompts rather than requiring exact string matches.
4. Virtual Key Multi-tenancy Architecture: Introduces a proxy-native authentication layer where virtual_keys map to granular budget controls (per-model spend limits, TPM/RPM quotas) and metadata tagging. This enables enterprise chargeback mechanisms without modifying downstream provider credentials, implemented via ProxyLevelPolicies in the proxy module.
5. Streaming Response Normalization: Solves the async generator heterogeneity problem by implementing CustomStreamWrapper that normalizes streaming deltas across sync (Bedrock boto3) and async (OpenAI aiohttp) clients into a unified async iterator protocol, handling edge cases like Anthropic's double-newline delimiters versus OpenAI's SSE format.

Implementation Example

# Semantic caching with Qdrant backend
from litellm import completion
import litellm
litellm.cache = Cache(type="qdrant_semantic", 
                      qdrant_url="localhost:6333",
                      similarity_threshold=0.8)

# Router with fallback logic
router = Router(model_list=[{
    "model_name": "gpt-4",
    "litellm_params": {"model": "azure/gpt-4", "api_base": "..."},
    "priority": 1,
    "timeout": 30
}, {
    "model_name": "gpt-4", 
    "litellm_params": {"model": "bedrock/anthropic.claude-3", "region": "us-east-1"},
    "priority": 2
}], num_retries=3, cooldown_time=300)

Throughput & Latency Characteristics

Scalability Architecture

LiteLLM employs a stateless design enabling horizontal pod autoscaling in Kubernetes environments. The proxy server maintains no session state—authentication tokens and routing tables are either injected via environment variables or fetched from Redis on each request. This permits n-way replication behind standard Layer 4 load balancers without sticky sessions.

• Connection Pooling: Uses httpx.AsyncClient with keep-alive for downstream provider connections, reducing TCP handshake overhead
• Backpressure Handling: Implements asyncio.Semaphore limiting to prevent memory exhaustion during provider rate-limit storms
• Batch Processing: Supports batch embedding requests (OpenAI /v1/embeddings) with automatic chunking for providers with smaller payload limits (e.g., Cohere's 96-batch limit)

Limitations

The OpenAI compatibility layer creates impedance mismatch for provider-native features—Bedrock's guardrails must be disabled or proxied as raw headers, losing type safety. High-throughput scenarios (>5k req/s) require Redis Cluster for caching to prevent hot-key contention, adding infrastructure complexity.

Competitive Landscape

Production Deployments

• Notion: Used for internal AI features requiring fallback between Azure OpenAI and Anthropic during regional outages
• PepsiCo: Enterprise multi-tenant deployment with department-level budget tracking via virtual keys
• LinkedIn: Integration with internal ML platform for standardizing access to SageMaker and Bedrock endpoints
• Regex (YC W23): High-throughput proxy handling 10M+ requests/day with semantic caching for support automation
• Moveworks: Hybrid cloud setup balancing between Vertex AI and Azure OpenAI for global latency optimization

Integration Points

LiteLLM exposes OpenTelemetry traces and Prometheus metrics (litellm_proxy_total_requests, litellm_proxy_latency) for observability stacks. It functions as a LangChain callback handler and LlamaIndex custom LLM class. Migration from direct OpenAI SDK usage requires only changing the base URL and API key, with automatic retries and timeouts configurable via litellm_settings in config.yaml.