LiteLLM is an open source AI Gateway and Python SDK maintained by BerriAI. It provides OpenAI compatible interfaces, proxy gateways, virtual keys, budget control, load balancing, rollback, logs and MCP Gateway capabilities for 100+ LLM Providers. It is suitable for managing multiple model calls into a unified infrastructure.

LiteLLM

Core parameters and statistics

LiteLLM is officially positioned as an open source AI Gateway: it calls 100+ LLM Providers in the OpenAI format, and centralizes authentication, cost, routing, load balancing, logs and management planes in a gateway layer through Proxy Server. It is not responsible for training models, nor is it a chat product for end users; its core value is to allow engineering teams to manage multiple model calls from "SDK code scattered in various business services" into a unified infrastructure.

Projects	Public Information
Product form	Python SDK, Proxy Server / AI Gateway, enterprise hosting and self-hosting
Official repository	BerriAI/litellm
Public version	v1.89.2 (2026-06-18, GitHub Releases/PyPI)
Provider coverage	Official README describes 100+ LLMs
API compatible	OpenAI format / OpenAI-compatible errors
Gateway capabilities	Virtual keys, spend tracking, guardrails, load balancing, admin dashboard
Supported endpoints	Chat Completions, Responses, Embeddings, Images, Audio, Batches, Rerank, Messages and more
Community size	About 50,971 stars, 9,019 forks (GitHub API, 2026-06-21)
Coding language	Python-based
Open source license	GitHub API does not give the SPDX logo; the warehouse has a LICENSE file, and the specific terms are subject to the real-time files of the warehouse

Positioning Boundary: LiteLLM is more like a "model access control layer", not a RAG framework, Agent orchestration framework or model training platform. It is suitable for solving multi-provider access, key isolation, budget governance and observability issues; if you only call one model and have no team budget and audit requirements, the complexity of using the official SDK directly is lower.

User and market recognition

LiteLLM's market recognition comes primarily from developer community and infrastructure adoption signals. GitHub API shows that BerriAI/litellm has about 50,971 stars and 9,019 forks, indicating that it has entered the mainstream LLMOps and AI Gateway tool stack discussions. The warehouse README also displays OSS adopters, including logos or names such as Stripe, Google ADK, Greptile, OpenHands, Netflix, OpenAI Agents SDK, etc. This information shows that its interface compatibility and gateway governance capabilities have been paid attention to by multiple engineering ecosystems, but the specific deployment scale, number of contract customers, and revenue have not been disclosed.

Community signal: More than 50K stars means strong external verification for infrastructure projects, especially Provider adaptation projects that need to continuously deal with API differences, error types, authentication methods and model naming changes. A high fork number also indicates that the team has more needs for self-hosting, secondary development, and internal branches of the enterprise.

Prerequisites for Adoption: The value of LiteLLM increases with the number of model vendors, number of business teams, and budget governance needs. A small team with a single application and a single provider may not feel the benefits of the gateway layer; organizations with multiple teams, multiple keys, multi-cloud models and cost aggregation requirements are more likely to reap the benefits.

Cost advantage

LiteLLM does not directly reduce the unit price of the underlying model. Its cost advantage comes from "unified entrance + budget and routing control". When multiple teams use OpenAI, Anthropic, Bedrock, Vertex AI, Azure OpenAI, or local inference services at the same time, the gateway can put call records, virtual keys, team budgets, fallback strategies, and cost aggregation on the same layer to reduce duplicate access and out-of-control consumption.

C side/individual: Individual developers can use open source SDK or local Proxy. The software itself can be tried for free. The main cost comes from the underlying LLM API called, server resources and maintenance time. If it is just a local script call, the value of LiteLLM lies in unifying Provider writing and exception handling.

API/Developer: The explicit cost for the development team is still the model API fee, and the implicit cost is multi-Provider SDK maintenance, error handling, log access and key management. LiteLLM reduces the amount of repetitive engineering through OpenAI compatible format Router, callbacks and virtual keys, and is suitable for integrating models into public services.

Enterprise/Private: The enterprise version involves terms such as SSO, SCIM, OIDC, RBAC, audit logs, enterprise support, hosted Proxy or self-hosted deployment. The public page is mainly for contacting sales. The final price SLA and deployment responsibility boundary are subject to the official real-time quotation and contract.

Cost Hierarchy	Cost Structure	The Role of LiteLLM
Open source for local use	Underlying LLM API, server, operation and maintenance	Unified Provider interface and exception handling
Team gateway	Proxy running resources, database, log system, model calling	Budget, virtual key, team collection, routing
Enterprise procurement	Enterprise authorization, SSO/SCIM/OIDC support, auditing, hosting services	Large-scale governance and compliance capabilities

Main functions

The function of LiteLLM does not revolve around the model capability itself, but around "call entry management".

• OpenAI Compatible Unified Interface: Business code can call 100+ Providers in OpenAI format, reducing migration costs caused by differences in vendor SDKs.
• Proxy Server / AI Gateway: Centralize model calls into one service entrance to facilitate unified authentication, logs, budgets and team policies.
• Virtual Keys: Use the LiteLLM management plane to issue virtual keys to avoid directly distributing the upstream Provider key to the business team.
• Spend Tracking and Budget Control: Track consumption by key, user, team or project, suitable for cost aggregation and quota management.
• Router, Retry and Fallback: Configure routing, retry and fallback between different models or deployments to reduce the impact of single supplier failure on the business.
• Load Balancing: Allocate requests across multiple deployments, suitable for Azure/OpenAI multi-deployment or multi-region scenarios.
• Guardrails and log callbacks: Connect with third-party observation systems through callbacks to integrate security policies, auditing and experimental data into existing platforms.
• Admin Dashboard: Provides a visual management entrance for team keys, budget providers and usage records.
• MCP Gateway: Official release notes have shown a team-level MCP Tools control visual showing that its gateway boundary extends from the LLM API to Agent tool access governance.

The focus of acceptance of these functions is not "whether a request can be issued", but whether high concurrency, failure rollback, budget capping, log integrity and permission isolation can run stably in real business links.

Model and version evolution

The evolution of LiteLLM can be seen as three main lines: SDK unified call Proxy gateway management, enterprise and Agent tool management.

SDK and unified calling stage

• 2023-07-27: GitHub repository created, the project uses Python SDK to unify the calling format of different LLM Providers. The core value of this stage is "write less vendor adaptation code".

Proxy Server and AI Gateway stage

• 2024-2025: README and documentation gradually position LiteLLM as an AI Gateway, with capabilities extended to virtual keys, spend tracking, load balancing, guardrails, and admin dashboard. It goes from a library to a deployable model access control layer.

Enterprise Governance and MCP Gateway Stage

• v1.78.0 (2025-09-12): Official release notes show the team-level tool control visual of MCP Gateway, indicating that LiteLLM begins to integrate MCP tools, vector stores, member permissions and logging settings into the same governance interface.
• v1.89.2 (2026-06-18): The current public version of GitHub Releases and PyPI, showing that the project is still being released frequently. Production environments should pin stable versions, use the official stable image strategy, and verify providers, log callbacks, and budget rules in staging.

Technical advantages

Compatibility layer mechanism: LiteLLM converts the authentication, model naming, request parameters, response structure and exception types of different Providers into a unified format. The effect is that business code is more likely to remain stable, suitable for teams that need to switch between OpenAI, Anthropic, Gemini, Bedrock, Vertex AI, Azure OpenAI and native models.

Gateway Centralized Governance: Proxy Server separates model calls from application code and handles keys, budgets, teams, logs, and routing in a unified manner. The effect is that platform teams can govern all AI applications horizontally, rather than tracking model consumption on a service-by-service basis.

Routing and fallback mechanism: Router supports retry, fallback and load balancing between multiple deployments. The effect is that when suppliers limit traffic, regional failures or model price changes, the team can adjust the traffic through configuration without having to change the business code immediately.

Observation and secure connections: callbacks, guardrails, logging settings, and admin dashboard put model calls into regular project monitoring links. The effect is that requests, costs, delays and failure reasons can be traced, which is suitable for enterprises that need auditing and cost review.

Costs and Constraints: The gateway layer adds deployment, upgrade, database, network latency, and policy maintenance costs. The LiteLLM README mentions a benchmark signal of 8ms P95 latency at 1k RPS, but the actual latency still depends on the deployment environment, network path, log callback and upstream Provider response.

How to use

LiteLLM has two main paths: embedded in an application as a Python SDK, or deployed as a team-level AI Gateway as a Proxy Server.

How to use	Entrance	Suitable scenarios	Points of concern
Python SDK	pip install litellm / PyPI	Single application, multiple Provider quick access	Provider key, exception handling, cost recording
Proxy Server	litellm[proxy], Docker, Helm	Multi-team unified gateway	Database master key, virtual key, log, budget
Enterprise / Hosted Proxy	Official Enterprise Portal	Enterprise Governance, Hosted SSO, Auditing	SLA, Data Boundaries, Support Response, Contract Terms
MCP Gateway	Official documentation and release notes	Agent tool access control	Tool permissions, team isolation, log integrity

A typical landing path can be divided into three steps. The first step is to use the SDK or local Proxy to run through the same business request from 2 to 3 Providers to verify whether the OpenAI compatible format can cover the parameter requirements. The second step is to connect the team key, budget and logs to the Proxy and check whether the consumption collection is accurate. The third step is to expand Router, fallback, guardrails, SSO and MCP Gateway to avoid pushing all governance items to the production link from the beginning.

Product Pricing

LiteLLM's public commercial structure can be divided into three layers: open source usage, team self-hosting and enterprise services. The open source SDK and Proxy are free to use, and the underlying model API fee is charged by the connected Provider; the enterprise version, hosted Proxy, SSO, SCIM/OIDC, support services and SLA are subject to the official real-time page and sales contract.

• Open source layer: suitable for individual developers and small teams, used to unify model calling and local verification. Explicit software subscription cost is $0, actual costs come from model API, machine resources, and maintenance.
• Team self-hosted tier: suitable for organizations that already have platform engineering capabilities. Costs include database, gateway operation, logging system, monitoring, upgrades and in-house support.
• Enterprise Tier: For organizations requiring SSO, SCIM, OIDC, RBAC, audit logs, managed services, or dedicated support. The public page usually requires contacting sales, and the specific price is not disclosed.

Pricing evaluation can’t just look at the software price. The benefits of LiteLLM should be weighed against multi-provider access man-hours, risk of budget runaway, model switching costs, and log audit costs.

Application scenarios

• Multi-model unified entrance: The AI platform team provides a unified base_url and virtual key for multiple business lines. The business side maintains OpenAI compatible calls, and the platform side uniformly configures the upstream Provider and model routing.
• Cost governance and budget capping: Enterprises can track call costs by team, project, user or key, set budgets and quotas, and reduce billing opacity caused by shared provider keys.
• High-availability model routing: Fallback and load balancing between OpenAI, Azure OpenAI, Bedrock, Vertex AI or local models to reduce the impact of single-point supplier failure.
• Compliance and Audit: Incorporate LLM requests, responses, delays, errors, costs and user attributions into the log system to facilitate security audits and problem review.
• Agent Tool Governance: Through MCP Gateway and team permission management, Agent callable tools are included in access control to prevent tool permissions from spreading with business code.

Each scenario requires verification of key systems: provider credential management, log masking, failure retry strategies, user experience when the budget is in effect, and the difference in output quality caused by the fallback model.

Applicable people

• AI Platform and Infrastructure Team: It is necessary to unify multiple business line model calls into a gateway, paying attention to cost, permissions, availability and auditing.
• Backend Engineering Team: Need to reduce vendor SDK differences, maintain OpenAI compatible interfaces, while leaving room to switch providers.
• Enterprise Security and Finance Team: Need to see clearly who is calling what model, how much budget is spent, and whether the permission boundaries are reached.
• Agent Platform Team: The MCP tool vector store, member permissions and log policies need to be included in team-level control.

The boundaries of incompatibility are also clear: single-person projects, single-model calls, extremely low-latency sensitive links, or teams without operation and maintenance capabilities do not necessarily need to introduce the gateway layer. For heavily regulated industries, LiteLLM is only one of the governance components, and data residency, log masking, key rotation, vendor compliance, and enterprise contract terms still need to be independently confirmed.

Summary and Outlook

LiteLLM's core competitiveness lies in upgrading "multi-model access" to "model access management". It uses an OpenAI compatible format to reduce business access costs, uses Proxy Server, virtual keys, budget, Router, logging and admin dashboard to support team-level governance, and then extends agent tool permissions through MCP Gateway. For organizations that already have multiple providers, multiple teams, and cost auditing needs, it can become a critical control point in the AI ​​application infrastructure.

The current limitations mainly focus on four points: first, the SPDX logo of the open source license cannot be clearly identified through the GitHub API, and the final terms must be subject to the warehouse LICENSE file; second, the enterprise price, hosting service details, and SLA are not disclosed and require business confirmation; third, the gateway layer will bring additional deployment and operation and maintenance complexity; fourth, the Provider API changes frequently, and production must establish version freezing, regression testing, and log audit processes.

The implementation suggestion is to pilot low-risk business traffic first: select more than two Providers, a unified business interface, a team budget and a set of log callbacks as the minimum requirements, and verify the call success rate, P95 delay, cost aggregation accuracy and fallback triggering behavior. After the pilot is stable, it will be expanded to enterprise SSO, SCIM/OIDC, MCP Gateway and cross-team permission governance.

Related tools: Hugging Face, replicate

Version Info

• LiteLLM v1.89.2 ：The current public version shown in both GitHub Releases and PyPI continues the high-frequency iteration of LiteLLM AI Gateway, Proxy Server, Provider adaptation and enterprise governance capabilities. （2026-06-18）
• LiteLLM v1.88.3 ：The stable version v1.89.x released the day before reflects that the project maintains an intensive patching rhythm in provider compatibility, agent stability and gateway governance. （2026-06-17）
• LiteLLM v1.78.0 ：Official release notes The public version node includes product visuals related to MCP Gateway and team-level tool control, showing that LiteLLM has expanded from LLM Proxy to Agent tool governance. （2025-09-12）
• LiteLLM GitHub public repository ：When the GitHub repository was created, the public project was based on Python SDK and OpenAI compatible multi-provider calls, and later evolved into Proxy Server and AI Gateway. （2023-07-27）