provider-architecture

name

provider-architecture

description

LiteLLM-RS provider system in two tiers - data-driven OpenAI-compatible catalog entries auto-routed through OpenAILikeProvider, plus code-based provider modules implementing the LLMProvider trait behind the closed Provider enum. Covers unified ProviderError handling, connection pooling, capabilities, and model metadata. Use when adding a new provider, editing registry/catalog.rs, or implementing/debugging the LLMProvider trait.

Provider Architecture Guide

Architecture Overview

Providers come in two tiers behind one implementation trait:

Tier 1 — Catalog-only (zero code). An OpenAI-compatible endpoint that differs only in base URL, auth env var, and advertised capabilities/models. It has no Rust module: one static entry in src/core/providers/registry/catalog.rs fully describes it, and the factory builds an OpenAILikeProvider from that data at runtime. This is the primary path for new integrations.

Tier 2 — Code-based. A provider needing custom request/response transformation, custom auth signing, non-standard streaming, or rich model metadata lives in src/core/providers/<name>/, implements LLMProvider, and is wired into routing.

Routing does not use trait objects. Router deployments store the closed Provider enum (src/core/providers/mod.rs), which dispatches to concrete provider structs. LLMProvider (src/core/traits/provider/llm_provider/trait_definition.rs) is the interface every variant implements — implementing the trait alone does not make a provider routeable; enum variant, dispatch arm, and factory wiring are crate-level changes. Trait objects appear only at the edges: the error mapper (Box<dyn ErrorMapper<ProviderError>>) and the streaming return type (Pin<Box<dyn Stream<Item = Result<ChatChunk, ProviderError>> + Send>>).

Enumerating Current Providers

Do not rely on memorized counts — they change often. Count from source:

# Tier 1: one def_chat()/def_local_chat() call per entry
# (each count includes the helper fn definition itself, so subtract 1)
grep -c 'def_chat(' src/core/providers/registry/catalog.rs
grep -c 'def_local_chat(' src/core/providers/registry/catalog.rs

# Tier 2: code-based provider modules (base/factory/macros/registry are infrastructure)
ls -d src/core/providers/*/ | grep -vE '/(base|factory|macros|registry)/'

---

Adding a Tier 1 Provider (Primary Path)

Two edits, nothing else:

// src/core/providers/registry/catalog.rs
def_chat(
    "myprovider",
    "My Provider",
    "https://api.myprovider.com/v1",
    "MYPROVIDER_API_KEY",
),

// src/core/providers/mod.rs — annotation comment alongside the other Tier 1 notes
// myprovider: Tier 1 -> registry/catalog.rs

At runtime create_provider (src/core/providers/factory/mod.rs) matches the selector against the catalog, builds an OpenAILikeConfig via ProviderDefinition::to_openai_like_config, and constructs openai_like::OpenAILikeProvider::new_for_catalog(oai_config, def.capabilities) into the Provider::OpenAILike variant.

Keyless local servers use def_local_chat (AuthType::None, skip_api_key = true):

def_local_chat("myrunner", "My Runner", "http://localhost:1234/v1"),

Each entry is a ProviderDefinition (src/core/providers/registry/definition.rs). Defaults from def_chat can be overridden with struct-update syntax:

• alternate_auth_env_vars — env vars checked after auth_env_var (see together)
• model_prefix: Some("xai/") — selector/model prefix stripping (see xai)
• custom capabilities profile replacing the default OPENAI_LIKE_CATALOG_CAPABILITIES (ChatCompletion, ChatCompletionStream, ToolCalling, FunctionCalling)
• name aliases resolve via canonical_catalog_name() in catalog.rs (e.g. "zhipuai" -> "zhipu")

Registry API (src/core/providers/registry/, re-exported in registry/mod.rs): is_tier1_provider(name), get_definition(name), canonical_catalog_name(name), PROVIDER_CATALOG.

---

Core Trait Definition: LLMProvider

There are no associated types. Every fallible method returns the unified ProviderError directly.

// src/core/traits/provider/llm_provider/trait_definition.rs
pub trait LLMProvider: Send + Sync + Debug + 'static {
    // ===== Required =====
    fn name(&self) -> &str;
    fn capabilities(&self) -> &'static [ProviderCapability];
    fn models(&self) -> &[ModelInfo];

    fn get_supported_openai_params(&self, model: &str) -> &'static [&'static str];
    async fn map_openai_params(
        &self,
        params: HashMap<String, Value>,
        model: &str,
    ) -> Result<HashMap<String, Value>, ProviderError>;
    async fn transform_request(
        &self,
        request: ChatRequest,
        context: RequestContext,
    ) -> Result<Value, ProviderError>;
    async fn transform_response(
        &self,
        raw_response: &[],
        model: &,
        request_id: &,
    )  <ChatResponse, ProviderError>;
     (&)  < ErrorMapper<ProviderError>>;
      (
        &,
        request: ChatRequest,
        context: RequestContext,
    )  <ChatResponse, ProviderError>;
      (&)  HealthStatus;
      (
        &,
        model: &,
        input_tokens: ,
        output_tokens: ,
    )  <, ProviderError>;

    
     (&)  &  {  }
     (&, capability: &ProviderCapability)   {
        .().(capability)
    }
     (&, model: &)   {
        .().().(|m| m.id == model)
    }
    
    
    
      (&, request: ChatRequest, context: RequestContext)
         <Pin<< Stream<Item = <ChatChunk, ProviderError>> + >>, ProviderError>;
      (&, request: EmbeddingRequest, context: RequestContext)
         <EmbeddingResponse, ProviderError>;
      (&, request: ImageGenerationRequest, context: RequestContext)
         <ImageGenerationResponse, ProviderError>;
      (&, request: TranscriptionRequest, context: RequestContext)
         <TranscriptionResponse, ProviderError>;
      (&, request: TranslationRequest, context: RequestContext)
         <TranslationResponse, ProviderError>;
      (&, request: SpeechRequest, context: RequestContext)
         <SpeechResponse, ProviderError>;
      (&)  <std::time::Duration, ProviderError>; 
      (&)  <, ProviderError>;                   
      (&, text: &)  <, ProviderError>;        
}

Optional dispatch methods (streaming, embeddings, images, audio) must still be gated by the matching ProviderCapability — route selection checks supports_capability() before calling them.

---

Tier 2: Code-Based Provider Pattern

Directory Structure

Real examples: src/core/providers/cloudflare/ (small) and src/core/providers/openai_like/.

src/core/providers/my_provider/
├── mod.rs           # Module exports, LLMProvider impl may live here too
├── config.rs        # Config struct (often via define_provider_config!)
├── provider.rs      # Provider struct + LLMProvider impl
├── model_info.rs    # Static model metadata
├── streaming.rs     # SSE parsing (optional)
└── error.rs         # Error helpers/mappers (optional; legacy name = ProviderError alias)

Configuration

BaseConfig (src/core/providers/base/config.rs) carries api_key, api_base, endpoint_access, timeout (secs), max_retries, headers, organization, api_version, with env fallbacks ({PROVIDER}_API_KEY, {PROVIDER}_API_BASE, ...) via from_env(provider) / for_provider(provider).

Implement the ProviderConfig trait (src/core/traits/provider/config.rs) — note the accessor names differ from BaseConfig's helpers:

pub trait ProviderConfig: Send + Sync + Clone + Debug + 'static {
    fn validate(&self) -> Result<(), String>;
    fn api_key(&self) -> Option<&str>;
    fn api_base(&self) -> Option<&str>;
    fn timeout(&self) -> std::time::Duration;
    fn max_retries(&self) -> u32;
    fn endpoint_access(&self) -> ProviderEndpointAccess { ProviderEndpointAccess::PublicOnly }
    fn use_ssrf_safe_client(&self) -> bool { false }
    // validate_standard(provider_name): shared key/timeout/retries checks
}

The define_provider_config! macro (exported from base/config.rs) generates the struct, builders (with_api_key, with_base_url, with_timeout), from_env(), get_api_key()/get_api_base(), and the ProviderConfig impl in one call.

Provider Implementation

Modeled on src/core/providers/cloudflare/provider.rs:

use std::sync::Arc;
use crate::core::providers::base::{header, BaseConfig, GlobalPoolManager, HttpMethod};
use crate::core::providers::ProviderError;
use crate::core::traits::error_mapper::trait_def::ErrorMapper;
use crate::core::traits::error_mapper::DefaultErrorMapper;
use crate::core::traits::provider::llm_provider::trait_definition::LLMProvider;
use crate::core::traits::provider::ProviderConfig;

pub struct MyProvider {
    config: MyProviderConfig,
    pool_manager: Arc<GlobalPoolManager>,
    models: Vec<ModelInfo>,
}

impl MyProvider {
    pub fn new(config: MyProviderConfig) -> Result<Self, ProviderError> {
        config.validate()
            .map_err(|e| ProviderError::configuration(PROVIDER_NAME, e))?;
        let http_config = BaseConfig {
            api_key: config.api_key().map(str::to_owned),
            api_base: config.api_base().map(str::to_owned),
            endpoint_access: config.endpoint_access(),
            timeout: config.timeout().as_secs(),
            max_retries: config.max_retries(),
            ..BaseConfig::default()
        };
        Ok(Self {
            config,
            pool_manager: Arc::(GlobalPoolManager::(
                PROVIDER_NAME,
                http_config,
            )?),
            models: (),
        })
    }
}

    {
     (&)  &  { PROVIDER_NAME }
     (&)  &  { PROVIDER_NAME }

      (
        &,
        request: ChatRequest,
        context: RequestContext,
    )  <ChatResponse, ProviderError> {
          = .config.()
            .(|| ProviderError::(PROVIDER_NAME, ))?;
          = [(, ())];
        headers.((, .()));

        
          = .(request, context).?;
          = .pool_manager
            .(&url, HttpMethod::POST, headers, (body))
            .?;

          = response.();
         !status.() {
              = response.().
                .(|e| ProviderError::(PROVIDER_NAME, e.()))?;
             (.().(status.(), &body_text));
        }
          = response.().
            .(|e| ProviderError::(PROVIDER_NAME, e.()))?;
        .(&raw, &model, &request_id).
    }

     (&)  < ErrorMapper<ProviderError>> {
        ::(DefaultErrorMapper)
    }
    
}

Registration (routing) Requirements

Adding a Tier 2 provider means crate-level wiring, in this order of authority:

1. Add src/core/providers/<name>/, its module declaration, and a closed Provider enum variant under the same feature gate in src/core/providers/mod.rs.
2. Add the variant to all four dispatch_provider! @expand arms (sync, async_err, value, async_direct) and to Provider::name() and provider_type().
3. Add a ProviderType variant and all_non_custom_provider_types() entry in provider_type.rs.
4. Add canonical name, aliases, catalog_backed and the correct ProviderDispatchKind to PROVIDER_TYPE_REGISTRY in registry/types.rs. For a feature-gated native implementation, use or add a cfg-sensitive dispatch-kind helper that reports the correct enabled and disabled modes; registry entries themselves do not have a feature field.
5. Add the config builder in factory/builder.rs and match branch in factory/registry.rs, with matching feature gates. Keep cfg gates synchronized across module, Provider, dispatch and factory wiring.
6. Update provider-type/registry lifecycle, factory support and feature-on/off tests so aliases, support state and construction stay in sync.

---

Connection Pooling

The recommended new_for_provider path runs requests through BaseHttpClient and ProviderHttpClient. ProviderHttpClient keeps a process-wide client cache keyed by endpoint policy, request timeout, and mode (ordinary, streaming, or no-redirect). Those clients use HttpClientPoolConfig::default() (src/utils/net/http.rs): 100 maximum idle connections per host, 90-second idle timeout, 10-second connect timeout, and 60-second TCP keepalive.

PoolConfig in src/core/providers/base/connection_pool.rs (there is no pool.rs) instead configures the legacy global ConnectionPool used by the unbound new() / shared() path. These constants do not tune the policy-bound clients above:

pub struct PoolConfig;
impl PoolConfig {
    pub const TIMEOUT_SECS: u64 = 600;
    pub const POOL_SIZE: usize = 80;      // pool_max_idle_per_host
    pub const KEEPALIVE_SECS: u64 = 90;   // pool_idle_timeout
}

// Provider implementations use new_for_provider(provider, BaseConfig) so
// endpoint-access and timeout policy are installed. new()/shared() omit that policy.
pub async fn execute_request(
    &self,
    url: &str,
    method: HttpMethod,               // GET | POST | PUT | DELETE
    headers: Vec<HeaderPair>,         // (Cow<'static, str>, Cow<'static, str>)
    body: Option<serde_json::Value>,  // serialized as JSON; not generic
) -> Result<reqwest::Response, ProviderError>;

pub async fn execute_streaming_request(
    &self,
    url: &str,
    headers: Vec<HeaderPair>,
    body: serde_json::Value,
    legacy_provider: &'static str,
) -> Result<reqwest::Response, ProviderError>;

Build header pairs zero-copy where possible: header(key, value) (static key, owned value), header_static(key, value), header_owned(key, value). Streaming callers get a separate client without a total-body timeout via streaming_unbounded_client(); header phase bounded by STREAMING_HEADER_TIMEOUT_SECS, error bodies by read_streaming_error_body (10s / 64KiB caps).

---

Model Information

ModelInfo (src/core/types/model.rs) is a plain serializable struct with Default:

pub struct ModelInfo {
    pub id: String,
    pub name: String,
    pub provider: String,
    pub max_context_length: u32,
    pub max_output_length: Option<u32>,
    pub supports_streaming: bool,
    pub supports_tools: bool,
    pub supports_multimodal: bool,
    pub input_cost_per_1k_tokens: Option<f64>,
    pub output_cost_per_1k_tokens: Option<f64>,
    pub currency: String,
    pub capabilities: Vec<ProviderCapability>,
    pub created_at: Option<SystemTime>,
    pub updated_at: Option<SystemTime>,
    pub metadata: HashMap<String, serde_json::Value>,
}

Costs are per 1K tokens (catalog pricing constants in registry/catalog.rs are per million and divided by 1_000 when converted). Return &self.models from LLMProvider::models(); supports_model() scans that slice.

---

Provider Capabilities

ProviderCapability (src/core/types/model.rs) — the full current variant list:

pub enum ProviderCapability {
    ChatCompletion,
    ChatCompletionStream,
    Embeddings,
    ImageGeneration,
    ImageEdit,
    ImageVariation,
    AudioTranscription,
    AudioTranslation,
    TextToSpeech,
    Moderation,
    Rerank,
    ToolCalling,
    FunctionCalling,
    CodeExecution,
    FileUpload,
    FineTuning,
    BatchProcessing,
    RealtimeApi,
    GeminiGenerateContent,
}

Return a static slice (a temporary array literal cannot back &'static [...]):

const MY_CAPABILITIES: &[ProviderCapability] = &[
    ProviderCapability::ChatCompletion,
    ProviderCapability::ChatCompletionStream,
    ProviderCapability::ToolCalling,
];

fn capabilities(&self) -> &'static [ProviderCapability] {
    MY_CAPABILITIES
}

---

Unified Error Handling

ProviderError (src/core/providers/unified_provider_error.rs, re-exported as crate::core::providers::ProviderError and unified_provider::ProviderError) is the single error type across the trait. Construct with factory methods (src/core/providers/unified_provider_methods.rs) rather than struct literals — some variants carry extra optional fields (e.g. RateLimit rpm/tpm limits):

ProviderError::authentication(provider, msg)
ProviderError::rate_limit(provider, retry_after: Option<u64>)
ProviderError::rate_limit_with_retry(provider, msg, retry_after)
ProviderError::model_not_found(provider, model)
ProviderError::invalid_request(provider, msg)
ProviderError::network(provider, msg)
ProviderError::timeout(provider, msg)
ProviderError::api_error(provider, status: u16, msg)
ProviderError::provider_unavailable(provider, msg)
ProviderError::not_supported(provider, feature)
ProviderError::configuration(provider, msg)
ProviderError::serialization(provider, msg)

ErrorMapper<E> (src/core/traits/error_mapper/trait_def.rs) converts HTTP statuses and JSON error bodies into errors: required map_http_error(u16, &str); defaulted map_json_error, map_network_error, map_parsing_error, map_timeout_error. Ready-made mappers:

• GenericErrorMapper — core::traits::error_mapper::types, aliased DefaultErrorMapper at core::traits::error_mapper (what most providers return from get_error_mapper())
• OpenAIErrorMapper, AnthropicErrorMapper — core::traits::error_mapper::implementations

Legacy per-provider error enums were removed; surviving names like AnthropicError or GeminiError are pub type ... = ProviderError aliases (see reference/migration-from-legacy-errors.md).

---

References

• reference/best-practices-and-checklist.md — error factory conventions, coding practices, and new-provider checklist (Tier 1 and Tier 2)
• reference/migration-from-legacy-errors.md — how legacy per-provider error types map onto unified ProviderError