Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
FreeLLMAPI is a self-hosted OpenAI-compatible proxy that aggregates free-tier API keys from ~14 AI providers (Google, Groq, Cerebras, SambaNova, NVIDIA, Mistral, OpenRouter, GitHub Models, Hugging Face, Cohere, Cloudflare, Zhipu, Moonshot, MiniMax) behind a single /v1/chat/completions endpoint. It handles automatic failover on 429/5xx, per-key rate tracking, sticky sessions for multi-turn conversations, and AES-256-GCM encrypted key storage.
Installation
Prerequisites: Node.js 20+, npm.
git clone https://github.com/tashfeenahmed/freellmapi.git
cd freellmapi
npm install
# Generate encryption key and set up environmentcp .env.example .envecho"ENCRYPTION_KEY=$(node -e "console.log(require('crypto').randomBytes(32).toString('hex'))")" >> .env
npm run dev
npm run build
node server/dist/index.js
Never commit .env. The ENCRYPTION_KEY protects all stored provider API keys.
Key Commands
npm run dev # Start Express server + Vite dashboard in watch mode
npm run build # Compile TypeScript server + build React dashboard
npm run lint # ESLint across server/ and client/
npm run test# Run test suite
Provider Setup
Open the dashboard at http://localhost:5173 (dev) or http://localhost:3001 (prod).
Navigate to Keys page.
Add raw API keys for each provider you have. Keys are encrypted before SQLite storage.
Navigate to Fallback Chain to reorder provider priority.
Copy your unified freellmapi-… bearer token from the Keys page header.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:3001/v1",
api_key="freellmapi-your-unified-key", # from dashboard Keys page
)
# Let the router pick the best available provider
response = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Explain async/await in Python in two sentences."}],
)
print(response.choices[0].message.content)
# Which provider actually served this request:print("Routed via:", response.headers.get("x-routed-via"))
Request a specific model
# Request a specific model — router finds a provider that has it
response = client.chat.completions.create(
model="gemini-2.5-flash",
messages=[{"role": "user", "content": "Write a haiku about SQLite."}],
)
Streaming
stream = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "List 5 TypeScript best practices."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
Tool calling works across all supported providers. OpenAI-compatible providers receive requests verbatim; Gemini requests are automatically translated to functionDeclarations/functionResponse format and back.
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:3001/v1",
api_key="freellmapi-your-unified-key",
)
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a city.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "City name"},
},
"required": ["city"],
},
},
}
]
# Step 1: Model requests a tool call
first = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
tools=tools,
tool_choice="required",
)
call = first.choices[0].message.tool_calls[0]
print(f"Tool requested: {call.function.name}({call.function.arguments})")
# Step 2: Execute the tool locally, feed result back
final = client.chat.completions.create(
model="auto",
messages=[
{"role": "user", "content": "What's the weather in Karachi?"},
first.choices[0].message, # assistant message with tool_calls
{
"role": "tool",
"tool_call_id": call.id,
"content": '{"temp_c": 32, "condition": "sunny"}',
},
],
tools=tools,
)
print(final.choices[0].message.content)
Streaming tool calls
stream = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
tools=tools,
tool_choice="required",
stream=True,
)
tool_call_chunks = []
for chunk in stream:
delta = chunk.choices[0].delta
if delta.tool_calls:
tool_call_chunks.extend(delta.tool_calls)
if chunk.choices[0].finish_reason == "tool_calls":
print("Tool call complete — assemble chunks and execute")
Multi-turn Conversations (Sticky Sessions)
The proxy keeps multi-turn conversations on the same model for 30 minutes to avoid hallucination spikes from mid-conversation model switches. Pass a consistent session_id in requests if the provider supports it, or rely on the proxy's automatic session tracking.
messages = [{"role": "system", "content": "You are a helpful coding assistant."}]
# Turn 1
messages.append({"role": "user", "content": "Write a Python function to flatten a nested list."})
resp1 = client.chat.completions.create(model="auto", messages=messages)
assistant_msg = resp1.choices[0].message
messages.append({"role": "assistant", "content": assistant_msg.content})
print(assistant_msg.content)
# Turn 2 — sticky session keeps same provider
messages.append({"role": "user", "content": "Now add type hints to that function."})
resp2 = client.chat.completions.create(model="auto", messages=messages)
print(resp2.choices[0].message.content)
LangChain Integration
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
import os
llm = ChatOpenAI(
model="auto",
openai_api_base="http://localhost:3001/v1",
openai_api_key=os.environ["FREELLMAPI_KEY"],
streaming=True,
)
response = llm.invoke([HumanMessage(content="Summarise the CAP theorem in one paragraph.")])
print(response.content)
Response Headers
Every response includes diagnostic headers:
Header
Description
X-Routed-Via
<platform>/<model> — which provider served the request
X-Fallback-Attempts
Number of providers tried before success (only present if > 0)
response = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "hi"}],
)
# Headers are on the raw httpx response:
raw = response._response # openai SDK exposes underlying httpx responseprint(raw.headers.get("x-routed-via")) # e.g. "groq/llama-4-scout"print(raw.headers.get("x-fallback-attempts")) # e.g. "2"
How the Router Works
Request arrives
│
▼
Router scans fallback chain (priority order)
│
├─ For each model: is there a healthy key under all rate caps?
│ RPM / RPD / TPM / TPD tracked per (platform, model, key)
│
├─ Picks first viable (platform, model, key) tuple
│
├─ Decrypts key in-memory, calls provider SDK
│
└─ On 429 / 5xx / timeout:
Put key on cooldown → retry next model (up to 20 attempts)
Rate limit tracking: The router tracks RPM, RPD, TPM, and TPD counters per (platform, model, key) triple. When a key hits a cap it's cooled down automatically and the next viable key/model is tried.
Health checks: Background probes classify each key as healthy, rate_limited, invalid, or error. The router skips non-healthy keys without making a live request.
Dashboard Pages
Page
Purpose
Keys
Add/remove provider credentials, view health status, copy unified API key
Fallback Chain
Drag to reorder provider priority
Playground
Interactive chat showing which provider served each message + latency
