fireworks-ai-inference

# Fireworks AI -- Fast Inference & Fine-Tuning Fastest open-model inference platform with serverless and on-demand GPU deployments, OpenAI-compatible API, and built-in fine-tuning (SFT, DPO, RL). ## When to Use Fireworks AI **Use Fireworks AI when:** - Need fast serverless inference for open-source models (Llama, Qwen, DeepSeek, Mixtral) - Want OpenAI SDK drop-in replacement with open models - Need fine-tuning without managing infrastructure (SFT, DPO, RL) - Require structured output / JSON mode / function calling with open models - Need dedicated GPU deployments with predictable latency - Require SOC2 or HIPAA compliance - Want prompt caching and batch inference for cost savings **Use alternatives instead:** | Need | Use Instead | |------|-------------| | Self-hosted inference (full control) | **vLLM**, **TensorRT-LLM** | | Cheapest serverless inference | **Groq** (free tier), **Together AI** | | Managed LoRA fine-tuning (no infra) | **Tinker** | | Closed-model APIs (GPT-4, Claude) | **OpenAI**, **Anthropic** direct | | GPU instances with SSH access | **Lambda Labs**, **RunPod** | | Multi-cloud orchestration | **SkyPilot** | ## Credential Setup Credentials are auto-injected by openscience when connected via the dashboard. ```bash # Verify credentials [ -n "$FIREWORKS_API_KEY" ] && echo "FIREWORKS_API_KEY set" || echo "NOT SET" ``` If not set: add your Fireworks AI key in Customize → Models or export `FIREWORKS_API_KEY` locally. ## Quick Start ### Install ```python pip install fireworks-ai openai ``` ### Set API key ```python import os os.environ["FIREWORKS_API_KEY"] = "fw_..." # from https://fireworks.ai/api-keys ``` ### Basic chat completion ```python from openai import OpenAI client = OpenAI( base_url="https://api.fireworks.ai/inference/v1", api_key=os.environ["FIREWORKS_API_KEY"], ) response = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p3-70b-instruct", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain gradient descent in two sentences."}, ], max_tokens=256, temperature=0.7, ) print(response.choices[0].message.content) ``` ## Inference ### Chat completions **Endpoint:** `POST https://api.fireworks.ai/inference/v1/chat/completions` ```python from openai import OpenAI client = OpenAI( base_url="https://api.fireworks.ai/inference/v1", api_key=os.environ["FIREWORKS_API_KEY"], ) response = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p3-70b-instruct", messages=[ {"role": "user", "content": "Write a Python quicksort function."}, ], max_tokens=512, temperature=0.0, ) print(response.choices[0].message.content) ``` ### Streaming ```python stream = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p3-70b-instruct", messages=[{"role": "user", "content": "Explain transformers."}], stream=True, max_tokens=512, ) for chunk in stream: delta = chunk.choices[0].delta.content if delta: print(delta, end="", flush=True) ``` ### Function calling / tool use ```python tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p3-70b-instruct", messages=[{"role": "user", "content": "What is the weather in San Francisco?"}], tools=tools, tool_choice="auto", ) tool_call = response.choices[0].message.tool_calls[0] print(tool_call.function.name, tool_call.function.arguments) ``` ### Structured output / JSON mode ```python response = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p3-70b-instruct", messages=[{"role": "user", "content": "List 3 planets with mass and diameter."}], response_format={"type": "json_object"}, max_tokens=512, ) import json data = json.loads(response.choices[0].message.content) ``` For strict schema enforcement, use `response_format` with a JSON schema: ```python response = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p3-70b-instruct", messages=[{"role": "user", "content": "Extract name and age from: John is 30."}], response_format={ "type": "json_object", "schema": { "type": "object", "properties": { "name": {"type": "string"}, "age": {"type": "integer"}, }, "required": ["name", "age"], }, }, ) ``` ### Vision (multimodal) ```python response = client.chat.completions.create( model="accounts/fireworks/models/llama-v3p2-11b-vision-instruct", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe this image."}, { "type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}, }, ], } ], max_tokens=256, ) print(response.choices[0].message.content) ``` ## Fine-Tuning Fireworks supports three fine-tuning methods: Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Fine-Tuning (RL). All use LoRA adapters by default. ### SFT -- Supervised Fine-Tuning **Data format (JSONL):** Each line is a conversation with `messages` array: ```json {"messages": [{"role": "system", "content": "You are a coding assistant."}, {"role": "user", "content": "Write a Python hello world."}, {"role": "assistant", "content": "print('Hello, world!')"}]} {"messages": [{"role": "user", "content": "What is 2+2?"}, {"role": "assistant", "content": "4"}]} ``` **Create SFT job via API:** ```python import requests url = "https://api.fireworks.ai/v1/accounts/{account_id}/supervisedFineTuningJobs" headers = { "Authorization": f"Bearer {os.environ['FIREWORKS_API_KEY']}", "Content-Type": "application/json", } payload = { "displayName": "my-sft-job", "model": "accounts/fireworks/models/llama-v3p3-70b-instruct", "dataset": "accounts/{account_id}/datasets/{dataset_id}", "hyperparameters": { "epochs": 3, "learning_rate": 1e-4, "batch_size": 8, "lora_rank": 16, }, } response = requests.post(url, headers=headers, json=payload) job = response.json() print(f"Job ID: {job['name']}") ``` **Monitor job:** ```python job_url = f"https://api.fireworks.ai/v1/{job['name']}" status = requests.get(job_url, headers=headers).json() print(f"State: {status['state']}, Progress: {status.get('progress', 'N/A')}") ``` ### DPO -- Direct Preference Optimization **Data format (JSONL):** Each line has `chosen` and `rejected` conversations: ```json {"chosen": [{"role": "user", "content": "Explain ML"}, {"role": "assistant", "content": "Machine learning is..."}], "rejected": [{"role": "user", "content": "Explain ML"}, {"role": "assistant", "content": "ML is complicated..."}]} ``` **Create DPO job:** ```python url = "https://api.fireworks.ai/v1/accounts/{account_id}/dpoFineTuningJobs" payload = { "displayName": "my-dpo-job", "model": "accounts/fireworks/models/llama-v3p3-70b-instruct", "dataset": "accounts/{account_id}/datasets/{dataset_id}", "hyperparameters": { "epochs": 2, "learning_rate": 5e-5, "beta": 0.1, }, } response = requests.post(url, headers=headers, json=payload) ``` ### RL -- Reinforcement Fine-Tuning Reinforcement fine-tuning uses a reward model or reward function to optimize the base model. Jobs run on on-demand GPUs and are billed at GPU-hour rates. **Create RL job:** ```python url = "https://api.fireworks.ai/v1/accounts/{account_id}/reinforcementFineTuningJobs" payload = { "displayName": "my-rl-job", "baseModel": "accounts/fireworks/models/llama-v3p3-70b-instruct", "rewardModel": "accounts/{account_id}/models/{reward_model_id}", "dataset": "accounts/{account_id}/datasets/{dataset_id}", } response = requests.post(url, headers=headers, json=payload) ``` ### Fine-tuning pricing | Model Size | SFT ($/hr) | DPO ($/hr) | |------------|------------|------------| | Up to 16B | $0.50 | $1.00 | | 16B - 80B | $3.00 | $6.00 | | 80B - 300B | $6.00 | $12.00 | | 300B+ | $10.00 | $20.00 | RL fine-tuning is billed at on-demand GPU rates. ## On-Demand GPU Deployments Dedicated GPU deployments provide predictable latency, no rate limits, and support for custom/fine-tuned models. Billed per GPU-second. ### Create deployment ```python import requests url = "https://api.fireworks.ai/v1/accounts/{account_id}/deployments" headers = {