together-ai-inference

# Together AI — Serverless Inference & Fine-Tuning Together AI is an AI cloud platform providing serverless inference on 200+ open-source models through an OpenAI-compatible API. It supports chat completions, embeddings, fine-tuning, image generation, and batch processing at `https://api.together.xyz/v1`. ## When to Use Together AI **Use Together AI when:** - You need fast serverless inference on open-source models (Llama, DeepSeek, Qwen, Mistral) - You want an OpenAI-compatible API so you can swap providers with a single line change - You need to fine-tune open-source models without managing GPU infrastructure - You want cost-effective inference with pay-per-token pricing - You need function calling, JSON mode, or structured outputs from open-source models - You want batch processing at 50% lower cost for non-urgent workloads - You need embeddings or image generation alongside chat completions **Use alternatives instead:** | Need | Use Instead | |------|-------------| | Managed LoRA fine-tuning with training platform | **Tinker** | | Self-hosted inference with full control | **vLLM**, **TensorRT-LLM** | | Dedicated GPU instances | **Lambda Labs**, **RunPod** | | Serverless GPU with custom containers | **Modal** | | Multi-cloud cost optimization | **SkyPilot** | | Proprietary models (GPT-4o, Claude) | **OpenAI**, **Anthropic** directly | ## Credential Setup Credentials are auto-injected by openscience when connected via the dashboard. ```bash # Verify credentials [ -n "$TOGETHER_API_KEY" ] && echo "TOGETHER_API_KEY set" || echo "NOT SET" ``` If not set: add your Together AI key in Customize → Models or export `TOGETHER_API_KEY` locally. ## Quick Start ### Install ```python pip install together openai ``` ### Set API Key ```python import os os.environ["TOGETHER_API_KEY"] = "your-api-key" # Or export in shell: # export TOGETHER_API_KEY="your-api-key" ``` Get your API key from https://api.together.xyz/settings/api-keys ### Basic Chat Completion ```python from together import Together client = Together() response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct-Reference", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain gradient descent in one paragraph."}, ], max_tokens=256, temperature=0.7, ) print(response.choices[0].message.content) ``` ## Inference ### Chat Completions The primary endpoint for conversational AI. Supports system/user/assistant messages, temperature, top_p, top_k, repetition_penalty, and stop sequences. ```python from together import Together client = Together() response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3", messages=[ {"role": "system", "content": "You are an expert ML researcher."}, {"role": "user", "content": "Compare LoRA vs full fine-tuning."}, ], max_tokens=512, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.1, stop=["</s>"], ) print(response.choices[0].message.content) print(f"Tokens used: {response.usage.total_tokens}") ``` ### Streaming ```python from together import Together client = Together() stream = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct-Reference", messages=[{"role": "user", "content": "Write a haiku about transformers."}], max_tokens=128, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta.content if delta: print(delta, end="", flush=True) print() ``` ### Function Calling Supported on select models including Llama, DeepSeek, Qwen, and Mistral variants. ```python from together import Together import json client = Together() tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and state, e.g. San Francisco, CA", }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], }, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct-Reference", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What's the weather in San Francisco?"}, ], tools=tools, tool_choice="auto", ) tool_calls = response.choices[0].message.tool_calls if tool_calls: call = tool_calls[0] print(f"Function: {call.function.name}") print(f"Arguments: {call.function.arguments}") ``` ### JSON Mode (Structured Outputs) Force the model to return valid JSON conforming to a schema. ```python import json from together import Together client = Together() schema = { "type": "object", "properties": { "name": {"type": "string"}, "age": {"type": "integer"}, "skills": {"type": "array", "items": {"type": "string"}}, }, "required": ["name", "age", "skills"], } response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct-Reference", messages=[ { "role": "system", "content": f"Respond only in JSON matching this schema: {json.dumps(schema)}", }, {"role": "user", "content": "Create a profile for a senior ML engineer."}, ], response_format={ "type": "json_object", "schema": schema, }, ) data = json.loads(response.choices[0].message.content) print(json.dumps(data, indent=2)) ``` ### Vision Models ```python from together import Together client = Together() response = client.chat.completions.create( model="meta-llama/Llama-4-Scout-17B-16E-Instruct-VLM", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe this image in detail."}, { "type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}, }, ], } ], max_tokens=512, ) print(response.choices[0].message.content) ``` ## Fine-Tuning Together AI supports both LoRA and full fine-tuning on a wide range of open-source models. ### Data Format Training data must be in JSONL format with chat-style messages: ```jsonl {"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}, {"role": "assistant", "content": "The capital of France is Paris."}]} {"messages": [{"role": "user", "content": "Explain photosynthesis."}, {"role": "assistant", "content": "Photosynthesis is the process by which plants convert sunlight, water, and carbon dioxide into glucose and oxygen."}]} ``` Each line is one training example. The `system` role is optional. The model learns to generate the `assistant` responses. ### Upload Training Data ```python from together import Together client = Together() # Upload training file file = client.files.upload(file="training_data.jsonl") print(f"File ID: {file.id}") ``` ### Create Fine-Tuning Job ```python from together import Together client = Together() # Create LoRA fine-tuning job job = client.fine_tuning.create( training_file="file-abc123", model="meta-llama/Llama-3.3-70B-Instruct-Reference", n_epochs=3, learning_rate=1e-5, batch_size=4, lora=True, lora_r=16, lora_alpha=32, lora_dropout=0.05, suffix="my-custom-model", ) print(f"Job ID: {job.id}") print(f"Status: {job.status}") ``` ### Monitor Fine-Tuning ```python from together import Together client = Together() # Check job status job = client.fine_tuning.retrieve(id="ft-abc123") print(f"Status: {job.status}") # List all jobs jobs = client.fine_tuning.list() for j in jobs: print(f"{j.id}: {j.status}") # List events (training logs) events = client.fine_tuning.list_events(id="ft-abc123") for event in events: print(event) # Cancel a job client.fine_tuning.cancel(id="ft-abc123") ``` ### CLI Fine-Tuning ```bash # Install CLI pip install --upgrade together # Upload data together files upload training_data.jsonl # Create fine-tuning job together fine-tuning create \ --training-file file-abc123 \ -m meta-llama/Meta-Llama-3.1-8B-Instruct-Reference # Check status together fine-tuning status ft-abc123 # List checkpoints together fine-tuning list-checkpoints ft-abc123 # Download fine-tuned model together fine-tuning download --ft-id ft-abc123 ``` ### Supported Fine-Tuning Models (Selection) | Model | LoRA | Full | |-------|------|------| | meta-llama/Meta-Llama-3.1-8B-Instruct-Reference | Yes | Yes | | meta-llama/Llama-3.3-70B-Instruct-Reference | Yes | Yes | | meta-llama/Llama-4-Scout-17B-16E-Instruct | Yes | No | | deepseek-ai/DeepSeek-V3 | Yes | No | | deepseek-ai/DeepSeek-R1 | Yes | No | | Qwen/Qwen3-8B | Yes | Yes | | Qwen/Qwen3-32B | Yes | Yes | | Qwen/Qwen3-235B-A22B | Yes | No | | google/gemma-3-27b-it | Yes | Yes | | google/gemma-3-4b-it | Yes | Yes | See the full list at https://docs.together.ai/docs/fine-tuning-models ## Embeddings ### Generate Embeddings ```python from together import Together client = Together() response = client.embeddings.create( model="BAAI/bge-large-en-v1.5", input="What is the meaning of life?", ) embedding = response.data[0].embedding print(f"Dimensions: {len(embedding)}") print(f"First 5 values: {embedding[:5]}") ``` ### Batch Embeddings ```python from together import Together client = Together() texts = [ "Machine learning is a subset of AI.", "Deep learning uses neural networks.", "Transformers revolutionized NLP.", ] response = client.embeddings.create( model="BAAI/bge-large-en-v1.5", input=texts, ) for i, item in enumerate(response.data): print(f"Text {i}: {len(item.embedding)} dimensions") ``` ### Available Embedding Models | Model ID | Dimensions | Best For |