togethercomputer

On-demand and reserved GPU clusters (H100, H200, B200) on Together AI with Kubernetes or Slurm orchestration, shared storage, credential management, and cluster scaling for ML and HPC jobs. Reach for it when the user needs multi-node compute or infrastructure…

Text-to-speech and speech-to-text via Together AI, including REST, streaming, and realtime WebSocket TTS, plus transcription, translation, diarization, timestamps, and live STT. Reach for it whenever the user needs audio in or audio out on Together AI rather…

Real-time and streaming text generation via Together AI's OpenAI-compatible chat/completions API, including multi-turn conversations, tool and function calling, structured JSON outputs, and reasoning models. Reach for it whenever the user wants to build or…

LLM-as-a-judge evaluation framework on Together AI. Classify, score, and compare model outputs, select judge models, use external-provider judges or targets, poll results and download reports. Reach for it whenever the user wants to benchmark outputs, grade…

Custom Dockerized inference workers on Together AI's managed GPU infrastructure. Build with Sprocket SDK, configure with Jig CLI, submit async queue jobs, and poll results. Reach for it whenever the user needs container-level control rather than a standard…

Deploy and operate models on dedicated GPUs with Together AI's Dedicated Model Inference (DMI, the v2 dedicated endpoints API): beta endpoints, deployments, deployment profiles and hardware configs, autoscaling, traffic splitting, A/B tests, shadow…

Dense vector embeddings, semantic search, RAG pipelines, and reranking via Together AI. Generate embeddings with open-source models and rerank results behind dedicated endpoints. Reach for it whenever the user needs vector representations or retrieval quality…

LoRA, full fine-tuning, DPO preference tuning, VLM training, function-calling tuning, reasoning tuning, and BYOM uploads on Together AI. Reach for it whenever the user wants to adapt a model on custom data rather than only run inference, evaluate outputs, or…

Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module

Add a new model to the SGLang Cookbook (docs/, Mintlify), config-driven format — instantiate the model-agnostic template into a per-model config (+ benchmarks) JSX under src/snippets/configs/, an MDX page, the docs.json nav entry, NEW-tag hygiene, and the…

Unified LLM torch-profiler triage skill for `sglang`, `vllm`, `TensorRT-LLM`, and `TokenSpeed`. Use it to inspect an existing `trace.json(.gz)` or profile directory, or to drive live profiling against a running server when supported and return one three-table…

How SGLang's runtime configuration and process-global state are organized (RuntimeContext tiers, publish + namespace config bags, the pristine ServerArgs seed, override entry points, resource/stream/buffer leases, per-forward flags), the CI guardrails that…

Use when adding a new diffusion model or Diffusers pipeline to SGLang.

Use when benchmarking denoise latency or profiling a diffusion bottleneck in SGLang.

Use when quantizing a diffusion DiT with NVIDIA ModelOpt and making the resulting FP8 or NVFP4 checkpoint loadable, verifiable, and benchmarkable in SGLang Diffusion.

Use when choosing the fastest SGLang Diffusion flags for a model, GPU, and VRAM budget.

Mint a new SigV4 S3 access key (access-key-id + secret) on the Together User Management Service (UMS) via its gRPC Key API, so you can authenticate to the s3-proxy. Discovers the project/creator from an existing key, then calls CreateProjectKeyV1 with…

Deploy (or clean-reinstall) Together T4 (gns control-plane + s3-proxy data-plane) and CS3 (s3-cache-proxy) on the transporter k3s cluster, backed by a Rook-Ceph RGW running on a separate cluster. Covers the two-cluster topology, the two data paths,…

Seed or repoint the GNS (Global Namespace Service) site/store routing in its Postgres so buckets resolve to a live Ceph RGW endpoint. Use after a fresh gns install or when the store URL is stale/unreachable (e.g. an old in-cluster rook DNS name that doesn't…

Run an end-to-end S3 round-trip (create-bucket, put, get, list) through the Together s3-proxy LoadBalancer to verify the full T4 path — UMS auth → GNS routing → Ceph RGW — actually serves data. Includes how to diagnose auth failures from the proxy logs…

Write structured long-form blog posts with a consistent structure and SEO optimization.

Create social media content including Twitter/X threads, LinkedIn posts, and short-form updates.