gpu-runner

name

gpu-runner

description

Execute model inference on GPU cloud providers. Handles code generation, deployment, execution, and result collection across HF Inference API/Endpoints, Colab, Modal, beam.cloud, Vast.ai, and RunPod. Use when running models on GPU, deploying to cloud, executing notebooks, or troubleshooting GPU execution failures. Triggers on "run on GPU", "execute model", "deploy to modal", "colab notebook", "beam deploy", "HF inference", "HF endpoints", "vast", "runpod".

GPU Runner

You are executing model inference on the appropriate GPU cloud provider.

Your Goal

Given a model, its requirements, and a chosen provider:

1. Write inference code tailored to the model
2. Execute it on the selected provider
3. Collect outputs (text, images, audio, metrics)
4. Handle errors and retry with alternatives if needed

Provider Selection (if not pre-selected)

Check .env for available credentials, then sort by cheapest hourly cost:

1. HF Inference API — Free with HF Pro. Requires HF_TOKEN. Catalog models only.
2. HF Inference Endpoints — Any HF model on dedicated GPU. HF_TOKEN only. $0.50–2.50/hr.
3. Colab Pro — Chrome MCP. No token needed. $9.99/month subscription. Up to ~30B.
4. Modal — Requires MODAL_TOKEN_ID + MODAL_TOKEN_SECRET. $30/month free tier. $0.59–3.95/hr.
5. beam.cloud — Requires BEAM_TOKEN. Existing credit. $0.54–3.50/hr.
6. Vast.ai — Requires VAST_API_KEY. Marketplace pricing (cheapest GPUs). $0.10–2.00/hr.
7. RunPod — Requires RUNPOD_API_KEY. Pods + Serverless. $0.34–2.69/hr.