runpod

name	runpod
description	Manage GPU pods, deploy serverless endpoints, and run inference workloads on RunPod
version	1.0.0
author	kai-agent
metadata	{"kai":{"tags":["kai","compute","runpod","gpu","pods","serverless","inference"]}}

RunPod

Skill for creating and managing GPU pods, deploying serverless inference endpoints, and running compute workloads on RunPod.

Authentication

This skill uses the RUNPOD_API_KEY environment variable. If it is not set, ask your admin to add it in Agent Settings. You can generate a key at https://www.runpod.io/console/user/settings.

Install

pip install runpod

Available GPU types

GPU	VRAM	Best for
RTX 3090	24 GB	Quantized models up to 13B
RTX 4090	24 GB	Fast inference, quantized models
A40	48 GB	Medium models, fine-tuning
A100 (40 GB)	40 GB	Large models, high throughput
A100 (80 GB)	80 GB	70B quantized, large batch inference
H100 (80 GB)	80 GB	Maximum throughput, largest models

GPU Pods

Create a pod

import runpod

runpod.api_key = os.environ["RUNPOD_API_KEY"]

pod = runpod.create_pod(
    name="vllm-llama3-8b",
    image_name="runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04",
    gpu_type_id="NVIDIA RTX A6000",
    gpu_count=1,
    volume_in_gb=50,
    container_disk_in_gb=20,
    ports="8000/http",
    env={
        "HF_TOKEN": os.environ.get("HF_TOKEN", ""),
    },
)
print(f"Pod ID: {pod['id']}")
print(f"Status: {pod['desiredStatus']}")

List available GPU types

gpu_types = runpod.get_gpus()
for gpu in gpu_types:
    print(f"{gpu['id']}: {gpu['displayName']} - {gpu['memoryInGb']}GB")

Get pod status

pod = runpod.get_pod(pod_id="your-pod-id")
print(f"Status: {pod['desiredStatus']}")
print(f"Runtime: {pod.get('runtime')}")

List all pods

pods = runpod.get_pods()
for pod in pods:
    print(f"{pod['id']}: {pod['name']} - {pod['desiredStatus']}")

Stop a pod

runpod.stop_pod(pod_id="your-pod-id")

Resume a stopped pod

runpod.resume_pod(pod_id="your-pod-id", gpu_count=1)

Terminate a pod

runpod.terminate_pod(pod_id="your-pod-id")

SSH access

When a pod is running, you can SSH into it:

# Get SSH command from pod details
ssh root@<pod-ip> -p <ssh-port> -i ~/.ssh/your_key

Or use the RunPod CLI:

runpodctl ssh --pod-id <pod-id>

Transfer files

# Upload
runpodctl send ./model-weights.tar.gz --pod-id <pod-id>

# Download from pod
runpodctl receive <pod-id>:/workspace/results.json ./results.json

Serverless endpoints

Create a serverless endpoint

Serverless endpoints auto-scale from zero and are billed per second of compute.

endpoint = runpod.create_endpoint(
    name="llama3-inference",
    template_id="your-template-id",
    gpu_ids="NVIDIA RTX A6000",
    workers_min=0,
    workers_max=3,
    idle_timeout=5,  # seconds before scaling down
)
print(f"Endpoint ID: {endpoint['id']}")

Build a serverless handler

Create a handler.py for your custom inference logic:

import runpod

def handler(event):
    """Serverless handler for inference requests."""
    input_data = event["input"]
    prompt = input_data.get("prompt", "")
    max_tokens = input_data.get("max_tokens", 256)

    # Your inference logic here (vLLM, TGI, custom model)
    result = run_inference(prompt, max_tokens)

    return {"output": result}

runpod.serverless.start({"handler": handler})

Dockerfile for serverless

FROM runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY handler.py .
COPY model/ ./model/

CMD ["python", "-u", "handler.py"]

Call a serverless endpoint

import runpod

endpoint = runpod.Endpoint("your-endpoint-id")

# Synchronous (waits for result)
result = endpoint.run_sync(
    {"input": {"prompt": "Explain tensor parallelism:", "max_tokens": 256}},
    timeout=60,
)
print(result)

# Asynchronous
run = endpoint.run({"input": {"prompt": "Explain KV-cache:", "max_tokens": 256}})
status = endpoint.status(run["id"])
# Poll until complete
output = endpoint.output(run["id"])

Stream results from a serverless endpoint

for chunk in endpoint.stream({"input": {"prompt": "Write a story:", "max_tokens": 1024}}):
    print(chunk, end="", flush=True)

Templates

Create a template (reusable pod config)

Templates define a pre-configured container image and settings:

template = runpod.create_template(
    name="vllm-serving",
    image_name="vllm/vllm-openai:latest",
    container_disk_in_gb=20,
    volume_in_gb=100,
    ports="8000/http",
    env={
        "MODEL_NAME": "meta-llama/Llama-3.1-8B-Instruct",
        "MAX_MODEL_LEN": "4096",
    },
)

Inference optimization patterns

1. Use serverless for variable traffic: Scale to zero when idle, scale up automatically under load.
2. Pre-bake models into Docker images: Avoid downloading weights on every cold start by including them in the container image.
3. Use network volumes: Mount a persistent volume with model weights shared across pods.
4. Right-size GPU memory: Use 24 GB GPUs for quantized sub-13B models, 80 GB for 70B+ models.
5. Batch requests: In serverless handlers, implement request batching to maximize GPU utilization.
6. Community Cloud vs. Secure Cloud: Community Cloud is cheaper but less reliable. Use Secure Cloud for production.

Rules

1. ALWAYS terminate pods when done: GPU pods bill per minute even when idle
2. Set idle_timeout on serverless endpoints to control scale-down behavior
3. Use templates for reproducible deployments instead of configuring pods manually each time
4. Never hardcode API keys in Docker images or handler code
5. Monitor GPU utilization: If utilization is consistently low, downsize the GPU or increase batch sizes