modal

name	modal
description	Define, deploy, and inspect Modal — serverless GPU functions, endpoints, volumes, and existing workspace deployments; also the default GPU runtime for Kai evaluators
version	2.1.0
author	kai-agent
metadata	{"kai":{"tags":["kai","compute","modal","gpu","serverless","inference","deployment","evaluator","inspection"]}}

Modal

Skill for defining new serverless GPU functions on Modal, inspecting existing workspace deployments, and using Modal as the GPU runtime for Kai code-optimization evaluators.

Three things this skill is for:

1. Define + deploy new GPU functions, web endpoints, volumes, secrets, schedules.
2. Inspect existing workspace deployments — enumerate apps, read their config, correlate with source repos, find optimization opportunities.
3. Run as a Kai evaluator — when start_code_optimization needs a GPU fitness function, the default path is a Modal scoring function invoked by a thin evaluator bridge (see kai-evolve/evaluator-creation).

Authentication

There are two modes — check which one applies before doing anything else.

Mode A — Kai workspace integration (preferred)

If the workspace has Modal connected via Agent Settings, the sandbox is already wired to Modal through a backend proxy. You do not configure anything. The following env vars are present automatically:

• MODAL_SERVER_URL — points at the backend's gRPC proxy, not Modal directly
• MODAL_TOKEN_ID / MODAL_TOKEN_SECRET — proxy credentials (not the real Modal tokens)

Just import modal and go. The proxy swaps the proxy credentials for the real workspace-scoped Modal tokens before forwarding each request, so real tokens never enter the sandbox. The proxy is rate-limited per workspace (~500 req/min), so prefer batch calls (.map(), @app.cls with warm containers) over tight loops of single invocations.

Check for this mode with:

import os
connected = bool(os.environ.get("MODAL_SERVER_URL"))

Mode B — User-provided tokens

If MODAL_SERVER_URL is not set but MODAL_TOKEN_ID and MODAL_TOKEN_SECRET are, those are real Modal tokens the user pasted. Treat them as sensitive — don't echo them, don't write them to disk.

If neither is set, tell the user: "I need Modal connected. Either connect it in Agent Settings (recommended — no tokens leave the backend) or paste a token from https://modal.com/settings." Don't try modal token set interactively — this environment isn't a TTY.

Install

pip install modal

Core concepts

Modal runs Python functions in containers on cloud GPUs. You define the environment declaratively in code, and Modal handles provisioning, scaling, and teardown.

Define a GPU function

Basic GPU function

import modal

app = modal.App("inference-benchmark")

image = modal.Image.debian_slim(python_version="3.11").pip_install(
    "vllm", "torch", "transformers"
)

@app.function(gpu="A100", image=image, timeout=600)
def run_benchmark(model_name: str, num_requests: int = 100):
    from vllm import LLM, SamplingParams

    llm = LLM(model=model_name)
    params = SamplingParams(temperature=0.7, max_tokens=256)

    prompts = ["Explain quantization:" for _ in range(num_requests)]
    outputs = llm.generate(prompts, params)

    return {
        "total_tokens": sum(len(o.outputs[0].token_ids) for o in outputs),
        "num_requests": num_requests,
    }

Specify GPU type and count

@app.function(gpu="A100-80GB")      # 80GB A100
@app.function(gpu="H100")           # H100
@app.function(gpu="A10G")           # A10G (cheaper, good for smaller models)
@app.function(gpu=modal.gpu.A100(count=2))  # Multi-GPU
@app.function(gpu="T4")             # T4 (cheapest, good for quantized models)

Run locally

modal run my_app.py

Run a specific function with arguments

modal run my_app.py::run_benchmark --model-name "meta-llama/Llama-3.1-8B-Instruct" --num-requests 500

Deploy a web endpoint

FastAPI-style endpoint

@app.function(gpu="A100", image=image, container_idle_timeout=300)
@modal.web_endpoint(method="POST")
def generate(request: dict):
    from vllm import LLM, SamplingParams

    llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
    params = SamplingParams(
        temperature=request.get("temperature", 0.7),
        max_tokens=request.get("max_tokens", 256),
    )
    outputs = llm.generate([request["prompt"]], params)
    return {"text": outputs[0].outputs[0].text}

ASGI app (full FastAPI)

from fastapi import FastAPI

web_app = FastAPI()

@web_app.post("/v1/completions")
async def completions(request: dict):
    # Your inference logic here
    return {"result": "..."}

@app.function(gpu="A100", image=image)
@modal.asgi_app()
def serve():
    return web_app

Deploy

modal deploy my_app.py

This returns a persistent URL. The endpoint auto-scales from zero.

Volumes (persistent storage)

Create and use a volume

model_volume = modal.Volume.from_name("model-cache", create_if_missing=True)

@app.function(
    gpu="A100",
    image=image,
    volumes={"/models": model_volume},
)
def download_model(model_name: str):
    from huggingface_hub import snapshot_download

    snapshot_download(
        repo_id=model_name,
        local_dir=f"/models/{model_name}",
    )
    model_volume.commit()  # Persist changes

Reuse cached models across functions

@app.function(
    gpu="A100",
    image=image,
    volumes={"/models": model_volume},
)
def run_inference(model_name: str, prompt: str):
    from vllm import LLM, SamplingParams

    llm = LLM(model=f"/models/{model_name}")
    outputs = llm.generate([prompt], SamplingParams(max_tokens=256))
    return outputs[0].outputs[0].text

Secrets management

Create a secret in Modal

modal secret create huggingface-secret HF_TOKEN=hf_xxxxx

Use secrets in functions

@app.function(
    gpu="A100",
    image=image,
    secrets=[modal.Secret.from_name("huggingface-secret")],
)
def download_gated_model():
    import os
    # HF_TOKEN is now available as an environment variable
    token = os.environ["HF_TOKEN"]
    # Download gated model...

Scheduling (cron jobs)

@app.function(schedule=modal.Period(hours=6), image=image)
def refresh_benchmarks():
    # Re-run inference benchmarks every 6 hours
    pass

Parallel execution

Map over inputs

@app.function(gpu="A10G", image=image)
def benchmark_single(config: dict):
    # Run a single benchmark configuration
    return {"config": config, "throughput": result}

@app.local_entrypoint()
def main():
    configs = [
        {"model": "llama-8b", "quant": "awq"},
        {"model": "llama-8b", "quant": "gptq"},
        {"model": "llama-8b", "quant": "fp16"},
    ]
    results = list(benchmark_single.map(configs))
    for r in results:
        print(r)

Container lifecycle (model preloading)

Use modal.enter() to load the model once when the container starts, avoiding reload on each request:

@app.cls(gpu="A100", image=image, volumes={"/models": model_volume})
class InferenceService:
    @modal.enter()
    def load_model(self):
        from vllm import LLM
        self.llm = LLM(model="/models/meta-llama/Llama-3.1-8B-Instruct")

    @modal.method()
    def generate(self, prompt: str, max_tokens: int = 256):
        from vllm import SamplingParams
        params = SamplingParams(max_tokens=max_tokens)
        outputs = self.llm.generate([prompt], params)
        return outputs[0].outputs[0].text

    @modal.web_endpoint(method="POST")
    def api(self, request: dict):
        return {"text": self.generate(request["prompt"])}

Inference optimization patterns

1. Volume caching: Download models once to a Volume, then mount it across functions to skip repeated downloads.
2. Container keep-alive: Set container_idle_timeout=300 to keep warm containers for faster cold starts.
3. Parallel benchmarking: Use .map() to benchmark multiple configurations across GPUs in parallel.
4. Class-based services: Use @app.cls with @modal.enter() to load the model once and serve many requests.
5. Right-size GPUs: Use A10G or T4 for quantized models, A100/H100 only when you need the memory or bandwidth.

Discover existing deployments

Defining new apps is only half the skill — most workspaces already have Modal apps in production, and you should be able to walk through what's there before recommending anything.

Enumerate deployed apps

modal app list

Returns every app in the workspace. Each row gives you App name, App ID, State (deployed / stopped), and the last-deploy timestamp. Use this at onboarding and whenever the user mentions Modal.

Programmatic enumeration (preferred inside the agent)

import modal
from modal.client import Client

client = Client.from_env()
apps = list(modal.App.list(client=client))  # returns [App, ...]

for app in apps:
    print(app.name, app.app_id, app.state)

Inspect a specific app's config

modal app show <app-name>

Or via SDK — this is what you want when auditing a deployment's GPU choice, timeouts, concurrency, and attached volumes:

from modal import App

app = App.lookup("my-app")               # existing deployed app
for fn_name in app.registered_functions: # function inventory
    fn = app.registered_functions[fn_name]
    print({
        "name": fn_name,
        "gpu": fn.gpu,
        "memory_mb": fn.memory,
        "timeout_s": fn.timeout,
        "container_idle_timeout_s": fn.container_idle_timeout,
        "keep_warm": fn.keep_warm,
        "volumes": [v.label for v in (fn.volumes or [])],
        "secrets": [s.label for s in (fn.secrets or [])],
    })

Logs (what's actually running)

modal app logs <app-name>              # tail recent logs
modal app logs <app-name> --since 1h   # last hour

Use logs to spot error patterns, cold-start latency, timeout spikes — concrete evidence for optimization proposals.

Correlate with repos

Every Modal app is usually defined in the user's source — decorators (@app.function, @app.cls, @modal.enter()) are searchable. To map a live app back to its source:

1. browse_repository_files(workspaceId, repoId) + grep for modal.App("<app-name>") to find the definition file.
2. Cross-reference the live function config (from app show) against the decorators in source to find drift (deployed has gpu="A100" but source uses "H100" → stale deployment).
3. Check for anti-patterns in source: missing @modal.enter(), missing container_idle_timeout, no Volume caching, single invocations where .map() would batch.

When you find an optimization opportunity, file a lifecycle_actions_create with linkedItems pointing to both the Modal app and the source file.

Inspection anti-patterns to flag

Use this checklist when scanning deployed Modal apps:

Anti-pattern	How to detect	Remediation
Over-provisioned GPU	fn.gpu == "A100" but model < 30B params and not latency-critical	Downsize to A10G or T4
Cold start tax	No @modal.enter() + model loaded in request handler	Move model load into @modal.enter(); add container_idle_timeout=300
Model re-downloaded	Model download in function body, no Volume mounted	Add modal.Volume.from_name(..., create_if_missing=True) with /models mount
Tight single-call loops	Caller invokes fn.remote() in a Python for	Replace with fn.map(inputs)
Hardcoded secrets	os.environ["HF_TOKEN"] = "hf_..." in source	Use modal.Secret.from_name("...")
Low timeout causing retries	Logs show recurring timeout errors	Increase timeout=; profile the slow path

Using Modal as a Kai evaluator runtime

When start_code_optimization needs a GPU for fitness scoring, the default path is:

1. Define a scorer as a Modal function (GPU type and image scoped to the target workload).
2. Deploy it with modal deploy scorer.py — registers it in Modal's function registry.
3. Write a thin evaluator bridge that calls it via modal.Function.lookup(...).remote(...).
4. Upload the bridge to Kai via create_evaluator_from_code.

Each iteration Kai runs, evaluate(program_path) reads the candidate, ships it to Modal, and returns the score. Zero user infrastructure, no HTTP endpoint to deploy, the proxy keeps credentials out of the sandbox.

Full template + setup is in the kai-evolve/evaluator-creation skill, "GPU Evaluator Pattern" section. Load it when you need a GPU evaluator — that's where the decision tree and concrete code templates live.

Prereq: Mode A (workspace integration) or Mode B (user-pasted tokens) must be active. If MODAL_SERVER_URL isn't set and MODAL_TOKEN_* aren't set, stop and ask the user to connect Modal before proposing GPU optimization. Don't fall back silently to the non-Modal paths — those are last resorts, not peers.

Rules

1. ALWAYS call volume.commit() after writing to a Volume
2. Use container_idle_timeout on endpoints to balance cost vs. cold start latency
3. Prefer @app.cls with @modal.enter() for inference services to avoid reloading the model per request
4. Never hardcode secrets in code: use modal.Secret and environment variables
5. Pin your image dependencies to avoid nondeterministic builds