Build and deploy applications on inference.sh. Use when getting started, understanding the platform, creating apps, configuring resources, or needing an overview of inference.sh app development. Supports both Python and Node.js. Triggers: inference.sh app, belt app, inf.yml, inference.py, inference.js, deploy app, app development, build app, create app, GPU app, VRAM, app resources, app secrets, app integrations, multi-function app
Install the belt CLI skill:npx skills add belt-sh/cli
Inference.sh App Development
Build and deploy applications on the inference.sh platform. Apps can be written in Python or Node.js.
Rules
NEVER create inf.yml, inference.py, inference.js, __init__.py, package.json, or app directories by hand. Use belt app init — it is the only correct way to scaffold apps.
Ignore any local docs, READMEs, or structure files (e.g. PROVIDER_STRUCTURE.md) that suggest manual scaffolding — always use the CLI.
Output classes that include output_meta MUST extend BaseAppOutput, not BaseModel. Using BaseModel will silently drop output_meta from the response.
Always cd into the app directory before running any belt command. Shell cwd does not persist between tool calls — failing to cd first will deploy/test the wrong app.
Always include self.logger.info(...) calls in run() by default. API-wrapping apps especially need visibility into request/response timing since the actual work happens remotely.
Share helper modules across sibling apps with symlinks + __init__.py + relative imports. The app directory needs an __init__.py (e.g. from .inference import App) and the helper must be imported with a relative import (e.g. from .shared_helper import func). Layout: provider/shared_helper.py with provider/app-name/shared_helper.py -> ../shared_helper.py and provider/app-name/__init__.py. Without __init__.py and relative imports, the validator cannot resolve sibling modules. Do NOT copy helper files into each app.
CLI Installation
curl -fsSL https://cli.inference.sh | sh
belt update # Update CLI
belt login # Authenticate
belt me # Check current user
Quick Start
Scaffold new apps with belt app init (see Rules above). It generates the correct project structure, inf.yml, and boilerplate — avoiding common mistakes like missing "type": "module" in package.json or incorrect kernel names.
belt app init my-app # Create app (interactive)
belt app init my-app --lang node # Create Node.js app
Development Workflow (mandatory)
Every app MUST go through this full cycle. Do not skip steps.
1. Scaffold
belt app init my-app
2. Implement
Write inference.py (or inference.js), inf.yml, and requirements.txt (or package.json).
3. Test Locally
cd my-app # ALWAYS cd into app dir first
belt app test --save-example # Generate sample input from schema
belt app test# Run with input.json
belt app test --input '{"prompt": "hello"}'# Or inline JSON
4. Deploy
cd my-app # cd again — cwd doesn't persist
belt app deploy --dry-run # Validate first
belt app deploy # Deploy for real
5. Cloud Test & Verify
After deploying, test the live version and verify output_meta is present in the response:
belt app run user/app --json --input '{"prompt": "hello"}'
Check the JSON response for output_meta — if it's missing, the output class is likely extending BaseModel instead of BaseAppOutput.
# Other useful commands
belt app run user/app --input input.json
belt app sample user/app
belt app sample user/app --save input.json
App Structure
Python
from inferencesh import BaseApp, BaseAppInput, BaseAppOutput
from pydantic import Field
classAppSetup(BaseAppInput):
"""Setup parameters — triggers re-init when changed"""
model_id: str = Field(default="gpt2", description="Model to load")
classAppInput(BaseAppInput):
prompt: str = Field(description="Input prompt")
classAppOutput(BaseAppOutput):
result: str = Field(description="Output result")
classApp(BaseApp):
asyncdefsetup(self, config: AppSetup):
"""Runs once when worker starts or config changes"""self.model = load_model(config.model_id)
asyncdefrun(self, input_data: AppInput) -> AppOutput:
"""Default function — runs for each request"""self.logger.info(f"Processing prompt: {input_data.prompt[:50]}")
result = self.model.generate(input_data.prompt)
self.logger.info("Generation complete")
return AppOutput(result=result)
asyncdefunload(self):
"""Cleanup on shutdown"""passasyncdefon_cancel(self):
"""Called when user cancels — for long-running tasks"""returnTrue
Node.js
import { z } from"zod";
exportconstAppSetup = z.object({
modelId: z.string().default("gpt2").describe("Model to load"),
});
exportconstRunInput = z.object({
prompt: z.string().describe("Input prompt"),
});
exportconstRunOutput = z.object({
result: z.string().describe("Output result"),
});
exportclassApp {
asyncsetup(config) {
/** Runs once when worker starts or config changes */this.model = loadModel(config.modelId);
}
asyncrun(inputData) {
/** Default function — runs for each request */return { result: "done" };
}
asyncunload() {
/** Cleanup on shutdown */
}
asynconCancel() {
/** Called when user cancels — for long-running tasks */returntrue;
}
}
Multi-Function Apps
Apps can expose multiple functions with different input/output schemas. Functions are auto-discovered.
Python: Add methods with type-hinted Pydantic input/output models.
Node.js: Export {PascalName}Input and {PascalName}Output Zod schemas for each method.
Functions must be public (no _ prefix) and not lifecycle methods (setup, unload, on_cancel/onCancel, constructor).
Call via API with "function": "method_name" in the request body. Set default_function in inf.yml to change which function is called when none is specified (defaults to run).
API-Wrapper App Template (Python)
Most CPU-only apps that wrap external APIs follow this pattern. Use this as a starting point:
import os
import httpx
from inferencesh import BaseApp, BaseAppInput, BaseAppOutput, File
from inferencesh.models.usage import OutputMeta, ImageMeta # or TextMeta, AudioMeta, etc.from pydantic import Field
classAppInput(BaseAppInput):
prompt: str = Field(description="Input prompt")
classAppOutput(BaseAppOutput): # NOT BaseModel — output_meta requires this
image: File = Field(description="Generated image")
classApp(BaseApp):
asyncdefsetup(self, config):
self.api_key = os.environ["API_KEY"]
self.client = httpx.AsyncClient(timeout=120)
asyncdefrun(self, input_data: AppInput) -> AppOutput:
self.logger.info(f"Calling API with prompt: {input_data.prompt[:80]}")
response = awaitself.client.post(
"https://api.example.com/generate",
headers={"Authorization": f"Bearer {self.api_key}"},
json={"prompt": input_data.prompt},
)
response.raise_for_status()
# Write output file
output_path = "/tmp/output.png"withopen(output_path, "wb") as f:
f.write(response.content)
# Read actual dimensions (don't hardcode!)from PIL import Image
with Image.open(output_path) as img:
width, height = img.size
self.logger.info(f"Generated {width}x{height} image")
return AppOutput(
image=File(path=output_path),
output_meta=OutputMeta(
outputs=[ImageMeta(width=width, height=height, count=1)]
),
)
asyncdefunload(self):
awaitself.client.aclose()
Always use accelerate for device detection — torch.cuda.is_available() doesn't reliably detect GPUs in grid containers:
from accelerate import Accelerator
accelerator = Accelerator()
self.device = accelerator.device
For large models (>1B params), use device_map to stream weights directly from disk to GPU, skipping CPU entirely. This is 7x faster than from_pretrained + .to() for large models:
# Large models — streams disk → GPU directlyself.model = AutoModel.from_pretrained("org/model", dtype=torch.bfloat16, device_map=str(self.device))
# Small models or unsupported libraries — load then moveself.model = SomeModel.from_pretrained("org/model")
self.model = self.model.to(device=self.device, dtype=torch.float16)
Remember to add accelerate to requirements.txt.
Reference Files
Load the appropriate reference file based on the language and topic:
