// Debug and troubleshoot inference.sh apps. Use when facing import errors, CUDA issues, memory problems, or deployment failures.
Configure inf.yml for inference.sh apps. Use when setting GPU, VRAM, RAM, categories, environment variables, packages.txt, or resource requirements.
Handle graceful cancellation in inference.sh apps. Use when implementing long-running tasks that users might cancel.
Handle API keys and sensitive values in inference.sh apps. Use when adding secrets, accessing environment variables, or securing credentials.
Optimize inference.sh app performance. Use when handling memory, devices, model loading, mixed precision, or flash attention.
Build and deploy applications on inference.sh. Use when getting started, understanding the platform, or needing an overview of inference.sh development.
Track usage with output metadata in inference.sh apps. Use when implementing billing, counting tokens, or reporting image/video/audio generation metrics.
| name | debugging-issues |
| description | Debug and troubleshoot inference.sh apps. Use when facing import errors, CUDA issues, memory problems, or deployment failures. |
Common issues and solutions for inference.sh apps.
import sys, os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
For local packages: -e ./local_package in requirements.txt
torch.float16 or bfloat16model.gradient_checkpointing_enable()torch.cuda.empty_cache() after requestsvram in inf.ymlimport gc, torch
async def run(self, input_data):
result = self.process(input_data)
if torch.cuda.is_available():
torch.cuda.empty_cache()
gc.collect()
return result
input_tensor = input_tensor.to(self.device)
secrets:
- key: HF_TOKEN
description: HuggingFace token for gated models
tempfile.NamedTemporaryFile(suffix='.jpg', delete=False)
import logging
logging.basicConfig(level=logging.DEBUG)
š Full docs: inference.sh/docs/extend/troubleshooting