nim-model-testing

name	nim-model-testing
description	Test multiple NIM models against live API keys to determine which models are available, their latency, and token usage
source	auto-skill
extracted_at	2026-05-30T06:33:38.348Z

NIM Model Testing

When API keys are configured for an LLM provider (NVIDIA NIM), test multiple models to determine which are available on the user's tier, their latency, and token costs. This is done via a live API call per model with a minimal prompt.

Critical: Fetch the real model catalog first

Do not guess model IDs. NVIDIA's NIM catalog has specific naming conventions that differ from common model names. Always fetch the live catalog first:

curl -s -H "Authorization: Bearer $NVIDIA_NIM_KEY_ADAM" \
  https://integrate.api.nvidia.com/v1/models | python -c "
import sys,json
data=json.load(sys.stdin)
models=data.get('data',[])
print(f'Total models: {len(models)}')
for m in sorted(models, key=lambda x: x['id']):
    print(m['id'])
"

This returns the actual available models (e.g., 118 models). Key naming patterns discovered:

• GLM: z-ai/glm-5.1 (not zhipuai/glm-4-9b)
• MiniMax: minimaxai/minimax-m2.7 (not minimax/minimax-m1)
• StepFun: stepfun-ai/step-3.5-flash (not stepfun/step-1)
• GPT-OSS: openai/gpt-oss-120b (correct)
• Qwen: qwen/qwen3.5-122b-a10b (not qwen/qwen2.5-72b-instruct)

Procedure

1. Build the model list from the live catalog

After fetching the catalog, filter to chat/instruct models and build your test list:

NIM_MODELS_TO_TEST = [
    # Meta Llama
    "meta/llama-3.1-8b-instruct",
    "meta/llama-3.1-70b-instruct",
    "meta/llama-3.3-70b-instruct",
    "meta/llama-4-maverick-17b-128e-instruct",
    # Mistral / Mixtral
    "mistralai/mixtral-8x7b-instruct-v0.1",
    "mistralai/mistral-medium-3.5-128b",
    "mistralai/mistral-small-4-119b-2603",
    "mistralai/mistral-large-3-675b-instruct-2512",
    "mistralai/mistral-nemotron",
    # NVIDIA Nemotron
    "nvidia/nemotron-3-super-120b-a12b",
    "nvidia/llama-3.3-nemotron-super-49b-v1",
    # Qwen
    "qwen/qwen3.5-122b-a10b",
    "qwen/qwen3-coder-480b-a35b-instruct",
    "qwen/qwen3-next-80b-a3b-instruct",
    # GLM (Zhipu AI)
    "z-ai/glm-5.1",
    # GPT-OSS (OpenAI)
    "openai/gpt-oss-120b",
    "openai/gpt-oss-20b",
    # MiniMax
    "minimaxai/minimax-m2.7",
    # StepFun
    "stepfun-ai/step-3.5-flash",
    "stepfun-ai/step-3.7-flash",
    # DeepSeek
    "deepseek-ai/deepseek-v4-flash",
    "deepseek-ai/deepseek-v4-pro",
    # Microsoft Phi
    "microsoft/phi-3.5-moe-instruct",
    "microsoft/phi-4-mini-instruct",
    # Google Gemma
    "google/gemma-3-12b-it",
    "google/gemma-4-31b-it",
    # Writer Palmyra
    "writer/palmyra-creative-122b",
    # Moonshot
    "moonshotai/kimi-k2.6",
    # ByteDance
    "bytedance/seed-oss-36b-instruct",
]

2. Implement the test function

def test_models_live(agent: str = "Adam") -> list[dict]:
    key_env = "NVIDIA_NIM_KEY_ADAM" if agent == "Adam" else "NVIDIA_NIM_KEY_EVE"
    api_key = os.environ.get(key_env, "")

    if not api_key:
        return [{"model": "all", "status": "error", "message": f"No API key for {agent}"}]

    results = []
    base_url = "https://integrate.api.nvidia.com/v1"

    for model in NIM_MODELS_TO_TEST:
        result = {
            "model": model,
            "status": "testing",
            "latency_ms": None,
            "prompt_tokens": None,
            "completion_tokens": None,
            "total_tokens": None,
            "error": None,
        }

        try:
            payload = {
                "model": model,
                "messages": [{"role": "user", "content": "Reply with exactly: OK"}],
                "temperature": 0.1,
                "max_tokens": 10,
            }

            url = f"{base_url}/chat/completions"
            data = json.dumps(payload).encode("utf-8")
            req = urllib.request.Request(url, data=data, method="POST")
            req.add_header("Content-Type", "application/json")
            req.add_header("Authorization", f"Bearer {api_key}")

            start = time.monotonic()
            with urllib.request.urlopen(req, timeout=30) as response:
                raw = response.read().decode("utf-8")
                api_result = json.loads(raw)

            elapsed_ms = round((time.monotonic() - start) * 1000)
            usage = api_result.get("usage", {})
            choices = api_result.get("choices", [])

            result["status"] = "success"
            result["latency_ms"] = elapsed_ms
            result["prompt_tokens"] = usage.get("prompt_tokens")
            result["completion_tokens"] = usage.get("completion_tokens")
            result["total_tokens"] = usage.get("total_tokens")
            result["response_preview"] = choices[0].get("message", {}).get("content", "")[:50] if choices else ""

        except urllib.error.HTTPError as e:
            result["status"] = "http_error"
            result["error"] = f"HTTP {e.code}: {e.reason}"
        except urllib.error.URLError as e:
            result["status"] = "connection_error"
            result["error"] = str(e.reason)
        except Exception as e:
            result["status"] = "error"
            result["error"] = f"{type(e).__name__}"

        results.append(result)

    return results

3. Add API endpoint

@app.get("/api/test-models")
def get_test_models(agent: str = "Adam"):
    """Test multiple NIM models with live API calls. Takes ~30s per model."""
    return {"agent": agent, "results": test_models_live(agent)}

4. Create accessible test page

frontend/test-models.html — requirements:

• Large font (18px+), high contrast
• Agent selector dropdown (Adam/Eve)
• Large "Test All Models" button — disabled during testing
• Progress bar — shows testing in progress
• Summary cards — Working count, Failed count, Fastest latency
• Results table — Model, Status, Latency, Tokens, Response Preview, Error
• Color-coded status — green for success, red for errors

5. Key rules

• Minimal prompt — "Reply with exactly: OK" keeps token cost low
• Low temperature — 0.1 for deterministic responses
• Low max_tokens — 10 to minimize cost
• 30-second timeout per model — prevents hanging
• Never log API keys — only key_present: true/false
• Per-agent testing — Adam and Eve may have different keys and different available models
• HTTP 404 means model not available — common for models not on the user's tier

6. Typical results

Working models (on a standard NIM free tier):

• meta/llama-3.1-8b-instruct — ~400ms, ~42 tokens
• meta/llama-3.3-70b-instruct — ~300-500ms, ~42 tokens
• mistralai/mixtral-8x7b-instruct-v0.1 — ~500ms, ~24 tokens
• meta/llama-3.1-70b-instruct — ~400ms-13s, ~42 tokens

404 models (not available on free tier):

• meta/llama-3.1-405b-instruct
• mistralai/mistral-large-2-instruct
• google/gemma-2-9b-it
• google/gemma-2-27b-it
• nvidia/nemotron-4-340b-instruct
• qwen/qwen2.5-72b-instruct

Anti-Patterns

• ❌ Using long prompts for model testing (wastes tokens)
• ❌ High temperature or max_tokens (unnecessary cost)
• ❌ Testing without checking if key is configured first
• ❌ Logging API keys during model tests
• ❌ Assuming all models are available on every tier
• ✅ Use minimal prompt: "Reply with exactly: OK"
• ✅ Set temperature=0.1, max_tokens=10
• ✅ Show clear success/failure per model in UI
• ✅ Display latency and token usage for working models
• ✅ Test both agents separately (they may have different keys)