| name | vlm-configuration |
| description | Configure the Vision Language Model (VLM) used for analyzing experiment plots. Use when the user asks to change the VLM provider or model, adjust temperature or max_tokens, enable or disable thinking mode, or troubleshoot VLM-related behavior in config.yaml. |
VLM Configuration
The VLM (Vision Language Model) is configured in config.yaml under the vlm section.
Config Location
config.yaml (in the project root)
Current Config Structure
vlm:
provider: "nvidia"
model: "nvidia/ising-calibration-1-35b-a3b"
api_key_env: "NVIDIA_API_KEY"
temperature: 0.2
max_tokens: 32768
enable_thinking: false
Supported Models (NVIDIA API)
| Model | Provider | Notes |
|---|
nvidia/ising-calibration-1-35b-a3b | NVIDIA | Default, fine-tuned for calibration |
google/gemma-4-31b-it | Google | General-purpose 31B |
meta/llama-3.2-90b-vision-instruct | Meta | Most capable |
meta/llama-3.2-11b-vision-instruct | Meta | Balanced |
microsoft/phi-3.5-vision-instruct | Microsoft | Lightweight |
Environment Variables
The VLM uses NVIDIA_API_KEY (same key as the main LLM).
To Change VLM
- Read current config.yaml
- Edit the
vlm section with new values
- Changes take effect immediately on next
vlm_inspect call (no restart needed)
