| name | model-routing |
| description | Dynamic model routing for task-specific optimization. Selects optimal model based on task type, priority, and cost. |
| version | 1.0.0 |
| author | Hermes Agent |
| license | MIT |
| platforms | ["linux","macos"] |
| metadata | {"hermes":{"tags":["llm","routing","optimization","cost","speed"]}} |
Model Routing
Dynamic model routing for real business workloads. Use stable profile lanes first, task-level overrides second, and delegation/specialized overrides only when the work clearly justifies them.
Overview
This skill's operating model is:
- Profile lane first — every profile should have a stable default model pair.
- Task override second — only override the lane when the task clearly breaks the profile's normal workload.
- Delegation override third — for coding bursts, deep research, or other specialized execution, explicitly set the worker/delegation model instead of letting subagents inherit accidentally.
The goal is not "pick the perfect model for every single prompt." The goal is a routing system that is simple, predictable, and aligned with business stakes.
Older Ollama/GLM/MiniMax benchmark sections below are retained as historical evidence and failure-mode documentation. They are not the primary live routing policy.
Current operating model (2026-06-29)
Tier plan (beta: ollama-cloud substitutes for Nous until 7/4 cutover)
Authoritative policy: ~/.hermes/profiles/sirvir/skills/sirvir/references/fleet-routing-policy.md
Three-tier lane routing with hybrid provider strategy:
Premium lane
Use for business-critical reasoning, research, strategic planning, and orchestration.
Profiles:
Model pair (beta):
- main:
ollama-cloud / glm-5.2
- compression:
ollama-cloud / glm-5.2
- vision:
nvidia / minimaxai/minimax-m3 (NIM free)
- web_extract:
nvidia / minimaxai/minimax-m3 (NIM free)
After 7/4 cutover:
- main:
nous / glm-5.2
- compression:
nous / glm-5.2
Default lane
Use for implementation, operations, infra management, and production work.
Profiles:
rollout-autosirvirgarry-builds
Model pair:
- main:
ollama-cloud / deepseek-v4-pro
- compression:
ollama-cloud / deepseek-v4-pro
- vision:
nvidia / minimaxai/minimax-m3 (NIM free)
- web_extract:
nvidia / minimaxai/minimax-m3 (NIM free)
Cheap lane
Use for comms, formatting, low-stakes drafting, and routine admin.
Profiles:
commsforge-framepersonal
Model pair:
- main:
ollama-cloud / deepseek-v4-flash
- compression:
ollama-cloud / deepseek-v4-flash
- vision:
nvidia / minimaxai/minimax-m3 (NIM free)
- web_extract:
nvidia / minimaxai/minimax-m3 (NIM free)
Fallback (all tiers)
openai-codex / gpt-5.4 — expires 7/4, will be re-evaluated at cutover
Why this lane split
- Premium profiles get the strongest text model (glm-5.2) for reasoning and compression
- Default profiles get deepseek-v4-pro — solid quality at same cost tier as premium on ollama-cloud
- Cheap profiles get deepseek-v4-flash — cheapest acceptable lane for low-stakes work
- Vision and web_extract standardized fleet-wide on nvidia/minimax-m3 (NIM free, good multimodal)
- Compression split off from vision/web_extract — different requirements (text quality vs multimodal)
default acts as premium command center for synthesis and routing
Coding delegation override
For coding bursts dispatched via delegate_task, set delegation.model explicitly:
kimi-k2.7-code — preferred coding model (6/25 benchmark: 10.1s avg, 100% success, clean output)qwen3-coder:480b — alternative (3.45s in June benchmark, but variable latency on ollama cloud)glm-5.2 — fallback if neither coder model is available
Always clear delegation.model after the coding session to avoid wrong-model inheritance.
Workload enhancement rules
1. Use lanes as the baseline operating system
Do not start from task-type routing every time. Start from the profile's lane and only override when needed.
2. Use task routing for exceptions, not for every prompt
Examples:
- a light-lane profile doing a strategic memo may deserve a premium override
- a premium profile doing repetitive low-stakes cleanup may stay on its default lane unless cost pressure is real
3. Keep one aux model everywhere unless there is a strong reason not to
Current standard aux:
Why:
- 1M context
- vision-capable
- cheap relative to premium mains
- operationally simpler than maintaining different aux rules per profile
4. Treat delegation as a separate execution layer
The parent chat model and the delegated worker model do not have to be the same.
Use explicit delegation overrides for:
- coding bursts
- PR review / implementation sessions
- bulk research extraction
- other specialized execution where inheritance would pick the wrong model
5. Optimize for stability over theoretical perfection
A stable routing system with a few deliberate overrides is more valuable than constantly reshuffling models based on minor benchmark deltas.
Workload classes
Use these business-aligned classes before falling back to narrow benchmark categories:
| Workload class | Typical work | Default routing |
|---|
| Executive / strategic | Decisions, synthesis, planning, tradeoffs | Premium lane |
| Client / business deliverable | Proposals, research briefs, financial analysis, important brand copy | Premium lane |
| Operational execution | Follow-ups, triage, formatting, coordination, admin | Light lane |
| Specialized coding / execution | Repo work, debugging, PR review, builds, engineering bursts | Delegation override / specialized model |
Special use case models
These are not the baseline profile models. They are explicit overrides for special workloads.
| Use case | Recommended model | Role | Notes |
|---|
| Coding delegation / production code | kimi-k2.7-code | specialized worker | Preferred when dispatching coding-heavy subagents or implementation bursts |
| Vision-heavy analysis | qwen/qwen3.5-flash-02-23 | aux / multimodal helper | Default shared aux because it is cheap, vision-capable, and operationally simple |
| Long-context / document-heavy analysis | premium lane first; escalate to long-context specialist only when needed | conditional override | Do not escalate automatically; only when the profile lane is clearly insufficient |
| Math / proof-style reasoning | use only as an explicit override after validation | experimental specialist | Historical candidates exist below, but treat them as opt-in, not default routing |
| Chain-of-thought specialists | use only as an explicit override after validation | experimental specialist | Slower and more expensive; avoid for normal reasoning |
Rule:
- If a special-use model is not clearly better for the current job, stay on the profile lane.
Pitfall: reviewing a routing skill is not the same as editing it
If the user asks to "look at" or "reflect on" the routing skill, treat the skill as decision-support context first. Do not edit the skill just because it was consulted.
Correct sequence:
- Read the routing skill and summarize what still applies vs what is outdated.
- Propose the workload or routing improvement in plain language.
- Wait for explicit user approval before editing
SKILL.md, sync scripts, or profile copies.
Why this matters:
- consulting a skill and modifying a skill are different actions
- unsolicited edits create churn and can make the user feel like analysis turned into mutation without consent
- the best first deliverable is usually a cleaned-up recommendation, not an immediate patch
Pitfall: profile renames require routing-skill maintenance
When a business profile is renamed, update the model-routing skill immediately if it carries any hardcoded profile lists or examples.
Minimum follow-up after a rename:
- Update every
PROFILES = [...] list in this skill's sync scripts.
- Update the live routing map in
SKILL.md to use the new slug.
- Re-sync the skill to sub-profile copies so future sessions do not keep teaching the old name.
- If the renamed profile has a managed gateway, restart it and verify the new profile slug is the one being supervised.
Example learned here:
construction was renamed to forge-frame- the sync scripts still referenced
construction
- the gateway supervision slot also needed re-registration under the new slug before restart
Watchdog alignment rule
The Provider Health Watchdog is a separate operational layer from the routing policy in this skill.
Current rule:
- if the watchdog still manages an older provider failover path (for example
ollama-cloud -> openai-codex), keep its primary-model map aligned to the LIVE deployed stack
- do not update the watchdog to a future routing plan until that plan is actually wired into profile config and credentials
- when the routing plan changes materially, update all three together in the same turn:
- this skill's live routing map
~/.hermes/profiles/sirvir/skills/sirvir/scripts/provider_watchdog.py~/.hermes/profiles/sirvir/skills/sirvir/scripts/provider_watchdog.py
This avoids a bad state where the skill teaches one routing policy, the watchdog restores another, and cron silently undoes intended model changes.
Pitfall: review requests are not permission to edit the skill
When the user asks to review, reflect on, or critique the routing skill, treat that as an analysis request first.
Correct sequence:
- Read the current skill and summarize what it implies for the live workload.
- Call out what is outdated, conflicting, or operationally expensive.
- Propose the cleaner routing policy.
- Only patch the skill after the user explicitly asks to save or update it.
Why this matters:
- unsolicited edits create churn and can make the user feel like analysis turned into mutation without consent
- the best first deliverable is usually a cleaned-up recommendation, not an immediate patch
When to Use
- When defining or reviewing profile-level routing
- Before dispatching specialized delegated work
- When deciding whether a task should stay on its profile lane or use an override
- When cost, quality, or operational simplicity need to be traded off deliberately
Quick Start
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --task financial --priority speed
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --prompt "Write brand voice for..." --priority quality
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --list
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/router_integration.py --task coding --priority quality --hermes-cmd
Business Domains → Task Types
Map business use cases to optimal models. Not just task-type based — consider domain context.
| Business Domain | Use Cases | Task Type | Priority |
|---|
| Marketing | Campaign strategy, social planning, SEO | creative | balanced |
| Marketing | Image/video prompts (multimodal) | vision | quality |
| Sales | Outreach, follow-ups, CRM enrichment | creative | speed |
| Sales | Pipeline analytics | reasoning | balanced |
| Operations | Scheduling, logistics, fleet coordination | operations | speed |
| Operations | Capacity planning, forecasting | financial | balanced |
| Creative | Social media, video scripts, design prompts | creative | quality |
| Creative | Brand voice, tone guides | creative | quality |
| Financial | Modeling, IRR, valuation | financial | quality |
| Financial | Audit, compliance review | financial | quality |
| Financial | Quick calculations, ratios | quick | speed |
| Technical | Architecture, debugging, features | coding | balanced |
| Technical | Client deliverables (production) | coding | quality + high-stakes |
Task Types
| Type | Description | Keywords |
|---|
| financial | Revenue, EBITDA, IRR, valuation | "revenue", "profit", "cash flow", "investment" |
| creative | Brand, copy, marketing, social | "brand voice", "email", "headline", "copy" |
| coding | Implementation, debugging, API | "function", "algorithm", "debug", "refactor" |
| operations | Scheduling, logistics, routing | "schedule", "dispatch", "optimize", "capacity" |
| quick | Simple questions, conversions | "what is", "calculate", "convert", "capital of" |
| reasoning | Analysis, comparison, strategy | "analyze", "compare", "should i", "strategy" |
Priorities
| Priority | Optimization | Best For |
|---|
| speed | Latency first | Client calls, real-time work |
| balanced | Equal weights | General use |
| cost | Subscription preference | High volume, drafts |
| quality | Output quality | Final deliverables |
Tier Strategy
Models fall into three cost-quality tiers. Always benchmark before assigning a model to a tier — don't assume based on name or reputation.
| Tier | Models | Best For | Token Cost | Notes |
|---|
| Cheap | deepseek-v4-flash | Simple queries, personal tasks, quick lookups | Lowest | Hallucinates on real thinking tasks. Don't use for financial, coding, or anything requiring accuracy. |
| Great+cheap | minimax-m2.5 | Conversation, orchestration, operations, quick tasks | Low | ⚠️ 34.8s on financial prompts, 50% success on coding. Good for ops/quick only. Always benchmark before expanding scope. |
| Quality | deepseek-v4-pro, kimi-k2:1t, qwen3-coder:480b | Highest | MAF/research/garry-builds defaults. Token-expensive but reliable. | |
| Quality (CONFIRMED 6/23) | glm-5.2 | Highest | Wins 3/4 categories in 3-way benchmark vs DS4Pro + M3 — 6.8× faster on coding, 2.5× faster on financial, sub-second on quick. Now the default profile model as of 6/24. | |
| Quality (WORKER 6/24) | deepseek-v4-pro | High | Replaced M3 on all 7 Diwan worker profiles. Beats M3 on latency in all 4 categories (1.3-2.9x). Same ollama-cloud cost tier. See references/benchmark-3way-2026-06-23.md. | |
Tier Assignment Rules (Hard Rules — Not Suggestions)
- BENCHMARK BEFORE ASSIGNING — never add a model to routing tables without running it through
benchmark_fast.py first. A model that sounds "great and cheap" (e.g. minimax-m2.5) may be 34s on financial or 50% on coding. Do not assume from hearsay or marketing. Run the test.
- Test across ALL task types — a model that excels at quick ops may fail at financial or coding. Run the full benchmark suite across financial, creative, coding, and quick categories, not just one.
- Update routing tables immediately — after benchmarking, update the SKILL.md routing tables and sync to all sub-profiles. Stale tables cause bad routing decisions.
- Document failure modes explicitly — if a model fails a category (high latency, low success, or partial success), note it in the routing table so future sessions don't repeat the mistake.
- Prefer the model name from the provider's API — "MiMo v2.5 Pro" might not match
minimax-m2.5 on Ollama. Verify the exact model ID from /v1/models before benchmarking.
Pitfall: Assigning Without Testing
Scenario: Someone recommends a model as "great and cheap" for a specific use case. You add it to routing tables without testing.
What goes wrong: The model may have 34s latency on financial prompts, 50% success on coding, or other hidden failure modes that only surface under your actual workloads.
Fix: Always run benchmark_fast.py first and inspect the results before adding any model to routing tables. If the user asks "why is this model in the table?", the answer should be "because we benchmarked it", not "because it sounded good".
Pitfall: Not checking the routing skill before dispatching coding subagents
Symptom (verified 2026-06-25): Controller dispatched delegate_task subagents for a production dashboard build without checking the model-routing skill. Subagents inherited glm-5.2 (the orchestrator model) instead of qwen3-coder:480b (the benchmarked coding champion at 3.45s). User caught it.
Why it happens: delegate_task can't override the model per-call. If delegation.model is empty in config.yaml, children inherit the parent. The controller didn't load the routing skill before starting the build, so it didn't know to set delegation.model first.
Prevention — before dispatching ANY subagent for coding/financial/creative work:
- Load the model-routing skill (
skill_view(name='model-routing')) and check the routing table for the task type
- Set
delegation.model and delegation.provider in config.yaml to the routed model for that task type
- Verify the config took effect with
hermes config get delegation.model
- Then dispatch — all subagents in that session will use the correct model
- After the build session, reset
delegation.model to empty if the next batch of work is a different task type
Rule: the routing skill is a pre-flight check for delegation, not just for /model slash commands. If you're about to delegate_task and you haven't checked the routing table, you're guessing.
Auxiliary Model Architecture
Hermes supports separating the main chat model from auxiliary models used for side tasks (summarization, web page extraction, browser screenshots, session title generation, context compression).
This is a recommended pattern for token efficiency — use a quality model for thinking/conversation and a cheap model for grunt work.
Recommendation (updated 2026-06-29)
Tier plan (beta: ollama-cloud substitutes for Nous until 7/4 cutover):
- Premium main (default, maf, research):
ollama-cloud / glm-5.2 — conversation, orchestration, routing, reasoning, compression
- Default main (rollout-auto, sirvir, garry-builds):
ollama-cloud / deepseek-v4-pro — domain work, implementation, operations
- Cheap main (comms, forge-frame, personal):
ollama-cloud / deepseek-v4-flash — comms, formatting, low-stakes admin
- Compression (all profiles): same as tier main model — split off from vision/web_extract
- Vision + web_extract (all profiles):
nvidia / minimaxai/minimax-m3 (NIM free, good multimodal)
- Utility aux (skills_hub, approval, mcp, title_generation):
nvidia / minimaxai/minimax-m3
- Workflow aux (triage_specifier, kanban_decomposer, profile_describer, curator):
auto (inherit main model)
- Kanban decomposer + triage specifier: inherit main model via
auto
- Fallback (all tiers):
openai-codex / gpt-5.4 (expires 7/4)
Why: GLM-5.2 wins 3/4 benchmark categories (financial, coding, quick) including sub-second latency on quick tasks. DS v4 Pro is the solid middle lane. DS v4 Flash is the cheapest acceptable lane. M3 retained for vision/web_extract because it's free via NIM and verbose output is acceptable for side tasks. Compression moved off M3 to the tier text model because compression needs reliable instruction-following, not multimodal capability.
After 7/4 cutover:
- Premium main + compression:
nous / glm-5.2
- Default and cheap lanes may stay on ollama-cloud or move to NIM depending on cost analysis
- Vision + web_extract: stay on
nvidia / minimax-m3
Configuration
By default, all auxiliary tasks use the main chat model (auxiliary.*.provider: "auto"). To override:
auxiliary:
ag:
provider: ollama-cloud
model: deepseek-v4-flash
web:
provider: ollama-cloud
model: deepseek-v4-flash
vision:
provider: ollama-cloud
model: qwen3-vl:235b
compress:
provider: ollama-cloud
model: minimax-m3
For per-profile configuration, add the auxiliary section to each profile's config.yaml.
When to Apply
- All profiles — every profile benefits from aux model separation
- Especially quality-tier profiles (maf, research, garry-builds) — these default to expensive models and burn tokens fastest on side tasks
- Skip if the profile's main model is already cheap (personal with deepseek-v4-flash doesn't gain anything)
Reference
See Hermes docs: https://hermes-agent.nousresearch.com/docs/user-guide/configuration#auxiliary-models
Historical benchmark archive
This section is preserved for benchmark evidence, prior routing decisions, and debugging context. It is not the live routing policy. For live operation, use ## Current operating model (2026-06-26) above.
Based on older June benchmark runs and older provider assumptions.
Historical model assignments (as of 2026-06-25)
| Role | Model | Provider | Profiles | Notes |
|---|
| Default/orchestrator | glm-5.2 | ollama-cloud | default | Wins 3/4 benchmark categories. Fallback: openai-codex/gpt-5.4 |
| Diwan workers (×7) | deepseek-v4-pro | ollama-cloud | comms, construction, garry-builds, maf, personal, research, rollout-auto | Replaced M3 on 6/24. Beats M3 on all 4 categories. Fallback: openai-codex/gpt-5.4 |
| Coding (delegation) | kimi-k2.7-code | ollama-cloud | — | 6/25 benchmark winner. 10.1s avg, 100% success, 1446 avg tokens. Set via delegation.model when all subagents are coding. Clear after session. |
| Kanban decomposer | glm-5.2 | ollama-cloud | — | Pinned via config |
| Triage specifier | glm-5.2 | ollama-cloud | — | Pinned via config |
| Auxiliary vision | minimax-m3 | ollama-cloud | all | Verbose output acceptable for vision tasks |
| Auxiliary web_extract | minimax-m3 | ollama-cloud | all | Verbose output acceptable for extraction |
| Delegate_task subagents | (inherits parent or delegation.model) | — | — | Set delegation.model for task-specific routing; clear after use |
Coding benchmark (6/25, 3 prompts × 3 models):
| Model | Avg latency | Avg tokens | Success |
|---|
kimi-k2.7-code | 10.1s | 1,446 | 100% |
glm-5.2 | 9.9s | 1,819 | 100% |
qwen3-coder:480b | 58.5s | 1,103 | 100% |
kimi-k2.7-code is the recommended coding model: clean output, good on async SQLAlchemy, more token-efficient than glm-5.2. qwen3-coder:480b was fastest in June (3.45s) but slow on 6/25 (85.5s on one prompt) — ollama cloud latency varies.
Historical change history:
- 6/25: Coding benchmark re-run with kimi-k2.7-code, qwen3-coder:480b, glm-5.2. kimi-k2.7-code recommended for delegation coding tasks.
- 6/24: All 7 Diwan profiles flipped from
minimax-m3 to deepseek-v4-pro based on 3-way benchmark showing DS beats M3 on all 4 categories at same cost tier.
- 6/23: 3-way benchmark confirmed GLM-5.2 as top performer (3/4 categories). Default profile already on GLM-5.2.
- 6/16: All profiles on M3 + DS4F aux strategy (now superseded).
Speed Priority
| Task | Model | Provider | Latency | Notes |
|---|
| Financial | glm-5.2 | ollama-cloud | 3.49s | 2.5x faster than DS4Pro, 3.4x faster than M3 |
| Creative | deepseek-v4-pro | ollama-cloud | 4.57s | 1.3x faster than M3, most concise (628 tok) |
| Coding | kimi-k2.7-code | ollama-cloud | 10.1s avg | 6/25 benchmark: 4.5s basic, 7.7s API, 18.1s system. Was glm-5.2 in 3-way (qwen not tested). |
| Operations | deepseek-v4-pro | ollama-cloud | — | Workers now on DS4Pro; ops is reasoning-adjacent |
| Quick | glm-5.2 | ollama-cloud | 0.78s | Sub-second, lightest tokens (96) |
| Reasoning | deepseek-v4-pro | ollama-cloud | — | Workers on DS4Pro; DS4Pro wins reasoning in monthly matrix |
Cost Priority (Ollama — Flat Subscription, as of 2026-06-25)
All models below are on ollama-cloud (tier 1, flat subscription). Cost is identical across models.
| Task | Model | Provider | Latency | Notes |
|---|
| Financial | glm-5.2 | ollama-cloud | 3.49s | Fastest + fewest tokens (1035) |
| Creative | deepseek-v4-pro | ollama-cloud | 4.57s | Fastest + fewest tokens (628) |
| Coding | kimi-k2.7-code | ollama-cloud | Best coding model (6/25 benchmark, 10.1s avg, 1446 tok avg). GLM-5.2 close (9.9s, 1819 tok). qwen3-coder:480b was 58.5s avg — not viable. | |
| Operations | deepseek-v4-pro | ollama-cloud | — | Workers on DS4Pro |
| Quick | glm-5.2 | ollama-cloud | 0.78s | Sub-second, 96 tokens |
| Reasoning | deepseek-v4-pro | ollama-cloud | — | Retained from monthly matrix |
Balanced (as of 2026-06-25)
| Task | Model | Provider | Notes |
|---|
| Financial | glm-5.2 | ollama-cloud | Fastest + cleanest output. Default profile already uses GLM-5.2 |
| Creative | deepseek-v4-pro | ollama-cloud | Most concise (628 tok). Workers on DS4Pro |
| Coding | qwen3-coder:480b | ollama-cloud | 3.45s, 780 tok, 100% success in June benchmark — actually the fastest coding model, not GLM-5.2. GLM-5.2 (6.75s) only "won" the 3-way because qwen3-coder was excluded from that run. |
| Operations | deepseek-v4-pro | ollama-cloud | Workers on DS4Pro |
| Quick | glm-5.2 | ollama-cloud | Sub-second, lightest tokens |
| Reasoning | deepseek-v4-pro | ollama-cloud | Workers on DS4Pro |
Model strategy (6/25):
- Default/orchestrator:
glm-5.2 — fastest on 3/4 categories, already set
- 7 Diwan workers:
deepseek-v4-pro — beats M3 on all 4 categories, same cost tier
- Coding delegation:
kimi-k2.7-code — 6/25 coding benchmark winner (10.1s avg, 100% success, clean output). Set delegation.model when dispatching coding subagents; clear after session.
- Auxiliary (vision, web_extract):
minimax-m3 — verbose but acceptable for side tasks
- Kanban decomposer + triage specifier:
glm-5.2
- Fallback: all profiles →
openai-codex / gpt-5.4 if ollama-cloud goes down
- delegate_task subagents: set
delegation.model for task-specific routing; clear after use to avoid wrong-model delegation. Pre-flight rule: check routing table before dispatching coding subagents — set delegation.model: qwen3-coder:480b + delegation.provider: ollama-cloud for coding work. kimi-k2.7-code is Garry's preferred coding model to test. Available coding models on ollama: qwen3-coder:480b, kimi-k2.7-code, qwen3-coder-next, qwen3.5:397b.
Appendix A — Historical routing patterns and tooling
The rest of this skill is reference material. Use it when re-benchmarking, debugging older routing decisions, or reviving legacy tooling.
Appendix map:
- Appendix A — older routing patterns, switching flows, and cost reasoning
- Appendix B — benchmark maintenance cadence
- Appendix C — historical pitfalls and troubleshooting
- Appendix D — specialized override reference
- Appendix E — files and reference artifacts
- Appendix F — evaluation methodology
Do not treat these appendices as the default live routing policy unless the current operating model section above explicitly points you here.
A1. High-Stakes Mode
For client deliverables, production code, or high-impact work:
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --task financial --high-stakes --priority quality
High-stakes mode:
- Prefers OpenAI (established quality record)
- Boosts quality weight in scoring
- Avoids experimental models
A2. Integration with Hermes
Manual Switching
/model gpt-5.5
/model kimi-k2:1t
/model qwen3-coder:480b
Programmatic (from Skill)
import sys
sys.path.insert(0, "~/.hermes/profiles/sirvir/skills/sirvir")
from model_router import select_model
model, details = select_model("financial", priority="speed")
model, details = select_model("creative", priority="cost")
model, details = select_model("auto", priority="balanced", prompt="Write brand voice...")
Hermes Slash Command (from Skill)
import sys, subprocess
sys.path.insert(0, "~/.hermes/profiles/sirvir/skills/sirvir")
from model_router import select_model
model, details = select_model("financial", priority="quality")
print(f"/model {details['provider']}:{model}")
JSON Output for Scripts
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --task coding --priority speed --json
Output:
{
"model": "gpt-5.4-mini",
"provider": "openai",
"speed_score": 8.87,
"cost_score": 7.0,
"quality_score": 10.0,
"weighted_score": 8.76,
"avg_latency": "1.26s",
"alternatives": [...]
}
A3. Cost Analysis
Subscription vs API (Fair Comparison)
Critical: Do NOT compare API tokens across providers without normalizing. DeepSeek is ~100x cheaper than OpenAI per token.
| Usage/Month | Ollama Pro | DeepSeek API | OpenAI API |
|---|
| 1M tokens | $20 | $1-2 | $5-15 |
| 5M tokens | $20 | $5-10 | $25-75 |
| 15M tokens | $20 | $15-30 | $75-200 |
| 50M tokens | $20 | $50-100 | $250-500 |
Break-even: ~5M tokens/month for Ollama Pro vs API
Decision Framework
| Scenario | Best Choice | Rationale |
|---|
| High volume (15M+ tokens/mo) | Ollama Pro | Flat $20/mo beats any API |
| Burst usage (spikes + quiet) | API | Only pay for what you use |
| Client deliverables (time-sensitive) | OpenAI Plus | Speed premium worth it |
| Drafts / internal work | Ollama Pro | Cost-efficient for iterations |
| Math / Financial models | Test DeepSeek API | May be cheaper than Ollama sub |
Pitfall: Comparing DeepSeek API cost to OpenAI Plus subscription is unfair. DeepSeek API is pay-per-token; OpenAI Plus is flat $20/mo. Fair comparisons:
- DeepSeek API vs OpenAI API (both pay-per-token, DeepSeek is ~100x cheaper)
- Ollama Pro ($20/mo) vs OpenAI Plus ($20/mo) (both flat subscription)
- Ollama Pro vs your actual token usage across ALL API providers
Your Usage Pattern
Agent operations (daily chat, coding, analysis): 10-30M tokens/month
Recommendation: Ollama Pro subscription for cost-efficiency
Pitfall: Incomplete Benchmark Set Produces Misleading Routing Winner
Symptom (verified 2026-06-25): The 6/23 3-way benchmark (DS v4 Pro vs M3 vs GLM-5.2) declared GLM-5.2 the coding winner. But it excluded qwen3-coder:480b and kimi-k2.7-code — the two models most relevant to coding. The routing table said "GLM-5.2 wins coding" because the models that would have beaten it weren't in the test.
What went wrong: Garry asked "did you use the model router to the coding model of choice?" The agent checked the routing table, saw GLM-5.2 listed as coding winner, and dispatched coding subagents on GLM-5.2. But the broader June benchmark had qwen3-coder:480b at 3.45s — faster than GLM-5.2's 6.75s. The 3-way was narrower than the prior benchmark, so its "winner" was misleading.
Fix: When a benchmark is used to update routing tables, always note which models were excluded. If a specialized model (e.g., a coding-specific model) exists in the inventory but wasn't in the benchmark, the routing table should say "won among tested models" not "is the best." When in doubt, re-run with the missing models included.
Rule: A routing table entry is only as trustworthy as the breadth of its benchmark. Before relying on a "winner," check whether the obvious candidates for that task type were actually tested.
Pitfall: Model ID Drift — Deprecated Models Return 410 Gone
Symptom (verified 2026-06-25): kimi-k2:1t was listed in the June benchmark results. When re-running the benchmark on 6/25, kimi-k2:1t returned HTTP 410 Gone. The model had been deprecated and replaced by kimi-k2.7-code.
Fix: Before running any benchmark, verify model IDs against the live API:
python3 -c "
import urllib.request, json
env = open('~/.hermes/.env').read()
key = [l.split('=',1)[1].strip() for l in env.split('\n') if l.startswith('OLLAMA_API_KEY=***None
req = urllib.request.Request('https://ollama.com/v1/models', headers={'Authorization': f'Bearer {key}'})
data = json.loads(urllib.request.urlopen(req, timeout=30).read())
for m in sorted([x['id'] for x in data.get('data',[])]):
print(m)
"
Rule: Model IDs on hosted providers change without notice. Always verify against /v1/models before benchmarking or updating routing tables. A stale model ID wastes a full benchmark run and produces a 0% success rate that looks like a model quality failure but is actually just a 410.
Pitfall: Latency-fallback score (5.0) is too generous vs measured-but-slow models
Symptom (verified 2026-06-22, glm-5.2 vs gpt-5.5 coding): When using priority=speed, the router picked gpt-5.5 (score 5.97) over glm-5.2 (score 4.22) even though glm-5.2 was measured at 6.75s. gpt-5.5 had no benchmark data → latency_score = 5.0 (fallback). glm-5.2's measured 6.75s → latency_score = max(1.0, 10 - (6.75/3)*9) = max(1.0, -10.25) = 1.0 (clamped to floor).
Why it stays silent: A model with no data scores 5.0; a model measured at 6.75s scores 1.0. The router treats "I don't know how fast this is" as faster than "I measured it at 6.75s." For speed priority (60% latency weight), this is a major routing inversion.
Why it matters: gpt-5.5's quality baseline (9.5) further compounds the bias — high-quality + no-data beats measured-actually-fast.
Two fixes (apply both):
-
Tighten the no-data latency fallback from 5.0 to 3.0 in model_router.py line ~158:
if avg_latency is not None:
latency_score = max(1.0, min(10.0, 10.0 - (avg_latency / 3.0) * 9.0))
else:
latency_score = 5.0
latency_score = 3.0
-
Re-benchmark gpt-5.5 in the next monthly run so it has actual data instead of relying on the fallback. The monthly cron (979b95e18555) covers this.
Rule of thumb: for priority=speed, prefer measured-data models over unmeasured-high-quality-baseline models. The fallback is a placeholder, not a recommendation.
3-way head-to-head benchmark pattern
When a new model is "candidates" tier, don't add it to routing tables on a single 1v1 benchmark against the current default — the new model may win on speed but lose on quality, or vice versa. Run a 3-way benchmark: new model + current default + nearest competitor. Same 4 prompts (financial, creative, coding, quick), all three models, all four categories.
Why 3-way, not 1v1:
- 1v1 winner may be relative (X beats Y but Z beats both)
- 3-way forces relative comparison and surfaces category-specific winners
- Sample outputs side-by-side reveal qualitative differences (verbosity, formatting) the numbers hide
- Quality baselines (
QUALITY_BASELINE dict) shift with new data
The 3-way benchmark script is at ~/.hermes/profiles/sirvir/skills/sirvir/benchmark_glm52_minimal.py — drop-in for any new model. Set MODELS = [<new>, <current_default>, <nearest_competitor>] and run.
Output structure: 4 categories × 3 models = 12 requests, all in one run. Side-by-side latency + tokens + sample output (first 400 chars). Total runtime ~2-3 minutes for fast models, 5-10 for slow ones.
Pattern (verified 2026-06-22, glm-5.2 vs deepseek-v4-pro vs minimax-m3):
- glm-5.2 won 3/4 categories (financial, coding, quick)
- deepseek-v4-pro won 1/4 (creative — most concise + fastest)
- minimax-m3 lost all 4 but was tested for completeness
- Routing table updated: glm-5.2 as primary for financial/coding/quick, DS v4 Pro for creative
Don't: add a new model to "Quality" tier without running the 3-way. The 6/22 glm-5.2 candidate status was earned by this benchmark, not by marketing or hearsay.
Appendix B — Benchmark maintenance
Note: Garry specified benchmarks should run biweekly (every 2 weeks), but the current cron 979b95e18555 is set to monthly (first Sunday). This should be updated to biweekly cadence. The cron schedule needs changing from 0 6 1-7 * 0 (first Sunday monthly) to a biweekly pattern like 0 6 * * 0 with repeat: 0 (every Sunday) or a true biweekly expression. Verify with Garry before changing.
Benchmark runs automatically first Sunday of each month at 6 AM.
Pitfall: degraded monthly benchmark runs are provider-health signals, not clean routing verdicts
Symptom (verified 2026-06-28): The monthly benchmark_fast.py run completed, but one provider tier returned HTTP 429 Too Many Requests on every benchmarked request. Only the other provider's models produced successful rows. If you feed that result directly into the router, the routing table can collapse to "healthy provider wins every category" even though the outcome reflects temporary provider unavailability rather than true model quality.
What to do:
- Still archive the run — it is real evidence.
- Label the run explicitly as degraded or provider-health-limited.
- Compare against the last healthy benchmark snapshot, not just the immediately previous file, before changing normal-mode recommendations.
- Split recommendations into two layers:
- temporary degraded-mode routing — what to use right now while the provider is failing
- normal-mode routing to restore after recovery — preserve the last known good winners from the healthy benchmark
- Call out any categories that lost coverage entirely because all candidates for that category were on the degraded provider.
Rule: a monthly benchmark can update routing in two different ways:
- quality/latency update when multiple providers/models completed normally
- provider-health update when one provider fails broadly
Do not treat a provider-health outage as proof that the surviving provider permanently displaced the prior winner.
Reference: references/degraded-benchmark-provider-health.md
Results delivery: Discord (gateway channel), not Telegram.
Results archive:
~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast_results_YYYY-MM.json~/.hermes/profiles/sirvir/skills/sirvir/references/model_routing_matrix.md (updated)
To force update:
cd ~/.hermes/profiles/sirvir/skills/sirvir
python3 benchmark_fast.py
Appendix C — Historical troubleshooting and pitfalls
No benchmark data
Error: No benchmark results found
Fix: Run benchmark first
cd ~/.hermes/profiles/sirvir/skills/sirvir
python3 benchmark_fast.py
Model not available
Error: Model 'gpt-5.5' not found
Fix: Check provider auth
grep OLLAMA_API_KEY ~/.hermes/.env
hermes auth list
Recommendations outdated
If recommendations seem wrong after running benchmark:
- Check benchmark results age
- Re-run benchmark
- Verify no failures in benchmark output
Fallback provider not triggering (Layer 1 dead)
Symptom: fallback_providers is set in config.yaml but Hermes never falls back when the primary provider returns 429.
Root cause: hermes config set fallback_providers '["openai-codex"]' serializes the value as a YAML string (fallback_providers: '["openai-codex"]') instead of a proper YAML list. Hermes reads a string, not a list of providers, so the fallback chain is empty.
Fix: Edit the YAML directly to use list format:
sed -i "s/fallback_providers: '\\(\\.*\\)'/fallback_providers:\n- \1/" ~/.hermes/config.yaml
Or for sub-profiles, ensure the format is:
fallback_providers:
- openai-codex
Not:
fallback_providers: '["openai-codex"]'
Verify:
grep -A2 "fallback_providers" ~/.hermes/config.yaml
See references/fallback-providers-yaml-format-bug.md for full reproduction and fix.
Cron job runs once then stops
Symptom: A cron job with schedule: 30m runs exactly once and never again. The job state shows completed instead of scheduled.
Root cause: Cron jobs default to repeat: once even when a recurring schedule is set. The repeat field must be explicitly set to 0 (forever) for recurring behavior.
Fix when creating:
hermes cron create 30m --name "My Job" --script my_script.sh --no-agent
hermes cron update <job_id> --repeat 0
Verify:
hermes cron list | grep -A5 "My Job"
Cron job created but never runs
Symptom: A cron job was created, hermes cron list shows it, but it never fires. Hours later, the system still hasn't run it.
Root cause: The cron job was created with repeat: once (the default) instead of repeat: forever. After the first run, the job enters state: completed and never fires again. This is easy to miss because the schedule field shows the recurring interval but repeat controls whether it actually repeats.
Fix: After creating any recurring cron job, immediately verify with hermes cron list:
- The job appears in the list
state=scheduled (not completed)repeat=forevernext_run_at has a future timestampenabled=true
If repeat=once, update it: hermes cron update <job_id> --repeat 0
Prevention: Always set repeat=0 explicitly when creating recurring cron jobs, or update immediately after creation.
Appendix D — Special use case reference (historical / specialized overrides)
Use this section only when the baseline lane policy is clearly insufficient. These are explicit overrides for specialized tasks, not default profile routing. Most are not benchmarked recently — use with caution.
| Task Type | Primary | Fallback Chain | Notes |
|---|
| Vision (images) | qwen3-vl:235b | Ask user to describe → text model | NOT TESTED |
| Math proofs | minimax-m2.7 | deepseek-v4-pro → gpt-5.5 | Math-specialized (NOT TESTED) |
| Long context (>50K) | kimi-k2:1t | gpt-5.5 | 128K context window |
| Chain-of-thought | kimi-k2-thinking | kimi-k2:1t | Reasoning-specialized (NOT TESTED) |
| Ultra-cheap | deepseek-v4-flash | API only | NOT TESTED - avoid gemma3:4b |
| Production code | kimi-k2.7-code | glm-5.2 → gpt-5.5 | Quality first. 6/25 benchmark confirmed kimi-k2.7-code writes cleanest production code. |
Fallback Handling
Vision:
qwen3-vl:235b (Ollama)
→ If fails → Ask user to describe → Use text model
Math:
minimax-m2.7 (Ollama, NOT TESTED)
→ deepseek-v4-pro (Ollama, TESTED)
→ gpt-5.5 (OpenAI, TESTED)
Long Context:
kimi-k2:1t (Ollama, 128K)
→ gpt-5.5 (OpenAI, 200K)
Chain of Thought:
kimi-k2-thinking (Ollama, NOT TESTED)
→ kimi-k2:1t (Ollama, TESTED)
When to Use Special Overrides
- Vision: User uploads image/screenshot, asks for visual analysis
- Math: Symbolic math, proofs, multi-step calculations (not simple arithmetic — use
quick for that)
- Long Context: Document >50 pages, large codebases, multi-file analysis
- Chain-of-thought: Complex reasoning requiring explicit intermediate steps
Pitfall: Do NOT use chain-of-thought or other special overrides for simple tasks — they are slower, more expensive, and add operational complexity. Stay on the profile lane unless the workload clearly demands otherwise.
Appendix E — Files and reference artifacts
~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py — Core routing logic~/.hermes/profiles/sirvir/skills/sirvir/scripts/router_integration.py — Hermes integration~/.hermes/profiles/sirvir/skills/sirvir/references/model_routing_matrix.md — Full routing matrix (auto-generated by model_router.py --list)~/.hermes/profiles/sirvir/skills/sirvir/references/benchmark_fast_results.json — Current benchmark data~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast_report.md — Human-readable report~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast.py — Benchmark runner script~/.hermes/profiles/sirvir/skills/sirvir/benchmark_coding_3way.py — Coding-only 3-way benchmark script (created 6/25)~/.hermes/profiles/sirvir/skills/sirvir/scripts/provider_watchdog.py — Proactive health watchdog (cron d85df0885c22, every 30 min)scripts/sync-skills-to-profiles.py — Sync SKILL.md to all 7 sub-profile copies after updates
Reference Files
references/model-inventory-2026-06-07.md — Tested vs available models, cost analysis, selection policyreferences/ollama-pro-inventory-2026-06-07.md — Full Ollama Pro model inventory with token economicsreferences/benchmark-results-2026-06-07.json — Machine-readable benchmark datareferences/model-router-architecture-2026-06-15.md — Router build-out: dead skill → live scripts, config changesreferences/provider-watchdog-failsafe.md — Two-layer fail-safe: Hermes fallback chain + proactive health watchdog cronreferences/benchmark-scoring-cross-category-bug.md — Router scored models by (provider, model) instead of (provider, model, category), causing cross-category score overwritesreferences/benchmark-3way-2026-06-23.md — 3-way benchmark results: DS v4 Pro vs M3 vs GLM-5.2 (12/12 requests, all categories). Drove the 6/24 decision to flip Diwan workers from M3 to DS v4 Pro.references/benchmark-coding-3way-2026-06-25.md — Coding-specific 3-way benchmark: kimi-k2.7-code vs qwen3-coder:480b vs glm-5.2 (9/9 requests, 3 coding prompts). Drove the 6/25 decision to use kimi-k2.7-code for coding delegation.references/benchmark-coding-3way-2026-06-25.md — Coding-focused 3-way: kimi-k2.7-code vs qwen3-coder:480b vs glm-5.2 (9/9 requests, 3 coding prompts). Drove the 6/25 decision to set delegation.model to kimi-k2.7-code for coding subagents.references/coding-benchmark-comparison-2026-06-25.md — Why GLM-5.2 didn't actually win coding (qwen3-coder:480b was excluded from the 3-way). delegate_task routing lesson.references/benchmark-coding-qwen-vs-glm-2026-06-25.md — Coding model comparison showing qwen3-coder:480b (3.45s) actually beats GLM-5.2 (6.75s). Corrected the false "GLM-5.2 wins coding" from the incomplete 3-way benchmark.references/fallback-providers-yaml-format-bug.md — YAML string vs list bug that killed Layer 1 fallbackreferences/cron-job-repeat-once-pitfall.md — Cron jobs default to one-shot; how to make them recurring
E1. See Also
llm-benchmark skill — How to run benchmarks, test prompts, token economics- Cron job
979b95e18555 ("Monthly Model Benchmark") — runs benchmark_fast.py first Sunday of each month at 6 AM UTC, delivers to Discord
Appendix F — Evaluation Methodology
Core principle: Evaluate ALL use cases up front, not ad-hoc. Before committing to a model strategy, map every business domain→task type and pre-select the optimal model for each. This prevents wasted tokens on mismatched work.
Pre-Planning Checklist
- Inventory all use cases — List every business domain (marketing, sales, operations, creative, financial, technical)
- Map task types — Which models handle each domain? (creative → kimi-k2:1t, financial → gpt-5.5, coding → qwen3-coder:480b)
- Calculate token economics — Subscription break-even vs API pay-per-token for expected volume
- Benchmark before deploying — Run actual prompts from YOUR workflows, not generic tests
- Set routing rules — Define fallback chains (primary → backup → last resort)
Dimensions to Measure
| Dimension | What to Test | Why It Matters |
|---|
| Latency | Time to first token, total completion | Developer productivity, real-time work |
| Accuracy | Correctness on domain-specific tasks | Client deliverables, financial models |
| Cost | Tokens × price per token OR flat subscription | Budget constraints |
| Subscription vs API | Flat rate vs pay-per-use | High volume → subscription; burst → API |
| Quality | Polish, tone, formatting | Client-facing work |
Common Evaluation Mistakes
-
Unfair cost comparisons — DeepSeek API is ~100x cheaper than OpenAI API per token. Compare subscriptions to subscriptions, APIs to APIs.
-
Single-model thinking — One model doesn't fit all. Match model to use case: kimi-k2:1t for creative, gpt-5.5 for financial, qwen3-coder:480b for code.
-
Not re-evaluating — Model rankings change monthly. The cron job at ~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast.py runs the first Sunday of each month.
-
Ad-hoc routing — Picking models task-by-task wastes tokens. Pre-plan all use cases.
Use Case → Task Type Mapping
Business context determines task type, not just prompt keywords.
| Business Domain | Example Prompts | Task Type |
|---|
| Marketing campaign | "Write brand voice guide for..." | creative |
| Financial model | "Calculate EBITDA given..." | financial |
| Code review | "Review this PR for..." | coding |
| Dispatch routing | "Optimize schedule for..." | operations |
| Quick question | "What's the capital of..." | quick |
| Strategy analysis | "Compare approaches for..." | reasoning |
