메뉴
Dynamic model routing for task-specific optimization. Selects optimal model based on task type, priority, and cost.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
VRAM scaling skill. Wraps `serve vram`, `serve downscale`, and the scaling ladder. Documents the 3-tier (Beefy/Modest/Thin) hardware detection, the 7-step Beefy scaling ladder, the Modest (5-step) and Thin (4-step) ladders, when to trigger downscale, and how to recover when VRAM frees up. Includes the optimization priority ladder (262K → 30 tok/s → 1M → max speed). Complements the turbofit core skill — turbofit launches models, sirvir-scale adapts them to live conditions.
Token usage monitoring and budget skill. Documents how to read Hermes state.db for real usage data, track spending against a monthly budget, set alert thresholds (75% yellow, 90% orange, 100% red), and suggest upgrades/downgrades based on utilization. Complements the turbofit core skill — turbofit serves models, sirvir-budget tracks what they cost.
Model fleet manager and intelligence engine for Hermes Agent. Autonomous model lifecycle manager — local serving, benchmarking, auto-scaling, API fallback, HuggingFace scanning, creator tracking, backend optimization, budget monitoring, and external app endpoint serving. Optimization priority: 262K context → 30 tok/s → 1M context → max speed.
Opinionated unified LLM backend (turbofit v5.1). Picks the best main + aux model for your hardware — local or API — launches them detached, wires Hermes-Agent config, and adapts to live VRAM pressure via a scaling ladder. Three hardware tiers: Beefy (local+local), Modest (API+local), Thin (API+API). `serve auto main` auto-detects GPU and suggests the right setup. API fallback is always available (free: DeepSeek V4 Pro + Kimi K2.6). Replaces llama-launch, omni-va, and ad-hoc llama-server scripts. Catalog schema supports per-model binary pinning (atomic fork vs stock), named flag presets (nextn, draft-mtp, turbo4-kv, vision-mmproj), tier ladder (s/sf/sd/f/c), and the 64K Hermes context floor is enforced everywhere. End-user UX is `serve auto main` and the user is done.
Model benchmarking skill. Wraps `serve bench` and lm-eval-harness to run standardized benchmarks (MMLU, GPQA, SWE-bench, HumanEval, AIME) on local fleet models and monitored API models. Documents the benchmark workflow, score interpretation thresholds, and how benchmark results feed back into tier assignments in model-database.yaml. Complements the turbofit core skill — turbofit launches and serves models, sirvir-bench measures them.
Model research skill. Wraps HuggingFace scanning, OpenRouter pricing sync, and creator quality tracking. Documents how to scan HuggingFace for new GGUF models matching fleet archetypes, assess creator quality, update the model database, and sync to GitHub. Includes the creator quality database schema and the daily scan workflow. Complements the turbofit core skill — turbofit serves models, sirvir-research finds and vets them.
| name | model-routing |
| description | Dynamic model routing for task-specific optimization. Selects optimal model based on task type, priority, and cost. |
| version | 1.0.0 |
| author | Hermes Agent |
| license | MIT |
| platforms | ["linux","macos"] |
| metadata | {"hermes":{"tags":["llm","routing","optimization","cost","speed"]}} |
Dynamic model routing for real business workloads. Use stable profile lanes first, task-level overrides second, and delegation/specialized overrides only when the work clearly justifies them.
This skill's operating model is:
The goal is not "pick the perfect model for every single prompt." The goal is a routing system that is simple, predictable, and aligned with business stakes.
Older Ollama/GLM/MiniMax benchmark sections below are retained as historical evidence and failure-mode documentation. They are not the primary live routing policy.
Authoritative policy: ~/.hermes/profiles/sirvir/skills/sirvir/references/fleet-routing-policy.md
Three-tier lane routing with hybrid provider strategy:
Use for business-critical reasoning, research, strategic planning, and orchestration.
Profiles:
defaultmafresearchModel pair (beta):
ollama-cloud / glm-5.2ollama-cloud / glm-5.2nvidia / minimaxai/minimax-m3 (NIM free)nvidia / minimaxai/minimax-m3 (NIM free)After 7/4 cutover:
nous / glm-5.2nous / glm-5.2Use for implementation, operations, infra management, and production work.
Profiles:
rollout-autosirvirgarry-buildsModel pair:
ollama-cloud / deepseek-v4-proollama-cloud / deepseek-v4-pronvidia / minimaxai/minimax-m3 (NIM free)nvidia / minimaxai/minimax-m3 (NIM free)Use for comms, formatting, low-stakes drafting, and routine admin.
Profiles:
commsforge-framepersonalModel pair:
ollama-cloud / deepseek-v4-flashollama-cloud / deepseek-v4-flashnvidia / minimaxai/minimax-m3 (NIM free)nvidia / minimaxai/minimax-m3 (NIM free)openai-codex / gpt-5.4 — expires 7/4, will be re-evaluated at cutoverdefault acts as premium command center for synthesis and routingFor coding bursts dispatched via delegate_task, set delegation.model explicitly:
kimi-k2.7-code — preferred coding model (6/25 benchmark: 10.1s avg, 100% success, clean output)qwen3-coder:480b — alternative (3.45s in June benchmark, but variable latency on ollama cloud)glm-5.2 — fallback if neither coder model is availableAlways clear delegation.model after the coding session to avoid wrong-model inheritance.
Do not start from task-type routing every time. Start from the profile's lane and only override when needed.
Examples:
Current standard aux:
qwen/qwen3.5-flash-02-23Why:
The parent chat model and the delegated worker model do not have to be the same.
Use explicit delegation overrides for:
A stable routing system with a few deliberate overrides is more valuable than constantly reshuffling models based on minor benchmark deltas.
Use these business-aligned classes before falling back to narrow benchmark categories:
| Workload class | Typical work | Default routing |
|---|---|---|
| Executive / strategic | Decisions, synthesis, planning, tradeoffs | Premium lane |
| Client / business deliverable | Proposals, research briefs, financial analysis, important brand copy | Premium lane |
| Operational execution | Follow-ups, triage, formatting, coordination, admin | Light lane |
| Specialized coding / execution | Repo work, debugging, PR review, builds, engineering bursts | Delegation override / specialized model |
These are not the baseline profile models. They are explicit overrides for special workloads.
| Use case | Recommended model | Role | Notes |
|---|---|---|---|
| Coding delegation / production code | kimi-k2.7-code | specialized worker | Preferred when dispatching coding-heavy subagents or implementation bursts |
| Vision-heavy analysis | qwen/qwen3.5-flash-02-23 | aux / multimodal helper | Default shared aux because it is cheap, vision-capable, and operationally simple |
| Long-context / document-heavy analysis | premium lane first; escalate to long-context specialist only when needed | conditional override | Do not escalate automatically; only when the profile lane is clearly insufficient |
| Math / proof-style reasoning | use only as an explicit override after validation | experimental specialist | Historical candidates exist below, but treat them as opt-in, not default routing |
| Chain-of-thought specialists | use only as an explicit override after validation | experimental specialist | Slower and more expensive; avoid for normal reasoning |
Rule:
If the user asks to "look at" or "reflect on" the routing skill, treat the skill as decision-support context first. Do not edit the skill just because it was consulted.
Correct sequence:
SKILL.md, sync scripts, or profile copies.Why this matters:
When a business profile is renamed, update the model-routing skill immediately if it carries any hardcoded profile lists or examples.
Minimum follow-up after a rename:
PROFILES = [...] list in this skill's sync scripts.SKILL.md to use the new slug.Example learned here:
construction was renamed to forge-frameconstructionThe Provider Health Watchdog is a separate operational layer from the routing policy in this skill.
Current rule:
ollama-cloud -> openai-codex), keep its primary-model map aligned to the LIVE deployed stack~/.hermes/profiles/sirvir/skills/sirvir/scripts/provider_watchdog.py~/.hermes/profiles/sirvir/skills/sirvir/scripts/provider_watchdog.pyThis avoids a bad state where the skill teaches one routing policy, the watchdog restores another, and cron silently undoes intended model changes.
When the user asks to review, reflect on, or critique the routing skill, treat that as an analysis request first.
Correct sequence:
Why this matters:
# Route for a specific task type
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --task financial --priority speed
# Auto-detect task from prompt
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --prompt "Write brand voice for..." --priority quality
# List all routing tables
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --list
# Get Hermes slash command for switching
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/router_integration.py --task coding --priority quality --hermes-cmd
Map business use cases to optimal models. Not just task-type based — consider domain context.
| Business Domain | Use Cases | Task Type | Priority |
|---|---|---|---|
| Marketing | Campaign strategy, social planning, SEO | creative | balanced |
| Marketing | Image/video prompts (multimodal) | vision | quality |
| Sales | Outreach, follow-ups, CRM enrichment | creative | speed |
| Sales | Pipeline analytics | reasoning | balanced |
| Operations | Scheduling, logistics, fleet coordination | operations | speed |
| Operations | Capacity planning, forecasting | financial | balanced |
| Creative | Social media, video scripts, design prompts | creative | quality |
| Creative | Brand voice, tone guides | creative | quality |
| Financial | Modeling, IRR, valuation | financial | quality |
| Financial | Audit, compliance review | financial | quality |
| Financial | Quick calculations, ratios | quick | speed |
| Technical | Architecture, debugging, features | coding | balanced |
| Technical | Client deliverables (production) | coding | quality + high-stakes |
| Type | Description | Keywords |
|---|---|---|
| financial | Revenue, EBITDA, IRR, valuation | "revenue", "profit", "cash flow", "investment" |
| creative | Brand, copy, marketing, social | "brand voice", "email", "headline", "copy" |
| coding | Implementation, debugging, API | "function", "algorithm", "debug", "refactor" |
| operations | Scheduling, logistics, routing | "schedule", "dispatch", "optimize", "capacity" |
| quick | Simple questions, conversions | "what is", "calculate", "convert", "capital of" |
| reasoning | Analysis, comparison, strategy | "analyze", "compare", "should i", "strategy" |
| Priority | Optimization | Best For |
|---|---|---|
| speed | Latency first | Client calls, real-time work |
| balanced | Equal weights | General use |
| cost | Subscription preference | High volume, drafts |
| quality | Output quality | Final deliverables |
Models fall into three cost-quality tiers. Always benchmark before assigning a model to a tier — don't assume based on name or reputation.
| Tier | Models | Best For | Token Cost | Notes |
|---|---|---|---|---|
| Cheap | deepseek-v4-flash | Simple queries, personal tasks, quick lookups | Lowest | Hallucinates on real thinking tasks. Don't use for financial, coding, or anything requiring accuracy. |
| Great+cheap | minimax-m2.5 | Conversation, orchestration, operations, quick tasks | Low | ⚠️ 34.8s on financial prompts, 50% success on coding. Good for ops/quick only. Always benchmark before expanding scope. |
| Quality | deepseek-v4-pro, kimi-k2:1t, qwen3-coder:480b | Highest | MAF/research/garry-builds defaults. Token-expensive but reliable. | |
| Quality (CONFIRMED 6/23) | glm-5.2 | Highest | Wins 3/4 categories in 3-way benchmark vs DS4Pro + M3 — 6.8× faster on coding, 2.5× faster on financial, sub-second on quick. Now the default profile model as of 6/24. | |
| Quality (WORKER 6/24) | deepseek-v4-pro | High | Replaced M3 on all 7 Diwan worker profiles. Beats M3 on latency in all 4 categories (1.3-2.9x). Same ollama-cloud cost tier. See references/benchmark-3way-2026-06-23.md. |
benchmark_fast.py first. A model that sounds "great and cheap" (e.g. minimax-m2.5) may be 34s on financial or 50% on coding. Do not assume from hearsay or marketing. Run the test.minimax-m2.5 on Ollama. Verify the exact model ID from /v1/models before benchmarking.Scenario: Someone recommends a model as "great and cheap" for a specific use case. You add it to routing tables without testing.
What goes wrong: The model may have 34s latency on financial prompts, 50% success on coding, or other hidden failure modes that only surface under your actual workloads.
Fix: Always run benchmark_fast.py first and inspect the results before adding any model to routing tables. If the user asks "why is this model in the table?", the answer should be "because we benchmarked it", not "because it sounded good".
Symptom (verified 2026-06-25): Controller dispatched delegate_task subagents for a production dashboard build without checking the model-routing skill. Subagents inherited glm-5.2 (the orchestrator model) instead of qwen3-coder:480b (the benchmarked coding champion at 3.45s). User caught it.
Why it happens: delegate_task can't override the model per-call. If delegation.model is empty in config.yaml, children inherit the parent. The controller didn't load the routing skill before starting the build, so it didn't know to set delegation.model first.
Prevention — before dispatching ANY subagent for coding/financial/creative work:
skill_view(name='model-routing')) and check the routing table for the task typedelegation.model and delegation.provider in config.yaml to the routed model for that task typehermes config get delegation.modeldelegation.model to empty if the next batch of work is a different task typeRule: the routing skill is a pre-flight check for delegation, not just for /model slash commands. If you're about to delegate_task and you haven't checked the routing table, you're guessing.
Hermes supports separating the main chat model from auxiliary models used for side tasks (summarization, web page extraction, browser screenshots, session title generation, context compression).
This is a recommended pattern for token efficiency — use a quality model for thinking/conversation and a cheap model for grunt work.
Tier plan (beta: ollama-cloud substitutes for Nous until 7/4 cutover):
ollama-cloud / glm-5.2 — conversation, orchestration, routing, reasoning, compressionollama-cloud / deepseek-v4-pro — domain work, implementation, operationsollama-cloud / deepseek-v4-flash — comms, formatting, low-stakes adminnvidia / minimaxai/minimax-m3 (NIM free, good multimodal)nvidia / minimaxai/minimax-m3auto (inherit main model)autoopenai-codex / gpt-5.4 (expires 7/4)Why: GLM-5.2 wins 3/4 benchmark categories (financial, coding, quick) including sub-second latency on quick tasks. DS v4 Pro is the solid middle lane. DS v4 Flash is the cheapest acceptable lane. M3 retained for vision/web_extract because it's free via NIM and verbose output is acceptable for side tasks. Compression moved off M3 to the tier text model because compression needs reliable instruction-following, not multimodal capability.
After 7/4 cutover:
nous / glm-5.2nvidia / minimax-m3By default, all auxiliary tasks use the main chat model (auxiliary.*.provider: "auto"). To override:
auxiliary:
ag:
provider: ollama-cloud
model: deepseek-v4-flash
web:
provider: ollama-cloud
model: deepseek-v4-flash
vision:
provider: ollama-cloud
model: qwen3-vl:235b
compress:
provider: ollama-cloud
model: minimax-m3 # DS4F may not have 1M context — use M3
For per-profile configuration, add the auxiliary section to each profile's config.yaml.
See Hermes docs: https://hermes-agent.nousresearch.com/docs/user-guide/configuration#auxiliary-models
This section is preserved for benchmark evidence, prior routing decisions, and debugging context. It is not the live routing policy. For live operation, use ## Current operating model (2026-06-26) above.
Based on older June benchmark runs and older provider assumptions.
| Role | Model | Provider | Profiles | Notes |
|---|---|---|---|---|
| Default/orchestrator | glm-5.2 | ollama-cloud | default | Wins 3/4 benchmark categories. Fallback: openai-codex/gpt-5.4 |
| Diwan workers (×7) | deepseek-v4-pro | ollama-cloud | comms, construction, garry-builds, maf, personal, research, rollout-auto | Replaced M3 on 6/24. Beats M3 on all 4 categories. Fallback: openai-codex/gpt-5.4 |
| Coding (delegation) | kimi-k2.7-code | ollama-cloud | — | 6/25 benchmark winner. 10.1s avg, 100% success, 1446 avg tokens. Set via delegation.model when all subagents are coding. Clear after session. |
| Kanban decomposer | glm-5.2 | ollama-cloud | — | Pinned via config |
| Triage specifier | glm-5.2 | ollama-cloud | — | Pinned via config |
| Auxiliary vision | minimax-m3 | ollama-cloud | all | Verbose output acceptable for vision tasks |
| Auxiliary web_extract | minimax-m3 | ollama-cloud | all | Verbose output acceptable for extraction |
| Delegate_task subagents | (inherits parent or delegation.model) | — | — | Set delegation.model for task-specific routing; clear after use |
Coding benchmark (6/25, 3 prompts × 3 models):
| Model | Avg latency | Avg tokens | Success |
|---|---|---|---|
kimi-k2.7-code | 10.1s | 1,446 | 100% |
glm-5.2 | 9.9s | 1,819 | 100% |
qwen3-coder:480b | 58.5s | 1,103 | 100% |
kimi-k2.7-code is the recommended coding model: clean output, good on async SQLAlchemy, more token-efficient than glm-5.2. qwen3-coder:480b was fastest in June (3.45s) but slow on 6/25 (85.5s on one prompt) — ollama cloud latency varies.
Historical change history:
minimax-m3 to deepseek-v4-pro based on 3-way benchmark showing DS beats M3 on all 4 categories at same cost tier.| Task | Model | Provider | Latency | Notes |
|---|---|---|---|---|
| Financial | glm-5.2 | ollama-cloud | 3.49s | 2.5x faster than DS4Pro, 3.4x faster than M3 |
| Creative | deepseek-v4-pro | ollama-cloud | 4.57s | 1.3x faster than M3, most concise (628 tok) |
| Coding | kimi-k2.7-code | ollama-cloud | 10.1s avg | 6/25 benchmark: 4.5s basic, 7.7s API, 18.1s system. Was glm-5.2 in 3-way (qwen not tested). |
| Operations | deepseek-v4-pro | ollama-cloud | — | Workers now on DS4Pro; ops is reasoning-adjacent |
| Quick | glm-5.2 | ollama-cloud | 0.78s | Sub-second, lightest tokens (96) |
| Reasoning | deepseek-v4-pro | ollama-cloud | — | Workers on DS4Pro; DS4Pro wins reasoning in monthly matrix |
All models below are on ollama-cloud (tier 1, flat subscription). Cost is identical across models.
| Task | Model | Provider | Latency | Notes |
|---|---|---|---|---|
| Financial | glm-5.2 | ollama-cloud | 3.49s | Fastest + fewest tokens (1035) |
| Creative | deepseek-v4-pro | ollama-cloud | 4.57s | Fastest + fewest tokens (628) |
| Coding | kimi-k2.7-code | ollama-cloud | Best coding model (6/25 benchmark, 10.1s avg, 1446 tok avg). GLM-5.2 close (9.9s, 1819 tok). qwen3-coder:480b was 58.5s avg — not viable. | |
| Operations | deepseek-v4-pro | ollama-cloud | — | Workers on DS4Pro |
| Quick | glm-5.2 | ollama-cloud | 0.78s | Sub-second, 96 tokens |
| Reasoning | deepseek-v4-pro | ollama-cloud | — | Retained from monthly matrix |
| Task | Model | Provider | Notes |
|---|---|---|---|
| Financial | glm-5.2 | ollama-cloud | Fastest + cleanest output. Default profile already uses GLM-5.2 |
| Creative | deepseek-v4-pro | ollama-cloud | Most concise (628 tok). Workers on DS4Pro |
| Coding | qwen3-coder:480b | ollama-cloud | 3.45s, 780 tok, 100% success in June benchmark — actually the fastest coding model, not GLM-5.2. GLM-5.2 (6.75s) only "won" the 3-way because qwen3-coder was excluded from that run. |
| Operations | deepseek-v4-pro | ollama-cloud | Workers on DS4Pro |
| Quick | glm-5.2 | ollama-cloud | Sub-second, lightest tokens |
| Reasoning | deepseek-v4-pro | ollama-cloud | Workers on DS4Pro |
Model strategy (6/25):
glm-5.2 — fastest on 3/4 categories, already setdeepseek-v4-pro — beats M3 on all 4 categories, same cost tierkimi-k2.7-code — 6/25 coding benchmark winner (10.1s avg, 100% success, clean output). Set delegation.model when dispatching coding subagents; clear after session.minimax-m3 — verbose but acceptable for side tasksglm-5.2openai-codex / gpt-5.4 if ollama-cloud goes downdelegation.model for task-specific routing; clear after use to avoid wrong-model delegation. Pre-flight rule: check routing table before dispatching coding subagents — set delegation.model: qwen3-coder:480b + delegation.provider: ollama-cloud for coding work. kimi-k2.7-code is Garry's preferred coding model to test. Available coding models on ollama: qwen3-coder:480b, kimi-k2.7-code, qwen3-coder-next, qwen3.5:397b.The rest of this skill is reference material. Use it when re-benchmarking, debugging older routing decisions, or reviving legacy tooling.
Appendix map:
Do not treat these appendices as the default live routing policy unless the current operating model section above explicitly points you here.
For client deliverables, production code, or high-impact work:
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --task financial --high-stakes --priority quality
# → selects OpenAI for reliability
High-stakes mode:
# In Hermes session
/model gpt-5.5 # Switch to financial model
/model kimi-k2:1t # Switch to creative model
/model qwen3-coder:480b # Switch to coding model
import sys
sys.path.insert(0, "~/.hermes/profiles/sirvir/skills/sirvir")
from model_router import select_model
# For financial task with speed priority
model, details = select_model("financial", priority="speed")
# Returns: ("gpt-5.5", {"provider": "openai-codex", "avg_latency_s": 9.37, ...})
# For creative task with cost priority (currently blocked — Ollama quota)
model, details = select_model("creative", priority="cost")
# Returns: ("gpt-5.4", {"provider": "openai-codex", ...}) — falls back to OpenAI
# Auto-detect task from prompt
model, details = select_model("auto", priority="balanced", prompt="Write brand voice...")
import sys, subprocess
sys.path.insert(0, "~/.hermes/profiles/sirvir/skills/sirvir")
from model_router import select_model
model, details = select_model("financial", priority="quality")
# Use /model slash command to switch
print(f"/model {details['provider']}:{model}")
# → /model openai-codex:gpt-5.5
python ~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py --task coding --priority speed --json
Output:
{
"model": "gpt-5.4-mini",
"provider": "openai",
"speed_score": 8.87,
"cost_score": 7.0,
"quality_score": 10.0,
"weighted_score": 8.76,
"avg_latency": "1.26s",
"alternatives": [...]
}
Critical: Do NOT compare API tokens across providers without normalizing. DeepSeek is ~100x cheaper than OpenAI per token.
| Usage/Month | Ollama Pro | DeepSeek API | OpenAI API |
|---|---|---|---|
| 1M tokens | $20 | $1-2 | $5-15 |
| 5M tokens | $20 | $5-10 | $25-75 |
| 15M tokens | $20 | $15-30 | $75-200 |
| 50M tokens | $20 | $50-100 | $250-500 |
Break-even: ~5M tokens/month for Ollama Pro vs API
| Scenario | Best Choice | Rationale |
|---|---|---|
| High volume (15M+ tokens/mo) | Ollama Pro | Flat $20/mo beats any API |
| Burst usage (spikes + quiet) | API | Only pay for what you use |
| Client deliverables (time-sensitive) | OpenAI Plus | Speed premium worth it |
| Drafts / internal work | Ollama Pro | Cost-efficient for iterations |
| Math / Financial models | Test DeepSeek API | May be cheaper than Ollama sub |
Pitfall: Comparing DeepSeek API cost to OpenAI Plus subscription is unfair. DeepSeek API is pay-per-token; OpenAI Plus is flat $20/mo. Fair comparisons:
Agent operations (daily chat, coding, analysis): 10-30M tokens/month Recommendation: Ollama Pro subscription for cost-efficiency
Symptom (verified 2026-06-25): The 6/23 3-way benchmark (DS v4 Pro vs M3 vs GLM-5.2) declared GLM-5.2 the coding winner. But it excluded qwen3-coder:480b and kimi-k2.7-code — the two models most relevant to coding. The routing table said "GLM-5.2 wins coding" because the models that would have beaten it weren't in the test.
What went wrong: Garry asked "did you use the model router to the coding model of choice?" The agent checked the routing table, saw GLM-5.2 listed as coding winner, and dispatched coding subagents on GLM-5.2. But the broader June benchmark had qwen3-coder:480b at 3.45s — faster than GLM-5.2's 6.75s. The 3-way was narrower than the prior benchmark, so its "winner" was misleading.
Fix: When a benchmark is used to update routing tables, always note which models were excluded. If a specialized model (e.g., a coding-specific model) exists in the inventory but wasn't in the benchmark, the routing table should say "won among tested models" not "is the best." When in doubt, re-run with the missing models included.
Rule: A routing table entry is only as trustworthy as the breadth of its benchmark. Before relying on a "winner," check whether the obvious candidates for that task type were actually tested.
Symptom (verified 2026-06-25): kimi-k2:1t was listed in the June benchmark results. When re-running the benchmark on 6/25, kimi-k2:1t returned HTTP 410 Gone. The model had been deprecated and replaced by kimi-k2.7-code.
Fix: Before running any benchmark, verify model IDs against the live API:
python3 -c "
import urllib.request, json
env = open('~/.hermes/.env').read()
key = [l.split('=',1)[1].strip() for l in env.split('\n') if l.startswith('OLLAMA_API_KEY=***None
req = urllib.request.Request('https://ollama.com/v1/models', headers={'Authorization': f'Bearer {key}'})
data = json.loads(urllib.request.urlopen(req, timeout=30).read())
for m in sorted([x['id'] for x in data.get('data',[])]):
print(m)
"
Rule: Model IDs on hosted providers change without notice. Always verify against /v1/models before benchmarking or updating routing tables. A stale model ID wastes a full benchmark run and produces a 0% success rate that looks like a model quality failure but is actually just a 410.
Symptom (verified 2026-06-22, glm-5.2 vs gpt-5.5 coding): When using priority=speed, the router picked gpt-5.5 (score 5.97) over glm-5.2 (score 4.22) even though glm-5.2 was measured at 6.75s. gpt-5.5 had no benchmark data → latency_score = 5.0 (fallback). glm-5.2's measured 6.75s → latency_score = max(1.0, 10 - (6.75/3)*9) = max(1.0, -10.25) = 1.0 (clamped to floor).
Why it stays silent: A model with no data scores 5.0; a model measured at 6.75s scores 1.0. The router treats "I don't know how fast this is" as faster than "I measured it at 6.75s." For speed priority (60% latency weight), this is a major routing inversion.
Why it matters: gpt-5.5's quality baseline (9.5) further compounds the bias — high-quality + no-data beats measured-actually-fast.
Two fixes (apply both):
Tighten the no-data latency fallback from 5.0 to 3.0 in model_router.py line ~158:
if avg_latency is not None:
latency_score = max(1.0, min(10.0, 10.0 - (avg_latency / 3.0) * 9.0))
else:
latency_score = 5.0 # OLD
latency_score = 3.0 # NEW — measured-but-slow beats unmeasured-high-quality
Re-benchmark gpt-5.5 in the next monthly run so it has actual data instead of relying on the fallback. The monthly cron (979b95e18555) covers this.
Rule of thumb: for priority=speed, prefer measured-data models over unmeasured-high-quality-baseline models. The fallback is a placeholder, not a recommendation.
When a new model is "candidates" tier, don't add it to routing tables on a single 1v1 benchmark against the current default — the new model may win on speed but lose on quality, or vice versa. Run a 3-way benchmark: new model + current default + nearest competitor. Same 4 prompts (financial, creative, coding, quick), all three models, all four categories.
Why 3-way, not 1v1:
QUALITY_BASELINE dict) shift with new dataThe 3-way benchmark script is at ~/.hermes/profiles/sirvir/skills/sirvir/benchmark_glm52_minimal.py — drop-in for any new model. Set MODELS = [<new>, <current_default>, <nearest_competitor>] and run.
Output structure: 4 categories × 3 models = 12 requests, all in one run. Side-by-side latency + tokens + sample output (first 400 chars). Total runtime ~2-3 minutes for fast models, 5-10 for slow ones.
Pattern (verified 2026-06-22, glm-5.2 vs deepseek-v4-pro vs minimax-m3):
Don't: add a new model to "Quality" tier without running the 3-way. The 6/22 glm-5.2 candidate status was earned by this benchmark, not by marketing or hearsay.
Note: Garry specified benchmarks should run biweekly (every 2 weeks), but the current cron 979b95e18555 is set to monthly (first Sunday). This should be updated to biweekly cadence. The cron schedule needs changing from 0 6 1-7 * 0 (first Sunday monthly) to a biweekly pattern like 0 6 * * 0 with repeat: 0 (every Sunday) or a true biweekly expression. Verify with Garry before changing.
Benchmark runs automatically first Sunday of each month at 6 AM.
Symptom (verified 2026-06-28): The monthly benchmark_fast.py run completed, but one provider tier returned HTTP 429 Too Many Requests on every benchmarked request. Only the other provider's models produced successful rows. If you feed that result directly into the router, the routing table can collapse to "healthy provider wins every category" even though the outcome reflects temporary provider unavailability rather than true model quality.
What to do:
Rule: a monthly benchmark can update routing in two different ways:
Do not treat a provider-health outage as proof that the surviving provider permanently displaced the prior winner.
Reference: references/degraded-benchmark-provider-health.md
Results delivery: Discord (gateway channel), not Telegram.
Results archive:
~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast_results_YYYY-MM.json~/.hermes/profiles/sirvir/skills/sirvir/references/model_routing_matrix.md (updated)To force update:
cd ~/.hermes/profiles/sirvir/skills/sirvir
python3 benchmark_fast.py
Error: No benchmark results found
Fix: Run benchmark first
cd ~/.hermes/profiles/sirvir/skills/sirvir
python3 benchmark_fast.py
Error: Model 'gpt-5.5' not found
Fix: Check provider auth
# Ollama
grep OLLAMA_API_KEY ~/.hermes/.env
# OpenAI (OAuth)
hermes auth list
If recommendations seem wrong after running benchmark:
Symptom: fallback_providers is set in config.yaml but Hermes never falls back when the primary provider returns 429.
Root cause: hermes config set fallback_providers '["openai-codex"]' serializes the value as a YAML string (fallback_providers: '["openai-codex"]') instead of a proper YAML list. Hermes reads a string, not a list of providers, so the fallback chain is empty.
Fix: Edit the YAML directly to use list format:
sed -i "s/fallback_providers: '\\(\\.*\\)'/fallback_providers:\n- \1/" ~/.hermes/config.yaml
Or for sub-profiles, ensure the format is:
fallback_providers:
- openai-codex
Not:
fallback_providers: '["openai-codex"]' # WRONG — string, not list
Verify:
grep -A2 "fallback_providers" ~/.hermes/config.yaml
# Should show:
# fallback_providers:
# - openai-codex
See references/fallback-providers-yaml-format-bug.md for full reproduction and fix.
Symptom: A cron job with schedule: 30m runs exactly once and never again. The job state shows completed instead of scheduled.
Root cause: Cron jobs default to repeat: once even when a recurring schedule is set. The repeat field must be explicitly set to 0 (forever) for recurring behavior.
Fix when creating:
hermes cron create 30m --name "My Job" --script my_script.sh --no-agent
# Then immediately update to make it recurring:
hermes cron update <job_id> --repeat 0
Verify:
hermes cron list | grep -A5 "My Job"
# Check: repeat=forever, state=scheduled (not completed)
Symptom: A cron job was created, hermes cron list shows it, but it never fires. Hours later, the system still hasn't run it.
Root cause: The cron job was created with repeat: once (the default) instead of repeat: forever. After the first run, the job enters state: completed and never fires again. This is easy to miss because the schedule field shows the recurring interval but repeat controls whether it actually repeats.
Fix: After creating any recurring cron job, immediately verify with hermes cron list:
state=scheduled (not completed)repeat=forevernext_run_at has a future timestampenabled=trueIf repeat=once, update it: hermes cron update <job_id> --repeat 0
Prevention: Always set repeat=0 explicitly when creating recurring cron jobs, or update immediately after creation.
Use this section only when the baseline lane policy is clearly insufficient. These are explicit overrides for specialized tasks, not default profile routing. Most are not benchmarked recently — use with caution.
| Task Type | Primary | Fallback Chain | Notes |
|---|---|---|---|
| Vision (images) | qwen3-vl:235b | Ask user to describe → text model | NOT TESTED |
| Math proofs | minimax-m2.7 | deepseek-v4-pro → gpt-5.5 | Math-specialized (NOT TESTED) |
| Long context (>50K) | kimi-k2:1t | gpt-5.5 | 128K context window |
| Chain-of-thought | kimi-k2-thinking | kimi-k2:1t | Reasoning-specialized (NOT TESTED) |
| Ultra-cheap | deepseek-v4-flash | API only | NOT TESTED - avoid gemma3:4b |
| Production code | kimi-k2.7-code | glm-5.2 → gpt-5.5 | Quality first. 6/25 benchmark confirmed kimi-k2.7-code writes cleanest production code. |
Vision:
qwen3-vl:235b (Ollama)
→ If fails → Ask user to describe → Use text model
Math:
minimax-m2.7 (Ollama, NOT TESTED)
→ deepseek-v4-pro (Ollama, TESTED)
→ gpt-5.5 (OpenAI, TESTED)
Long Context:
kimi-k2:1t (Ollama, 128K)
→ gpt-5.5 (OpenAI, 200K)
Chain of Thought:
kimi-k2-thinking (Ollama, NOT TESTED)
→ kimi-k2:1t (Ollama, TESTED)
quick for that)Pitfall: Do NOT use chain-of-thought or other special overrides for simple tasks — they are slower, more expensive, and add operational complexity. Stay on the profile lane unless the workload clearly demands otherwise.
~/.hermes/profiles/sirvir/skills/sirvir/scripts/model_router.py — Core routing logic~/.hermes/profiles/sirvir/skills/sirvir/scripts/router_integration.py — Hermes integration~/.hermes/profiles/sirvir/skills/sirvir/references/model_routing_matrix.md — Full routing matrix (auto-generated by model_router.py --list)~/.hermes/profiles/sirvir/skills/sirvir/references/benchmark_fast_results.json — Current benchmark data~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast_report.md — Human-readable report~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast.py — Benchmark runner script~/.hermes/profiles/sirvir/skills/sirvir/benchmark_coding_3way.py — Coding-only 3-way benchmark script (created 6/25)~/.hermes/profiles/sirvir/skills/sirvir/scripts/provider_watchdog.py — Proactive health watchdog (cron d85df0885c22, every 30 min)scripts/sync-skills-to-profiles.py — Sync SKILL.md to all 7 sub-profile copies after updatesreferences/model-inventory-2026-06-07.md — Tested vs available models, cost analysis, selection policyreferences/ollama-pro-inventory-2026-06-07.md — Full Ollama Pro model inventory with token economicsreferences/benchmark-results-2026-06-07.json — Machine-readable benchmark datareferences/model-router-architecture-2026-06-15.md — Router build-out: dead skill → live scripts, config changesreferences/provider-watchdog-failsafe.md — Two-layer fail-safe: Hermes fallback chain + proactive health watchdog cronreferences/benchmark-scoring-cross-category-bug.md — Router scored models by (provider, model) instead of (provider, model, category), causing cross-category score overwritesreferences/benchmark-3way-2026-06-23.md — 3-way benchmark results: DS v4 Pro vs M3 vs GLM-5.2 (12/12 requests, all categories). Drove the 6/24 decision to flip Diwan workers from M3 to DS v4 Pro.references/benchmark-coding-3way-2026-06-25.md — Coding-specific 3-way benchmark: kimi-k2.7-code vs qwen3-coder:480b vs glm-5.2 (9/9 requests, 3 coding prompts). Drove the 6/25 decision to use kimi-k2.7-code for coding delegation.references/benchmark-coding-3way-2026-06-25.md — Coding-focused 3-way: kimi-k2.7-code vs qwen3-coder:480b vs glm-5.2 (9/9 requests, 3 coding prompts). Drove the 6/25 decision to set delegation.model to kimi-k2.7-code for coding subagents.references/coding-benchmark-comparison-2026-06-25.md — Why GLM-5.2 didn't actually win coding (qwen3-coder:480b was excluded from the 3-way). delegate_task routing lesson.references/benchmark-coding-qwen-vs-glm-2026-06-25.md — Coding model comparison showing qwen3-coder:480b (3.45s) actually beats GLM-5.2 (6.75s). Corrected the false "GLM-5.2 wins coding" from the incomplete 3-way benchmark.references/fallback-providers-yaml-format-bug.md — YAML string vs list bug that killed Layer 1 fallbackreferences/cron-job-repeat-once-pitfall.md — Cron jobs default to one-shot; how to make them recurringllm-benchmark skill — How to run benchmarks, test prompts, token economics979b95e18555 ("Monthly Model Benchmark") — runs benchmark_fast.py first Sunday of each month at 6 AM UTC, delivers to DiscordCore principle: Evaluate ALL use cases up front, not ad-hoc. Before committing to a model strategy, map every business domain→task type and pre-select the optimal model for each. This prevents wasted tokens on mismatched work.
| Dimension | What to Test | Why It Matters |
|---|---|---|
| Latency | Time to first token, total completion | Developer productivity, real-time work |
| Accuracy | Correctness on domain-specific tasks | Client deliverables, financial models |
| Cost | Tokens × price per token OR flat subscription | Budget constraints |
| Subscription vs API | Flat rate vs pay-per-use | High volume → subscription; burst → API |
| Quality | Polish, tone, formatting | Client-facing work |
Unfair cost comparisons — DeepSeek API is ~100x cheaper than OpenAI API per token. Compare subscriptions to subscriptions, APIs to APIs.
Single-model thinking — One model doesn't fit all. Match model to use case: kimi-k2:1t for creative, gpt-5.5 for financial, qwen3-coder:480b for code.
Not re-evaluating — Model rankings change monthly. The cron job at ~/.hermes/profiles/sirvir/skills/sirvir/benchmark_fast.py runs the first Sunday of each month.
Ad-hoc routing — Picking models task-by-task wastes tokens. Pre-plan all use cases.
Business context determines task type, not just prompt keywords.
| Business Domain | Example Prompts | Task Type |
|---|---|---|
| Marketing campaign | "Write brand voice guide for..." | creative |
| Financial model | "Calculate EBITDA given..." | financial |
| Code review | "Review this PR for..." | coding |
| Dispatch routing | "Optimize schedule for..." | operations |
| Quick question | "What's the capital of..." | quick |
| Strategy analysis | "Compare approaches for..." | reasoning |