claude-maintain-models

name	claude-maintain-models
description	Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, or wants to add a net-new AI provider to Kiln.
allowed-tools	Read Edit Write Bash Grep Glob Agent WebSearch WebFetch

Add a New AI Model to Kiln

Branch check first: if the request involves a provider Kiln does not support yet, start at Adding a Net-New Provider. That workflow spans core, server, UI, tests, and tooling, and is gated on a client release — the model-entry steps below are only its final, gated piece.

For a new model on an already-supported provider, integrating it into libs/core/kiln_ai/adapters/ml_model_list.py requires:

1. ModelName enum – add an enum member
2. built_in_models list – add a KilnModel(...) entry with providers
3. ModelFamily enum – only if the vendor is brand-new

After code changes, run paid integration tests, then draft a Discord post.

---

Global Rules

These apply throughout the entire workflow.

• Slug verification: NEVER guess or infer model slugs from naming patterns. Every model_id must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.
• Date awareness: These models are often released very recently. Web search for current info before assuming you know the details.

---

Phase 1 – Model Discovery (only when asked to find new/missing models)

If the user asks you to find new models, do NOT just web search "new AI models this week" — that only surfaces major releases. Instead, systematically check each family against both the LiteLLM catalog and models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.

1. Read the ModelFamily and ModelName enums to know what we already have.

2. Query both catalogs for each family (run in parallel where possible):

   LiteLLM catalog — filters out mirror providers to avoid duplicates:

   curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != "openrouter" and .provider != "bedrock" and .provider != "bedrock_converse" and .provider != "vertex_ai-anthropic_models" and .provider != "azure") | .id] | unique | .[]'
   

   models.dev — search all model IDs across all providers:

   curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"
   

   For details on a specific provider+model: curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'

3. Search terms (one query per term): claude, gpt, o1, o3, o4 (OpenAI reasoning), gemini, llama, deepseek, qwen, qwq, mistral, grok, kimi, glm, minimax, hunyuan, ernie, phi, gemma, seed, step, pangu

4. Union and cross-reference results from both catalogs against ModelName. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). Skip pure coding models (e.g. codestral, deepseek-coder, qwen-coder).

5. Run targeted web searches per family to catch very fresh releases not yet in either catalog:

   • "[family] new model [current year]"
   • "[family] release [current month] [current year]"

---

Phase 1B – Lagging-Provider Backfill Check (every run)

Some providers — Fireworks AI, Together AI, SiliconFlow — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM. Relying only on those two catalogs will both under-populate the provider list for the model you're adding now and miss the window to backfill recently-added models whose provider support has since grown.

Run this check on every invocation of the skill, regardless of whether you're in discovery mode or adding a specific model.

1. Pull the 10 most recently added models from git history. List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:

   git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E "^\+\s+name=ModelName\." | head -20
   

2. For the model you're adding (if any) AND each of those 10 models, cross-check Fireworks, Together, and SiliconFlow directly using the endpoints in the Lagging Providers Reference. Do NOT trust models.dev / LiteLLM as the final word for these three providers.

3. If a lagging provider now supports a recently-added model that isn't yet in its KilnModel entry, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.

---

Phase 2 – Gather Context

1. Read the predecessor model in ml_model_list.py (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.

2. Query the LiteLLM catalog for the new model. This is the primary slug source since Kiln uses LiteLLM. See the Slug Lookup Reference for query syntax and all verified sources.

3. Get the OpenRouter slug via:

   • curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"
   • Fallback: WebSearch for openrouter [model name] model id

4. Get the direct-provider slug (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the Slug Lookup Reference for provider-specific URLs.

5. Identify quirks — check the Provider Quirks Reference for the relevant provider, and web search for any new quirks:

   • Structured output mode (JSON schema vs function calling)?
   • Reasoning model (needs reasoning_capable, parsers, OpenRouter options)?
   • Vision/multimodal support? Which MIME types?
   • Provider-specific flags (temp_top_p_exclusive, etc.)?
   • Rate limit concerns (max_parallel_requests)?

6. Determine thinking levels — does the model support configurable reasoning effort? See Thinking Levels Reference for the full lookup chain. Key quick checks:

   • Check the vendor model page (e.g. OpenAI model pages say "Reasoning.effort supports: X, Y, Z")
   • Check OpenRouter supported_parameters — if reasoning is absent, skip thinking levels
   • R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts

---

Phase 3 – Code Changes

All changes go in libs/core/kiln_ai/adapters/ml_model_list.py.

3a. ModelName enum

• snake_case: claude_opus_4_6 = "claude_opus_4_6"
• Place before predecessor (newer first within group). If the vendor is brand-new there is no predecessor — start a new group at the end of the enum
• Follow existing grouping (all claude together, all gpt together, etc.)

3b. KilnModel entry in built_in_models

• Place per the ordering rules in 3c — this placement is user-visible, get it right
• Copy predecessor's structure and modify: name, friendly_name, model_id per provider, flags
• friendly_name must follow the existing naming pattern of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use "Claude {version} Sonnet" (e.g. "Claude 4.5 Sonnet"), not "Claude Sonnet {version}". Do NOT use the vendor's marketing name if it differs from Kiln's established convention.

Provider model_id formats:

Provider	Format	Notes
openrouter	vendor/model-name	Always verify via API
openai	Bare model name	Verify via OpenAI docs
anthropic	Variable — older models have date stamps, newer may not	Always verify via Anthropic docs
gemini_api	Bare name	Verify via Google AI Studio docs
fireworks_ai	accounts/fireworks/models/...	Verify via Fireworks docs
together_ai	Vendor path format	Verify via Together docs
vertex	Usually same as gemini_api	Verify via Vertex docs
siliconflow_cn	Vendor/model format	Verify via SiliconFlow docs
featherless_ai	HuggingFace repo id, case-sensitive (zai-org/GLM-5.2)	Verify via their /v1/models — see Featherless

Every single model_id must be verified from an authoritative source. No exceptions.

Setting flags — use catalog data + predecessor as dual signals:

The LiteLLM catalog and models.dev responses include capability flags (supports_vision, supports_function_calling, supports_reasoning, etc.). Use these as the primary signal for what to enable on the new model:

• If the catalog says supports_vision: true → enable supports_vision, multimodal_capable, and vision MIME types (see 2c)
• If the catalog says supports_function_calling: true → use StructuredOutputMode.json_schema (or function_calling depending on provider norms — check predecessor)
• If the catalog says supports_reasoning: true → the model can reason, but do NOT reflexively set reasoning_capable=True — default to reasoning_capable=False (see Reasoning Capable Default). Still add available_thinking_levels if it supports effort levels, and check parser/formatter flags.

Then cross-check against the predecessor. The predecessor tells you how Kiln configures a similar model (which structured_output_mode, which provider-specific flags, etc.). The catalog tells you what the model can do. Use both:

• Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.
• Predecessor has temp_top_p_exclusive but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.
• Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.

Common flags:

• structured_output_mode – how the model handles JSON output
• suggested_for_evals / suggested_for_data_gen – see zero-sum rule below
• multimodal_capable / supports_vision / supports_doc_extraction – see multimodal rules below
• reasoning_capable – for thinking/reasoning models. Default new models to reasoning_capable=False unless the model always emits its reasoning (see Reasoning Capable Default)
• temp_top_p_exclusive – Anthropic models that can't have both temp and top_p
• parser / formatter – for models needing special parsing (e.g. R1-style thinking)

2c. Multimodal capabilities

If the model supports non-text inputs, configure:

• multimodal_capable=True and supports_doc_extraction=True if it supports any MIME types
• supports_vision=True if it supports images
• multimodal_requires_pdf_as_image=True if vision-capable but no native PDF support (also add KilnMimeType.PDF to MIME list). Always set this on OpenRouter providers — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.
• Always include KilnMimeType.TXT and KilnMimeType.MD on any multimodal_capable model

Strategy: start broad, narrow based on test failures. Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.

Full MIME superset (Gemini uses all):

# documents
KilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD
# images
KilnMimeType.JPG, KilnMimeType.PNG
# audio
KilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG
# video
KilnMimeType.MP4, KilnMimeType.MOV

3c. Ordering — the list IS the UI

The order of built_in_models is exactly the order users see: model dropdowns group by provider and list each provider's models in list order, and the model library page lists models in list order. A misplaced entry ships a scrambled dropdown to every client via the remote config. Rules:

1. Families are contiguous. Every model of a family sits in one block. Never append a new family member at the bottom of the file or after another family — that strands it (past bugs: Mistral Small 4/3 ended up inside the Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).

2. Within a family, versions run newest → oldest, top to bottom. 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the family block. A new model of an existing version goes inside that version's group — NOT at the top of the family and NOT below older versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7 landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).

3. Within a version, big → small. Always.

   • Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite; Large > Medium > Small.
   • Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and VL Thinking blocks) stay intact, ordered descending internally.
   • Tier-grouped families follow the same rule for their tier blocks: Claude runs Fable, then Opus > Sonnet > Haiku, versions descending inside each tier. (It once ran Haiku-first, and several families ran sizes ascending — those were bugs, not conventions. Do not preserve an ascending run because it's "what the family already does.")

4. A net-new family's block goes at the END of built_in_models (this is the existing convention — the newest niche vendors sit at the bottom of the list). The "place before predecessor" rule only applies within an existing family; a new family has no predecessor.

5. Verify after editing — print the order you just shipped and eyeball the affected family:

   uv run python -c "
   from kiln_ai.adapters.ml_model_list import built_in_models
   for m in built_in_models: print(m.family, '|', m.friendly_name)"
   

3d. suggested_for_evals / suggested_for_data_gen

Only set these if the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).

Zero-sum rule: When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. Ask the user to confirm the swap before making changes.

3e. ModelFamily enum (only if needed)

Only add a new family if the vendor is completely new.

3f. Thinking Levels (available_thinking_levels / default_thinking_level)

If the model supports configurable reasoning effort (not just on/off), add available_thinking_levels and default_thinking_level to each provider entry. See Thinking Levels Reference for the full lookup chain and existing constants.

Quick rules:

• Reuse an existing _THINKING_LEVELS constant if the levels match exactly
• Create a new constant only if levels differ; name it {MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS
• default_thinking_level must be one of the values in available_thinking_levels

---

Phase 4 – Run Tests

Tests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.

Vertex AI authentication: Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run gcloud auth application-default login before trying. These failures are auth issues, not model config problems.

-k filter syntax: Always use bracket notation for model+provider filtering, never and:

• Good: -k "test_name[glm_5-fireworks_ai]" or -k "glm_5"
• Bad: -k "glm_5 and fireworks" — and is a pytest keyword expression that can match wrong tests

4.0 — If the test env can't build (blocked git dependency)

Kiln's core lib pins together to a git fork (libs/core/pyproject.toml: together = { git = "https://github.com/scosman/together-python" }). In a sandboxed environment whose GitHub access is scoped to kiln-ai/kiln only (e.g. Claude Code Web), uv sync fails to fetch that fork with a 403 from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only kiln-ai/kiln resolves.

Workaround to build the venv for testing (the together fork isn't exercised by model-integration tests, which route through LiteLLM):

1. Do NOT use uv sync --no-sources — it strips the workspace workspace = true sources too and makes kiln-root → kiln-ai unsatisfiable.
2. Instead, temporarily comment out ONLY the together = { git = ... } line in libs/core/pyproject.toml, then run uv sync (this resolves together from PyPI).
3. Run the tests.
4. Revert the workaround with git checkout -- libs/core/pyproject.toml uv.lock — this restores the commented-out together git-fork pin in libs/core/pyproject.toml (NOT the repo-root pyproject.toml, which the workaround never touches) and reverts uv.lock if it changed. Only ml_model_list.py (and any intended test-file edits) should remain modified.

Note: the PyPI together may pull slightly different transitive deps (e.g. a newer starlette), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your -k filters to the model files and ignore those.

4a. Parallel testing + API keys

Parallel testing is already on. There is no pytest.ini — pytest config lives in [tool.pytest.ini_options] in the root pyproject.toml, and addopts = "-n auto" is active. No edit and no revert are needed. (Only override to -n 8 if a provider rate-limits you.)

Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings. conftest.py has an autouse use_temp_settings_dir fixture that points Config.settings_path at a temp dir, so ~/.kiln_ai/settings.yaml is deliberately ignored during tests. A key the user added through the app's provider page will not be seen — the test fails with "Attempted to use X without an API key set", which looks like a config bug but isn't.

Bridge the key from the user's settings into the test environment without ever printing it:

export FEATHERLESS_AI_API_KEY="$(uv run python -c \
  "from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)" 2>/dev/null | tail -1)"

Use the provider's env_var name from libs/core/kiln_ai/utils/config.py. To check which keys are available before running, print booleans only — never the values:

uv run python -c "from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))"

Many paid tests also carry the ollama marker, so --runpaid alone silently skips them. Always pass --runpaid --ollama together, and use -rs to see skip reasons when a test you expected to run reports as skipped.

4b. Smoke test — verify slug works

Run a single test+provider combo first:

uv run pytest --runpaid --ollama -k "test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]"

If it fails, fix the slug/config before proceeding. Use --collect-only to find exact parameter IDs if unsure.

4c. Full test suite

uv run pytest --runpaid --ollama -k "MODEL_ENUM" -v 2>&1 | grep -E "PASSED|FAILED|ERROR|short test|=====|collected"

If tests fail — debug one at a time:

1. Pick ONE failing test, run it with -v for full output
2. Fix the config
3. Re-run that single test to verify
4. Only re-run the full suite once the single test passes

Anthropic API key gotcha: if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as KILN_ANTHROPIC_API_KEY instead of ANTHROPIC_API_KEY (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't export it globally:

ANTHROPIC_API_KEY="$KILN_ANTHROPIC_API_KEY" uv run pytest --runpaid ...

4d. Extraction tests (if supports_doc_extraction=True)

Tests are in libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py.

# See what will run:
uv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM

# Run them:
uv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k "MODEL_ENUM"

If a provider rejects a data type (400 error), remove that KilnMimeType and re-run.

4e. Confirm failures are actually yours

Before treating a failure as a problem with your change, check whether the same test already fails for an existing provider of that model. Several assertions are provider-independent and fail regardless.

Known example: test_structured_input_cot_prompt_builder asserts len(trace) == 5 unconditionally, which is incompatible with any provider setting reasoning_capable=True (that selects the single-call strategy, producing 3 messages). It fails for gpt_oss_120b on fireworks_ai on a clean tree.

uv run pytest --runpaid --ollama -q "path::test_name[MODEL-OTHER_PROVIDER]"

If it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.

4f. Test output format

Collect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:

• ✅ for passed tests
• ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason
• ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason
• List every test using the full pytest parametrize ID, grouped by provider
• Include extraction tests (Phase 4d) if they were run

---

Phase 5 – Create Pull Request

5.0 — Important context about Claude Code Web's stop hook

This skill is often run via Claude Code Web (Slack connector). That environment has a non-user-configurable stop hook which, at end of session, will:

• Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits
• Instruct the agent to commit and push any local work before stopping
• Explicitly tell the agent NOT to create a PR unless the user asked for one

The problems this causes:

1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned add-model/* branches on the remote.
2. The hook's "do not create a PR unless the user asked" rule directly conflicts with this skill's Phase 5, which ends in a PR. Running this skill is the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.

The user's desires, in priority order:

1. Ask before you push. If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code "just to satisfy the stop hook."
2. No abandoned branches. Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.
3. If the user says to abandon: revert your local changes (git restore / git clean the specific files you touched) and delete any branch you created (git checkout main && git branch -D add-model/MODEL_NAME) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.
4. On a successful run, push and open the PR as described in 5a/5b. Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says "don't create a PR." Only re-ask if tests failed or the user hasn't confirmed the results.

5.1 — Gate before pushing

Do NOT commit, push, or create a branch if any of the following are true:

• Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)
• The smoke test (4b) failed and wasn't resolved
• Any step in Phases 2–4 was skipped or incomplete
• You are unsure whether a ⚠️ flake is actually a real failure

If any of the above apply, stop and ask the user what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.

After all tests pass, commit the changes and open a PR against main.

5a. Commit and push

1. Create a new branch named add-model/MODEL_NAME (e.g. add-model/glm-5-1)
2. Stage only the changed files (typically just ml_model_list.py)
3. Commit with a concise message (e.g. "Add GLM 5.1 to model list (together_ai, siliconflow_cn)")
4. Push the branch

5b. Create the PR

Use gh pr create against main. The PR body must follow this exact format:

## What does this PR do?

 Test Results

[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]

[Model Name] ([provider]):
- [N] passed, [N] skipped[, [N] failed]
- [Any notable failures or flakes]

[Repeat for each model+provider combo]

---
[Model Name] ([provider]):
✅ test_data_gen_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]
✅ test_all_built_in_models_llm_as_judge[model_enum-provider]
✅ test_all_built_in_models_structured_output[model_enum-provider]
✅ test_all_built_in_models_structured_input[model_enum-provider]
✅ test_structured_output_cot_prompt_builder[model_enum-provider]
✅ test_all_models_providers_plaintext[model_enum-provider]
✅ test_cot_prompt_builder[model_enum-provider]
⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason
❌ test_name[model_enum-provider] — brief reason

[Repeat for each model+provider combo]

## Checklists

- [X] Tests have been run locally and passed
- [X] New tests have been added to any work in /lib

Rules for the PR body:

• Every test that ran must appear in the per-test dump, using the full pytest parametrize ID
• Group tests by [Model Name] ([provider]): headers
• The summary section at the top gives a quick pass/skip/fail count per model+provider
• The detailed section below the --- lists every individual test result
• Use ⚠️ for content quality flakes (not real failures), ❌ for real errors

---

Checklist

• ModelName enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)
• KilnModel entry added to built_in_models per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)
• Printed the resulting list order and eyeballed the affected family (3c verification command)
• friendly_name matches the naming pattern of sibling models in the same family
• If the provider is net-new: followed Adding a Net-New Provider, including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)
• ModelFamily enum updated (only if new family)
• All provider slugs verified from authoritative sources
• Flags inherited from predecessor and adjusted for quirks
• reasoning_capable defaulted to False for adaptive-reasoning models (only True for always-emits-reasoning models — see Reasoning Capable Default)
• Thinking levels configured if model supports reasoning effort (see Thinking Levels Reference)
• Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)
• Zero-sum applied if model is suggested for evals/data gen
• RAG config templates updated if the new model replaces one used in app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts
• API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)
• Smoke test passed

---

Reasoning Capable Default

Default newly-added models to reasoning_capable=False, even when the catalog reports supports_reasoning: true.

Most recent models use adaptive reasoning and sometimes return no reasoning at all. Kiln raises RuntimeError("Reasoning is required for this model, but no reasoning was returned.") whenever a provider has reasoning_capable=True and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.

reasoning_capable is orthogonal to available_thinking_levels — you keep the important behavior with it set to False:

• Thinking-level / effort selection still works (every GPT-5.x entry sets available_thinking_levels with reasoning_capable unset).
• Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).
• The test_thinking_level_reasoning_content paid test still runs and still asserts reasoning is present — it is gated on available_thinking_levels, not reasoning_capable. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels.

What you give up with reasoning_capable=False:

• The conditional "reasoning is present in intermediate_outputs" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).
• When a user attaches a chain-of-thought prompt, the model uses the two-call two_message_cot strategy instead of the single-call native single_turn_r1_thinking.

Keep reasoning_capable=True only for models that always emit reasoning in a native <think> format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.

Narrower alternative: if a model reliably reasons but you only hit the error on structured output, set reasoning_optional_for_structured_output=True (requires reasoning_capable=True) instead of disabling reasoning entirely.

---

Adding a Net-New Provider

Adding a provider Kiln has never supported is a bigger job than adding a model, and it has a hard sequencing constraint. Follow this section end to end.

The release-gating rule (do NOT skip)

The remote config is generated from main's built_in_models and reaches all existing clients immediately — but provider support (name map, connect flow, API key handling) only reaches users through a client app release. If model entries for a brand-new provider land on main before a client release with the provider plumbing is live, every deployed client shows the raw provider ID (e.g. featherless_ai) in the model library and offers models nobody can connect to. This happened with Featherless in Aug 2026 and the entries had to be rolled back.

Sequence it in two PRs:

1. PR 1 — provider plumbing only. The ModelProviderName enum member and everything else in the checklist below EXCEPT the model catalog. Merge whenever ready.
2. PR 2 — model catalog. ALL ml_model_list.py changes: ModelName members, ModelFamily (if the vendor is new), and the built_in_models entries. Open it, but merge only after a client release containing PR 1 is live. (Only built_in_models is published via the remote config, but the enums belong in the same PR as the entries they exist for.)

Touchpoint checklist (from the Featherless integration, #1618)

First, identify the provider's authentication model — the checklist below describes the common single-API-key pattern, but not every provider fits it. Existing variants to crib from: Bedrock stores an access key + secret pair, Fireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project ID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base URL with no credentials. Follow the closest existing analog's plumbing through config.py, provider_warnings, provider_api.py, and the connect page — the credential fields, validation call, and UI steps all change with the auth model.

libs/core:

• ModelProviderName enum — libs/core/kiln_ai/datamodel/datamodel_enums.py
• Credential storage — libs/core/kiln_ai/utils/config.py (for API-key providers: a new key with env_var; otherwise whatever fields the auth model needs)
• LiteLLM provider mapping — libs/core/kiln_ai/utils/litellm.py
• libs/core/kiln_ai/adapters/provider_tools.py — three spots: the provider_name_from_id match (friendly name; pyright flags a missed case), provider_warnings (missing-credential message, required_config_keys), and the adapter config (credential / base URL / headers plumbing)
• Model catalog — libs/core/kiln_ai/adapters/ml_model_list.py: ModelName members, ModelFamily if needed, and the built_in_models entries (PR 2 only, see gating rule)

Desktop server (app/desktop/studio_server/provider_api.py):

• connect_<provider> credential-validation endpoint (find a cheap authenticated call; see the Featherless connect function for a pattern when the provider has no authenticated GET to ping)
• Disconnect handling (clear the stored credentials)
• Tests in test_provider_api.py

Web UI (app/web_ui):

• src/lib/stores.ts — provider_name_map entry (friendly name)
• src/lib/ui/provider_image.ts + SVG in static/images/ — icon must match the monochrome convention: bare glyph, currentColor, no background tile, viewBox cropped to the artwork
• Connect page — src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte: provider card (name, description, and the auth flow — api_key_steps / api_key_fields for key providers, or the custom flow the auth model needs) plus connected-status wiring from settings keys
• src/lib/api_schema.d.ts — regenerate with make schema (needs the server running on :8757)

Tests & tooling:

• libs/core/kiln_ai/adapters/test_provider_tools.py, test_adapter_registry.py, model_adapters/test_litellm_adapter.py — extend the per-provider parametrized cases
• .agents/scripts/provider_utils.py — add the provider's model-catalog endpoint so agent tooling can enumerate its models

Known gap: the generated Copilot API client (app/desktop/studio_server/api_client/) mirrors the remote Copilot service's schema — it can't learn the new provider until that service updates. Note it in the PR rather than hand-editing generated code.

Friendly names everywhere

The raw enum value must never be user-visible. When you add the provider, verify a friendly name exists in both name maps (python provider_name_from_id and web provider_name_map) — pyright and typescript respectively force these when the enum gains a member, which is why the plumbing PR must not skip them.

---

Provider Quirks Reference

Anthropic

• Newer models (Opus 4.1+, Sonnet 4.5+) need temp_top_p_exclusive=True
• Opus 4.5+ uses json_schema; older Opus uses function_calling
• Extended thinking models: anthropic_extended_thinking=True + reasoning_capable=True

OpenAI

• Most GPT models use json_schema for structured output
• GPT-5.x models support available_thinking_levels — see Thinking Levels Reference
• Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort
• o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)

Google/Gemini

• gemini_reasoning_enabled=True for reasoning-capable models
• Gemini 3.x models support available_thinking_levels — see Thinking Levels Reference
• Rich multimodal support (audio, video, images, documents)

DeepSeek

• R1 models: parser=ModelParserID.r1_thinking + reasoning_capable=True
• V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN
• Some need r1_openrouter_options=True + require_openrouter_reasoning=True

OpenRouter (general)

• Slugs: vendor/model-name
• Reasoning models: may need require_openrouter_reasoning=True
• Some models: openrouter_skip_required_parameters=True
• Logprobs: logprobs_openrouter_options=True if supported
• Always multimodal_requires_pdf_as_image=True (OpenRouter's PDF routing breaks LiteLLM)

Featherless AI

Serverless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:

• json_instructions is the only usable structured output mode. LiteLLM's featherless_ai provider rejects response_format outright (UnsupportedParamsError), so json_schema, json_mode, and json_instruction_and_object are all unavailable. Routing it as a custom openai provider bypasses that gate, but was tested and is not reliable: GLM 5.2 accepts json_schema and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.
• Gated models return HTTP 403 model_gated_needs_oauth and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. Always filter is_gated before adding. Note all 20 official meta-llama/* repos are gated, so no Llama variant is usable.
• No cost reporting. Featherless models aren't in LiteLLM's price map (only two legacy Qwerky entries, on main too — not a version issue), and Featherless doesn't return cost in the usage object. Runs record tokens with cost: null.
• Not in models.dev or the LiteLLM catalog, so their /v1/models endpoint is the only authoritative source. See Lagging Providers.
• Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.

Qwen3 / Thinking Models

• Thinking variants: reasoning_capable=True, parser=ModelParserID.r1_thinking
• No-thinking variants: formatter=ModelFormatterID.qwen3_style_no_think
• SiliconFlow may need siliconflow_enable_thinking=True/False

---

Thinking Levels Reference

No API provides the available thinking levels programmatically — they must be manually sourced. Use this lookup chain in priority order:

Lookup Chain

1. Vendor model page (most authoritative)

   • OpenAI: Each model page includes "Reasoning.effort supports: X, Y, Z" in the description text. URL: https://developers.openai.com/api/docs/models/{model-id}
   • Anthropic: The effort docs list levels per model. Opus 4.6 supports low, medium, high, max; Sonnet 4.6 supports low, medium, high.
   • Google Gemini: The models API returns thinking: true/false (boolean only). Levels come from docs.

2. Vercel AI Gateway docs — clean structured tables per provider:

   • https://vercel.com/docs/ai-gateway/capabilities/reasoning/openai
   • https://vercel.com/docs/ai-gateway/capabilities/reasoning/anthropic
   • https://vercel.com/docs/ai-gateway/capabilities/reasoning/google

3. Inherit from predecessor — if the same family/tier model has a _THINKING_LEVELS dict, the new model very likely uses the same or a superset.

4. OpenRouter supported_parameters — check if reasoning is present:

   curl -s https://openrouter.ai/api/v1/models | jq '.data[] | select(.id == "SLUG") | .supported_parameters'
   

   If reasoning is absent, the model does not support effort levels — skip thinking levels entirely.

5. Smoke test — as a last resort, send a request with an invalid effort level and check the error message, which often enumerates the valid values.

Important Distinctions

• Effort-level models (GPT-5.x, Claude 4.x, Gemini 3.x) → add available_thinking_levels dicts
• R1-style thinking models (DeepSeek R1, Qwen thinking variants) → on/off thinking, NOT effort levels. Use reasoning_capable=True + parser=ModelParserID.r1_thinking. Do NOT add thinking level dicts.
• Chat/instant models (e.g. GPT-5.3 Instant) → may not support reasoning effort at all. Verify on the vendor model page.

Existing Constants

Reuse when levels match exactly. Create a new constant only if levels differ. This is not an exhaustive list.

Constant	Levels	Default	Used by
GPT_5_4_OPENAI_THINKING_LEVELS	none, low, medium, high, xhigh	none	GPT-5.4
GPT_5_4_PRO_OPENAI_THINKING_LEVELS	medium, high, xhigh	medium	GPT-5.4 Pro
GPT_5_2_OPENAI_THINKING_LEVELS	none, low, medium, high, xhigh	none	GPT-5.2, GPT-5.2 Chat
GPT_5_2_PRO_OPENAI_THINKING_LEVELS	medium, high, xhigh	medium	GPT-5.2 Pro
GPT_5_1_OPENAI_THINKING_LEVELS	none, low, medium, high	none	GPT-5.1
GPT_5_OPENAI_THINKING_LEVELS	minimal, low, medium, high	medium	GPT-5, GPT-5 Mini, GPT-5 Nano, GPT-5 Chat
GEMINI_3_PRO_THINKING_LEVELS	low, medium, high	high	Gemini 3 Pro, Gemini 3.1 Pro
GEMINI_3_FLASH_THINKING_LEVELS	minimal, low, medium, high	high	Gemini 3 Flash, Gemini 3.1 Flash Lite
CLAUDE_ANTHROPIC_EFFORT_THINKING_LEVELS	low, medium, high	high	Claude (Anthropic direct)
CLAUDE_OPENROUTER_THINKING_LEVELS	none, minimal, low, medium, high, xhigh	none	Claude (OpenRouter)

Sources That Do NOT Work

These were investigated and confirmed to lack thinking level data:

• OpenRouter API — only boolean reasoning in supported_parameters
• LiteLLM catalog — only supports_reasoning: true/false
• Google Gemini models API — only thinking: true/false
• OpenAI /v1/models endpoint — minimal object with no capability fields

---

Slug Lookup Reference

Use both LiteLLM and models.dev when looking up slugs — they complement each other. LiteLLM gives you the exact slugs Kiln will use (since Kiln runs on LiteLLM), while models.dev often has broader coverage of newer or niche models with pricing, context limits, and capability details.

LiteLLM Model Catalog (https://api.litellm.ai/model_catalog)

100 free requests/day, no key needed. Supports server-side filtering: model= (substring match), provider=, mode=, supports_vision=true, supports_reasoning=true, page_size=500.

# Find all variants of a model across providers:
curl -s 'https://api.litellm.ai/model_catalog?model=MODEL_NAME&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[] | {id, provider, mode, max_input_tokens, supports_vision, supports_reasoning, supports_function_calling}'

# List all models for a provider:
curl -s 'https://api.litellm.ai/model_catalog?provider=PROVIDER&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[].id'

models.dev (https://models.dev/api.json)

Mega JSON covering 50+ providers with model IDs, pricing, context limits, capabilities, and release dates. Large file — always use curl+jq, never WebFetch.

# Search all model IDs across all providers:
curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"

# List all model IDs for a specific provider:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models | keys[]'

# Get full details for a specific provider+model:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'

Other verified sources

• OpenRouter: curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"
• Anthropic: https://docs.anthropic.com/en/api/models/list
• Cerebras: https://inference-docs.cerebras.ai/models/overview

Lagging Providers

Fireworks, Together, and SiliconFlow typically expose new models on their own endpoints 1–2 weeks before models.dev / LiteLLM catch up. For these providers, always cross-check directly — both when adding a new model and when running the Phase 1B backfill check.

Fireworks AI — model pages are the most current source. WebFetch directly:

WebFetch https://fireworks.ai/models/fireworks/{model-slug}

Or browse the catalog at https://fireworks.ai/models. Kiln slug format: accounts/fireworks/models/{model-slug}.

Together AI — the /v1/models endpoint requires an API key. The key is set in the user's shell as TOGETHERAI_API_KEY (note the AI — this matches LiteLLM's together_ai env var; the plain TOGETHER_API_KEY is usually NOT set):

# List all Together model IDs matching a term:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | .id' | grep -i "SEARCH_TERM"

# Full record for a specific slug:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | select(.id == "SLUG")'

If TOGETHERAI_API_KEY isn't set, ask the user before prompting them to export it — don't fail silently onto models.dev.

Featherless AI — appears in neither models.dev nor the LiteLLM catalog (the catalog has only two stale Qwerky entries), so /v1/models is the sole authoritative source. It needs no API key to enumerate.

The response is enormous — ~22,000 models, the vast majority community fine-tunes. Never WebFetch it and never dump it raw. Save it once, then filter with jq:

curl -s https://api.featherless.ai/v1/models -o /tmp/feath.json
jq '.data | length' /tmp/feath.json

Restrict to official vendor orgs, or you'll drown in forks like DavidAU/Gemma3-27B-it-vl-GLM-4.7-Uncensored-Heretic:

# newest official models, excluding gated ones (gated = unusable, see Provider Quirks)
jq -r '.data[]
  | select(.is_gated | not)
  | select(.id | test("^(deepseek-ai|zai-org|Qwen|moonshotai|MiniMaxAI|mistralai|openai|google|microsoft)/"))
  | "\(.id)\tctx=\(.context_length)\t$\(.pricing.input)/\(.pricing.output)\ttools=\(.features.tool_use // false)\timg=\(.features.image_input // false)"' \
  /tmp/feath.json | sort

Look up one model, including its capability flags and exact pricing:

jq '.data[] | select(.id == "zai-org/GLM-5.2")' /tmp/feath.json

The highest-precision way to find backfill candidates is to check whether a slug Kiln already ships for together_ai / siliconflow_cn exists verbatim here — those providers use the same HuggingFace repo-id convention, so a hit needs no guessing:

jq -r '.data[] | select(.is_gated | not) | .id' /tmp/feath.json | sort > /tmp/feath_ids.txt
grep -Fx "zai-org/GLM-5.2" /tmp/feath_ids.txt   # exit 0 = safe to add

Slugs are case-sensitive (google/gemma-4-31B-it, not -31b-it). Always confirm is_gated is false and cross-check features.tool_use / features.image_input against the flags you set — don't claim capabilities the endpoint doesn't advertise.

SiliconFlow — WebFetch the public model catalog page, or a specific model page if you have the vendor/model path:

WebFetch https://siliconflow.com/models
WebFetch https://siliconflow.com/models/{vendor}/{model}

When you find a new reliable slug source, append it here.