update-synthetic-model

name	update-synthetic-model
description	Update model metadata for the pi-synthetic extension. Use when adding or refreshing entries in extensions/provider/models.ts. Start by running the model tests, inspect current hardcoded definitions, fetch live data from Synthetic and models.dev, then update the file proactively without asking the user which model to change.

Update Synthetic model

Update extensions/provider/models.ts from live data, not guesswork.

Default behavior

Take initiative.

Do not start by asking which model to update. First detect drift, then update whatever needs updating:

1. Run pnpm install to ensure local dependencies are up to date.
2. Run the model test to find mismatches and new models.
3. Read the current hardcoded definitions in extensions/provider/models.ts.
4. Fetch live model data from:
   • https://api.synthetic.new/openai/v1/models
   • https://models.dev/api.json
5. Reconcile the differences.
6. Edit extensions/provider/models.ts.
7. Re-run the relevant tests.

Only ask the user if there is a real blocker, such as missing credentials for runtime validation or conflicting evidence you cannot resolve.

Sources of truth

Use these in order:

1. Synthetic models endpoint: https://api.synthetic.new/openai/v1/models
2. Existing test failures from extensions/provider/models.test.ts
3. https://models.dev/api.json under .synthetic.models
4. Synthetic runtime behavior via direct chat/completions calls when needed
5. If a model is missing under Synthetic on models.dev, inspect the same model under other providers on models.dev only as supporting evidence

Required workflow

0) Install dependencies

Run pnpm install first to ensure local dependencies are up to date:

pnpm install

1) Start with tests

Run the targeted model test so you know what changed:

pnpm test -- extensions/provider/models.test.ts

Use the failures to identify:

• stale fields on existing models
• models that exist in code but no longer exist upstream
• new Synthetic models missing from SYNTHETIC_MODELS
• upstream backend changes in the model provider field

If the test passes, still check for drift manually by reading the current file and comparing with fresh endpoint data. Do not assume no work is needed just because tests pass.

2) Inspect current definitions

Read:

• extensions/provider/models.ts
• extensions/provider/models.test.ts

Use the current file shape and comments as the formatting baseline.

3) Fetch Synthetic endpoint data

Query the full model list, then inspect affected models.

Example:

curl -s https://api.synthetic.new/openai/v1/models \
  | jq '.data[] | select(.id=="hf:zai-org/GLM-4.7-Flash")'

Useful narrow query:

curl -s https://api.synthetic.new/openai/v1/models \
  | jq '.data[] | select(.id==$id) | {
      id,
      name,
      input_modalities,
      output_modalities,
      context_length,
      max_output_length,
      provider,
      pricing,
      supported_features
    }' --arg id 'hf:zai-org/GLM-4.7-Flash'

The endpoint provider field is the upstream backend Synthetic uses for that model. A value of synthetic means the model is hosted by Synthetic directly. Values such as fireworks or together mean Synthetic proxies the request to that backend.

4) Fetch models.dev data

Check the Synthetic provider entry first:

curl -sL -A 'Mozilla/5.0' https://models.dev/api.json \
  | jq '.synthetic.models["hf:zai-org/GLM-4.7"]'

If missing under Synthetic, inspect other providers:

curl -sL -A 'Mozilla/5.0' https://models.dev/api.json \
  | jq 'to_entries
    | map({provider: .key, model: .value.models["hf:zai-org/GLM-4.7-Flash"]})
    | map(select(.model != null))
    | map({provider, reasoning: .model.reasoning, input: .model.modalities.input, maxTokens: .model.max_output_tokens})'

Field mapping

Copy these directly from the Synthetic endpoint when available:

• id
• name
• provider -> provider (synthetic for Synthetic-hosted models, otherwise the proxied upstream backend such as fireworks or together)
• context_length -> contextWindow
• max_output_length -> maxTokens when present and trustworthy
• pricing.prompt -> cost.input per 1M
• pricing.completion -> cost.output per 1M
• pricing.input_cache_reads -> cost.cacheRead per 1M
• pricing.input_cache_writes -> cost.cacheWrite per 1M
• input_modalities -> input

Cross-check these from models.dev:

• reasoning
• modalities.input
• max_output_tokens / output token limit when Synthetic metadata is absent or suspicious

Decision rules

• Start from test failures, but update all clearly stale entries you find in the same pass.
• Add new models when the Synthetic endpoint exposes them and they fit the existing provider scope.
• Remove models only when they are truly gone from Synthetic, not because of a temporary fetch issue.
• Set input from the Synthetic endpoint first.
• Set pricing from the Synthetic endpoint.
• Set provider from the Synthetic endpoint. Do not infer hosting from the model name. synthetic means Synthetic-hosted; fireworks, together, or another value means Synthetic proxies to that backend.
• A model with provider other than "synthetic" will be hidden from users when the Proxied Models setting is disabled.
• Set contextWindow from the Synthetic endpoint.
• Set maxTokens from Synthetic when exposed; otherwise use models.dev Synthetic data.
• Set reasoning from:
  1. confirmed Synthetic runtime behavior
  2. else Synthetic endpoint supported_features
  3. else models.dev Synthetic entry
  4. else other providers on models.dev as weak evidence only
• Keep existing compat unless live behavior or current repo conventions show it should change.
• Do not ask the user which models to update unless there is a true ambiguity you cannot resolve.

Alias models

Synthetic exposes permanent aliases (IDs starting with syn:) that route to underlying concrete models. These are thin references — they have no provider, cost, contextWindow, or compat of their own.

Identifying aliases: Query the API for entries with hugging_face_id where the ID does not start with hf::

curl -s https://api.synthetic.new/openai/v1/models \
  | jq '.data[] | select(.hugging_face_id != null and (.id | startswith("hf:") | not)) | {id, name, hugging_face_id}'

Mapping:

• id -> id (keep the syn: prefix)
• name -> name
• hugging_face_id -> aliasFor, prefixed with hf: (e.g. hugging_face_id: "zai-org/GLM-5.1" becomes aliasFor: "hf:zai-org/GLM-5.1")

Rules:

• syn:* entries must be at the top of SYNTHETIC_MODELS to stay visible regardless of proxiedModels setting
• Alias entries are only { id, name, aliasFor }. Do not copy provider, cost, contextWindow, maxTokens, compat, or thinkingLevelMap from the API
• Alias metadata (cost, contextWindow, compat, thinkingLevelMap) is resolved at build time from the concrete target
• Alias provider is always forced to "synthetic" at build time regardless of the target's actual provider
• When Synthetic changes which model an alias routes to, update only the aliasFor field
• If an alias's API-reported metadata diverges from its target's metadata (e.g. pricing, context length), alert the user — this likely means the alias should become a concrete entry or the target has changed

Reasoning level classification for aliases:

• Classify reasoning for the concrete target model only, not the alias
• The alias inherits thinkingLevelMap and compat.supportsReasoningEffort from its target automatically
• Do not set thinkingLevelMap or compat on alias entries

Required runtime checks

Do not rely only on metadata for reasoning or multimodal support when the evidence is mixed or when you are adding a new model with unclear behavior.

Use the environment variable SYNTHETIC_API_KEY. Never print it.

Reasoning check

curl -sS https://api.synthetic.new/openai/v1/chat/completions \
  -H "Authorization: Bearer $SYNTHETIC_API_KEY" \
  -H 'Content-Type: application/json' \
  -d @- <<'JSON'
{
  "model": "hf:zai-org/GLM-4.7-Flash",
  "messages": [{"role": "user", "content": "Reply with ok"}],
  "reasoning_effort": "low",
  "max_completion_tokens": 64
}
JSON

Treat reasoning as supported if the request succeeds and clearly accepts reasoning mode.

Reasoning level classification

When adding or updating a reasoning model, determine whether it supports multiple reasoning levels or is binary on/off. This affects the thinkingLevelMap and compat.supportsReasoningEffort settings.

Test the model with reasoning_effort set to low, medium, and high:

for effort in low medium high; do
  echo "=== $effort ==="
  curl -sS https://api.synthetic.new/openai/v1/chat/completions \
    -H "Authorization: Bearer $SYNTHETIC_API_KEY" \
    -H 'Content-Type: application/json' \
    -d @- <<JSON
{
  "model": "MODEL_ID",
  "messages": [{"role": "user", "content": "What is 17*23? Reply with just the number."}],
  "reasoning_effort": "$effort",
  "max_completion_tokens": 256
}
JSON
done

Compare reasoning_content length and reasoning_tokens across the three levels:

• Multi-level: reasoning_content length differs substantially across levels (e.g. 14c vs 208c vs 586c). No thinkingLevelMap needed; Pi's default level map applies.
• Binary on/off: reasoning_content is either absent or roughly the same at all non-off levels. Set thinkingLevelMap to { off: "none", minimal: null, low: null, medium: "medium", high: null, xhigh: null } so Pi presents a single reasoning toggle.

Also test whether reasoning_effort: "none" actually disables reasoning:

curl -sS https://api.synthetic.new/openai/v1/chat/completions \
  -H "Authorization: Bearer $SYNTHETIC_API_KEY" \
  -H 'Content-Type: application/json' \
  -d @- <<'JSON'
{
  "model": "MODEL_ID",
  "messages": [{"role": "user", "content": "What is 17*23? Reply with just the number."}],
  "reasoning_effort": "none",
  "max_completion_tokens": 256
}
JSON

• If reasoning_content disappears and reasoning_tokens drops to 0: set off: "none" (Pi sends reasoning_effort: "none" when the user disables reasoning).
• If the model still produces reasoning_content with "none": set off: null (hides the "off" level from Pi's UI since the model cannot disable reasoning).

When adding a thinkingLevelMap, also add supportsReasoningEffort: true to compat so Pi sends the reasoning_effort parameter.

Image input check

curl -sS https://api.synthetic.new/openai/v1/chat/completions \
  -H "Authorization: Bearer $SYNTHETIC_API_KEY" \
  -H 'Content-Type: application/json' \
  -d @- <<'JSON'
{
  "model": "hf:zai-org/GLM-4.7-Flash",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image? Reply in 3 words max."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9WnR0i8AAAAASUVORK5CYII="}}
      ]
    }
  ],
  "max_completion_tokens": 32
}
JSON

If Synthetic rejects image input, keep input: ["text"].

Compat rules

extensions/provider/models.ts includes a required provider field and supports an optional compat object per model.

provider is maintenance metadata only. registerSyntheticProvider strips it before registering models with Pi, so it must describe Synthetic's upstream backend, not the Pi provider name users select.

Only add or change compat when live behavior, provider quirks, or current repo conventions require it.

Model-level fields:

• thinkingLevelMap — maps Pi thinking levels to provider-specific values; null hides a level from the UI

Compat fields:

• supportsDeveloperRole
• supportsReasoningEffort
• maxTokensField
• requiresToolResultName
• requiresMistralToolIds

Do not add compat by default.

Aliases inherit compat and thinkingLevelMap from their target. Do not add compat or thinkingLevelMap to alias entries.

Output expectations

When done:

1. Ensure extensions/provider/models.ts is updated, including correct provider values for Synthetic-hosted vs proxied models.
2. Re-run pnpm test -- extensions/provider/models.test.ts.
3. If the change is user-facing, prepare a changeset per repo conventions.
4. Commit the model update and changeset. Never use --no-verify.
5. If the pre-commit hooks fail (typecheck, lint, test), stash the model changes (git stash) and investigate the failing hook. Fix the underlying issue but do not commit the fix yourself — report the findings to the user and let them decide.
6. Summarize what changed, including newly added, removed, or materially corrected models.

Known repo paths

Use these exact paths in this repo:

• extensions/provider/models.ts
• extensions/provider/models.test.ts