Fixed workflow for developing AgentHub itself — adding or updating model support. Use when asked to support a new model or protocol version in this repository, sync llmsdk_docs, or implement a provider client. Covers doc syncing, live API capture, paired Python/TypeScript implementation, and model-scoped e2e testing.
Fixed workflow for developing AgentHub itself — adding or updating model support. Use when asked to support a new model or protocol version in this repository, sync llmsdk_docs, or implement a provider client. Covers doc syncing, live API capture, paired Python/TypeScript implementation, and model-scoped e2e testing.
AgentHub Development Workflow
Adding or updating model support follows the stages below, in order. Where a stage says stop and ask, pause and ask the user; do not continue until the issue is resolved, and never fill the gap yourself.
Directory map
llmsdk_docs/<model_version>/ Official docs snapshot, one folder per model generation (README.md + docs/)
api_captures/<protocol>/ Git-ignored raw API captures: request payloads + stream events
src_py/agenthub/<protocol>/ Python client, one folder per wire protocol
src_py/agenthub/auto_client.py Routes model names to protocol clients by explicit version
src_ts/src/<protocol>/ TypeScript client, mirrors the Python folder
src_ts/src/autoClient.ts TypeScript routing, mirrors auto_client.py
src_py/tests/test_client.py Parameterized e2e tests (env-gated AVAILABLE_MODELS)
src_ts/tests/client.test.ts Same for TypeScript
changelog/unreleased/ Where unshipped changes go; renamed to the version at release
changelog/<version>/ Release summary (README.md) plus one detail file per entry
changelog/README.md The entry format: metadata block, body rules, bilingual pairing
CHANGELOG.md One brief line per release linking into changelog/
<name>.zh.md Chinese counterpart, required for every file in changelog/
Stage 1 — Sync official docs into llmsdk_docs/
Sources must be the model vendor's official documentation site (e.g. docs.anthropic.com, platform.openai.com, ai.google.dev). Never use third-party mirrors, blog posts, or model memory.
Save the snapshot under llmsdk_docs/<model_version>/ following the existing folder conventions, and list the folder in llmsdk_docs/README.md. Running this workflow is the explicit request that the repository rule against editing llmsdk_docs/ asks for.
When the fetched docs differ from an existing snapshot, the new official docs win: update the old files in place.
The snapshot must be complete enough to implement from: request/response schemas, streaming event sequence, thinking output, tool calling, usage fields, and error responses.
Stop and ask if the official URL is uncertain or a page cannot be fetched. The user can paste the content manually.
Stage 2 — Capture a live API exchange into api_captures/
Gate: the provider's API key environment variable must be set and usable. Use the same environment variables and base URLs as src_py/tests/test_client.py (AVAILABLE_MODELS gating and _create_client). Stop and ask the user to supply the key if it is missing; the workflow must not continue without it.
Using the provider's official SDK, or raw HTTP exactly as documented, run one streaming tool-call request with thinking enabled, then send the tool result back so the capture also shows how assistant turns are re-sent.
Save the complete exchange unmodified under api_captures/<protocol>/ (git-ignored), e.g. round1.request.json plus round1.stream.jsonl with every raw stream event in order. Never save credentials.
When something in the exchange will not serialize to JSON (binary payloads, SDK objects holding non-JSON values), save its str() form and analyze from that. Keep the container parseable: wrap the text in a JSON object so the .jsonl stays one JSON value per line ({"unserializable_str": "<str(event)>"}, keeping the event in stream order), or put the whole exchange in a sibling .txt when nothing about it serializes. A str() capture is still the authoritative record of what the API returned. Never drop the event, hand-edit it into valid JSON, or fall back to the docs because serialization failed.
Stop and ask on any API error (invalid key, insufficient quota, rate limit). Do not mock the response or continue from docs alone.
The capture is the primary implementation reference and outranks the docs: where they disagree, implement what the API actually returned.
Then probe the replay, because the capture only shows what the API sends, not what it needs back. Re-send the assistant turn repeatedly, each time with one candidate field removed — the reasoning item's id, its content, the whole reasoning item, each field you were tempted to keep — and record which removals the API still accepts with the model's behavior intact. That result, not the shape of the response, decides what Stage 3 stores. Save the probe outcomes next to the capture.
Stage 3 — Implement the Python and TypeScript clients
One folder per wire protocol, named after the newest model generation that uses it. Diff the new protocol (capture + docs) against the closest existing folder:
Any difference between generations, even a single key name, means a separate folder per generation (e.g. claude4_6/ vs claude5/).
Only an identical wire protocol may share a folder; name it after the newest generation (rename and reroute if needed). This is how claude5/ serves Claude 4.7, 4.8, and 5.
When the old and new generations' implementations differ, keep the old model supported: leave its client folder and routing in place and add a new client folder for the new model. Never delete or rewire away an old model's client unless the user explicitly instructs it.
auto_client.py / autoClient.ts route model names by explicit version matching only, never a bare substring like "claude" in model.
Replay must be minimal, not exhaustive. A replayed assistant turn has to be accepted by the API with the model's behavior intact; it does not have to reproduce the wire item field for field. fidelity is an exception channel, not a mirror of the payload: a field earns a place there only when the Stage 2 probe shows the request fails or the model degrades without it. Provider-generated ids the API regenerates or ignores — a reasoning item id, an output-item id — stay out. Fields the API demands go in: GPT-5.5's phase, Claude's thinking signature, the exact reasoning field name a strict upstream requires.
Never copy into fidelity what a universal field already carries — the thinking text, a tool call's name or parsed arguments. Rebuild the wire item from the universal fields instead.
tool_call_id is not optional: always capture the provider's call id and replay it. Where the wire format carries both an item id and a call id, the call id is the one that correlates a result to its call.
Follow the reference client; do not redesign.gpt5_5/ is the shape for Responses-style protocols, openai/ for Chat Completions: same method order, same control flow, same names. A new client should read as a diff against its reference, because that is how it will be reviewed.
Keep it readable top to bottom. A reader should follow one request or response straight through the file without jumping between definitions, so inlining is the default: read the SDK's attributes directly (model_output.delta, model_output.item.call_id) and keep field access, usage arithmetic, and error text where they are used. Extract a helper only for a genuinely large, self-contained block — the kind that would bury the main flow if inlined, such as fetching and decoding an image — never for a few lines. Mirror the reference client's own private methods (, ) instead of inventing a layer beside them; a shim that accepts both dicts and SDK objects is never one of them, because the events are typed with the SDK's own types.
Stage 4 — Verify
Register the model in the env-gated AVAILABLE_MODELS lists of both test files with correct capability flags. Do not add model-specific test functions or files.
That AVAILABLE_MODELS entry is the only test change a new model is entitled to. src_py/tests/test_client.py and src_ts/tests/client.test.ts are shared contracts every client must already satisfy: never rewrite their bodies, assertions, prompts, or helpers to accommodate one model, and never branch inside them on a model name. A failing shared test means the client is wrong until proven otherwise. Stop and ask for explicit approval before any broader test edit, and get it per change — approval for one edit is not approval for the next.
AVAILABLE_MODELS keeps only the newest version of each model family per provider block (e.g. gemini-3.6-flash, not gemini-3.5-flash or 3.5-flash-lite as well). When a newer generation lands, replace the older entry — the old client folder stays supported and routed (see Stage 3) but is no longer e2e-tested.
Static checks: make lint in src_py/; npm run lint and npm run build in src_ts/.
Run only the new model's e2e tests; the full suites are slow and spend real API quota:
cd src_py && uv run pytest -vvv tests/test_client.py -k "<model-name>"
cd src_ts && npm run test -- -t "<model-name>"
Leave unrelated tests to CI.
Supported-model registry
src_py/agenthub/registry.py / src_ts/src/registry.ts list the supported models as entries of (model, base_url, client) plus input/output modalities, context window, and USD-stored pricing keyed by AgentHub's usage buckets. Keep both languages identical; the registry unit test constructs every entry through AutoLLMClient.
For OpenRouter-hosted entries, pull authoritative data from the live models API GET https://openrouter.ai/api/v1/models (docs: https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties): pricing.prompt/completion are USD per token (multiply by 1e6), plus context_length and architecture.input_modalities/output_modalities. The API lists chat models only — embedding models are absent and must be checked via their model pages.
SiliconFlow publishes no pricing API; declare official CNY list prices with the cny() initializer (converted to USD storage at 7 CNY/USD).
Record and ship
Unreleased changes all land in changelog/unreleased/, named for its state rather than a number because the version is not decided until release. Never create a numbered folder for unshipped work and never invent the next version number; release preparation renames unreleased/ to the decided version.
Write changelog/unreleased/YYYY-MM-DD-<slug>.md in the format changelog/README.md specifies: the metadata block (Date, Type, Scope, PR, Issue, Breaking) directly under the title, then ## What changed recording what the change did, plus the factual detail it introduced (config mappings, registry metadata, protocol differences) and ## Compatibility when it breaks something.
The entry records what was done, never the reasoning behind it and never the state of the codebase. ## Why, ## Problem, ## Decision, ## Alternatives considered, ## Verification, ## Risks do not belong on disk, and neither do claims like "X is not exported from Y": they read as fact, go stale as the code moves, and every later reader pays to re-check them.
The reasoning is still required — it goes where it stays tied to its moment. Report it in the conversation as the work happens (what the probes returned, which alternatives you weighed and why the others lost, what the verification actually covered, what you are unsure of), and write it into the PR description. Anyone who needs it later pulls the PR description, git log, or git blame.
Link the PR and any issue the change closes as full URLs (https://github.com/Prism-Shadow/agenthub/pull/N), in the metadata block and in the release-README line. A bare #N is not a link in a Markdown file. The PR number only exists once the PR is open, so open it first, then add both links in a follow-up commit on the same branch.
Write the Chinese counterpart changelog/<version>/YYYY-MM-DD-<slug>.zh.md in the same PR, mirroring the English file section for section. The metadata block stays English verbatim (only the Breaking reason is prose to translate), as do code identifiers, model ids, and links; changelog/README.md lists the standard heading renderings. An entry without its counterpart is unfinished.
Add one line at the top of that version's and the matching line in , whose link points at the entry; the root and keep one line per release, added at release preparation.
_convert_thinking_level_*
_convert_tool_choice
Stream on deltas only. Open a partial tool call on the item-added event (name plus call id), accumulate its arguments from the argument deltas, and emit the complete tool_call item at the terminal event, exactly as gpt5_5 does — every client must emit a complete tool_call, not just partials. List completion events (response.output_item.done and its equivalents) with the ignored types: never re-read a completed item or cross-check it against what the deltas produced. Leave provider error events (response.failed, response.error, error) to the unknown-event guard rather than translating them into AgentHub errors.
unused never leaves the client. A wire event a client has nothing universal to emit for gets event_type: "unused", and the streaming loop skips it — it is the client's own bookkeeping, not an event a caller may see. The base client drops one that escapes and raises on it under AGENTHUB_DEBUG, so a client that forgets the filter fails in CI rather than shipping empty events.
The message transform keeps the order of the content items. Whatever a turn produced — thinking, then text, then a tool call — has to leave the transform in that order. Where a protocol makes an item its own entry (a Responses reasoning / function_call, say) and the message text is collected separately, flush the collected text before appending the entry; a message appended after the items it preceded is what DeepSeek answers with "No tool output found for tool call". Anthropic Messages and Gemini keep one ordered block list per message, so appending in item order is enough; Chat Completions splits content and tool_calls into fields of one message and carries no interleaving at all.
UniConfig keys rarely map one-to-one onto provider config keys. Stop and ask: list every non-obvious mapping and confirm it with the user before coding. Never decide silently.
Every ThinkingLevel must stay usable on every client — never raise for a thinking level. Map each level to the closest level the model supports and degrade silently when a level has no exact equivalent (e.g. gemini3_7 maps NONE to MINIMAL, or to low on the models that reject minimal; kimi_k3 maps NONE to low because K3 cannot disable reasoning).
temperature and tool_choice (and other unsupported parameter values, e.g. prompt_caching) may reject with an exception, but must raise the AgentHub-specific UnsupportedParameterError from errors.py / errors.ts, never a bare ValueError/Error. Keep the message wording consistent with existing clients (containing "not support").
Implement Python and TypeScript together with identical behavior.
changelog/<version>/README.md
README.zh.md
[详情]
.zh.md
CHANGELOG.md
CHANGELOG.zh.md
Commit on a feature branch and open a PR with gh pr create --base dev; direct pushes to dev are rejected.
