test-reasoning

Stars7,464

Forks1,441

UpdatedApril 2, 2026 at 12:27

Validate that reasoning parameters are correctly serialized and sent to provider APIs. Use when the user asks to test reasoning serialization, run reasoning tests, verify reasoning config fields, or check that ReasoningConfig maps correctly to provider-specific JSON (OpenRouter, Anthropic, GitHub Copilot, Codex).

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

tailcallhq

tailcallhq/forgecode

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Software Quality Assurance Analysts and TestersComputer and Mathematical Occupations·SOC 15-1253

File Explorer

2 files

SKILL.md

readonly

name	test-reasoning
description	Validate that reasoning parameters are correctly serialized and sent to provider APIs. Use when the user asks to test reasoning serialization, run reasoning tests, verify reasoning config fields, or check that ReasoningConfig maps correctly to provider-specific JSON (OpenRouter, Anthropic, GitHub Copilot, Codex).

Test Reasoning Serialization

Validates that ReasoningConfig fields are correctly serialized into provider-specific JSON for OpenRouter, Anthropic, GitHub Copilot, and Codex.

Quick Start

Run all tests with the bundled script:

./scripts/test-reasoning.sh

The script builds forge in debug mode, runs each provider/model combination, captures the outgoing HTTP request body via FORGE_DEBUG_REQUESTS, and asserts the correct JSON fields.

Running a Single Test Manually

FORGE_DEBUG_REQUESTS="forge.request.json" \
FORGE_SESSION__PROVIDER_ID=<provider_id> \
FORGE_SESSION__MODEL_ID=<model_id> \
FORGE_REASONING__EFFORT=<effort> \
target/debug/forge -p "Hello!"

Then inspect .forge/forge.request.json for the expected fields.

Test Coverage

Provider	Model	Config fields	Expected JSON field
`open_router`	`openai/o4-mini`	`effort: none\|minimal\|low\|medium\|high\|xhigh`	`reasoning.effort`
`open_router`	`openai/o4-mini`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`openai/o4-mini`	`effort: high` + `exclude: true`	`reasoning.effort` + `.exclude`
`open_router`	`openai/o4-mini`	`enabled: true`	`reasoning.enabled`
`open_router`	`anthropic/claude-opus-4-5`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`moonshotai/kimi-k2`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`moonshotai/kimi-k2`	`effort: high`	`reasoning.effort`
`open_router`	`minimax/minimax-m2`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`minimax/minimax-m2`	`effort: high`	`reasoning.effort`
`anthropic`	`claude-opus-4-6`	`effort: low\|medium\|high\|max`	`output_config.effort`
`anthropic`	`claude-3-7-sonnet-20250219`	`enabled: true` + `max_tokens: 8000`	`thinking.type` + `budget_tokens`
`github_copilot`	`o4-mini`	`effort: none\|minimal\|low\|medium\|high\|xhigh`	`reasoning_effort` (top-level)
`codex`	`gpt-5.1-codex`	`effort: none\|minimal\|low\|medium\|high\|xhigh`	`reasoning.effort` + `.summary`
`codex`	`gpt-5.1-codex`	`effort: medium` + `exclude: true`	`reasoning.summary = "concise"`
all providers	one model each	`effort: invalid`	non-zero exit, no request written

Tests for unconfigured providers are skipped automatically. Invalid-effort tests run regardless of credentials — the rejection happens at config parse time before any provider interaction.

References

More from this repository

same repository

write-release-notes

tailcallhq/forgecode

Generate engaging, high-energy release notes for a given version tag. Fetches the release from GitHub, retrieves every linked PR's title and description, then synthesizes all changes into a polished, user-facing release note with an enthusiastic tone. Use when the user asks to write, generate, or create release notes for a version (e.g. "write release notes for v1.32.0", "generate release notes for the latest release", "create changelog for v2.0").

2026-04-137.5k

resolve-fixme

tailcallhq/forgecode

Find all FIXME comments across the codebase and fully implement the work they describe. Use when the user asks to fix, resolve, or address FIXME comments, or when running the "fixme" command. Runs a discovery script to find every FIXME, expands multiline comment blocks, groups related FIXMEs across files into a single implementation task, completes the full underlying code changes, removes the FIXME comments only after the work is done, and verifies that no FIXMEs remain.

2026-04-137.5k

github-pr-comments

tailcallhq/forgecode

Resolve inline code review comments on a GitHub PR. Use when asked to "resolve review comments", "address PR feedback", "fix PR comments", or "work through review comments". Fetches every inline comment with its surrounding code context, then applies each change systematically.

2026-04-027.5k

post-forge-feature

tailcallhq/forgecode

Generate a Twitter/X post highlighting a Forge feature. Use when the user asks to write a tweet, create a Twitter post, or promote a ForgeCode feature on social media. The post always accompanies an attached video demonstrating the feature.

2026-03-217.5k

github-pr-description

tailcallhq/forgecode

Generate and create pull request descriptions automatically using GitHub CLI. Use when the user asks to create a PR, generate a PR description, make a pull request, or submit changes for review. Analyzes git diff and commit history to create comprehensive, meaningful PR descriptions that explain what changed, why it matters, and how to test it.

2026-03-037.5k

create-github-issue

tailcallhq/forgecode

Create GitHub issues using GitHub CLI with support for templates, labels, assignees, milestones, and draft issues. Use when the user asks to create a GitHub issue, file a bug report, submit a feature request, or open an issue in a GitHub repository.

2026-02-127.5k

name	test-reasoning
description	Validate that reasoning parameters are correctly serialized and sent to provider APIs. Use when the user asks to test reasoning serialization, run reasoning tests, verify reasoning config fields, or check that ReasoningConfig maps correctly to provider-specific JSON (OpenRouter, Anthropic, GitHub Copilot, Codex).

Test Reasoning Serialization

Validates that ReasoningConfig fields are correctly serialized into provider-specific JSON for OpenRouter, Anthropic, GitHub Copilot, and Codex.

Quick Start

Run all tests with the bundled script:

./scripts/test-reasoning.sh

The script builds forge in debug mode, runs each provider/model combination, captures the outgoing HTTP request body via FORGE_DEBUG_REQUESTS, and asserts the correct JSON fields.

Running a Single Test Manually

FORGE_DEBUG_REQUESTS="forge.request.json" \
FORGE_SESSION__PROVIDER_ID=<provider_id> \
FORGE_SESSION__MODEL_ID=<model_id> \
FORGE_REASONING__EFFORT=<effort> \
target/debug/forge -p "Hello!"

Then inspect .forge/forge.request.json for the expected fields.

Test Coverage

Provider	Model	Config fields	Expected JSON field
`open_router`	`openai/o4-mini`	`effort: none\|minimal\|low\|medium\|high\|xhigh`	`reasoning.effort`
`open_router`	`openai/o4-mini`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`openai/o4-mini`	`effort: high` + `exclude: true`	`reasoning.effort` + `.exclude`
`open_router`	`openai/o4-mini`	`enabled: true`	`reasoning.enabled`
`open_router`	`anthropic/claude-opus-4-5`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`moonshotai/kimi-k2`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`moonshotai/kimi-k2`	`effort: high`	`reasoning.effort`
`open_router`	`minimax/minimax-m2`	`max_tokens: 4000`	`reasoning.max_tokens`
`open_router`	`minimax/minimax-m2`	`effort: high`	`reasoning.effort`
`anthropic`	`claude-opus-4-6`	`effort: low\|medium\|high\|max`	`output_config.effort`
`anthropic`	`claude-3-7-sonnet-20250219`	`enabled: true` + `max_tokens: 8000`	`thinking.type` + `budget_tokens`
`github_copilot`	`o4-mini`	`effort: none\|minimal\|low\|medium\|high\|xhigh`	`reasoning_effort` (top-level)
`codex`	`gpt-5.1-codex`	`effort: none\|minimal\|low\|medium\|high\|xhigh`	`reasoning.effort` + `.summary`
`codex`	`gpt-5.1-codex`	`effort: medium` + `exclude: true`	`reasoning.summary = "concise"`
all providers	one model each	`effort: invalid`	non-zero exit, no request written

Tests for unconfigured providers are skipped automatically. Invalid-effort tests run regardless of credentials — the rejection happens at config parse time before any provider interaction.