Skip to main content Home Creators adu2021 skillxiv thinkdial-reasoning-effort-control
thinkdial-reasoning-effort-control Control LLM reasoning effort through discrete modes (High/Medium/Low) using budget-aware supervised fine-tuning and adaptive reward shaping, enabling compression-performance tradeoffs.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/ADu2021/skillXiv --skill thinkdial-reasoning-effort-controlThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name thinkdial-reasoning-effort-control title ThinkDial: Controlling Reasoning Effort in Large Language Models version 0.0.2 engine skillxiv-v0.0.2-claude-opus-4.6 license MIT url https://arxiv.org/abs/2508.18773 keywords ["reasoning-budget","token-compression","controllable-inference","budget-aware-training","reward-shaping"] description Control LLM reasoning effort through discrete modes (High/Medium/Low) using budget-aware supervised fine-tuning and adaptive reward shaping, enabling compression-performance tradeoffs.
ThinkDial: Controlling Reasoning Effort in LLMs
Core Concept
ThinkDial enables dynamic control over reasoning computation in LLMs through discrete modes: High (full capability), Medium (50% token reduction), and Low (75% token reduction). The approach combines budget-aware supervised fine-tuning (embedding budget constraints early) with two-phase reinforcement learning (offline stability + online refinement). This enables practical compression-performance tradeoffs comparable to proprietary systems while remaining open-source.
Architecture Overview
Discrete Reasoning Modes : Three effort levels with clear token budgetsBudget-Aware SFT : Fine-tuning with explicit budget constraintsTwo-Phase RL : Offline + online for stable trainingAdaptive Reward Shaping : Budget-dependent rewardsMode Switching : Runtime selection of reasoning effort
Implementation Steps
1. Define Reasoning Modes and Budgets
Create discrete effort levels:
from dataclasses import dataclass
from enum import Enum
from typing import Dict
class ReasoningMode (Enum ):
HIGH = "high"
MEDIUM = "medium"
LOW = "low"
@dataclass
class BudgetConfig :
"""Token budget for reasoning mode."""
mode: ReasoningMode
max_tokens: int
thinking_proportion: float
description: str
class ThinkDialConfig :
"""Configuration for ThinkDial reasoning modes."""
BUDGETS = {
ReasoningMode.HIGH: BudgetConfig(
mode=ReasoningMode.HIGH,
max_tokens=1000 ,
thinking_proportion=0.7
"Full reasoning capability, highest quality"
500
0.5
"Balanced reasoning, moderate compression"
250
0.3
"Minimal reasoning, maximum compression"
@staticmethod
def
get_budget
mode: ReasoningMode
return
@staticmethod
def
get_budget_embedding
mode: ReasoningMode
str
"""Textual embedding of budget constraint for prompting."""
return
f"[BUDGET: {mode.value.upper()} - {budget.max_tokens} tokens max]"
2. Implement Budget-Aware SFT Fine-tune models with explicit budget constraints:
import torch
import torch.nn.functional as F
class BudgetAwareSFTTrainer :
"""Supervised fine-tuning with budget constraints."""
def __init__ (
self,
model: "LLM" ,
learning_rate: float = 1e-5
):
self .model = model
self .optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
def compute_budget_aware_loss (
self,
input_ids: torch.Tensor,
target_ids: torch.Tensor,
mode: ReasoningMode,
current_length: int = 0
) -> torch.Tensor:
"""
Compute SFT loss with budget awareness.
"""
budget_config = ThinkDialConfig.get_budget(mode)
remaining_budget = budget_config.max_tokens - current_length
outputs = self .model(input_ids, labels=target_ids)
logits = outputs.logits
token_loss = F.cross_entropy(
logits.view(-1 , logits.shape[-1 ]),
target_ids.view(-1 ),
reduction='none'
).view(target_ids.shape)
sequence_length = target_ids.shape[1 ]
if sequence_length > remaining_budget:
excess_tokens = sequence_length - remaining_budget
budget_penalty = excess_tokens * 0.5
token_loss = token_loss + budget_penalty / sequence_length
if mode == ReasoningMode.HIGH:
token_loss = token_loss * 0.8
elif mode == ReasoningMode.LOW:
token_loss = token_loss * 1.2
return token_loss.mean()
def train_budget_aware_sft (
self,
train_data: List [Dict ],
num_epochs: int = 3 ,
batch_size: int = 8
) -> Dict [str , list ]:
"""Train SFT with budget awareness."""
losses = {"high" : [], "medium" : [], "low" : []}
for epoch in range (num_epochs):
for batch_idx in range (0 , len (train_data), batch_size):
batch = train_data[batch_idx:batch_idx + batch_size]
batch_loss_by_mode = {m: [] for m in ReasoningMode}
for example in batch:
input_ids = torch.tensor(example["input_ids" ])
target_ids = torch.tensor(example["target_ids" ])
mode = example.get("mode" , ReasoningMode.HIGH)
loss = self .compute_budget_aware_loss(
input_ids.unsqueeze(0 ),
target_ids.unsqueeze(0 ),
mode
)
batch_loss_by_mode[mode].append(loss.item())
self .optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self .model.parameters(), 1.0 )
self .optimizer.step()
for mode in ReasoningMode:
if batch_loss_by_mode[mode]:
avg = sum (batch_loss_by_mode[mode]) / len (batch_loss_by_mode[mode])
losses[mode.value].append(avg)
return losses
3. Implement Two-Phase RL Training Offline + online RL with budget adaptation:
class TwoPhaseRLTrainer :
"""Two-phase RL: offline stable + online refined."""
def __init__ (
self,
model: "LLM" ,
offline_steps: int = 500 ,
online_steps: int = 200 ,
learning_rate: float = 1e-5
):
self .model = model
self .offline_steps = offline_steps
self .online_steps = online_steps
self .optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
def compute_budget_aware_reward (
self,
response: str ,
ground_truth: str ,
mode: ReasoningMode,
actual_tokens: int
) -> float :
"""
Compute reward considering both accuracy and budget compliance.
"""
accuracy = self ._compute_accuracy(response, ground_truth)
budget_config = ThinkDialConfig.get_budget(mode)
budget_used_ratio = actual_tokens / budget_config.max_tokens
if budget_used_ratio > 1.0 :
budget_compliance = 1.0 - (budget_used_ratio - 1.0 ) * 0.5
else :
budget_compliance = 1.0 + (1.0 - budget_used_ratio) * 0.1
budget_compliance = max (0.0 , budget_compliance)
reward = 0.8 * accuracy + 0.2 * budget_compliance
return reward
def offline_rl_phase (
self,
dataset: List [Dict ],
mode: ReasoningMode
) -> Dict [str , float ]:
"""
Offline RL: learn from fixed dataset.
Prioritize stability and budget constraint satisfaction.
"""
print (f"Offline RL Phase for {mode.value} mode" )
metrics = {"loss" : [], "reward" : []}
for step in range (self .offline_steps):
batch = self ._sample_batch(dataset, batch_size=4 )
batch_loss = 0.0
batch_reward = 0.0
for example in batch:
response = self .model.generate(
example["input" ],
mode=mode,
max_tokens=ThinkDialConfig.get_budget(mode).max_tokens
)
reward = self .compute_budget_aware_reward(
response,
example["ground_truth" ],
mode,
len (response.split())
)
log_prob = self .model.get_log_prob(response)
loss = -log_prob * reward
batch_loss += loss.item()
batch_reward += reward
avg_loss = batch_loss / len (batch)
avg_reward = batch_reward / len (batch)
self .optimizer.zero_grad()
(torch.tensor(avg_loss)).backward()
torch.nn.utils.clip_grad_norm_(self .model.parameters(), 1.0 )
self .optimizer.step()
metrics["loss" ].append(avg_loss)
metrics["reward" ].append(avg_reward)
if (step + 1 ) % 100 == 0 :
print (f" Step {step+1 } : Loss={avg_loss:.4 f} , Reward={avg_reward:.4 f} " )
return metrics
def online_rl_phase (
self,
tasks: List [Dict ],
mode: ReasoningMode,
reward_model: "RewardModel" = None
) -> Dict [str , float ]:
"""
Online RL: interact with environment.
Refine policy based on real feedback.
"""
print (f"Online RL Phase for {mode.value} mode" )
metrics = {"loss" : [], "reward" : [], "budget_compliance" : []}
for step in range (self .online_steps):
task = tasks[step % len (tasks)]
response = self .model.generate(
task["prompt" ],
mode=mode,
max_tokens=ThinkDialConfig.get_budget(mode).max_tokens
)
if reward_model:
reward = reward_model.score(response, task["expected" ])
else :
reward = self .compute_budget_aware_reward(
response,
task["expected" ],
mode,
len (response.split())
)
log_prob = self .model.get_log_prob(response)
loss = -log_prob * reward
self .optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self .model.parameters(), 1.0 )
self .optimizer.step()
budget_config = ThinkDialConfig.get_budget(mode)
tokens_used = len (response.split())
compliance = 1.0 if tokens_used <= budget_config.max_tokens else 0.0
metrics["loss" ].append(loss.item())
metrics["reward" ].append(reward)
metrics["budget_compliance" ].append(compliance)
if (step + 1 ) % 50 == 0 :
avg_loss = sum (metrics["loss" ][-50 :]) / 50
avg_reward = sum (metrics["reward" ][-50 :]) / 50
avg_compliance = sum (metrics["budget_compliance" ][-50 :]) / 50
print (f" Step {step+1 } : Loss={avg_loss:.4 f} , Reward={avg_reward:.4 f} , Compliance={avg_compliance:.2 %} " )
return metrics
4. Implement Mode Selection and Deployment Enable runtime mode switching:
class ThinkDialModel :
"""ThinkDial model with mode control."""
def __init__ (self, base_model: "LLM" ):
self .model = base_model
self .mode_selector = ModeSelector()
def generate_with_mode (
self,
prompt: str ,
mode: ReasoningMode = ReasoningMode.HIGH,
return_mode_info: bool = False
) -> str :
"""Generate response in specified mode."""
budget_config = ThinkDialConfig.get_budget(mode)
budget_embedding = ThinkDialConfig.get_budget_embedding(mode)
augmented_prompt = f"{budget_embedding} \n{prompt} "
response = self .model.generate(
augmented_prompt,
max_tokens=budget_config.max_tokens,
temperature=0.7
)
if return_mode_info:
return response, {
"mode" : mode.value,
"tokens_used" : len (response.split()),
"budget" : budget_config.max_tokens
}
return response
def auto_select_mode (
self,
prompt: str ,
time_budget_ms: int = None ,
compute_budget: float = None
) -> ReasoningMode:
"""Automatically select mode based on constraints."""
if time_budget_ms:
if time_budget_ms < 100 :
return ReasoningMode.LOW
elif time_budget_ms < 500 :
return ReasoningMode.MEDIUM
else :
return ReasoningMode.HIGH
if compute_budget:
if compute_budget < 0.3 :
return ReasoningMode.LOW
elif compute_budget < 0.7 :
return ReasoningMode.MEDIUM
else :
return ReasoningMode.HIGH
return ReasoningMode.HIGH
class ModeSelector (torch.nn.Module):
"""Learn to select appropriate mode for tasks."""
def __init__ (self, task_embedding_dim: int = 256 ):
super ().__init__()
self .task_encoder = torch.nn.Linear(task_embedding_dim, 128 )
self .mode_classifier = torch.nn.Sequential(
torch.nn.Linear(128 , 64 ),
torch.nn.ReLU(),
torch.nn.Linear(64 , len (ReasoningMode))
)
def select_mode (self, task_embedding: torch.Tensor ) -> ReasoningMode:
"""Select mode based on task."""
encoded = self .task_encoder(task_embedding)
logits = self .mode_classifier(encoded)
mode_idx = torch.argmax(logits, dim=-1 ).item()
return list (ReasoningMode)[mode_idx]
Practical Guidance
When to Use ThinkDial
Deployment with variable latency constraints
Mobile/edge devices with compute limits
Cost-sensitive inference (pay-per-token models)
Applications requiring quality-latency tradeoffs
Research into compression-performance curves
When NOT to Use
Scenarios requiring always maximum quality
Real-time applications (<10ms latency)
Models without budget-aware training
Key Hyperparameters
High budget : 1000 tokens (baseline)Medium budget : 500 tokens (50% compression)Low budget : 250 tokens (75% compression)accuracy_weight : 0.8 in rewardbudget_weight : 0.2 in reward
Performance Expectations
Medium Mode: ~95% of High mode quality
Low Mode: ~85-90% of High mode quality
Token Reduction: 50% (Medium), 75% (Low)
Latency Improvement: 2-3x (Medium), 4-5x (Low)
Inference Cost: Proportional to token reduction
Reference Researchers. (2024). ThinkDial: Controlling Reasoning Effort in LLMs. arXiv preprint arXiv:2508.18773.
