Menu
AEGISモデルのARC-Challenge改善、GSM8K健全性検証、GRPO報酬多目的化のための包括的Planモード。頑健な回答抽出、タイムアウト最適化、データ汚染チェック、複数seed評価、汎化性能向上を実装。
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
Implement comprehensive mathematical theorem proving capabilities with SFT+GRPO training, MCP/A2A agent integration, and imatrix quantization protection to surpass Boreas-phi3.5-instinct-jp in formal proof generation and scientific discovery. Use when building mathematical reasoning systems, formal verification tools, or AI-assisted theorem proving environments.
Execute GGUF quantization with imatrix protection, perform statistical benchmark evaluation with error bars, generate academic-style methodology documentation, and create comprehensive scorecards. Use when evaluating model quantization quality, comparing quantization methods, or generating publication-ready evaluation results with subagent execution and PowerShell progress visualization.
Advanced multimodal 'thinking' model pipeline with SO8T Grand Design, stability-constrained evolutionary training (EvoFreeze), and specialized OSINT/Military/Bio data collection.
Implement comprehensive mathematical theorem proving capabilities with SFT+GRPO training, MCP/A2A agent integration, and imatrix quantization protection to surpass Boreas-phi3.5-instinct-jp in formal proof generation and scientific discovery. Use when building mathematical reasoning systems, formal verification tools, or AI-assisted theorem proving environments.
Execute GGUF quantization with imatrix protection, perform statistical benchmark evaluation with error bars, generate academic-style methodology documentation, and create comprehensive scorecards. Use when evaluating model quantization quality, comparing quantization methods, or generating publication-ready evaluation results with subagent execution and PowerShell progress visualization.
Create, manage, and update multi-step execution plans for complex tasks in Cursor. Use when the user asks for a plan/roadmap/phases/checklist, or when work has multiple dependent steps, higher risk, or requires coordination. Skip for straightforward tasks. Integrates with Cursor's todo_write tool for plan tracking.
| name | plan-mode |
| description | AEGISモデルのARC-Challenge改善、GSM8K健全性検証、GRPO報酬多目的化のための包括的Planモード。頑健な回答抽出、タイムアウト最適化、データ汚染チェック、複数seed評価、汎化性能向上を実装。 |
AEGISモデルのベンチマーク性能を科学的に改善するための包括的Planモード。ABCテスト結果に基づき、ARC-Challenge、GSM8K、GRPO報酬の3つの優先領域を体系的に改善します。
from skills.plan_mode import AEGISImprovementPlan
# ARC-Challenge改善Plan実行
arc_plan = AEGISImprovementPlan()
arc_improvements = {
"extraction_logic": "robust",
"timeout_extension": 240,
"prompt_unification": True,
"response_validation": True
}
arc_results = arc_plan.execute_arc_improvement(arc_improvements)
print(f"ARC改善効果: {arc_results['score_improvement']:.1f}ポイント")
# GSM8K健全性チェック
gsm8k_checks = {
"contamination_check": True,
"multi_seed_evaluation": True,
"zero_shot_assessment": True,
"scoring_validation": True
}
sanity_results = arc_plan.execute_gsm8k_sanity_checks(gsm8k_checks)
print(f"データ汚染検出: {'あり' if sanity_results['contamination_found'] else 'なし'}")
print(f"安定性スコア: {sanity_results['stability_score']:.2f}")
# GRPO報酬最適化
grpo_optimization = {
"arc_format_rewards": True,
"math_reasoning_rewards": True,
"gsm8k_weight_reduction": 0.7,
"generalization_focus": True
}
grpo_results = arc_plan.execute_grpo_optimization(grpo_optimization)
print(f"汎化性能向上: {grpo_results['generalization_gain']:.1f}%")
# v2.5モデル構築とABCテスト
v25_config = {
"arxiv_biorxiv_integration": True,
"group_representation_transformer": True,
"tool_calling_enhancement": True,
"abc_test_automation": True
}
v25_results = arc_plan.execute_v25_integration(v25_config)
print(f"AEGIS v2.5 ABCテスト結果: {v25_results['abc_test_summary']}")
class RobustARCExtractor:
def extract_answer(self, response: str) -> str:
"""複数パターン対応の頑健な回答抽出"""
patterns = [
r"Answer:\s*([A-E])\b",
r"答え:\s*([A-E])\b",
r"\b([A-E])\b(?=\s*(?:\.|\)|$))",
r"選択肢\s*([A-E])\b",
r"option\s*([A-E])\b"
]
for pattern in patterns:
match = re.search(pattern, response, re.IGNORECASE)
if match:
return match.group(1).upper()
# フォールバック: 最初のA-E文字
choices = re.findall(r'\b([A-E])\b', response.upper())
return choices[0] if choices else ""
class DynamicTimeoutManager:
def calculate_optimal_timeout(self, question_complexity: float) -> int:
"""問題複雑度に基づくタイムアウト計算"""
base_timeout = 180
complexity_factor = min(question_complexity * 0.5, 1.0)
return int(base_timeout * (1 + complexity_factor))
class DataContaminationChecker:
def check_contamination(self, training_data: List[str], test_questions: List[str]) -> Dict:
"""学習データとテストデータの重複検査"""
exact_matches = 0
near_duplicates = 0
for test_q in test_questions:
for train_q in training_data:
similarity = self.calculate_similarity(test_q, train_q)
if similarity > 0.95:
exact_matches += 1
elif similarity > 0.8:
near_duplicates += 1
return {
"exact_matches": exact_matches,
"near_duplicates": near_duplicates,
"contamination_risk": "high" if exact_matches > 0 else "low"
}
class MultiSeedEvaluator:
def evaluate_stability(self, model, seeds: List[int], samples: int = 300) -> Dict:
"""複数seedでの性能安定性評価"""
performances = []
for seed in seeds:
torch.manual_seed(seed)
accuracy = self.evaluate_on_sample(model, samples)
performances.append(accuracy)
return {
"mean_performance": np.mean(performances),
"std_performance": np.std(performances),
"stability_score": 1.0 - (np.std(performances) / np.mean(performances)),
"variance_analysis": self.analyze_variance_sources(performances)
}
class MultiObjectiveGRPOReward:
def calculate_reward(self, response: str, ground_truth: str, task_type: str) -> float:
"""タスクタイプ別報酬計算"""
base_reward = self.calculate_correctness_reward(response, ground_truth)
if task_type == "arc_challenge":
format_reward = self.calculate_arc_format_reward(response)
reasoning_reward = self.calculate_reasoning_quality_reward(response)
return base_reward + 0.3 * format_reward + 0.2 * reasoning_reward
elif task_type == "math":
step_reward = self.calculate_step_by_step_reward(response)
final_answer_reward = self.calculate_final_answer_reward(response)
return base_reward + 0.4 * step_reward + 0.3 * final_answer_reward
elif task_type == "gsm8k":
# GSM8K報酬重み調整（過度最適化防止）
return 0.7 * base_reward # 重み70%に調整
return base_reward
class GeneralizationEnhancer:
def enhance_generalization(self, model, diverse_tasks: List[Task]) -> Dict:
"""多様なタスクでの汎化性能向上"""
improvements = {}
for task in diverse_tasks:
initial_performance = self.evaluate_task_performance(model, task)
self.fine_tune_on_task(model, task)
final_performance = self.evaluate_task_performance(model, task)
improvements[task.name] = final_performance - initial_performance
return {
"generalization_gain": np.mean(list(improvements.values())),
"task_specific_gains": improvements,
"overall_improvement": sum(improvements.values())
}
# ARC-Challenge改善分析と実装
python scripts/plan_mode/execute_arc_improvement.py \
--model_path AEGIS-Phi3.5mini-jp \
--sample_size 500 \
--timeout_start 180 \
--timeout_end 240 \
--extraction_logic robust \
--output_path results/arc_improvement_results.json
# GSM8Kの包括的健全性チェック
python scripts/plan_mode/execute_gsm8k_sanity.py \
--model_path AEGIS-Phi3.5mini-jp \
--training_data_path data/training/so8t_training.jsonl \
--test_data_path data/test/gsm8k_test.jsonl \
--seeds 42,123,456,789,999 \
--include_zero_shot \
--output_path results/gsm8k_sanity_results.json
# GRPO報酬多目的化
python scripts/plan_mode/execute_grpo_optimization.py \
--model_path AEGIS-Phi3.5mini-jp \
--arc_reward_weight 0.3 \
--math_reward_weight 0.4 \
--gsm8k_reward_weight 0.7 \
--generalization_focus \
--output_path results/grpo_optimization_results.json
# AEGIS v2.5構築とABCテスト
python scripts/plan_mode/execute_v25_integration.py \
--base_model AEGIS-Phi3.5mini-jp \
--arxiv_biorxiv_data data/arxiv_biorxiv_structured.jsonl \
--group_representation_transformer \
--tool_calling_enhancement \
--abc_test_execution \
--output_path results/aegis_v25_integration_results.json
改善対象ベンチマーク: ARC-Challenge, GSM8K, MATH 改善手法: 頑健抽出/タイムアウト最適化/データ検証/報酬多目的化 目標性能: AEGIS v2.5でノーベル賞級推論能力獲得
ユーザーのABCテスト分析に基づき、ARC-Challenge改善・GSM8K健全性検証・GRPO報酬多目的化を統合した包括的Planモードを実装。
これによりAEGISモデルのベンチマーク性能が飛躍的に向上し、真の汎化推論能力が獲得されます！ 🚀🔬📊
実装完了: 2026-01-17 23:50:00 Planモード: AEGISモデル改善総合計画 🎯🧠