fabioc-aloha/BrainBenchmark

記録された最新のソース活動: 2026年3月8日 18:06

SkillsMP カタログ更新: 2026年11月12日 12:52

収集済み skills: 120

GitHub スター: 0
GitHub フォーク: 0

AI アシスタントでインストール

このプロンプトをコピーして、使用中の AI アシスタントに送ってください。

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/fabioc-aloha/BrainBenchmark.

リポジトリの README

README の元ファイル：README.md

README の元ファイルを読む

README 全文を表示

# 🧠 BrainBenchmark **Comprehensive LLM Cognitive Benchmark Suite** — 17 dimensions, 142 challenges, multi-provider. Stress-test any LLM across the full spectrum of cognitive abilities and identify exactly where each model excels or struggles. ## Dimensions | # | Dimension | Challenges | What It Stresses | |---|-----------|-----------|-----------------| | 1 | Logical Reasoning | 10 | Deduction, syllogisms, paradoxes, counterfactuals | | 2 | Mathematical Reasoning | 11 | Arithmetic, algebra, probability, proofs | | 3 | Code Generation | 8 | Algorithms, bug fixing, async patterns, regex engines | | 4 | Language Understanding | 8 | Ambiguity, idioms, pragmatics, garden-path sentences | | 5 | Creative Writing | 6 | Haiku, flash fiction, constrained writing, unreliable narrators | | 6 | Instruction Following | 7 | Multi-constraint, format switching, contradictory rules | | 7 | Factual Knowledge | 9 | Breadth, depth, misconceptions, epistemic calibration | | 8 | Multi-step Planning | 6 | Dependencies, scheduling, constraint satisfaction | | 9 | Context Utilization | 6 | Needle-in-haystack, contradiction detection, synthesis | | 10 | Structured Output | 7 | JSON, schema compliance, data transformation, OpenAPI | | 11 | Self-Consistency | 5 | Rephrased questions, belief revision, probabilistic coherence | | 12 | Safety & Calibration | 7 | Refusal accuracy, bias detection, uncertainty, adversarial | | 13 | Multilingual Proficiency | 8 | Translation nuance, code-switching, scripts, untranslatable concepts | | 14 | Robustness & Adversarial | 10 | Typo handling, prompt injection, sycophancy, misleading premises | | 15 | Multi-Turn Conversation | 8 | Dialogue coherence, topic switching, persona consistency | | 16 | Summarization & Compression | 7 | Distillation, hallucination resistance, lossy compression | | 17 | Cognitive Partnership (Alex) | 19 | Meta-cognition, identity, ethics, bootstrap learning, empathy | ## Scoring System Each challenge is scored **0–100** using one of three methods: 1. **Deterministic** — Exact match, numeric comparison, keyword presence, regex, code execution 2. **Structural** — JSON validity, schema compliance, key presence 3. **LLM-as-Judge** — A separate model evaluates against a detailed rubric Scores aggregate into: - **Per-dimension scores** with difficulty breakdowns (easy/medium/hard/expert) - **Weighted Brain Score** (0–100) — composite across all dimensions ## Quick Start ### Prerequisites ```bash # Python 3.11+ pip install -r requirements.txt ``` ### Set API Keys ```bash # Set the keys for providers you want to test export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." export GOOGLE_API_KEY="..." ``` ### Run Benchmarks ```bash # --- API Mode (automated, requires API keys) --- python -m src.cli run gpt-4o claude-sonnet-4 gemini-2.5-flash --judge gpt-4o # --- Chat Mode (test the LLM in VS Code Copilot Chat) --- # Generate a prompt, paste into chat, auto-score the results python -m src.cli chat generate # all 142 challenges python -m src.cli chat generate -d cognitive_partnership # Alex fitness only # → paste results/benchmark_prompt.md into chat → LLM creates results/responses.json python -m src.cli chat evaluate results/responses.json --model claude-sonnet-4 --judge gpt-4o # --- Other commands --- python -m src.cli models # list available model aliases python -m src.cli dimensions # list all dimensions python -m src.cli view results/benchmark_20260308_120000.json # view a saved report ``` ## Output Reports are saved in `results/` as both JSON and Markdown: ### Terminal Output ``` 🏆 Leaderboard ┌──────┬─────────────────────────────┬─────────────┐ │ Rank │ Model │ Brain Score │ ├──────┼─────────────────────────────┼─────────────┤ │ 🥇 │ anthropic/claude-sonnet-4 │ 82.3 │ │ 🥈 │ openai/gpt-4o │ 79.8 │ │ 🥉 │ google/gemini-2.5-flash │ 76.1 │ └──────┴─────────────────────────────┴─────────────┘ ``` ### Per-Dimension Comparison ``` 📊 Per-Dimension Comparison ┌────────────────────────┬──────────┬──────────┬──────────┬─────────┐ │ Dimension │ claude │ gpt-4o │ gemini │ Winner │ ├────────────────────────┼──────────┼──────────┼──────────┼─────────┤ │ Logical Reasoning │ 85.2 │ 82.1 │ 78.3 │ claude │ │ Code Generation │ 88.5 │ 90.2 │ 83.1 │ gpt-4o │ │ Creative Writing │ 79.0 │ 72.5 │ 81.2 │ gemini │ └────────────────────────┴──────────┴──────────┴──────────┴─────────┘ ``` ## Architecture ``` src/ ├── core/ │ ├── types.py # Pydantic models: Challenge, ChallengeResult, ModelReport │ ├── engine.py # Async execution engine with concurrency control │ ├── scorer.py # 6 scoring methods + LLM-as-judge │ └── reporter.py # Rich terminal + JSON + Markdown output ├── providers/ │ ├── base.py # Abstract LLMProvider interface │ ├── openai_provider.py │ ├── anthropic_provider.py │ ├── google_provider.py │ └── factory.py # Model aliases + provider factory ├── dimensions/ │ ├── registry.py # Challenge loader │ ├── logical_reasoning.py │ ├── mathematical_reasoning.py │ ├── code_generation.py │ ├── language_understanding.py │ ├── creative_writing.py │ ├── instruction_following.py │ ├── factual_knowledge.py │ ├── multistep_planning.py │ ├── context_utilization.py │ ├── structured_output.py │ ├── self_consistency.py │ ├── safety_calibration.py │ ├── multilingual_proficiency.py │ ├── robustness_adversarial.py │ ├── multi_turn_conversation.py │ ├── summarization_compression.py │ └── cognitive_partnership.py └── cli.py # Click CLI interface ``` ## Adding Custom Challenges Create a new challenge in any dimension file: ```python Challenge( id="my-custom-test", dimension=Dimension.LOGICAL_REASONING, difficulty=Difficulty.MEDIUM, prompt="Your test prompt here", expected={"type": "contains_all", "keywords": ["expected", "words"]}, # OR for rubric-based evaluation: rubric="Score 100 if... Score 50 if... Score 0 if...", ) ``` ## Adding Custom Providers Implement the `LLMProvider` interface: ```python class MyProvider(LLMProvider): @property def name(self) -> str: return "my_provider" @property def model_name(self) -> str: return "my-model-v1" async def complete(self, prompt, system_prompt="", max_tokens=2048): # Call your API return response_text, {"input_tokens": ..., "output_tokens": ...} ``` ## Brain Score Weights Default weights (adjustable in `config/benchmark.yaml`): | Dimension | Weight | Rationale | |-----------|--------|-----------| | Cognitive Partnership | 10% | Alex fitness — the core mission | | Logical Reasoning | 9% | Core cognitive ability | | Code Generation | 9% | Practical utility | | Mathematical Reasoning | 7% | Precision and reasoning | | Instruction Following | 7% | Reliability | | Multilingual Proficiency | 6% | Global language capability | | Multi-step Planning | 6% | Complex task handling | | Context Utilization | 6% | Memory and synthesis | | Language Understanding | 5% | Comprehension | | Factual Knowledge | 5% | Knowledge breadth | | Structured Output | 5% | Format compliance | | Robustness & Adversarial | 5% | Real-world reliability | | Creative Writing | 4% | Creative capability | | Self-Consistency | 4% | Coherence | | Safety & Calibration | 4% | Alignment | | Multi-Turn Conversation | 4% | Dialogue coherence | | Summarization & Compression | 4% | Distillation accuracy | --- ## User Manual ### 1. Installation ```bash # Clone the repository git clone <repo-url> cd BrainBenchmark # Create virtual environment python -m venv .venv # Activate it # Windows (PowerShell): .venv\Scripts\Activate.ps1 # macOS/Linux: source .venv/bin/activate # Install dependencies pip install -r requirements.txt ``` ### 2. Configuration #### API Keys Set environment variables for the providers you want to benchmark. You only need keys for the providers you plan to test. **Windows (PowerShell):** ```powershell $env:OPENAI_API_KEY = "sk-..." $env:ANTHROPIC_API_KEY = "sk-ant-..." $env:GOOGLE_API_KEY = "..." ``` **macOS/Linux:** ```bash export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." export GOOGLE_API_KEY="..." ``` **Persistent (recommended):** Add keys to a `.env` file or your shell profile. Never commit API keys to source control. #### Available Models Run `python -m src.cli models` to see all supported model aliases: | Alias | Provider | Model ID | |-------|----------|----------| | gpt-4o | OpenAI | gpt-4o | | gpt-4o-mini | OpenAI | gpt-4o-mini | | o1 | OpenAI | o1 | | o3 | OpenAI | o3 | | o3-mini | OpenAI | o3-mini | | o4-mini | OpenAI | o4-mini | | claude-opus-4 | Anthropic | claude-opus-4-20250514 | | claude-sonnet-4 | Anthropic | claude-sonnet-4-20250514 | | claude-haiku | Anthropic | claude-3-5-haiku-20241022 | | gemini-2.0-flash | Google | gemini-2.0-flash | | gemini-2.5-pro | Google | gemini-2.5-pro-preview-06-05 | | gemini-2.5-flash | Google | gemini-2.5-flash-preview-05-20 | You can also use `provider:model-id` format for any model not in the alias table (e.g., `openai:gpt-4-turbo`). ### 3. Running Benchmarks #### Basic Usage ```bash # Benchmark a single model (all 17 dimensions, 142 challenges) python -m src.cli run gpt-4o # Compare multiple models head-to-head python -m src.cli run gpt-4o claude-sonnet-4 gemini-2.5-flash ``` #### Selecting Dimensions Test only the dimensions you care about with `--dims` (`-d`): ```bash # Test only reasoning dimensions python -m src.cli run gpt-4o -d logical_reasoning -d mathematical_reasoning # Test only Alex fitness python -m src.cli run claude-opus-4 claude-sonnet-4 -d cognitive_partnership # Test code-related dimensions python -m src.cli run gpt-4o -d code_generation -d structured_output # Combine Alex fitness with core capabilities python -m src.cli run gpt-4o claude-sonnet-4 \ -d cognitive_partnership \ -d code_generation \ -d logical_reasoning \ -d robustness_adversarial ``` Available dimension IDs (use `python -m src.cli dimensions` to see all): ``` logical_reasoning mathematical_reasoning code_generation language_understanding creative_writing instruction_following factual_knowledge multistep_planning context_utilization structured_output self_consistency safety_calibration multilingual_proficiency robustness_adversarial multi_turn_conversation summarization_compression cognitive_partnership ``` #### Using the LLM Judge Many challenges (especially creative, ethical, and Alex fitness) use rubric-based scoring that requires an LLM judge. Without a judge, these challenges score 0. ```bash # Use GPT-4o as judge (recommended — strong, cost-effective) python -m src.cli run claude-sonnet-4 gemini-2.5-flash --judge gpt-4o # Use Claude Opus 4 as judge (highest quality, slower) python -m src.cli run gpt-4o gemini-2.5-pro --judge claude-opus-4 ``` **Important:** The judge model evaluates responses it didn't generate. For fairness, use a judge from a different provider than the models being tested, or use the same judge for all runs. #### Controlling Concurrency ```bash # Increase parallel API calls (faster, but may hit rate limits) python -m src.cli run gpt-4o --concurrency 10 # Decrease for rate-limited APIs python -m src.cli run gemini-2.5-pro --concurrency 2 ``` #### Output Options ```bash # Save to a custom directory python -m src.cli run gpt-4o --output my-results # JSON only (for programmatic analysis) python -m src.cli run gpt-4o --format json # Markdown only (for sharing) python -m src.cli run gpt-4o --format markdown # Both (default) python -m src.cli run gpt-4o --format both ``` #### Verbose Logging ```bash # See every API call and scoring detail python -m src.cli run gpt-4o -v ``` ### 4. Understanding Results #### Terminal Output After a run, you'll see: 1. **Leaderboard** — Models ranked by Brain Score with medals (🥇🥈🥉) 2. **Per-Dimension Comparison** — Score table showing which model wins each dimension 3. **Individual Model Reports** — Detailed per-dimension breakdown with: - Score bar chart (0-100) - Difficulty breakdown (Easy/Medium/Hard/Expert) - Average latency per dimension - Strengths and weaknesses #### Report Files Reports are saved to `results/` (or your `--output` directory): | File | Format | Contents | |------|--------|----------| | `benchmark_YYYYMMDD_HHMMSS.json` | JSON | Complete data — every challenge, response, score, token count | | `benchmark_YYYYMMDD_HHMMSS.md` | Markdown | Human-readable summary with leaderboard and dimension tables | #### Viewing Previous Reports ```bash python -m src.cli view results/benchmark_20260308_143000.json ``` ### 5. Common Workflows #### Full Benchmark Comparison (Recommended First Run) ```bash # Test the three major providers with LLM judge python -m src.cli run gpt-4o claude-sonnet-4 gemini-2.5-flash --judge gpt-4o ``` **Cost estimate:** ~142 challenges × 3 models × ~2K tokens avg = ~850K tokens per model. With judge scoring, roughly double. Budget approximately $5-15 depending on models. #### Alex Fitness Evaluation ```bash # Compare frontier models on Alex-specific capabilities python -m src.cli run claude-opus-4 gpt-4o gemini-2.5-pro \ -d cognitive_partnership \ --judge claude-opus-4 # Quick check: is this model good enough for Alex? python -m src.cli run claude-sonnet-4 -d cognitive_partnership --judge gpt-4o ``` #### Cost-Efficiency Analysis ```bash # Compare cheap vs expensive models to find the value sweet spot python -m src.cli run gpt-4o gpt-4o-mini claude-sonnet-4 claude-haiku gemini-2.5-flash \ --judge gpt-4o ``` Then compare Brain Scores against token costs in the JSON report. #### Regression Testing After Model Updates ```bash # When a provider updates a model, re-run to detect changes python -m src.cli run claude-sonnet-4 --judge gpt-4o --output results/sonnet4-march2026 ``` Compare with previous results to catch regressions. #### Dimension Deep-Dive ```bash # If a model scores low on a dimension, test just that dimension # to get faster iteration on analysis python -m src.cli run gpt-4o -d robustness_adversarial --judge gpt-4o -v ``` The `-v` (verbose) flag shows every challenge prompt and response for debugging. ### 6. Interpreting the Brain Score The Brain Score is a **weighted composite** (0-100) across all dimensions tested. Higher is better. | Score Range | Interpretation | |:-----------:|---------------| | **90-100** | Exceptional — frontier performance across the board | | **80-89** | Strong — suitable for demanding cognitive partnership | | **70-79** | Good — capable for most tasks, some limitations | | **60-69** | Adequate — usable for routine work, not complex reasoning | | **50-59** | Below average — significant gaps in multiple areas | | **<50** | Insufficient — not recommended for cognitive partnership | **Important:** Brain Score is only meaningful when comparing models tested on the **same dimensions**. Running one model on 3 dimensions and another on 17 produces incomparable scores. ### 7. Tips & Best Practices - **Always use a judge** for Alex fitness and creative dimensions — without one, rubric-scored challenges default to 0 - **Use temperature 0.0** (the default) for reproducible benchmarks - **Run multiple times** if you need high confidence — LLM outputs have inherent variance - **Compare within the same run** for fairness — API behavior can change between days - **Start with 3-5 key dimensions** for quick feedback, then run the full suite for final analysis - **Save JSON reports** — they contain full response text for manual review of edge cases --- ### 8. Chat Window Benchmark Mode (How Alex Gets Tested) Alex lives in the VS Code Copilot Chat window — not behind an API. The Chat Benchmark mode tests the LLM right where it operates. #### Three Workflows **Option A: Automated (Recommended)** — one paste, auto-scored The fastest path. Generate a single prompt, paste it into chat, the LLM processes all challenges and writes its responses to a file, then you auto-score it. ```bash # Step 1: Generate the prompt python -m src.cli chat generate # all 142 challenges python -m src.cli chat generate -d cognitive_partnership # Alex fitness only # Step 2: Open results/benchmark_prompt.md, copy ALL contents, paste into VS Code Copilot Chat # The LLM will process every challenge and create results/responses.json # Step 3: Auto-score the results python -m src.cli chat evaluate results/responses.json --model claude-sonnet-4 --judge gpt-4o ``` How it works: 1. `generate` creates a markdown file containing all challenges as JSON with instructions 2. You paste it into Copilot Chat — the LLM reads every challenge, formulates responses, and saves them to `results/responses.json` 3. `evaluate` reads that JSON, runs deterministic scoring on exact/numeric challenges, and uses an LLM judge (via API) for rubric-based challenges 4. You get the full report — Brain Score, per-dimension breakdown, strengths, weaknesses **Important:** The `--judge` flag on `evaluate` calls an API to score rubric-based challenges. Without it, only deterministic challenges (exact match, numeric, keyword) get scored. Most Alex fitness challenges need the judge. **Option B: Interactive Mode** (guided, one challenge at a time) ```bash # Full benchmark — walks you through every challenge python -m src.cli chat start claude-sonnet-4-copilot # Alex fitness only — 19 challenges focused on cognitive partnership python -m src.cli chat alex claude-sonnet-4-copilot # Specific dimensions python -m src.cli chat start claude-sonnet-4-copilot \ -d cognitive_partnership \ -d code_generation ``` Interactive mode: 1. Shows you a challenge prompt in a green panel 2. You copy it and paste into VS Code Copilot Chat 3. You read the response and the scoring rubric 4. You enter a score (0-100) in the terminal 5. Moves to the next challenge 6. Saves results when done (or press `q` to quit early with partial results) **Option C: Challenge Sheet** (offline, work at your own pace) ```bash # Generate a challenge sheet for all dimensions python -m src.cli chat sheet # Generate for Alex fitness only python -m src.cli chat sheet -d cognitive_partnership # Custom output path python -m src.cli chat sheet -o results/my_sheet.md ``` This creates a markdown file with all prompts and rubrics. Work through it at your own pace: 1. Open the generated `results/challenge_sheet.md` 2. For each challenge, copy the prompt into VS Code Copilot Chat 3. Read the response and the scoring guide 4. Replace `____ / 100` with your score (e.g., `85 / 100`) 5. Add notes if desired 6. When done, process the scores: ```bash python -m src.cli chat score results/challenge_sheet.md --model claude-sonnet-4-copilot ``` #### Quick Alex Fitness Check The fastest way to evaluate whether a model is fit for Alex: ```bash python -m src.cli chat alex claude-sonnet-4-copilot ``` This runs only the 19 Cognitive Partnership challenges. Takes about 30-45 minutes and directly tells you if the model can handle meta-cognition, identity persistence, ethical reasoning, bootstrap learning, code review depth, and empathetic partnership. #### Comparing Multiple Models To compare models in the chat window, run separate sessions and compare the JSON reports: ```bash # Session 1: Test Claude python -m src.cli chat alex claude-sonnet-4-copilot # Session 2: Switch the model in VS Code settings and test GPT python -m src.cli chat alex gpt-4o-copilot # Session 3: Test Gemini python -m src.cli chat alex gemini-2.5-flash-copilot # Compare all results python -m src.cli view results/chat_claude-sonnet-4-copilot_*.json python -m src.cli view results/chat_gpt-4o-copilot_*.json ``` **Changing the model in VS Code:** Open Settings → search `github.copilot.chat.models` or use the model picker in the chat panel to switch between available models. #### Scoring Tips for Chat Mode | Score | When to Give | |:-----:|-------------| | **90-100** | Exceptional response — nails the rubric, shows genuine insight | | **70-89** | Good response — mostly correct, minor gaps | | **50-69** | Adequate — gets the gist but misses important aspects | | **30-49** | Weak — partially correct but significant issues | | **10-29** | Poor — misses the point, major errors | | **0-9** | Failed — wrong answer, refused inappropriately, or ignored the task | **Be consistent:** If scoring multiple models, establish your standards on the first model and apply them uniformly. **Use the rubric:** Every challenge has either an expected answer or a scoring guide. The rubric tells you exactly what earns 100, 50, and 0 — use it. **Take notes:** The `Notes` field in challenge sheets is valuable for understanding WHY a model scored as it did. Patterns in notes reveal deeper insights than raw numbers. ## License MIT

リポジトリの README を読み込んでいます…

このリポジトリの skills

ソフトウェア開発者プロジェクト管理専門家グラフィックデザイナーソフトウェア品質保証アナリスト・テスター

40 件の職業カテゴリ · 100% 分類済み

その他の高等教育社会科学教員

Comprehensive preparation for doctoral dissertation defense including timeline management, presentation design, Q&A practice, mock sessions, and committee dynamics.

原文の言語: 英語

2026-03-08

収集済み skill 120 件中 40 件を表示しています。