fabioc-aloha/BrainBenchmark

Latest recorded source activity: March 8, 2026 at 18:06

SkillsMP catalog refreshed: November 12, 2026 at 12:52

skills collected: 120

GitHub stars: 0
GitHub forks: 0

Install with an AI assistant

Copy this prompt into the AI assistant you're using.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/fabioc-aloha/BrainBenchmark.

Repository README

Repository README source: README.md

Read the source README

Expand full README

# 🧠 BrainBenchmark **Comprehensive LLM Cognitive Benchmark Suite** — 17 dimensions, 142 challenges, multi-provider. Stress-test any LLM across the full spectrum of cognitive abilities and identify exactly where each model excels or struggles. ## Dimensions | # | Dimension | Challenges | What It Stresses | |---|-----------|-----------|-----------------| | 1 | Logical Reasoning | 10 | Deduction, syllogisms, paradoxes, counterfactuals | | 2 | Mathematical Reasoning | 11 | Arithmetic, algebra, probability, proofs | | 3 | Code Generation | 8 | Algorithms, bug fixing, async patterns, regex engines | | 4 | Language Understanding | 8 | Ambiguity, idioms, pragmatics, garden-path sentences | | 5 | Creative Writing | 6 | Haiku, flash fiction, constrained writing, unreliable narrators | | 6 | Instruction Following | 7 | Multi-constraint, format switching, contradictory rules | | 7 | Factual Knowledge | 9 | Breadth, depth, misconceptions, epistemic calibration | | 8 | Multi-step Planning | 6 | Dependencies, scheduling, constraint satisfaction | | 9 | Context Utilization | 6 | Needle-in-haystack, contradiction detection, synthesis | | 10 | Structured Output | 7 | JSON, schema compliance, data transformation, OpenAPI | | 11 | Self-Consistency | 5 | Rephrased questions, belief revision, probabilistic coherence | | 12 | Safety & Calibration | 7 | Refusal accuracy, bias detection, uncertainty, adversarial | | 13 | Multilingual Proficiency | 8 | Translation nuance, code-switching, scripts, untranslatable concepts | | 14 | Robustness & Adversarial | 10 | Typo handling, prompt injection, sycophancy, misleading premises | | 15 | Multi-Turn Conversation | 8 | Dialogue coherence, topic switching, persona consistency | | 16 | Summarization & Compression | 7 | Distillation, hallucination resistance, lossy compression | | 17 | Cognitive Partnership (Alex) | 19 | Meta-cognition, identity, ethics, bootstrap learning, empathy | ## Scoring System Each challenge is scored **0–100** using one of three methods: 1. **Deterministic** — Exact match, numeric comparison, keyword presence, regex, code execution 2. **Structural** — JSON validity, schema compliance, key presence 3. **LLM-as-Judge** — A separate model evaluates against a detailed rubric Scores aggregate into: - **Per-dimension scores** with difficulty breakdowns (easy/medium/hard/expert) - **Weighted Brain Score** (0–100) — composite across all dimensions ## Quick Start ### Prerequisites ```bash # Python 3.11+ pip install -r requirements.txt ``` ### Set API Keys ```bash # Set the keys for providers you want to test export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." export GOOGLE_API_KEY="..." ``` ### Run Benchmarks ```bash # --- API Mode (automated, requires API keys) --- python -m src.cli run gpt-4o claude-sonnet-4 gemini-2.5-flash --judge gpt-4o # --- Chat Mode (test the LLM in VS Code Copilot Chat) --- # Generate a prompt, paste into chat, auto-score the results python -m src.cli chat generate # all 142 challenges python -m src.cli chat generate -d cognitive_partnership # Alex fitness only # → paste results/benchmark_prompt.md into chat → LLM creates results/responses.json python -m src.cli chat evaluate results/responses.json --model claude-sonnet-4 --judge gpt-4o # --- Other commands --- python -m src.cli models # list available model aliases python -m src.cli dimensions # list all dimensions python -m src.cli view results/benchmark_20260308_120000.json # view a saved report ``` ## Output Reports are saved in `results/` as both JSON and Markdown: ### Terminal Output ``` 🏆 Leaderboard ┌──────┬─────────────────────────────┬─────────────┐ │ Rank │ Model │ Brain Score │ ├──────┼─────────────────────────────┼─────────────┤ │ 🥇 │ anthropic/claude-sonnet-4 │ 82.3 │ │ 🥈 │ openai/gpt-4o │ 79.8 │ │ 🥉 │ google/gemini-2.5-flash │ 76.1 │ └──────┴─────────────────────────────┴─────────────┘ ``` ### Per-Dimension Comparison ``` 📊 Per-Dimension Comparison ┌────────────────────────┬──────────┬──────────┬──────────┬─────────┐ │ Dimension │ claude │ gpt-4o │ gemini │ Winner │ ├────────────────────────┼──────────┼──────────┼──────────┼─────────┤ │ Logical Reasoning │ 85.2 │ 82.1 │ 78.3 │ claude │ │ Code Generation │ 88.5 │ 90.2 │ 83.1 │ gpt-4o │ │ Creative Writing │ 79.0 │ 72.5 │ 81.2 │ gemini │ └────────────────────────┴──────────┴──────────┴──────────┴─────────┘ ``` ## Architecture ``` src/ ├── core/ │ ├── types.py # Pydantic models: Challenge, ChallengeResult, ModelReport │ ├── engine.py # Async execution engine with concurrency control │ ├── scorer.py # 6 scoring methods + LLM-as-judge │ └── reporter.py # Rich terminal + JSON + Markdown output ├── providers/ │ ├── base.py # Abstract LLMProvider interface │ ├── openai_provider.py │ ├── anthropic_provider.py │ ├── google_provider.py │ └── factory.py # Model aliases + provider factory ├── dimensions/ │ ├── registry.py # Challenge loader │ ├── logical_reasoning.py │ ├── mathematical_reasoning.py │ ├── code_generation.py │ ├── language_understanding.py │ ├── creative_writing.py │ ├── instruction_following.py │ ├── factual_knowledge.py │ ├── multistep_planning.py │ ├── context_utilization.py │ ├── structured_output.py │ ├── self_consistency.py │ ├── safety_calibration.py │ ├── multilingual_proficiency.py │ ├── robustness_adversarial.py │ ├── multi_turn_conversation.py │ ├── summarization_compression.py │ └── cognitive_partnership.py └── cli.py # Click CLI interface ``` ## Adding Custom Challenges Create a new challenge in any dimension file: ```python Challenge( id="my-custom-test", dimension=Dimension.LOGICAL_REASONING, difficulty=Difficulty.MEDIUM, prompt="Your test prompt here", expected={"type": "contains_all", "keywords": ["expected", "words"]}, # OR for rubric-based evaluation: rubric="Score 100 if... Score 50 if... Score 0 if...", ) ``` ## Adding Custom Providers Implement the `LLMProvider` interface: ```python class MyProvider(LLMProvider): @property def name(self) -> str: return "my_provider" @property def model_name(self) -> str: return "my-model-v1" async def complete(self, prompt, system_prompt="", max_tokens=2048): # Call your API return response_text, {"input_tokens": ..., "output_tokens": ...} ``` ## Brain Score Weights Default weights (adjustable in `config/benchmark.yaml`): | Dimension | Weight | Rationale | |-----------|--------|-----------| | Cognitive Partnership | 10% | Alex fitness — the core mission | | Logical Reasoning | 9% | Core cognitive ability | | Code Generation | 9% | Practical utility | | Mathematical Reasoning | 7% | Precision and reasoning | | Instruction Following | 7% | Reliability | | Multilingual Proficiency | 6% | Global language capability | | Multi-step Planning | 6% | Complex task handling | | Context Utilization | 6% | Memory and synthesis | | Language Understanding | 5% | Comprehension | | Factual Knowledge | 5% | Knowledge breadth | | Structured Output | 5% | Format compliance | | Robustness & Adversarial | 5% | Real-world reliability | | Creative Writing | 4% | Creative capability | | Self-Consistency | 4% | Coherence | | Safety & Calibration | 4% | Alignment | | Multi-Turn Conversation | 4% | Dialogue coherence | | Summarization & Compression | 4% | Distillation accuracy | --- ## User Manual ### 1. Installation ```bash # Clone the repository git clone <repo-url> cd BrainBenchmark # Create virtual environment python -m venv .venv # Activate it # Windows (PowerShell): .venv\Scripts\Activate.ps1 # macOS/Linux: source .venv/bin/activate # Install dependencies pip install -r requirements.txt ``` ### 2. Configuration #### API Keys Set environment variables for the providers you want to benchmark. You only need keys for the providers you plan to test. **Windows (PowerShell):** ```powershell $env:OPENAI_API_KEY = "sk-..." $env:ANTHROPIC_API_KEY = "sk-ant-..." $env:GOOGLE_API_KEY = "..." ``` **macOS/Linux:** ```bash export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." export GOOGLE_API_KEY="..." ``` **Persistent (recommended):** Add keys to a `.env` file or your shell profile. Never commit API keys to source control. #### Available Models Run `python -m src.cli models` to see all supported model aliases: | Alias | Provider | Model ID | |-------|----------|----------| | gpt-4o | OpenAI | gpt-4o | | gpt-4o-mini | OpenAI | gpt-4o-mini | | o1 | OpenAI | o1 | | o3 | OpenAI | o3 | | o3-mini | OpenAI | o3-mini | | o4-mini | OpenAI | o4-mini | | claude-opus-4 | Anthropic | claude-opus-4-20250514 | | claude-sonnet-4 | Anthropic | claude-sonnet-4-20250514 | | claude-haiku | Anthropic | claude-3-5-haiku-20241022 | | gemini-2.0-flash | Google | gemini-2.0-flash | | gemini-2.5-pro | Google | gemini-2.5-pro-preview-06-05 | | gemini-2.5-flash | Google | gemini-2.5-flash-preview-05-20 | You can also use `provider:model-id` format for any model not in the alias table (e.g., `openai:gpt-4-turbo`). ### 3. Running Benchmarks #### Basic Usage ```bash # Benchmark a single model (all 17 dimensions, 142 challenges) python -m src.cli run gpt-4o # Compare multiple models head-to-head python -m src.cli run gpt-4o claude-sonnet-4 gemini-2.5-flash ``` #### Selecting Dimensions Test only the dimensions you care about with `--dims` (`-d`): ```bash # Test only reasoning dimensions python -m src.cli run gpt-4o -d logical_reasoning -d mathematical_reasoning # Test only Alex fitness python -m src.cli run claude-opus-4 claude-sonnet-4 -d cognitive_partnership # Test code-related dimensions python -m src.cli run gpt-4o -d code_generation -d structured_output # Combine Alex fitness with core capabilities python -m src.cli run gpt-4o claude-sonnet-4 \ -d cognitive_partnership \ -d code_generation \ -d logical_reasoning \ -d robustness_adversarial ``` Available dimension IDs (use `python -m src.cli dimensions` to see all): ``` logical_reasoning mathematical_reasoning code_generation language_understanding creative_writing instruction_following factual_knowledge multistep_planning context_utilization structured_output self_consistency safety_calibration multilingual_proficiency robustness_adversarial multi_turn_conversation summarization_compression cognitive_partnership ``` #### Using the LLM Judge Many challenges (especially creative, ethical, and Alex fitness) use rubric-based scoring that requires an LLM judge. Without a judge, these challenges score 0. ```bash # Use GPT-4o as judge (recommended — strong, cost-effective) python -m src.cli run claude-sonnet-4 gemini-2.5-flash --judge gpt-4o # Use Claude Opus 4 as judge (highest quality, slower) python -m src.cli run gpt-4o gemini-2.5-pro --judge claude-opus-4 ``` **Important:** The judge model evaluates responses it didn't generate. For fairness, use a judge from a different provider than the models being tested, or use the same judge for all runs. #### Controlling Concurrency ```bash # Increase parallel API calls (faster, but may hit rate limits) python -m src.cli run gpt-4o --concurrency 10 # Decrease for rate-limited APIs python -m src.cli run gemini-2.5-pro --concurrency 2 ``` #### Output Options ```bash # Save to a custom directory python -m src.cli run gpt-4o --output my-results # JSON only (for programmatic analysis) python -m src.cli run gpt-4o --format json # Markdown only (for sharing) python -m src.cli run gpt-4o --format markdown # Both (default) python -m src.cli run gpt-4o --format both ``` #### Verbose Logging ```bash # See every API call and scoring detail python -m src.cli run gpt-4o -v ``` ### 4. Understanding Results #### Terminal Output After a run, you'll see: 1. **Leaderboard** — Models ranked by Brain Score with medals (🥇🥈🥉) 2. **Per-Dimension Comparison** — Score table showing which model wins each dimension 3. **Individual Model Reports** — Detailed per-dimension breakdown with: - Score bar chart (0-100) - Difficulty breakdown (Easy/Medium/Hard/Expert) - Average latency per dimension - Strengths and weaknesses #### Report Files Reports are saved to `results/` (or your `--output` directory): | File | Format | Contents | |------|--------|----------| | `benchmark_YYYYMMDD_HHMMSS.json` | JSON | Complete data — every challenge, response, score, token count | | `benchmark_YYYYMMDD_HHMMSS.md` | Markdown | Human-readable summary with leaderboard and dimension tables | #### Viewing Previous Reports ```bash python -m src.cli view results/benchmark_20260308_143000.json ``` ### 5. Common Workflows #### Full Benchmark Comparison (Recommended First Run) ```bash # Test the three major providers with LLM judge python -m src.cli run gpt-4o claude-sonnet-4 gemini-2.5-flash --judge gpt-4o ``` **Cost estimate:** ~142 challenges × 3 models × ~2K tokens avg = ~850K tokens per model. With judge scoring, roughly double. Budget approximately $5-15 depending on models. #### Alex Fitness Evaluation ```bash # Compare frontier models on Alex-specific capabilities python -m src.cli run claude-opus-4 gpt-4o gemini-2.5-pro \ -d cognitive_partnership \ --judge claude-opus-4 # Quick check: is this model good enough for Alex? python -m src.cli run claude-sonnet-4 -d cognitive_partnership --judge gpt-4o ``` #### Cost-Efficiency Analysis ```bash # Compare cheap vs expensive models to find the value sweet spot python -m src.cli run gpt-4o gpt-4o-mini claude-sonnet-4 claude-haiku gemini-2.5-flash \ --judge gpt-4o ``` Then compare Brain Scores against token costs in the JSON report. #### Regression Testing After Model Updates ```bash # When a provider updates a model, re-run to detect changes python -m src.cli run claude-sonnet-4 --judge gpt-4o --output results/sonnet4-march2026 ``` Compare with previous results to catch regressions. #### Dimension Deep-Dive ```bash # If a model scores low on a dimension, test just that dimension # to get faster iteration on analysis python -m src.cli run gpt-4o -d robustness_adversarial --judge gpt-4o -v ``` The `-v` (verbose) flag shows every challenge prompt and response for debugging. ### 6. Interpreting the Brain Score The Brain Score is a **weighted composite** (0-100) across all dimensions tested. Higher is better. | Score Range | Interpretation | |:-----------:|---------------| | **90-100** | Exceptional — frontier performance across the board | | **80-89** | Strong — suitable for demanding cognitive partnership | | **70-79** | Good — capable for most tasks, some limitations | | **60-69** | Adequate — usable for routine work, not complex reasoning | | **50-59** | Below average — significant gaps in multiple areas | | **<50** | Insufficient — not recommended for cognitive partnership | **Important:** Brain Score is only meaningful when comparing models tested on the **same dimensions**. Running one model on 3 dimensions and another on 17 produces incomparable scores. ### 7. Tips & Best Practices - **Always use a judge** for Alex fitness and creative dimensions — without one, rubric-scored challenges default to 0 - **Use temperature 0.0** (the default) for reproducible benchmarks - **Run multiple times** if you need high confidence — LLM outputs have inherent variance - **Compare within the same run** for fairness — API behavior can change between days - **Start with 3-5 key dimensions** for quick feedback, then run the full suite for final analysis - **Save JSON reports** — they contain full response text for manual review of edge cases --- ### 8. Chat Window Benchmark Mode (How Alex Gets Tested) Alex lives in the VS Code Copilot Chat window — not behind an API. The Chat Benchmark mode tests the LLM right where it operates. #### Three Workflows **Option A: Automated (Recommended)** — one paste, auto-scored The fastest path. Generate a single prompt, paste it into chat, the LLM processes all challenges and writes its responses to a file, then you auto-score it. ```bash # Step 1: Generate the prompt python -m src.cli chat generate # all 142 challenges python -m src.cli chat generate -d cognitive_partnership # Alex fitness only # Step 2: Open results/benchmark_prompt.md, copy ALL contents, paste into VS Code Copilot Chat # The LLM will process every challenge and create results/responses.json # Step 3: Auto-score the results python -m src.cli chat evaluate results/responses.json --model claude-sonnet-4 --judge gpt-4o ``` How it works: 1. `generate` creates a markdown file containing all challenges as JSON with instructions 2. You paste it into Copilot Chat — the LLM reads every challenge, formulates responses, and saves them to `results/responses.json` 3. `evaluate` reads that JSON, runs deterministic scoring on exact/numeric challenges, and uses an LLM judge (via API) for rubric-based challenges 4. You get the full report — Brain Score, per-dimension breakdown, strengths, weaknesses **Important:** The `--judge` flag on `evaluate` calls an API to score rubric-based challenges. Without it, only deterministic challenges (exact match, numeric, keyword) get scored. Most Alex fitness challenges need the judge. **Option B: Interactive Mode** (guided, one challenge at a time) ```bash # Full benchmark — walks you through every challenge python -m src.cli chat start claude-sonnet-4-copilot # Alex fitness only — 19 challenges focused on cognitive partnership python -m src.cli chat alex claude-sonnet-4-copilot # Specific dimensions python -m src.cli chat start claude-sonnet-4-copilot \ -d cognitive_partnership \ -d code_generation ``` Interactive mode: 1. Shows you a challenge prompt in a green panel 2. You copy it and paste into VS Code Copilot Chat 3. You read the response and the scoring rubric 4. You enter a score (0-100) in the terminal 5. Moves to the next challenge 6. Saves results when done (or press `q` to quit early with partial results) **Option C: Challenge Sheet** (offline, work at your own pace) ```bash # Generate a challenge sheet for all dimensions python -m src.cli chat sheet # Generate for Alex fitness only python -m src.cli chat sheet -d cognitive_partnership # Custom output path python -m src.cli chat sheet -o results/my_sheet.md ``` This creates a markdown file with all prompts and rubrics. Work through it at your own pace: 1. Open the generated `results/challenge_sheet.md` 2. For each challenge, copy the prompt into VS Code Copilot Chat 3. Read the response and the scoring guide 4. Replace `____ / 100` with your score (e.g., `85 / 100`) 5. Add notes if desired 6. When done, process the scores: ```bash python -m src.cli chat score results/challenge_sheet.md --model claude-sonnet-4-copilot ``` #### Quick Alex Fitness Check The fastest way to evaluate whether a model is fit for Alex: ```bash python -m src.cli chat alex claude-sonnet-4-copilot ``` This runs only the 19 Cognitive Partnership challenges. Takes about 30-45 minutes and directly tells you if the model can handle meta-cognition, identity persistence, ethical reasoning, bootstrap learning, code review depth, and empathetic partnership. #### Comparing Multiple Models To compare models in the chat window, run separate sessions and compare the JSON reports: ```bash # Session 1: Test Claude python -m src.cli chat alex claude-sonnet-4-copilot # Session 2: Switch the model in VS Code settings and test GPT python -m src.cli chat alex gpt-4o-copilot # Session 3: Test Gemini python -m src.cli chat alex gemini-2.5-flash-copilot # Compare all results python -m src.cli view results/chat_claude-sonnet-4-copilot_*.json python -m src.cli view results/chat_gpt-4o-copilot_*.json ``` **Changing the model in VS Code:** Open Settings → search `github.copilot.chat.models` or use the model picker in the chat panel to switch between available models. #### Scoring Tips for Chat Mode | Score | When to Give | |:-----:|-------------| | **90-100** | Exceptional response — nails the rubric, shows genuine insight | | **70-89** | Good response — mostly correct, minor gaps | | **50-69** | Adequate — gets the gist but misses important aspects | | **30-49** | Weak — partially correct but significant issues | | **10-29** | Poor — misses the point, major errors | | **0-9** | Failed — wrong answer, refused inappropriately, or ignored the task | **Be consistent:** If scoring multiple models, establish your standards on the first model and apply them uniformly. **Use the rubric:** Every challenge has either an expected answer or a scoring guide. The rubric tells you exactly what earns 100, 50, and 0 — use it. **Take notes:** The `Notes` field in challenge sheets is valuable for understanding WHY a model scored as it did. Patterns in notes reveal deeper insights than raw numbers. ## License MIT

Loading the repository README…

Skills in this repository

Systematic problem-solving and error analysis.

2026-03-08

deep-work-optimization

Exercise Trainers & Group Fitness InstructorsTour Guides & Escorts

Focus blocks, distraction management, and flow state triggers for cognitively demanding work

2026-03-08

dissertation-defense-skill

Social Sciences Teachers, Postsecondary, All Other

Comprehensive preparation for doctoral dissertation defense including timeline management, presentation design, Q&A practice, mock sessions, and committee dynamics.

2026-03-08

Showing 40 of 120 collected skills.