kaggle-benchmarks

# Skill: Writing Kaggle Benchmarks Tasks > This skill file teaches you how to write high-quality benchmark tasks using the `kaggle-benchmarks` Python library (version 0.7.0+). > Always verify patterns against the actual source code in `src/kaggle_benchmarks/` when in doubt. ## Quick Reference ```python import kaggle_benchmarks as kbench ``` | Symbol | Purpose | |--------|---------| | `kbench.task` / `kbench.benchmark` | Decorator to define a benchmark task | | `kbench.llm` | Default LLM actor (available when Kaggle is configured) | | `kbench.judge_llm` | Judge LLM for evaluation | | `kbench.llms` | Dict of all available models (e.g. `kbench.llms["google/gemini-2.5-flash"]`) | | `kbench.assertions` | Module with all assertion functions | | `kbench.chats` | Conversation/chat context management | | `kbench.ChatRoom` / `kbench.Participant` | Multi-agent conversation room with perspective-aware history | | `kbench.tools` | Built-in tools (Python runner, etc.) | | `kbench.user` / `kbench.actors.user` | Send user messages to conversation | | `kbench.system` / `kbench.actors.system` | Send system-level messages | | `kbench.last_reasoning_traces()` | Access reasoning traces from last prompt | | `kbench.content_types.images` | Image input helpers | | `kbench.content_types.videos` | Video input helpers | | `kbench.content_types.audios` | Audio input helpers | | `kbench.client` | Client for caching, storage | ## Minimal Examples ### Simple assertion check ```python import kaggle_benchmarks as kbench @kbench.task(name="geography_quiz") def geography_quiz(llm): response = llm.prompt("What is the longest river in the world?") kbench.assertions.assert_contains_regex( r"(?i)nile", response, expectation="Should mention the Nile river." ) geography_quiz.run(kbench.llm) ``` ### Evaluating a list of questions ```python import kaggle_benchmarks as kbench import pandas as pd @kbench.task(name="math_qa", store_task=False) def math_qa(llm, question, expected) -> bool: answer = llm.prompt(question + "\nAnswer with just the number.", schema=int) kbench.assertions.assert_equal(expected, answer) return answer == expected # %% df = pd.DataFrame([ {"question": "What is 15% of 200?", "expected": 30}, {"question": "What is 7 × 8?", "expected": 56}, ]) @kbench.task(name="math_benchmark") def math_benchmark(llm) -> float: results = math_qa.evaluate(llm=[llm], evaluation_data=df, n_jobs=2) scores = results.as_dataframe() return float(scores.result.mean()) math_benchmark.run(kbench.llm) ``` ## Key Rules - The first parameter of every task function **must** be the LLM actor. - If your task returns a value, you **MUST** add a return type annotation (`-> float`, `-> bool`, `-> dict`, etc.). - Use `kbench.assertions.*` instead of Python `assert` — library assertions are recorded and tracked. - Always check `assess_response_with_judge` for `None` before using the result. - Do NOT wrap `.run()` or `.evaluate()` calls inside `if __name__ == "__main__":`. Benchmark files are notebook-style scripts — all code runs at the top level. - Use `# %%` cell markers to create logical sections in benchmark files. - Prefer `# !pip install ...` (commented) over `!pip install ...` so the file works everywhere. - Use `store_task=False` for sub-tasks called inside other tasks. ## Common Mistakes to Avoid | Mistake | Correct Approach | |---------|-----------------| | Missing return type annotation on scoring task | Add `-> float`, `-> bool`, `-> dict`, etc. | | Using Python `assert` instead of `kbench.assertions.*` | Use library assertions — they're recorded and tracked | | Not checking `assess_response_with_judge` for `None` | Always check: `if assessment is None:` | | Using `kbench.llm` locally without Kaggle configured | Run `kaggle benchmarks init` to configure, or set env vars | | Forgetting `schema=` when needing structured output | Pass `schema=MyDataclass` to `llm.prompt()` | | Wrapping `.run()` / `.evaluate()` in `if __name__ == "__main__":` | Place them at module top level — benchmark files are scripts, not importable modules | | Using `user.send()` with image URLs | `user.send()` passes URLs as-is; prefer `llm.prompt(image=)` for auto-conversion | | Not isolating judge conversations | Use `with kbench.chats.new("judge"):` | | Multiple tasks sharing conversation history | Each `.run()` creates its own conversation | | Using `store_task=True` for sub-tasks | Set `store_task=False` for helper tasks called inside other tasks | | Using `!pip install` without commenting | Use `# !pip install -q pkg` — uncommented magics break local execution | | Forgetting `last_reasoning_traces()` can be `None` | Always check: `traces = kbench.last_reasoning_traces(); if traces: ...` | | Aggregating over all runs after `on_failure="continue"` | Filter first: `results.completed_runs.as_dataframe().result.mean()` — failed runs carry the `results.FAILED` sentinel which breaks `.mean()` / `.sum()` | | Using `max_attempts > 1` without `on_failure="continue"` | The default `"raise"` aborts on first failure, so retries never happen. Pair `max_attempts > 1` with `on_failure="continue"` and `enable_cache()` for selective retry. | --- ## §1. Import Styles There are two main import styles. **Prefer Style A** for clarity. ### Style A: Module import (Preferred) ```python import kaggle_benchmarks as kbench @kbench.task(name="my_task") def my_task(llm): response = llm.prompt("Question?") kbench.assertions.assert_true(True) ``` ### Style B: Direct imports ```python from kaggle_benchmarks import assertions, chats, llm, task, system, user @task("my_task") def my_task(llm): response = llm.prompt("Question?") assertions.assert_true(True) ``` Style B is shorter but risks name collisions (e.g., `llm` is both a module-level variable and a task parameter). ### File Structure: Cell Markers Benchmark files are Python scripts (`.py`), but use `# %%` cell markers to create logical sections. This makes them runnable as both standalone Python files and as interactive notebooks (via Jupyter/VS Code cell execution). ```python # %% import kaggle_benchmarks as kbench # %% @kbench.task() def my_task(llm): response = llm.prompt("Hello!") kbench.assertions.assert_not_empty(response) my_task.run(kbench.llm) # %% @kbench.task() def another_task(llm) -> float: ... ``` **IPython magics (`!pip install`, `%time`, etc.):** These work on Kaggle notebooks but NOT when running as standalone Python files. If you need a magic command (e.g., to install a dependency), comment it out so the file remains runnable locally: ```python # %% # !pip install -q pronouncing syllables # Uncomment on Kaggle import pronouncing ``` > **Rule:** Prefer `# !pip install ...` (commented) over `!pip install ...` so the file works everywhere. Only use uncommented magics when the file is exclusively for Kaggle notebook execution. **IMPORTANT — No `if __name__` guards.** Think of benchmark `.py` files as notebooks, not modules. They are never imported — they are always executed directly. ```python # ❌ WRONG — do not do this if __name__ == "__main__": my_task.run(kbench.llm) # ✅ CORRECT — top-level, in its own cell # %% my_task.run(kbench.llm) ``` --- ## §2. Defining Tasks ### `@kbench.task()` Parameters ```python @kbench.task( name="optional_name", # Defaults to function name, title-cased description="What it does", # Defaults to docstring version=1, # Task version store_task=True, # Set False for sub-tasks store_run=True, # Set False to skip storing results ) def my_task(llm): ... ``` `@kbench.benchmark()` is an exact alias for `@kbench.task()`. ### Task First Parameter The first parameter **must** be the LLM actor. It receives the model to test. ```python @kbench.task() def my_task(llm): # ✅ Correct ... @kbench.task() def my_task(llm, judge_llm): # ✅ Also fine — second LLM for judging ... ``` ### Task Additional Parameters Extra parameters are passed via `.run()` kwargs: ```python @kbench.task() def check_knowledge(llm, question, expected_answer): response = llm.prompt(question) kbench.assertions.assert_contains_regex( rf"(?i){expected_answer}", response ) check_knowledge.run(kbench.llm, question="Capital of Japan?", expected_answer="Tokyo") ``` ### Return Types **If your task returns a value, you MUST add a return type annotation.** | Annotation | Result Type | Meaning | |------------|-------------|---------| | (none) or `-> None` | PassFail | Pass if no exceptions, based on assertions | | `-> bool` | Boolean | True = pass, False = fail | | `-> float` | Score | Numerical score | | `-> int` | Numerical | Integer value | | `-> dict` | Dictionary | Arbitrary dict result | | `-> tuple[int, int]` | PassCount | Count (e.g., `(8, 10)`) | | `-> tuple[float, float]` | MetricWithCI | Value ± confidence interval | > **Note:** `-> None` is equivalent to omitting the annotation — both produce PassFail. ```python # Score task @kbench.task() def accuracy(llm) -> float: return 0.85 # Count task @kbench.task() def count_correct(llm) -> tuple[int, int]: return (8, 10) # 8 out of 10 passed # Dict task (for rich results) @kbench.task() def detailed_result(llm) -> dict: return {"accuracy": 0.9, "latency": 1.2, "is_correct": True} ``` --- ## §3. Running Tasks ### Running a Task ```python # Single run — returns a Run object run = my_task.run(kbench.llm) # With extra parameters run = my_task.run(kbench.llm, question="What is Python?") # Multiple models run1 = my_task.run(kbench.llm) # Default model run2 = my_task.run(kbench.judge_llm) # Judge model ``` **Available models (loaded from Kaggle environment):** - `kbench.llm` — default model - `kbench.judge_llm` — judge model - `kbench.llms` — list of ALL available models (useful for multi-model comparison) ### Run Object Properties The `Run` object returned by `.run()` has useful attributes: ```python run = my_task.run(kbench.llm) run.passed # bool — True if result + all assertions passed run.result # The returned value (type depends on task return annotation) run.assertion_results # list[AssertionResult] — all recorded assertions run.status # Status enum (PENDING, DONE, FAILED) run.chat # The conversation log ``` This is especially useful in sub-task composition: ```python runs = [subtask.run(llm, q=q) for q in questions] accuracy = sum(r.passed for r in runs) / len(runs) ``` ### Batch Evaluation: `.evaluate()` ```python import pandas as pd results = my_task.evaluate( llm=[kbench.llm], # List of models evaluation_data=df, # DataFrame of test cases n_jobs=3, # Parallel workers (default: 1) timeout=120, # Per-job timeout in seconds max_attempts=3, # Retry count retry_delay=15, # Seconds between retries on_failure="raise", # "raise" (default) or "continue" stop_condition=lambda runs: len(runs) == df.shape[0], # Early stop remove_run_files=True, # Clean up after ) # Access results results.as_dataframe() ```