vowel-core

# Vowel — YAML Evaluation Framework vowel is a YAML-based evaluation framework for testing Python functions. Write specs, run them, check results. ```bash pip install vowel ``` --- ## YAML Specification ### Basic Structure ```yaml # Top-level fixture definitions (optional) fixtures: fixture_name: setup: module.setup_func teardown: module.teardown_func # optional scope: function # function | module | session kwargs: # optional key: value # Function evaluation specs function_name: fixture: # fixture dependencies (optional) - fixture_name evals: # global evaluators (optional) EvaluatorName: param: value dataset: # test cases (required) - case: input: <value> # single parameter # OR inputs: { a: 1, b: 2 } # named parameters (dict) # OR inputs: [1, 2, 3] # positional parameters (list) expected: <value> # expected output (optional) raises: ExceptionType # expected exception (optional) match: "regex pattern" # exception message match (optional) duration: 50 # max ms for this case (optional) ``` ### Input Types ```yaml # Single parameter: func(x) single_param: dataset: - case: input: 42 expected: 84 # Multiple positional: func(a, b, c) multi_positional: dataset: - case: inputs: [1, 2, 3] expected: 6 # Named parameters: func(x=1, y=2) named_params: dataset: - case: inputs: { x: 10, y: 20 } expected: 30 ``` ### Function Sources vowel resolves functions from 4 sources: ```yaml # 1. Python builtins len: dataset: - case: { input: [1, 2, 3], expected: 3 } # 2. Standard library (module.function) math.sqrt: dataset: - case: { input: 16, expected: 4.0 } os.path.join: dataset: - case: { inputs: ["/home", "user"], expected: "/home/user" } # 3. Your own functions (via functions= or with_functions()) my_function: dataset: - case: { input: "hello", expected: "HELLO" } # 4. Local module functions (module.function) utils.uppercase: dataset: - case: { input: "hello", expected: "HELLO" } ``` ### Fixtures in YAML ```yaml fixtures: db: setup: myapp.fixtures.setup_db teardown: myapp.fixtures.close_db scope: module kwargs: db_name: test_db cache: setup: myapp.fixtures.setup_cache scope: session temp_dir: setup: myapp.fixtures.create_temp_dir teardown: myapp.fixtures.remove_temp_dir scope: function query_user: fixture: - db dataset: - case: inputs: { user_id: 1 } expected: { name: "Alice", email: "alice@test.com" } save_with_cache: fixture: - db - cache evals: Type: type: "bool" dataset: - case: inputs: { key: "user:1", value: "Alice" } expected: true ``` Functions MUST use keyword-only args for fixtures: ```python def query_user(user_id: int, *, db: dict) -> dict | None: return db["users"].get(user_id) def save_with_cache(key: str, value: str, *, db: dict, cache: dict) -> bool: db["data"][key] = value cache[key] = value return True ``` ### Input Serializers in YAML **Schema mode** — automatic dict → type: ```yaml get_user_info: dataset: - case: input: { id: 1, name: "Alice", email: "alice@test.com" } expected: "User Alice has email alice@test.com" ``` ```python from pydantic import BaseModel from vowel import RunEvals class User(BaseModel): id: int name: str email: str summary = ( RunEvals.from_file("user_evals.yml") .with_functions({"get_user_info": get_user_info}) .with_serializer({"get_user_info": User}) .run() ) ``` **Dict schema** — per-parameter types: ```python .with_serializer({"process": {"user": User, "config": Config}}) ``` **Serial fn mode** — full control: ```python from datetime import date def parse_date(data: dict) -> date: raw = data.get("input") or data.get("inputs") return date.fromisoformat(raw) summary = ( RunEvals.from_source(spec) .with_functions({"get_year": get_year}) .with_serializer(serial_fn={"get_year": parse_date}) .run() ) ``` Return types from serial_fn: | Return Type | Behavior | |-------------|----------| | **Single value** | Passed as single argument | | **Tuple** | Unpacked as positional arguments | | **Dict** | Unpacked as keyword arguments | ### Raises Variants ```yaml # Specific exception type - case: input: "bad" raises: ZeroDivisionError match: "division by zero" # optional regex on message # Dotted exception paths (compared by short name) - case: input: "bad" raises: mypackage.errors.APIError # Any exception must be raised - case: input: "bad" raises: any # Any exception OR normal return (both pass) - case: input: "maybe" raises: any? ``` --- ## Evaluators 8 built-in evaluators. ### Expected Value (Exact Match) ```yaml add: dataset: - case: inputs: { x: 2, y: 3 } expected: 5 ``` ### Type Checking ```yaml divide: evals: Type: type: "float" strict: true # no type promotion (int won't pass for float) dataset: - case: inputs: { a: 10, b: 3 } ``` Supported: `int`, `float`, `str`, `bool`, `list`, `dict`, `int | float`, `str | None`, etc. ### Assertion ```yaml is_positive: evals: Assertion: assertion: "output == (input > 0)" dataset: - case: { input: 5 } - case: { input: -3 } ``` **Available variables:** | Variable | Description | |----------|-------------| | `input` | Input value(s): single value, list, or dict | | `output` | Function return value | | `expected` | Expected value (if provided) | | `duration` | Execution time in seconds | **Examples:** ```yaml assertion: "output > 0" assertion: "output == input * 2" assertion: "abs(output - expected) < 0.001" assertion: "output == input[0] + input[1]" # list inputs assertion: "output == input['x'] * input['y']" # dict inputs ``` ### Duration (Performance) Function-level (seconds): ```yaml fast_function: evals: Duration: duration: 0.5 dataset: - case: { input: 1000 } ``` Case-level (milliseconds): ```yaml compute: dataset: - case: input: 100 duration: 50 ``` ### Pattern Matching (Regex) ```yaml format_phone: evals: Pattern: pattern: "^\\+?[0-9]{10,14}$" case_sensitive: true dataset: - case: { input: "5551234567" } ``` ### Contains Input ```yaml echo: evals: ContainsInput: case_sensitive: false as_strings: true dataset: - case: { input: "Hello" } ``` ### Raises (Exception Testing) ```yaml divide: dataset: - case: inputs: { a: 10, b: 0 } raises: ZeroDivisionError match: "division by zero" - case: input: "bad" raises: any - case: input: "maybe" raises: any? ``` ### LLM Judge ```yaml summarize: evals: LLMJudge: rubric: "Summary captures main points without losing critical information" include: [input, expected_output] config: model: "openai:gpt-4o-mini" temperature: 0.0 dataset: - case: input: "Long article text..." ``` ### Combining Evaluators ```yaml factorial: evals: Assertion: assertion: "output > 0" Type: type: "int" Duration: duration: 1.0 dataset: - case: { input: 0, expected: 1 } - case: { input: 5, expected: 120 } - case: { input: 10, expected: 3628800 } ``` --- ## Fixtures Fixtures inject external dependencies (databases, temp files, caches) as keyword-only arguments. ### Three Patterns ```python # 1. Generator (pytest-style yield) import tempfile, os def temp_file(): fd, path = tempfile.mkstemp() yield path # injected into function os.close(fd) os.remove(path) # 2. Tuple (setup + teardown) def setup_db(): return Database.connect() def teardown_db(conn): conn.close() # Usage: .with_fixtures({"db": (setup_db, teardown_db)}) # 3. Simple (setup only) def sample_data(): return {"users": ["alice", "bob"]} # Usage: .with_fixtures({"data": sample_data}) ``` ### Fixture Scopes | Scope | Behavior | |-------|----------| | `function` (default) | Setup/teardown for each test case | | `module` | Setup once per eval spec, teardown after all cases | | `session` | Setup once per `run_evals()` call, teardown at end | ### Function Signature Pattern ```python def query_user(user_id: int, *, db: dict) -> dict | None: return db["users"].get(user_id) ``` ### CLI Inspection ```bash vowel evals.yml --list-fixtures # list fixtures with scope/usage vowel evals.yml --fixture-tree # show dependency tree ``` --- ## CLI Reference ```bash vowel [OPTIONS] [YAML_FILE] ``` | Option | Short | Description | |--------|-------|-------------| | `--debug` | | Enable debug mode with stack traces | | `--dir` | `-d` | Run all YAML files in directory (recursive) | | `--filter` | `-f` | Only run specific function(s) (comma-separated) | | `--cov` / `--coverage` | | Required coverage percent (default: 100) | | `--ci` | | CI mode — exit 1 if coverage not met | | `--quiet` | `-q` | Minimal output | | `--watch` | `-w` | Watch mode: re-run on file changes |