vowel-core

name	vowel-core
description	Write, run, and debug YAML-based evaluation specs for Python functions using the vowel framework. This skill contains the complete YAML specification, all evaluators, fixtures, serializers, CLI, and the RunEvals fluent API. Use when the user asks about testing Python functions with vowel, writing eval YAML files, running evaluations, or using fixtures/serializers.
license	Apache-2.0
metadata	{"author":"fswair","version":"2.0"}

Vowel — YAML Evaluation Framework

vowel is a YAML-based evaluation framework for testing Python functions. Write specs, run them, check results.

pip install vowel

---

YAML Specification

Basic Structure

# Top-level fixture definitions (optional)
fixtures:
  fixture_name:
    setup: module.setup_func
    teardown: module.teardown_func   # optional
    scope: function                  # function | module | session
    kwargs:                          # optional
      key: value

# Function evaluation specs
function_name:
  fixture:                           # fixture dependencies (optional)
    - fixture_name
  evals:                             # global evaluators (optional)
    EvaluatorName:
      param: value
  dataset:                           # test cases (required)
    - case:
        input: <value>               # single parameter
        # OR
        inputs: { a: 1, b: 2 }      # named parameters (dict)
        # OR
        inputs: [1, 2, 3]           # positional parameters (list)
        expected: <value>            # expected output (optional)
        raises: ExceptionType        # expected exception (optional)
        match: "regex pattern"       # exception message match (optional)
        duration: 50                 # max ms for this case (optional)

Input Types

# Single parameter: func(x)
single_param:
  dataset:
    - case:
        input: 42
        expected: 84

# Multiple positional: func(a, b, c)
multi_positional:
  dataset:
    - case:
        inputs: [1, 2, 3]
        expected: 6

# Named parameters: func(x=1, y=2)
named_params:
  dataset:
    - case:
        inputs: { x: 10, y: 20 }
        expected: 30

Function Sources

vowel resolves functions from 4 sources:

# 1. Python builtins
len:
  dataset:
    - case: { input: [1, 2, 3], expected: 3 }

# 2. Standard library (module.function)
math.sqrt:
  dataset:
    - case: { input: 16, expected: 4.0 }

os.path.join:
  dataset:
    - case: { inputs: ["/home", "user"], expected: "/home/user" }

# 3. Your own functions (via functions= or with_functions())
my_function:
  dataset:
    - case: { input: "hello", expected: "HELLO" }

# 4. Local module functions (module.function)
utils.uppercase:
  dataset:
    - case: { input: "hello", expected: "HELLO" }

Fixtures in YAML

fixtures:
  db:
    setup: myapp.fixtures.setup_db
    teardown: myapp.fixtures.close_db
    scope: module
    kwargs:
      db_name: test_db

  cache:
    setup: myapp.fixtures.setup_cache
    scope: session

  temp_dir:
    setup: myapp.fixtures.create_temp_dir
    teardown: myapp.fixtures.remove_temp_dir
    scope: function

query_user:
  fixture:
    - db
  dataset:
    - case:
        inputs: { user_id: 1 }
        expected: { name: "Alice", email: "[e-mail]" }

save_with_cache:
  fixture:
    - db
    - cache
  evals:
    Type:
      type: "bool"
  dataset:
    - case:
        inputs: { key: "user:1", value: "Alice" }
        expected: true

Functions MUST use keyword-only args for fixtures:

def query_user(user_id: int, *, db: dict) -> dict | None:
    return db["users"].get(user_id)

def save_with_cache(key: str, value: str, *, db: dict, cache: dict) -> bool:
    db["data"][key] = value
    cache[key] = value
    return True

Input Serializers in YAML

Schema mode — automatic dict → type:

get_user_info:
  dataset:
    - case:
        input: { id: 1, name: "Alice", email: "[e-mail]" }
        expected: "User Alice has email [e-mail]"

from pydantic import BaseModel
from vowel import RunEvals

class User(BaseModel):
    id: int
    name: str
    email: str

summary = (
    RunEvals.from_file("user_evals.yml")
    .with_functions({"get_user_info": get_user_info})
    .with_serializer({"get_user_info": User})
    .run()
)

Dict schema — per-parameter types:

.with_serializer({"process": {"user": User, "config": Config}})

Serial fn mode — full control:

from datetime import date

def parse_date(data: dict) -> date:
    raw = data.get("input") or data.get("inputs")
    return date.fromisoformat(raw)

summary = (
    RunEvals.from_source(spec)
    .with_functions({"get_year": get_year})
    .with_serializer(serial_fn={"get_year": parse_date})
    .run()
)

Return types from serial_fn:

Return Type	Behavior
Single value	Passed as single argument
Tuple	Unpacked as positional arguments
Dict	Unpacked as keyword arguments

Raises Variants

# Specific exception type
- case:
    input: "bad"
    raises: ZeroDivisionError
    match: "division by zero"       # optional regex on message

# Dotted exception paths (compared by short name)
- case:
    input: "bad"
    raises: mypackage.errors.APIError

# Any exception must be raised
- case:
    input: "bad"
    raises: any

# Any exception OR normal return (both pass)
- case:
    input: "maybe"
    raises: any?

---

Evaluators

8 built-in evaluators.

Expected Value (Exact Match)

add:
  dataset:
    - case:
        inputs: { x: 2, y: 3 }
        expected: 5

Type Checking

divide:
  evals:
    Type:
      type: "float"
      strict: true       # no type promotion (int won't pass for float)
  dataset:
    - case:
        inputs: { a: 10, b: 3 }

Supported: int, float, str, bool, list, dict, int | float, str | None, etc.

Assertion

is_positive:
  evals:
    Assertion:
      assertion: "output == (input > 0)"
  dataset:
    - case: { input: 5 }
    - case: { input: -3 }

Available variables:

Variable	Description
input	Input value(s): single value, list, or dict
output	Function return value
expected	Expected value (if provided)
duration	Execution time in seconds

Examples:

assertion: "output > 0"
assertion: "output == input * 2"
assertion: "abs(output - expected) < 0.001"
assertion: "output == input[0] + input[1]"       # list inputs
assertion: "output == input['x'] * input['y']"    # dict inputs

Duration (Performance)

Function-level (seconds):

fast_function:
  evals:
    Duration:
      duration: 0.5
  dataset:
    - case: { input: 1000 }

Case-level (milliseconds):

compute:
  dataset:
    - case:
        input: 100
        duration: 50

Pattern Matching (Regex)

format_phone:
  evals:
    Pattern:
      pattern: "^\\+?[0-9]{10,14}$"
      case_sensitive: true
  dataset:
    - case: { input: "5551234567" }

Contains Input

echo:
  evals:
    ContainsInput:
      case_sensitive: false
      as_strings: true
  dataset:
    - case: { input: "Hello" }

Raises (Exception Testing)

divide:
  dataset:
    - case:
        inputs: { a: 10, b: 0 }
        raises: ZeroDivisionError
        match: "division by zero"

    - case:
        input: "bad"
        raises: any

    - case:
        input: "maybe"
        raises: any?

LLM Judge

summarize:
  evals:
    LLMJudge:
      rubric: "Summary captures main points without losing critical information"
      include: [input, expected_output]
      config:
        model: "openai:gpt-4o-mini"
        temperature: 0.0
  dataset:
    - case:
        input: "Long article text..."

Combining Evaluators

factorial:
  evals:
    Assertion:
      assertion: "output > 0"
    Type:
      type: "int"
    Duration:
      duration: 1.0
  dataset:
    - case: { input: 0, expected: 1 }
    - case: { input: 5, expected: 120 }
    - case: { input: 10, expected: 3628800 }

---

Fixtures

Fixtures inject external dependencies (databases, temp files, caches) as keyword-only arguments.

Three Patterns

# 1. Generator (pytest-style yield)
import tempfile, os

def temp_file():
    fd, path = tempfile.mkstemp()
    yield path          # injected into function
    os.close(fd)
    os.remove(path)

# 2. Tuple (setup + teardown)
def setup_db():
    return Database.connect()

def teardown_db(conn):
    conn.close()

# Usage: .with_fixtures({"db": (setup_db, teardown_db)})

# 3. Simple (setup only)
def sample_data():
    return {"users": ["alice", "bob"]}

# Usage: .with_fixtures({"data": sample_data})

Fixture Scopes

Scope	Behavior
function (default)	Setup/teardown for each test case
module	Setup once per eval spec, teardown after all cases
session	Setup once per run_evals() call, teardown at end

Function Signature Pattern

def query_user(user_id: int, *, db: dict) -> dict | None:
    return db["users"].get(user_id)

CLI Inspection

vowel evals.yml --list-fixtures    # list fixtures with scope/usage
vowel evals.yml --fixture-tree     # show dependency tree

---

CLI Reference

vowel [OPTIONS] [YAML_FILE]

Option	Short	Description
--debug		Enable debug mode with stack traces
--dir	-d	Run all YAML files in directory (recursive)
--filter	-f	Only run specific function(s) (comma-separated)
--cov / --coverage		Required coverage percent (default: 100)
--ci		CI mode — exit 1 if coverage not met
--quiet	-q	Minimal output
--watch	-w	Watch mode: re-run on file changes
--verbose	-v	Detailed summary: spec overview, per-case breakdown, pass rate
--hide-report		Hide pydantic_evals report output
--dry-run		Show test plan without running
--list-fixtures		List fixtures with setup/teardown/scope/usage
--fixture-tree		Show fixture dependency tree
--export-json PATH		Export results to JSON file

Common Commands

vowel evals.yml                         # run single file
vowel -d ./tests                        # run all YAML in directory
vowel evals.yml -f add,divide           # filter functions
vowel evals.yml --ci --cov 90           # CI mode with coverage threshold
vowel evals.yml -w                      # watch mode
vowel evals.yml -v --hide-report        # verbose, clean output
vowel evals.yml --export-json out.json  # export results

---

Fluent API (RunEvals)

from vowel import RunEvals

summary = (
    RunEvals.from_file("evals.yml")          # or .from_source(yaml_str)
    .with_functions({"add": add})             # provide functions
    .with_fixtures({"db": setup_db})          # provide fixtures
    .with_serializer({"func": MyModel})       # input serializers
    .filter(["add"])                           # filter to specific functions
    .debug()                                   # enable debug mode
    .ignore_duration()                         # skip duration evaluator
    .run()                                     # execute
)

EvalSummary

summary.all_passed       # bool
summary.success_count    # int
summary.failed_count     # int
summary.error_count      # int
summary.total_count      # int
summary.coverage         # float (0.0-1.0)
summary.has_errors       # bool
summary.has_failed_cases # bool
summary.meets_coverage(0.9)
summary.print()
summary.to_json()
summary.xml()

---

Troubleshooting

Error	Solution
Function not found	Use module.func path or provide via with_functions()
Undefined fixture	Define in YAML fixtures: block or with_fixtures()
Type compatibility	Use with_serializer() for non-primitive types
Assertion failure	Check variables: input, output, expected, duration
Coverage not met	Add test cases or lower threshold: --cov 90