testing

name	testing
description	Author Python test suites graders execute: pytest unit tests with fixtures and tmp_path, parametrized edge cases, mocking external boundaries with unittest.mock, property-based invariants via hypothesis, integration markers with conditional skips, coverage measurement with thresholds, and unittest.TestCase for legacy suites.
metadata	{"dependencies":["pytest","hypothesis","coverage"]}

Testing Recipe

Tasks produce a tests/ tree (and sometimes pytest.ini / coverage config) that a grader runs offline with pytest or coverage.

Step 1 — Pin the input/output mapping before writing tests

List (input, expected output) pairs and error inputs. Each row becomes one test (Step 2) or one parametrize row (Step 4). If you cannot enumerate expected outputs, you do not understand the contract.

# parse_amount(s) -> Decimal:  "$1,200" -> Decimal("1200")
#   "" raises ValueError("empty");  "abc" raises ValueError("not numeric")

Step 2 — Write the simplest passing pytest test

One file per module, one function per behaviour. The name spells out what's broken on failure. Use bare assert; pytest rewrites it.

# tests/test_amount.py
import pytest
from decimal import Decimal
from myapp.amount import parse_amount

def test_parse_amount_strips_currency_and_commas():
    assert parse_amount("$1,200") == Decimal("1200")

def test_parse_amount_rejects_empty_string():
    with pytest.raises(ValueError, match="empty"):
        parse_amount("")

Step 3 — Lift shared setup into pytest.fixture; use tmp_path for files

A fixture yields the resource and (optionally) tears down after yield. Default scope is function — fresh per test. tmp_path (built-in, pathlib.Path) gives each test an isolated, auto-cleaned directory.

@pytest.fixture
def config_file(tmp_path):
    path = tmp_path / "cfg.json"
    path.write_text('{"threshold": 5}')
    return path

def test_loader_reads_threshold(config_file):
    assert load_config(config_file).threshold == 5

For genuinely read-only expensive setup use scope="session".

Step 4 — Sweep edge cases with @pytest.mark.parametrize

One test body, many input rows. pytest.param(..., id="...") names each row in the report. Parametrize variations of the same behaviour; if assertion logic differs, write a separate test.

@pytest.mark.parametrize("raw,expected", [
    pytest.param("1.50",   Decimal("1.50"), id="plain"),
    pytest.param("$1,200", Decimal("1200"), id="currency"),
    pytest.param("  3 ",   Decimal("3"),    id="whitespace"),
])
def test_parse_amount_valid(raw, expected):
    assert parse_amount(raw) == expected

@pytest.mark.parametrize("bad", ["", "abc", "1.2.3"])
def test_parse_amount_rejects(bad):
    with pytest.raises(ValueError):
        parse_amount(bad)

Step 5 — Mock external calls at the boundary

Mock the outermost I/O surface (HTTP, DB, filesystem); let your code run for real. patch("module.symbol", ...) replaces the name where it is looked up — patch myapp.client.requests.get, not requests.get. MagicMock for sync, AsyncMock for awaitable. Don't mock what you own.

from unittest.mock import patch, MagicMock, AsyncMock

def test_fetch_user_parses_json():
    response = MagicMock(status_code=200)
    response.json.return_value = {"id": 1, "name": "Alice"}
    with patch("myapp.client.requests.get", return_value=response) as m:
        assert fetch_user(1)["name"] == "Alice"
    m.assert_called_once_with("https://api.example.com/users/1", timeout=5)

async def test_fetch_user_async_retries_on_503():
    ok  = MagicMock(status_code=200, json=MagicMock(return_value={"id": 1}))
    bad = MagicMock(status_code=503)
    with patch("myapp.client.httpx.AsyncClient.get",
               new_callable=AsyncMock, side_effect=[bad, ok]):
        assert (await fetch_user_async(1))["id"] == 1

Step 6 — Add property-based tests with hypothesis for invariants

State a law the code obeys; hypothesis searches the input space and shrinks failures to a minimal counter-example. Useful properties: round-trip (decode(encode(x)) == x), idempotence (f(f(x)) == f(x)), monotonicity, never-raises on valid input. @settings(max_examples=...) bounds runtime.

from hypothesis import given, strategies as st, settings

@given(st.decimals(min_value=0, max_value=10**9, allow_nan=False,
                   allow_infinity=False, places=2))
@settings(max_examples=200, deadline=None)
def test_format_then_parse_roundtrips(d):
    assert parse_amount(format_amount(d)) == d

@given(st.text())
def test_parse_amount_never_crashes_unhandled(s):
    try: parse_amount(s)
    except (ValueError, TypeError): pass

Step 7 — Mark integration tests and skip them by default

Tag tests that touch a real DB / broker / subprocess; register the marker in pytest.ini; gate them behind an env var. Default run: pytest -m "not integration". Full: RUN_INTEGRATION=1 pytest.

# pytest.ini
[pytest]
markers = integration: external services; opt-in via RUN_INTEGRATION=1
addopts = -ra

@pytest.mark.integration
@pytest.mark.skipif(os.getenv("RUN_INTEGRATION") != "1", reason="opt-in")
def test_pipeline_end_to_end(tmp_path):
    out = run(input_dir=tmp_path / "in", output_dir=tmp_path / "out")
    assert (tmp_path / "out" / "report.json").exists()

Step 8 — Measure coverage and fail under a threshold

coverage run -m pytest records execution; --fail-under=N fails CI when it drops. branch = true counts taken/not-taken branches. 100% lines with weak asserts proves nothing — read the HTML.

coverage run --branch --source=myapp -m pytest
coverage report -m --fail-under=80
coverage html

# pyproject.toml
[tool.coverage.run]
branch = true
source = ["myapp"]
[tool.coverage.report]
exclude_lines = ["pragma: no cover", "if __name__ == .__main__.:"]
fail_under = 80

Step 9 — Use unittest.TestCase only for legacy suites

Pytest runs TestCase natively, but inside one you must use self.assertEqual / self.assertRaisesRegex and setUp/tearDown instead of fixtures. Prefer pytest for new code.

import unittest
class ParseAmountTests(unittest.TestCase):
    def test_valid_inputs(self):
        for raw, expected in [("1.50", Decimal("1.50")), ("$3", Decimal("3"))]:
            with self.subTest(raw=raw):
                self.assertEqual(parse_amount(raw), expected)
    def test_invalid_raises(self):
        with self.assertRaisesRegex(ValueError, "empty"):
            parse_amount("")

Step 10 — Run the suite the way the grader will

pytest -x                       # stop on first failure
pytest -k "amount and not slow" # filter by name
pytest -q --tb=short            # compact output
pytest --lf                     # rerun only last-failed
pytest tests/test_amount.py::test_parse_amount_rejects_empty_string

Self-check

• Each test name identifies the bug from the failure line alone.
• Tests are order-independent; external I/O mocked or marked integration.
• Parametrize rows share assertion logic; differing logic = separate tests.
• pytest.raises uses match= to pin the message.
• Coverage threshold passes; missed lines in HTML are intentional.

Common Failures

• Patching where a symbol is defined instead of where imported — mock never fires.
• scope="session" fixture that mutates state: later tests see leftovers.
• Hypothesis strategy effectively constant (st.just(5)) — no search.
• tmp_path (pathlib.Path) vs legacy tmpdir (py.path.local) mixed.
• High line coverage with assert result is not None — the number lies.