run-integration-tests

name

run-integration-tests

description

Use this skill WHENEVER you are about to run the integration tests of this project — the whole suite, a single module, or one test. This includes any time you would otherwise type `pytest -m integration`, `uv run pytest -m integration`, run anything under `tests/integration/`, or want to check live Blockscout/Chainscout/BENS connectivity. Integration tests make real network calls with no hard HTTP timeout, so one unresponsive endpoint can hang a plain pytest run for an unbounded time and block you. This skill runs each test in an isolated subprocess with a per-test timeout, kills any hung test, and always returns a bounded report flagging which tests timed out or are slow. Prefer it over invoking pytest directly for integration tests.

Run Integration Tests (timeout-protected)

Why use this instead of plain pytest

Integration tests hit live external APIs and the HTTP client has no hard request timeout. When an endpoint is slow or unresponsive, a single test can hang indefinitely — a plain pytest -m integration run then blocks for an unbounded time and so do you. (This is exactly how this project's suite started taking 20+ minutes: a handful of tests hung for ~3 minutes each before being killed manually.)

The bundled runner solves this by executing each test in its own subprocess with a per-test wall-clock timeout. A hung test is killed and reported as TIMEOUT; the run continues. You always get a complete, bounded report instead of stalling.

Usage

The runner lives at scripts/run_integration_tests.py (relative to the project root, alongside the project's other tooling). Invoke it from the project root with that path, or use its absolute path. The script finds the project root on its own by walking up to the nearest tests/integration directory, so it can be launched from anywhere.

Run it with the project venv (the script auto-detects whether to use pytest directly inside the devcontainer or uv run pytest on the host):

uv run python scripts/run_integration_tests.py [TARGET ...] [--timeout N] [--slow-threshold S]

Inside the devcontainer (/.dockerenv exists) you may drop uv run and call python ... directly — but uv run python ... works in both environments, so prefer it when unsure.

The three granularities

TARGET is passed straight to pytest's collector, so the same command covers every scope:

1. Whole suite (no target → defaults to tests/integration):

uv run python scripts/run_integration_tests.py

2. A single module (a file, or a whole subdirectory):

# one file
uv run python scripts/run_integration_tests.py tests/integration/block/test_get_block_info_real.py
# a category directory
uv run python scripts/run_integration_tests.py tests/integration/transaction

3. A single test (a file::test node id):

uv run python scripts/run_integration_tests.py "tests/integration/block/test_get_block_info_real.py::test_get_block_info_integration"

You can also pass several targets at once.

Options

• --timeout N — per-test wall-clock limit in seconds (default 120). Lower it (e.g. --timeout 30) when you just want to fail fast on hangs; raise it only if a test legitimately needs longer.
• --slow-threshold S — flag completed tests slower than S seconds in the summary (default 10).
• --marker EXPR — pytest marker to select (default integration).
• --list — only list the tests that would run, then exit. Useful to scope a run before launching it.

Reading the output

Each test prints a live line as it finishes:

[  7/83] PASS         2.5s  tests/integration/...::test_x
[  8/83] TIMEOUT    120.0s  tests/integration/...::test_y

The summary at the end reports counts and three actionable lists:

• TIMED OUT — tests killed at the limit. These block the suite and are the first thing to investigate (usually a flaky/slow endpoint or a missing HTTP timeout in the tool code, not a bug in the test).
• FAILED — real assertion/error failures.
• SKIPPED (reason) — tests that skipped, each with the reason pytest reported (network unavailable, missing API key, external service down, …). A skip is an environment signal, not a pass, so read these before treating a run as clean.
• SLOW — tests that completed but exceeded --slow-threshold; candidates for optimization.

The script exits non-zero if anything failed or timed out, so it composes with other tooling.

Tips

• For routine development, scope to the module or test you're working on — it's much faster than the full suite and still timeout-protected.
• A TIMEOUT is not automatically a broken test. Live endpoints are sometimes temporarily slow; re-running often shows the same test passing in a couple of seconds. Treat persistent timeouts (across reruns) as the real signal.
• The runner isolates each test in its own pytest process, which adds a small per-test startup cost. That overhead is the deliberate price of guaranteed non-blocking behavior; it is negligible compared to a single hung test.