compare-python-typecheckers

name	compare-python-typecheckers
description	Run a Python file or snippet through the real third-party type checkers (ty, pyrefly, basedpyright, mypy, zuban) via uvx and collate their output into one Markdown report. Use when evaluating how a type should be inferred, debugging PyCharm code-insight / type-engine behaviour, or checking where the external checkers agree or disagree on a snippet.

Compare Python type checkers

When reasoning about how PyCharm should infer a type or whether something ought to be a type error, cross-checking against the real third-party checkers is the fastest way to gain insight — and to see where they disagree. This skill runs them all on a scratch file or snippet and collates the output into a single report.

Run it

${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py invokes each checker through uvx (fetched on demand — no install; the network is used on first run) and prints a Markdown report: a summary table (exit code / verdict / timing per tool) followed by each tool's raw output.

# Inline snippet:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py -c 'def f(x: int) -> int:
    return x

f("a")'

# An existing file:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py path/to/test.py

# A subset, written to a file:
uv run ${CLAUDE_SKILL_DIR}/scripts/compare_typecheckers.py test.py --tools ty,mypy -o /tmp/report.md

Flags: -c/--code (inline snippet), -t/--tools ty,mypy,… (subset), -o/--output (write to file), --timeout (per-checker seconds, default 180). The script exits 0 whenever it produced a report — findings live in the report, not the exit code.

Reading the result

• The summary table shows each tool's exit code (0 = nothing flagged, non-zero = the checker reported an issue or failed to run), a verdict, and time.
• Output formats differ per tool, so read each section, not just the table:
  • ty (Astral) and pyrefly (Meta) each have their own diagnostic style.
  • basedpyright is pyright-based and also emits extra diagnostics such as reportUnusedCallResult.
  • mypy and zuban share the same message format (zuban is mypy-compatible).
• The checkers track their latest releases through uvx, so behaviour can shift over time — record the date when capturing results in a YouTrack issue or test.

Manual one-offs

The exact per-tool invocations, if you want to run just one by hand (note basedpyright takes the file positionally — no check subcommand):

Tool	Command
ty	uvx ty check test.py
pyrefly	uvx pyrefly check test.py
basedpyright	uvx basedpyright test.py
mypy	uvx mypy test.py
zuban	uvx zuban check test.py