tenacious-python-code

# Tenacious Python Code ## Overview Write Python that is easy to trust a month later: typed, small in scope, deterministic where it matters, and boring in the good way. Favor standard-library solutions first. Add a dependency only when it removes real complexity or matches a pattern already established in the repo. Read `references/python-patterns.md` for implementation conventions and `references/repo-shape.md` for how those conventions map to this benchmark project. ## Workflow ### 1. Start from the artifact, not the abstraction Identify what the file needs to do in this repo: - score benchmark tasks - generate or transform dataset records - validate partitions and contamination checks - prepare training examples - summarize metrics or ablation outputs Keep the design shaped around that job. Avoid building framework-like layers for one-off scripts. ### 2. Choose the smallest solid structure Use: - a few pure functions for transformations - a small dataclass or `TypedDict` for record structure - a `main()` entrypoint for scripts - one level of orchestration, not three Reach for classes only when there is durable state or multiple related operations sharing invariants. ### 3. Type the boundaries Add type hints to: - public functions - dataclass fields - return values from parsing and scoring helpers - dictionary-like records that cross file boundaries Prefer: - `Path` over raw path strings - `dataclass` for lightweight structured data - `TypedDict` when interacting with JSON-like task payloads - `Iterable` or `Sequence` when the exact container type does not matter Avoid type noise that makes simple code harder to read. ### 4. Make scripts reproducible For benchmark, training, and scoring utilities: - accept explicit inputs and output locations - avoid hidden global state - pin random seeds when randomness exists - sort outputs when order should be stable - write JSON with consistent formatting - log enough context to reproduce results Do not bake credentials, model names, or local absolute paths into code. ### 5. Separate IO from logic Keep parsing, filesystem access, and network calls thin. Put the actual scoring, filtering, normalization, and comparison logic into testable functions. Good shape: - `load_*()` functions read files - `build_*()` or `score_*()` functions perform logic - `write_*()` functions emit results - `main()` wires them together ### 6. Handle errors like a teammate Raise clear exceptions for programmer errors. For user-facing scripts, catch them near `main()` and print actionable messages. Include the file path, task ID, or field name in error messages when possible. Avoid broad `except Exception:` unless re-raising with useful context. ### 7. Keep data formats explicit When working with JSON or JSONL benchmark data: - document required fields near the parsing code - validate important keys early - tolerate optional metadata only when the downstream logic truly can - preserve unknown fields unless the script is intentionally normalizing schema If a schema is evolving, prefer additive changes over silent shape changes. ### 8. Write code that is easy to test Prefer deterministic inputs and pure helpers so tests can cover: - rubric scoring - record normalization - partition counting - overlap or contamination checks - training-data formatting For small scripts, a few focused tests or a smoke-test command is enough. Broaden coverage when the code touches shared scoring behavior or publication artifacts. ### 9. Modern style defaults Prefer: - Python 3.11+ features already supported by the project - f-strings - `pathlib.Path` - `collections.Counter`, `defaultdict`, and standard-library parsing tools - comprehensions when they stay readable - `argparse` for CLIs Avoid: - giant functions - hidden mutation across helpers - manual stringly-typed path handling - ad hoc shelling out when Python can do the job directly - clever one-liners that obscure benchmark logic ## Review Checklist Before finishing Python changes, check: 1. Are names specific to benchmark or training intent? 2. Are inputs, outputs, and side effects obvious? 3. Is the core logic testable without touching the filesystem? 4. Are types present where another engineer would need them? 5. Will rerunning this script produce stable output? 6. Would a new teammate understand failures from the error messages? ## Practical Defaults For This Repo - Keep benchmark scripts as plain modules or CLIs, not services. - Use JSON or JSONL for task artifacts unless an existing format already wins. - Keep scoring rules explicit and inspectable. - Prefer small helper modules over one massive `utils.py`. - Add comments only where benchmark methodology or scoring logic is non-obvious.