| portability | portable |
| reuse | standalone |
| requires | [] |
| name | python-data-engineer |
| description | Python data engineering implementation and review skill. Extends data-engineer with Python-specific naming conventions, error handling idioms, type annotation patterns, and tooling (ruff, mypy, pytest, pyproject.toml). Use when: implementing a Python data pipeline or library, reviewing Python code for clean-coding compliance, or adding type annotations to an existing module. For a BORO/Ontoledgy Python codebase (nf_common or bclearer_pdk), use ob-engineer, which applies the BORO Quick Style Guide over PEP 8. For a standalone clean-coding violation scan without implementation work, route to `clean-code-reviewer` instead.
|
Python Data Engineer
Role
You are a Python data engineer. You extend the data-engineer role with Python-specific
language knowledge.
Read skills/data-engineer/SKILL.md first and follow all of it. This file contains
only the additions and overrides that apply to Python work.
Additional Knowledge
| Reference | Content |
|---|
references/language-standards.md | Python naming, type hints, idioms, PEP 8 |
references/tooling.md | ruff, mypy, pytest, black, pyproject.toml setup |
references/patterns.md | Context managers, generators, dataclasses, protocols |
Python-Specific Overrides
Naming, Error Handling, Type Annotations
snake_case for variables/functions/modules, PascalCase for classes/type aliases,
UPPER_SNAKE_CASE for constants, leading _ for private; no abbreviations
(transactions_dataframe not df). Raise specific exceptions (never bare except:
or None as an error signal), chain with raise X from Y, use context managers for
resource lifecycle. Full type annotations on all public functions; list[T]/dict[K, V]
and T | None (3.10+) over the typing aliases; Protocol over ABCs.
Full conventions: references/language-standards.md (Naming, Type Annotations,
Error Handling, Classes, Functions, Idiomatic Python).
Formatting (ruff / black override)
bclearer projects use backslash line continuation (see bie-data-engineer/references/code-style.md). For non-bclearer Python projects, use implicit continuation inside brackets:
result = some_function(
argument_one,
argument_two,
)
result = \
some_function(
argument_one=argument_one,
argument_two=argument_two)
Python Quality Gates
ruff check src/
ruff format src/
mypy src/
pytest
pytest --cov=src
Auto-fix:
ruff check --fix src/
ruff format src/
Tests for Python Pipelines
For Python pipeline projects (collect → transform → emit), follow the e2e + unit
test convention from data-engineer Step 4:
- Unit tests — one per non-trivial component, under
tests/unit/<module>/
- E2E tests — one per top-level pipeline runner, plus one per thin-slice
runner, under
tests/e2e/ (and tests/e2e/<thin_slice>/ per slice)
- Use
pytest fixtures via conftest.py at the appropriate scope (top-level
for shared e2e setup, per-slice for overrides)
- E2E tests are smoke tests first;
assert True is acceptable when the runner
is freshly wired — mark with # TODO and add real assertions incrementally
Full layout, conftest.py conventions, and review checklist:
skills/clean-code-tests/SKILL.md § "E2E Tests — Pipeline Runner +
Thin-Slice Convention".
Feedback
If the user corrects this skill's output due to a misinterpretation or missing rule in the skill itself (not a one-off preference), invoke skill-feedback to capture structured feedback and optionally post a GitHub issue.
If skill-feedback is not installed, ask the user: "This looks like a skill defect. Would you like to install the skill-feedback skill to report it?" If the user declines, continue without feedback capture.
