python-pypi-package-builder

# Python PyPI Package Builder Skill A complete, battle-tested guide for building, testing, linting, versioning, typing, and publishing a production-grade Python library to PyPI — from first commit to community-ready release. > **AI Agent Instruction:** Read this entire file before writing a single line of code or > creating any file. Every decision — layout, backend, versioning strategy, patterns, CI — > has a decision rule here. Follow the decision trees in order. This skill applies to any > Python package type (utility, SDK, CLI, plugin, data library). Do not skip sections. --- ## Quick Navigation | Section in this file | What it covers | |---|---| | [1. Skill Trigger](#1-skill-trigger) | When to load this skill | | [2. Package Type Decision](#2-package-type-decision) | Identify what you are building | | [3. Folder Structure Decision](#3-folder-structure-decision) | src/ vs flat vs monorepo | | [4. Build Backend Decision](#4-build-backend-decision) | setuptools / hatchling / flit / poetry | | [5. PyPA Packaging Flow](#5-pypa-packaging-flow) | The canonical publish pipeline | | [6. Project Structure Templates](#6-project-structure-templates) | Full layouts for every option | | [7. Versioning Strategy](#7-versioning-strategy) | PEP 440, semver, dynamic vs static | | Reference file | What it covers | |---|---| | `references/pyproject-toml.md` | All four backend templates, `setuptools_scm`, `py.typed`, tool configs | | `references/library-patterns.md` | OOP/SOLID, type hints, core class design, factory, protocols, CLI | | `references/testing-quality.md` | `conftest.py`, unit/backend/async tests, ruff/mypy/pre-commit | | `references/ci-publishing.md` | `ci.yml`, `publish.yml`, Trusted Publishing, TestPyPI, CHANGELOG, release checklist | | `references/community-docs.md` | README, docstrings, CONTRIBUTING, SECURITY, anti-patterns, master checklist | | `references/architecture-patterns.md` | Backend system (plugin/strategy), config layer, transport layer, CLI, backend injection | | `references/versioning-strategy.md` | PEP 440, SemVer, pre-release, setuptools_scm deep-dive, flit static, decision engine | | `references/release-governance.md` | Branch strategy, branch protection, OIDC, tag author validation, prevent invalid tags | | `references/tooling-ruff.md` | Ruff-only setup (replaces black/isort), mypy config, pre-commit, asyncio_mode=auto | **Scaffold script:** run `python skills/python-pypi-package-builder/scripts/scaffold.py --name your-package-name` to generate the entire directory layout, stub files, and `pyproject.toml` in one command. --- ## 1. Skill Trigger Load this skill whenever the user wants to: - Create, scaffold, or publish a Python package or library to PyPI - Build a pip-installable SDK, utility, CLI tool, or framework extension - Set up `pyproject.toml`, linting, mypy, pre-commit, or GitHub Actions for a Python project - Understand versioning (`setuptools_scm`, PEP 440, semver, static versioning) - Understand PyPA specs: `py.typed`, `MANIFEST.in`, `RECORD`, classifiers - Publish to PyPI using Trusted Publishing (OIDC) or API tokens - Refactor an existing package to follow modern Python packaging standards - Add type hints, protocols, ABCs, or dataclasses to a Python library - Apply OOP/SOLID design patterns to a Python package - Choose between build backends (setuptools, hatchling, flit, poetry) **Also trigger for phrases like:** "build a Python SDK", "publish my library", "set up PyPI CI", "create a pip package", "how do I publish to PyPI", "pyproject.toml help", "PEP 561 typed", "setuptools_scm version", "semver Python", "PEP 440", "git tag release", "Trusted Publishing". --- ## 2. Package Type Decision Identify what the user is building **before** writing any code. Each type has distinct patterns. ### Decision Table | Type | Core Pattern | Entry Point | Key Deps | Example Packages | |---|---|---|---|---| | **Utility library** | Module of pure functions + helpers | Import API only | Minimal | `arrow`, `humanize`, `boltons`, `more-itertools` | | **API client / SDK** | Class with methods, auth, retry logic | Import API only | `httpx` or `requests` | `boto3`, `stripe-python`, `openai` | | **CLI tool** | Command functions + argument parser | `[project.scripts]` or `[project.entry-points]` | `click` or `typer` | `black`, `ruff`, `httpie`, `rich` | | **Framework plugin** | Plugin class, hook registration | `[project.entry-points."framework.plugin"]` | Framework dep | `pytest-*`, `django-*`, `flask-*` | | **Data processing library** | Classes + functional pipeline | Import API only | Optional: `numpy`, `pandas` | `pydantic`, `marshmallow`, `cerberus` | | **Mixed / generic** | Combination of above | Varies | Varies | Many real-world packages | **Decision Rule:** Ask the user if unclear. A package can combine types (e.g., SDK with a CLI entry point) — use the primary type for structural decisions and add secondary type patterns on top. For implementation patterns of each type, see `references/library-patterns.md`. ### Package Naming Rules - PyPI name: all lowercase, hyphens — `my-python-library` - Python import name: underscores — `my_python_library` - Check availability: https://pypi.org/search/ before starting - Avoid shadowing popular packages (verify `pip install <name>` fails first) --- ## 3. Folder Structure Decision ### Decision Tree ``` Does the package have 5+ internal modules OR multiple contributors OR complex sub-packages? ├── YES → Use src/ layout │ Reason: prevents accidental import of uninstalled code during development; │ separates source from project root files; PyPA-recommended for large projects. │ ├── NO → Is it a single-module, focused package (e.g., one file + helpers)? │ ├── YES → Use flat layout │ └── NO (medium complexity) → Use flat layout, migrate to src/ if it grows │ └── Is it multiple related packages under one namespace (e.g., myorg.http, myorg.db)? └── YES → Use namespace/monorepo layout ``` ### Quick Rule Summary | Situation | Use | |---|---| | New project, unknown future size | `src/` layout (safest default) | | Single-purpose, 1–4 modules | Flat layout | | Large library, many contributors | `src/` layout | | Multiple packages in one repo | Namespace / monorepo | | Migrating old flat project | Keep flat; migrate to `src/` at next major version | --- ## 4. Build Backend Decision ### Decision Tree ``` Does the user need version derived automatically from git tags? ├── YES → Use setuptools + setuptools_scm │ (git tag v1.0.0 → that IS your release workflow) │ └── NO → Does the user want an all-in-one tool (deps + build + publish)? ├── YES → Use poetry (v2+ supports standard [project] table) │ └── NO → Is the package pure Python with no C extensions? ├── YES, minimal config preferred → Use flit │ (zero config, auto-discovers version from __version__) │ └── YES, modern & fast preferred → Use hatchling (zero-config, plugin system, no setup.py needed) Does the package have C/Cython/Fortran extensions? └── YES → MUST use setuptools (only backend with full native extension support) ``` ### Backend Comparison | Backend | Version source | Config | C extensions | Best for | |---|---|---|---|---| | `setuptools` + `setuptools_scm` | git tags (automatic) | `pyproject.toml` + optional `setup.py` shim | Yes | Projects with git-tag releases; any complexity | | `hatchling` | manual or plugin | `pyproject.toml` only | No | New pure-Python projects; fast, modern | | `flit` | `__version__` in `__init__.py` | `pyproject.toml` only | No | Very simple, single-module packages | | `poetry` | `pyproject.toml` field | `pyproject.toml` only | No | Teams wanting integrated dep management | For all four complete `pyproject.toml` templates, see `references/pyproject-toml.md`. --- ## 5. PyPA Packaging Flow This is the canonical end-to-end flow from source code to user install. **Every step must be understood before publishing.** ``` 1. SOURCE TREE Your code in version control (git) └── pyproject.toml describes metadata + build system 2. BUILD python -m build └── Produces two artifacts in dist/: ├── *.tar.gz → source distribution (sdist) └── *.whl → built distribution (wheel) — preferred by pip 3. VALIDATE twine check dist/* └── Checks metadata, README rendering, and PyPI compatibility 4. TEST PUBLISH (first release only) twine upload --repository testpypi dist/* └── Verify: pip install --index-url https://test.pypi.org/simple/ your-package 5. PUBLISH twine upload dist/* ← manual fallback OR GitHub Actions publish.yml ← recommended (Trusted Publishing / OIDC) 6. USER INSTALL pip install your-package pip install "your-package[extra]" ``` ### Key PyPA Concepts | Concept | What it means | |---|---| | **sdist** | Source distribution — your source + metadata; used when no wheel is available | | **wheel (.whl)** | Pre-built binary — pip extracts directly into site-packages; no build step | | **PEP 517/518** | Standard build system interface via `pyproject.toml [build-system]` table | | **PEP 621** | Standard `[project]` table in `pyproject.toml`; all modern backends support it | | **PEP 639** | `license` key as SPDX string (e.g., `"MIT"`, `"Apache-2.0"`) — not `{text = "MIT"}` | | **PEP 561** | `py.typed` empty marker file — tells mypy/IDEs this package ships type information | For complete CI workflow and publishing setup, see `references/ci-publishing.md`. --- ## 6. Project Structure Templates ### A. src/ Layout (Recommended default for new projects) ``` your-package/ ├── src/ │ └── your_package/ │ ├── __init__.py # Public API: __all__, __version__ │ ├── py.typed # PEP 561 marker — EMPTY FILE │ ├── core.py # Primary implementation │ ├── client.py # (API client type) or remove │ ├── cli.py # (CLI type) click/typer commands, or remove │ ├── config.py # Settings / configuration dataclass │ ├── exceptions.py # Custom exception hierarchy │ ├── models.py # Data classes, Pydantic models, TypedDicts │ ├── utils.py # Internal helpers (prefix _utils if private) │ ├── types.py # Shared type aliases and TypeVars │ └── backends/ # (Plugin pattern) — remove if not needed │ ├── __init__.py # Protocol / ABC interface definition │ ├── memory.py # Default zero-dep implementation │ └── redis.py # Optional heavy implementation ├── tests/ │ ├── __init__.py │ ├── conftest.py # Shared fixtures │ ├── unit/ │ │ ├── __init__.py │ │ ├── test_core.py │ │ ├── test_config.py │ │ └── test_models.py │ ├── integration/ │ │ ├── __init__.py │ │ └── test_backends.py │ └── e2e/ # Optional: end-to-end tests │ └── __init__.py ├── docs/ # Optional: mkdocs or sphinx ├── scripts/ │ └── scaffold.py ├── .github/ │ ├── workflows/ │ │ ├── ci.yml │ │ └── publish.yml │ └── ISSUE_TEMPLATE/ │ ├── bug_report.md │ └── feature_request.md ├── .pre-commit-config.yaml ├── pyproject.toml ├── CHANGELOG.md ├── CONTRIBUTING.md ├── SECURITY.md ├── LICENSE ├── README.md └── .gitignore ``` ### B. Flat Layout (Small / focused packages) ``` your-package/ ├── your_package/ # ← at root, not inside src/ │ ├── __init__.py │ ├── py.typed │ └── ... (same internal structure) ├── tests/ └── ... (same top-level files) ``` ### C. Namespace / Monorepo Layout (Multiple related packages) ``` your-org/ ├── packages/ │ ├── your-org-core/ │ │ ├── src/your_org/core/ │ │ └── pyproject.toml │ ├── your-org-http/ │ │ ├── src/your_org/http/ │ │ └── pyproject.toml │ └── your-org-cli/ │ ├── src/your_org/cli/ │ └── pyproject.toml ├── .github/workflows/ └── README.md ``` Each sub-package has its own `pyproject.toml`. They share the `your_org` namespace via PEP 420 implicit namespace packages (no `__init__.py` in the namespace root). ### Internal Module Guidelines | File | Purpose | When to include | |---|---|---| | `__init__.py` | Public API surface; re-exports; `__version__` | Always | | `py.typed` | PEP 561 typed-package marker (empty) | Always | | `core.py` | Primary class / main logic | Always | | `config.py` | Settings dataclass or Pydantic model | When configurable | | `exceptions.py` | Exception hierarchy (`YourBaseError` → specifics) | Always | | `models.py` | Data models / DTOs / TypedDicts | When data-heavy | | `utils.py` | Internal helpers (not part of public API) | As needed | | `types.py` | Shared `TypeVar`, `TypeAlias`, `Protocol` definitions | When complex typing | | `cli.py` | CLI entry points (click/typer) | CLI type only | | `backends/` | Plugin/strategy pattern | When swappable implementations | | `_compat.py` | Python version compatibility shims | When 3.9–3.13 compat needed | --- ## 7. Versioning Strategy ### PEP 440 — The Standard ``` Canonical form: N[.N]+[{a|b|rc}N][.postN][.devN] Examples: 1.0.0 Stable release 1.0.0a1 Alpha (pre-release) 1.0.0b2 Beta 1.0.0rc1 Release candidate 1.0.0.post1 Post-release (e.g., packaging fix only) 1.0.0.dev1 Development snapshot (not for PyPI) ``` ### Semantic Versioning (recommended) ``` MAJOR.MINOR.PATCH MAJOR: Breaking API change (remove/rename public function/class/arg) MINOR: New feature, fully backward-compatible PATCH: Bug fix, no API change ``` ### Dynamic versioning with setuptools_scm (recommended for git-tag workflows) ```bash # How it works: git tag v1.0.0 → installed version = 1.0.0 git tag v1.1.0 → installed version = 1.1.0 (commits after tag) → version = 1.1.0.post1 (suffix stripped for PyPI) # In code — NEVER hardcode when using setuptools_scm: from importlib.metadata import version, PackageNotFoundError try: __version__ = version("your-package")