prepare-providers-documentation

<!-- SPDX-License-Identifier: Apache-2.0 https://www.apache.org/licenses/LICENSE-2.0 --> # Prepare Providers Documentation (AI-driven) This skill replaces the manual commit-by-commit classification step that the release manager normally performs when running `breeze release-management prepare-provider-documentation`. Instead of asking the release manager to type `d`/`b`/`f`/`x`/`m`/`s`/`v` for each commit, the skill drives the classification itself — inspecting every PR (with extra care for potentially breaking changes), scoping multi-provider PRs to the slice that touched the current provider, and asking the release manager only when genuinely uncertain. The skill keeps the existing breeze tooling as the source of truth for template generation. Claude only owns the classification + version bump + changelog entries; everything else (`__init__.py`, `README.rst`, `pyproject.toml`, `conf.py`, `get_provider_info.py`, `index.rst`) is still regenerated by `breeze release-management prepare-provider-documentation --reapply-templates-only`. > [!IMPORTANT] > This is a release-manager workflow. It mutates `provider.yaml` and > `changelog.rst` for many providers in one pass. Always run on a clean > working tree (or in a dedicated branch) and let the release manager review > the diff before committing. --- ## When to Use This Skill Use during the regular provider release cycle, in place of either of: ```shell breeze release-management prepare-provider-documentation breeze release-management prepare-provider-documentation --incremental-update ``` …when the release manager wants Claude to classify the changes instead of doing it by hand. The skill covers the same scope: classifying changes, bumping versions, generating changelog sections, reapplying templates, and folding new commits into an already-prepared release PR (incremental update). Two entry points: - **Initial run** — classify everything from scratch for a new release. Follow Phases 1–5 below. - **Incremental update** — extend an existing release PR with commits that landed on `main` since the changelog was first generated (typical when rebasing a release PR before merging). Skip ahead to the **Incremental Update** section after Phase 5. Either entry point runs in one of two **release shapes**. Establish which one applies before starting — the incremental flow behaves differently in each: - **Wave** (the default) — the regular cycle that releases *every* provider with pending changes. The provider list is an **output** of the run, so it may legitimately grow between the initial cut and the merge. - **Ad-hoc** — a release deliberately scoped to a fixed provider list (the release manager named a subset, or set `DISTRIBUTIONS_LIST`). The provider list is an **input** and must not grow on its own. Ask the release manager if it is not obvious, and record the answer — Incremental Phases 2 and 3.6 branch on it. Do **not** use this skill for: - `--only-min-version-update` runs (these don't need classification — just run breeze directly). - Releasing from a non-`main` base branch unless you also pass the right `--base-branch` to the breeze invocations described below. - Removing providers (state changes belong in a separate PR). --- ## Inputs You Need Before Starting Ask the release manager (and confirm by reading the answers back) for: 1. **`RELEASE_DATE`** in `YYYY-MM-DD` (or `YYYY-MM-DD_NN`) format, e.g. `2026-04-26`. This is what breeze stamps into `providers/.last_release_date.txt`. 2. **Base branch** — defaults to `main`. Only override when releasing from a provider-specific branch (e.g. `provider-cncf-kubernetes/v4-4`). 3. **Subset of providers**, if any. By default, classify every provider that has pending changes since its last release tag. If the release manager wants a subset (or has set `DISTRIBUTIONS_LIST`), use that list. 4. **Include flags**: whether to include `--include-not-ready-providers` and/or `--include-removed-providers`. Set the environment for the session: ```bash export RELEASE_DATE=<date> # Optional, scopes everything to a subset export DISTRIBUTIONS_LIST="<provider1> <provider2> ..." ``` Make sure the `apache-https-for-providers` git remote exists and is up to date — running breeze the first time below will recreate and fetch it. --- ## Workflow The skill runs in five phases. Mark tasks with `TaskCreate` for each phase and tick them off as you go — the release manager wants to see progress. ### Phase 1 — Discover and pre-classify pending changes (deterministic) The source of truth for "what changed since last release" is the same git query breeze uses internally: commits between the latest release tag for that provider (`providers-<id>/<version>`) and `apache-https-for-providers/<base-branch>`, restricted to the provider's own folders. Run the **deterministic classifier** — it discovers every provider with pending changes **and** pre-classifies each commit with hard-coded, high-confidence rules, flagging only the genuinely ambiguous ones as `needs_llm`. No random answers, nothing to discard: ```bash breeze release-management classify-provider-changes \ --base-branch main \ --output-file /tmp/provider-changes.json # scope to a subset by appending provider ids, e.g. ... amazon cncf.kubernetes ``` The JSON it writes: ```json { "base_branch": "main", "providers": { "amazon": { "current_version": "9.29.0", "commits": [ {"hash": "c2dbd7a75a", "pr": "67987", "subject": "Fix IDC domain S3 path resolution", "classification": "needs_llm", "reason": "no high-confidence deterministic rule matched"}, {"hash": "abc123", "pr": "68087", "subject": "Bump the edge-ui-package-updates group ...", "classification": "misc", "reason": "dependency bump (subject starts with 'Bump')"} ] } } } ``` How to read it: - Providers under `providers` have pending changes (these need attention). - `classification ∈ {documentation, skip, misc}` are **decided by rules — take them as-is**, no sub-agent needed (doc-only → `documentation`, test/example only → `skip`, `Bump …` dependency bump → `misc`). - `classification == needs_llm` → **Phase 3 decides** with a sub-agent. These are the only commits that need LLM analysis. - A provider with a `note`/`error` (e.g. a brand-new provider with no prior release tag) → treat as an **initial release** and classify by hand. > [!NOTE] > The classifier is deliberately conservative: `Fix …`/`Add …` subjects are > **not** auto-classified (an "Add …" can be a breaking change), so they come > back as `needs_llm`. The rules live in `classify_change_deterministically` > (`dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py`). The range the classifier reports is not fixed for the whole run — it widens once a provider actually gets a version bump: > [!WARNING] > **A version bump supersedes the provider's doc-only marker — re-run discovery > after Phase 4a.** When `docs/.latest-doc-only-change.txt` is present the > classifier starts the range at that commit, so anything older (earlier doc-only > changes, tooling churn) is hidden for as long as the provider has no pending > release. That is correct while nothing ships — but the moment the provider gets > a real version bump the marker no longer applies, and those commits are part of > what the release publishes. > > Re-run `classify-provider-changes` **after** bumping versions and fold in every > commit that newly appears; otherwise the release silently drops doc-only > entries that were only ever suppressed by the marker. The tell is a provider > whose commit count grows between two runs without any new merges. Observed in > the 2026-08-01 wave: `cohere` gained #69649, `dbt.cloud`/`presto`/`trino` gained > #69478, and `papermill` gained #68322 — all invisible in the first pass. Then regenerate the auto-generated build files (this does **no** classification, so nothing random is produced): ```bash breeze release-management prepare-provider-documentation \ --reapply-templates-only --release-date "$RELEASE_DATE" git checkout -- $(git diff --name-only -- '**/provider.yaml' '**/changelog.rst') ``` This leaves the regenerated build files (`__init__.py`, `README.rst`, `pyproject.toml`, `conf.py`, `get_provider_info.py`, `index.rst`) in place and discards only the changelog/version files Claude is about to rewrite itself. ### Phase 2 — Per-provider commit list For each provider in **Success** from Phase 1, get the same commit list that breeze would have shown. From the repo root: ```bash PROVIDER_ID=<dotted.id> # e.g. amazon, cncf.kubernetes PROVIDER_PATH=$(echo "$PROVIDER_ID" | tr '.' '/') # folder path: cncf/kubernetes PROVIDER_TAG=$(echo "$PROVIDER_ID" | tr '.' '-') # tag segment: cncf-kubernetes # Pick the latest *final* release tag. Two gotchas the tag pattern must handle: # * dotted provider ids use HYPHENS in tag names (providers-cncf-kubernetes/<ver>), # even though the source folder uses slashes — build the tag prefix from # PROVIDER_TAG, not PROVIDER_PATH; # * skip the sentinel upper-bound tags (providers-<id>/99.98.0, /99.99.0) and rc # tags — git's default version sort orders "1.2.0rc1" AFTER "1.2.0", so a bare # `head -n1` would otherwise select a sentinel or a release candidate. LAST_TAG=$(git tag --list "providers-${PROVIDER_TAG}/*" --sort=-v:refname \ | grep -vE '/99\.9[0-9]\.' | grep -vE 'rc[0-9]+$' | head -n1) git log --pretty=format:'%H %h %cd %s' --date=short \ "${LAST_TAG}..apache-https-for-providers/main" \ -- "providers/${PROVIDER_PATH}/" ``` > [!WARNING] > This git query is a convenience for building the per-provider commit list, but > the **authoritative set is what breeze prints in the Phase 1 "Commit" tables** > for each provider. The tag-based range can still diverge from breeze when a > provider's most recent *final* tag is not the last actually-published release > (for example, a wave commit bumped the version on `main` but the published > baseline is older), which makes breeze include repo-wide commits this query > misses. When the two disagree, trust breeze's list and reconcile against it > before classifying. Capture the full hash, short hash, date, subject, and `#NNNN` PR number for each commit. Note that some old providers also have legacy paths under `airflow/providers/<id>/` — include those when present (consult `provider_details.possible_old_provider_paths` semantics by checking the provider's `provider.yaml` history if needed). ### Phase 3 — Classify the PRs (inline, or batched per-provider sub-agents) For each commit, classify it into one of: | Code | Meaning | Version bump | | ---- | ---------------------------------------------- | -------------- | | `d` | Documentation-only | none (patch if combined) | | `b` | Bug fix | patch | | `f` | Feature | minor | | `x` | Breaking change | major | | `m` | Misc (deps, refactors, internal only) | patch | | `s` | Skip (test/CI/example only — no user impact) | none | | `v` | Min Airflow version bump | minor (treated as misc + bump) | #### Take the deterministic classifications from Phase 1 `classify-provider-changes` (Phase 1) already classified every commit it could with hard-coded rules. Read `/tmp/provider-changes.json` and: - Use any commit whose `classification` is `documentation`, `skip`, or `misc` **as-is** — these map to `d`, `s`, `m` respectively; no sub-agent needed. - Only commits with `classification: needs_llm` go to a sub-agent (below). The deterministic rules (doc-only → `d`, test/example-only → `s`, `Bump …` dependency bump → `m`) are exactly the cheap cases — now computed once by breeze (`classify_change_deterministically`) instead of re-derived here. If you ever need the min-Airflow-bump case (`v`), that one is still a `needs_llm` judgement: a sub-agent should flag it when a PR bumps the provider's minimum Airflow version. > [!NOTE] > A `needs_llm` commit can still be classified as `skip` by the LLM. The > deterministic classifier only catches test/example-only changes by looking > at which files were modified in the commit. PRs that change build tooling, > packaging infrastructure, or breeze templates (files under `dev/breeze/`, > `scripts/`, etc.) and only regenerate template-generated files in the > provider slice (e.g. `README.rst`, `index.rst`, `pyproject.toml`) are NOT > caught by the deterministic rules but should still be classified as `skip` > — they have zero user-facing impact on the provider package itself. That rule cuts one way only. Read the next one before applying it. > [!WARNING] > The converse trap: a PR is **not** `skip` merely because its subject > describes tooling. `provider.yaml` is a **source** file, never a regenerated > one — anything it declares (`connection-types`, `conn-fields`, > `ui-field-behaviour`, `extra-links`, `hook-class-names`, dependencies) ships > to users through `get_provider_info.py`, so a change to it is `misc` at > minimum. `get_provider_info.py` moving *because `provider.yaml` moved* is > therefore not "regenerated metadata" in the sense above. Worked example: > `Fix conn-fields check crash for nested provider packages (#70224)` fixes a > prek check under `scripts/ci/prek/`, and in the same commit declares a new > `Verify SSL` connection-form field for `atlassian.jira` — `misc`, not `skip`. > Classify from the provider-scoped diff, never from the subject. #### Classify the `needs_llm` commits — batched per provider, not one agent per PR Only the commits the classifier returned as `needs_llm` still need a sub-agent. Classification is the token-heavy part of this skill, so spend sub-agents sparingly. Do **not** spawn one sub-agent per PR — that is one agent per commit and balloons to hundreds of agents on a normal release wave. Pick the smallest fan-out that fits the volume: - **Few `needs_llm` commits remain (≲ 15 across all providers) → classify inline.** Read each PR and its provider-scoped diff yourself, in this context. Spawn no sub-agents at all. - **More than that → one sub-agent per provider.** Each agent classifies that provider's *entire* remaining `needs_llm` list in a single pass. This is the natural unit: multi-provider PRs are classified independently per provider anyway (see Cross-Cutting Rules), and one provider-scoped agent amortizes the breaking-change checklist across all of that provider's commits instead of paying a fresh agent spin-up per commit. Only split a provider across more than one agent when its remaining list is large (> ~25 commits) — chunk it then. This keeps the sub-agent count at roughly the number of providers with pending changes, not the number of commits. Use the `Explore` agent type — they need read-only access. Brief each sub-agent with its provider and the whole batch of commits it owns: ``` Classify a batch of Apache Airflow provider PRs for ONE provider. Provider: <provider-id> (path: providers/<provider-path>/) Commits to classify (<N>) — one row per PR: #<NNNN> <full-hash> <subject> #<MMMM> <full-hash> <subject> …(this provider's full remaining list) For EACH commit above: 1. Read the PR's title, body, and labels: `gh pr view <NNNN> --json title,body,labels,files` 2. Read the diff for the slice of the PR that touched providers/<provider-path>/ only: