authoring-ci-workflows

name

authoring-ci-workflows

description

Use when adding or editing a GitHub Actions workflow, composite action, or reusable workflow under `.github/` — new CI jobs, triggers, matrices, checkout/clone tuning, action pinning, GitHub App token auth, concurrency groups, `timeout-minutes`, `paths` filters, caching, or runner choice. Covers PostHog's workflow-authoring conventions and the reasons behind them: the 500-runs/10s dispatch cap, shallow vs full clone, per-SHA push concurrency, dedicated App-token rate-limit buckets, and fork-safe secrets on a public repo. Points to the linters (`bin/hogli lint:workflows`, actionlint) that enforce the mechanical rules, and to the narrower skills for production deploys, secrets, and Depot runners. Not for debugging red CI (use debugging-ci-failures) or wiring a new secret end to end (use managing-github-actions-secrets).

Authoring CI workflows

Before you propose a change to CI, check things already tried for the idea. It records what was measured, and why some good-sounding changes were reverted or rejected.

Conventions for .github/workflows/** and .github/actions/**. The linters own the mechanical rules (below); this skill is the judgment calls they can't enforce.

Before you write

• Copy from a canonical file rather than from memory. ci-paths-filter.yml is the smallest complete example (triggers, concurrency, timeout, app token, Depot runner); ci-backend.yml is the reference for the heavy patterns (bounded-depth checkout, per-SHA concurrency, draft/ready, sharding).
• Related skills — reach for these instead of duplicating them here:
  • /gating-production-deploys — any job that pushes a prod image or dispatches a Charts deploy.
  • /managing-github-actions-secrets — creating the GitHub App / secret a workflow reads.
  • /depot-github-runners — Depot runner labels and sizing.
  • /debugging-ci-failures — CI is red and you need to know why.

What the linters already enforce

Run bin/hogli lint:workflows and actionlint before pushing — they gate CI, and they (not this list) are the source of truth for what's enforced. Today that's: timeout-minutes on every job, the canonical PR concurrency block, a repo-wide budget for unscoped PR event dispatches, dorny/paths-filter negation safety, justification for full-depth checkouts, cache-write gating, semgrep service coverage, required-check gate hygiene, and generic GHA correctness (bad secrets.* / needs: refs, deprecated ::set-output, unknown runner labels). Third-party action digests are bumped by Renovate.

The dispatch budget (500 runs / 10s / repo)

GitHub caps workflow-run dispatch at 500 runs per 10s per repo; overflow fails as startup_failure and takes unrelated runs in the same window down with it (a stack restack pushing many branches is the usual trigger). Minimize runs dispatched, not just work done — draft status doesn't help, runs dispatch before skip logic applies.

• A reusable-workflow call counts as one run. Small always-fire PR workflows should be jobs under a single workflow_call parent, not their own dispatches (see pr-updated.yml / pr-opened.yml folded behind their parent — fold pr housekeeping into one dispatch). Event-type scoping moves to job-level if: guards:

  jobs:
    turbo:
      if: contains(fromJSON('["opened", "synchronize", "reopened"]'), github.event.action)
      uses: ./.github/workflows/ci-turbo.yml
  

• Prefer a trigger-level paths: filter over dispatch-then-skip: a run that only starts to no-op still spends a dispatch (gate container workflows on trigger paths).

  on:
    pull_request:
      paths:
        - '.github/workflows/ci-x.yml'
        - 'path/to/product/**'
    workflow_dispatch:
  

• Judgment call — trigger paths: vs a runtime dorny/paths-filter job. Use trigger paths: for a workflow that is skippable as a whole. Never put a trigger paths: on a workflow whose check is required by branch protection: a required check that doesn't dispatch on a PR leaves it stuck "waiting for status" and unmergeable. Keep those firing on every PR and gate internally with a changes job (also the right call when several jobs branch on different path sets). Heavy matrices (ci-backend, ci-nodejs) do exactly this — deliberate.

• Delete dead dispatchers outright. A disabled-but-still-triggered workflow keeps dispatching no-op runs against the cap — remove the trigger, don't just disable it.

Concurrency

Every PR-triggered workflow gets the canonical block:

concurrency:
  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
  cancel-in-progress: ${{ github.event_name == 'pull_request' }}

• Cancel superseded PR runs; never cancel across master pushes. WF002 rejects a bare cancel-in-progress: true on any push-triggered workflow. Where latest-wins is genuinely right (a cache warmer), say so with # hogli-lint: allow-master-cancel -- <reason>.

• Use github.ref as the fallback, never github.run_id — run_id is unique per run, so it silently gives every push its own group and dedup is lost.

• Publish-on-push workflows must not let two master pushes race :latest / a deploy dispatch. Key the push arm per-SHA (see ci-backend.yml):

  group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.sha || github.head_ref || github.ref }}
  

Required-check gates

The "gate" is the collate job that emits the required status check by reading needs.*.result. By convention its display name ends in Pass (Django Tests Pass, Visual regression tests pass), but WF007 also finds gates structurally when a step reads needs.<dep>.result, because the convention is not universally followed. A job that inspects results without gating anything opts out with # hogli-lint: not-a-required-gate — <reason> above the job key. Gates and the workers they inspect need opposite conditions:

Job	Condition	Why
Gate	if: always()	It must run and emit an explicit verdict, even when everything upstream died.
Workers	if: !cancelled()	So a superseded run actually stops instead of holding the concurrency slot.

The gate condition must be exactly always(), with optional ${{ }} wrapping. Adding another predicate can skip the required check, so always() && <condition> is rejected.

!cancelled() is identical to always() on any run that is not cancelled, so failure-path reporting still works; only cancelled runs skip. Measured on a live superseded run (evidence): an always() worker dispatched and ran to completion after the cancel, while the !cancelled() worker never started and reported cancelled (not skipped), so the gate still fails closed.

Four rules for the gate body:

1. Allowlist every dependency, never denylist. Assert success/skipped and fail everything else. A dependency tested only against == 'failure' lets cancelled through, and one bad dependency is enough — a gate that clears four correctly and one with a bare failure test is still wrong. The trap is the changes detector: clearing it with == 'failure' and then reading needs.changes.outputs.* reports green on cancellation, because those outputs are empty and the gate takes its "nothing to test" exit.
2. needs every job that produces coverage. If a job's failure would only cascade into a downstream job being skipped, the gate reads that as a pass and you get a green check with zero tests run. Name the upstream job explicitly.
3. Legitimate skips must still pass. A frontend-only PR skips backend jobs by design.
4. Every dependency's result must reach a fail-closed allowlist guard. One inline if per dependency is the clearest form, but a shared shell helper or an env: block is equally fine: WF007 traces each result through assignments, ${!var} indirection, and helper argument positions within that step. The guard must compare with !=, join multiple allowed values with &&, and unconditionally exit 1 when entered. Comparisons in another step, comments, logs, or branches that do not exit nonzero prove nothing and are rejected. A result whose guard WF007 cannot follow is reported rather than assumed safe, so an unusual routing may need the checks moved inline.

WF007 enforces 1, 4, and the always() condition, and it takes the dependency list from needs: as well as the step body, so a job you wired into needs: and then forgot to test is reported rather than silently trusted. The half of rule 2 it cannot check is whether you named the right jobs in needs: to begin with: "reporting job" and "coverage job" look identical to a linter, so that one is on you and the reviewer.

Checkout / clone — sparse first, then shallow

This repo is 45k tracked files and 4.6 GiB of packed objects, so what you materialize costs more than how much history you fetch. Measured checkout-step durations, from the GitHub API on real runs:

Pattern	depot-ubuntu-24.04	GitHub-hosted ubuntu
sparse-checkout of a few paths, cone mode off	0–7s	0–7s
plain checkout (depth 1)	11–13s	22–44s
fetch-depth: 1000 + filter: blob:none	53–59s	—

• Biggest lever: check out only the paths the job reads. Sparse-checkout is not just for single files — a job that runs a local composite action, reads a JSON config, or lints one directory should name those paths and nothing else.

  - uses: actions/checkout@<sha> # v6
    with:
      sparse-checkout: |
        .github/actions/paths-filter
        .github/clickhouse-versions.json
      sparse-checkout-cone-mode: false
  

• Always set sparse-checkout-cone-mode: false. Cone mode additionally materializes every file in the repo root — here 70 files and 21.5 MB, .test_durations alone 18.5 MB — which is most of what you were trying to avoid. Cone mode also only takes whole directories, so it drags in all of bin/ when you wanted one script.

• filter: blob:none is counterproductive if the job then materializes the tree. It removes blobs from the fetch, but git checkout immediately lazy-fetches every blob in HEAD in a second round trip, which is slower than having fetched them in the pack. That lazy fetch also intermittently fails its per-blob credential lookup with could not read Username for github.com (#59779, blocked a merge until retried). Pair blob:none with sparse-checkout so the lazy fetch is a handful of blobs, or drop it and take the plain depth-1 checkout.

• Default: plain actions/checkout (depth 1). Add nothing.

• Diffing against the PR base: you need real history, so bound the depth, filter blobs, and sparse-checkout the files the job reads:

  - uses: actions/checkout@<sha> # v6
    with:
      fetch-depth: 1000
      filter: blob:none
      sparse-checkout: 
       
         
       
    
         
           
  

Pinning and tool versions

• Pin every third-party action to a full 40-char commit SHA with a # vX.Y.Z comment. A moved tag can ship malicious code; pinning is also reproducible and skips a per-run GitHub-API version lookup. The only sanctioned exception is a debug-only action. In-repo composites use a local path with no ref (uses: ./.github/actions/pnpm-install).
• Node version comes from .nvmrc — node-version-file: .nvmrc, never a hardcoded node-version:. Sparse-checkout .nvmrc if the job has no checkout.
• Pin setup-uv's version: — an unpinned setup-uv calls the GitHub API on every job and burns the rate limit.

Network fetches

Downloads from outside the runner need retries, or a transient reset becomes a red check with no findings (actionlint died on curl: (35)).

curl -fsSL --retry 5 --retry-all-errors --retry-max-time 60 --connect-timeout 10 -o "$out" "$url"

• --retry-all-errors is the part that catches a reset; plain --retry covers only timeouts and 408/429/5xx, and --retry-connrefused adds ECONNREFUSED, not ECONNRESET.
• Drop it on GitHub API calls: with -f it also retries 403 and 404, spending five more requests on an already-empty token bucket.
• No --retry-delay (it replaces exponential backoff with a fixed wait). Keep -f, or an error page lands in your output file at exit 0.
• Don't retry anything non-idempotent (webhook posts, telemetry), or where a shell loop or readiness wait already retries.

Tokens — dedicated App tokens for high-volume calls

GITHUB_TOKEN shares one ~15k req/hr bucket across every job of every run in the repo; it goes hot at merge peaks and change-detection jobs fail before real work starts. A dedicated GitHub App installation is its own bucket — rate-limit headroom plus blast-radius isolation.

- uses: actions/create-github-app-token@<sha> # v3.1.1
  id: app-token
  # forks can't read org secrets — fall back to github.token
  if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
  with:
    client-id: ${{ secrets.GH_APP_POSTHOG_PATHS_FILTER_APP_ID }}
    private-key: ${{ secrets.GH_APP_POSTHOG_PATHS_FILTER_PRIVATE_KEY }}

# a later step consumes the token (falling back to github.token on forks):
- uses: some-action@<sha>
  with:
    token: ${{ steps.app-token.outputs.token || github.token }}

• Right-size, don't over-isolate. One heavy consumer (change detection on a hot matrix) deserves its own app; a long tail of light workflows can share GITHUB_TOKEN. Convention: GH_APP_<PURPOSE>_APP_ID + GH_APP_<PURPOSE>_PRIVATE_KEY.
• Cross-repo tokens set explicit owner: + repositories: (least privilege).
• Creating the app + secret is out of scope here — use /managing-github-actions-secrets.

Forks and untrusted PRs (public repo)

Fork pull_request runs (and Dependabot) get a read-only GITHUB_TOKEN and no secrets. Make those runs pass, and never let untrusted code reach a secret.

• Guard secret-needing steps with if: github.event.pull_request.head.repo.full_name == github.repository, and degrade rather than fail (|| github.token, or the raw test outcome).
• Secret-injecting builds (BuildKit --secret, registry login) must skip forks — gate both the changes job and any always() build job (block fork PRs from rust image build).
• Comment or label only on same-repo PRs — the fork token can't write.
• To act on a fork PR with secrets/write (reviewer or label bots), use pull_request_target: base-repo permissions, but it must never check out and run fork code. That's why those workflows can't fold into a pull_request parent.
• First-time contributors need maintainer approval before workflows run (action_required) — expected.

Timeouts

Every job sets timeout-minutes, sized ~2-3x observed max; gate/aggregation jobs get ~5m. The default is 6 hours — a hung job burns paid minutes silently. Caveat: timeout-minutes is invalid on a job that only uses: a reusable workflow — put the timeout inside the called workflow instead.

Caching

Route through the shared composites rather than hand-rolling actions/cache: ./.github/actions/pnpm-install (single pnpm-<os>-<lockhash> key, save gated to master), astral-sh/setup-uv with enable-cache: true, Depot cache via ./.github/actions/build-n-cache-image. One canonical key per artifact; gate saves to master or key deliberately per-ref. PR-scoped cache writes nobody else can read just fragment the 10 GB LRU cap.

Any job that runs manage.py migrate against a fresh Postgres must restore the master schema dump first, keeping the migrate as a seconds-long top-up. A from-scratch replay of the full migration history grows with every migration merged and already costs more than most jobs' timeout-minutes, so an uncached migrate is a timeout that hasn't fired yet (agent-skills cancelled at 30 min with the checks green). Copy the three steps (compute keys, actions/cache/restore, prime) from ci-agent-skills.yml for compose-stack jobs or ci-rust-flags-integration.yml for service-container jobs; hogli db:restore-schema-fresh reads TARGET_DB to pick the database. A miss falls through to the full migrate, so the restore is never a correctness risk. The only sanctioned exception is a job whose purpose is validating the migration history itself (ci-backend's check-migrations), where a restored dump would mask what it checks.

Runners

depot-ubuntu-<version>[-<vCPU>] for build/compute-heavy jobs (the -4/-8 suffix bumps CPU from the 2-vCPU default); GitHub-hosted for light jobs. New Depot labels must be added to the allow-list in .github/actionlint.yaml or actionlint fails. Details: /depot-github-runners.

Draft vs ready-for-review

Most commits land before a PR is marked ready, and drafts can't merge — so heavy suites should run a narrowed subset on drafts and the full matrix on ready_for_review (the merge gate). Add ready_for_review to the pull_request types, and make aggregator "... Tests Pass" jobs treat skipped as success so drafts still report. Foot-gun: if the job that selects tests is cancelled mid-flight, its mode output is empty — normalize empty-mode on a draft to skip, or the draft grabs the full matrix and serializes the ready run behind it.

Backwards-compat with unrebased PRs

A workflow edit hits every open PR the instant it merges (it runs against PR-merged-with-master), but companion changes — a new dependency, file, or config — only reach a branch when it rebases. If the workflow starts requiring something unrebased branches lack, every in-flight PR fails before its tests run. Make new behavior degrade gracefully when the prerequisite is absent, or gate it. Roll out a new blocking lint the same way: ship continue-on-error, clear the inbox, promote to blocking.

New-workflow checklist

• Triggers scoped: trigger paths: where the whole workflow is skippable; a required check must still fire on every PR (never paths-gate it into never dispatching).
• Canonical concurrency: block (per-SHA push arm if it publishes on push).
• timeout-minutes on every job (except reusable-caller jobs).
• Checkout names only the paths the job reads (sparse-checkout + cone mode off), or is shallow; bounded 1000 + blob:none only for base diffing.
• Third-party actions SHA-pinned; Node from .nvmrc; setup-uv version pinned.
• External fetches retry (--retry-all-errors), except where a repeat has a side effect.
• High-volume API calls on a dedicated App token with || github.token fork fallback.
• Fork PRs handled: secret-needing steps guarded with the same-repo if:; no secret-injecting build runs on forks.
• Caching through the shared composites; writes gated to master.
• Any job running manage.py migrate restores the master schema dump first, with the migrate as top-up (see Caching).
• Prod image push / deploy dispatch gated per /gating-production-deploys.
• bin/hogli lint:workflows and actionlint pass locally.