ci-cd-workflow-spec

# CI/CD Workflow Specification General rules for clean, secure, reproducible CI/CD workflows. The supported first-class adapters are GitHub Actions YAML and GitLab CI YAML; their syntax, file paths, and keywords implement this policy but do not replace it. For policy that spans tools, defer to the sibling standards rather than restating it here: General rules for clean, secure, reproducible CI/CD workflows. GitHub Actions is the current implementation context, so its file paths and syntax appear in examples; treat them as adapters, not as the policy itself. Keep the rules, but rename every example workflow, script, job, and role for the target repository. For policy that spans tools, defer to the sibling standards rather than restating it here: Read [AI Workspace Infra Repository Map](../references/ai-workspace-infra-repository-map.md) first. Apply the target repository's current workflow conventions; do not copy a legacy workflow's inline scripts, secrets, or broad trigger scope into a new delivery path. - Environment routing (SIT/UAT/Prod) & Vault OIDC role names → `multi-environment-delivery-and-release` - No-inline-scripts / code purity for HCL & playbooks → `infrastructure-as-code-spec`, `config-as-code-spec` - Branching, PR targets, and committed-secret response → `project-development-standard` ## 1. Supported pipeline adapters Use GitHub Actions or GitLab CI for new primary pipelines. Both are YAML-based, work with hosted or self-managed/open-source-compatible installations, and can share portable scripts, OCI artifacts, Terraform plans, CMDB files, and Vault OIDC contracts. | Concern | GitHub Actions | GitLab CI | | --- | --- | --- | | Pipeline file | `.github/workflows/*.yml` | `.gitlab-ci.yml` and reviewed local includes | | External logic | `.github/scripts/` | repository-local `ci/scripts/` or equivalent | | Job dependency | `needs` | `stages` + `needs` | | Reuse | `workflow_call` | local/reviewed `include`, child pipeline, or component | | OIDC to Vault | `permissions: id-token: write` | job `id_tokens` with a narrowly scoped `aud` claim | | Serialization | `concurrency` | `resource_group` | | Artifact handoff | `upload-artifact` / `download-artifact` | `artifacts` plus explicit `needs:artifacts` | - Keep provider-neutral deployment logic in repository-controlled scripts; YAML selects stages, dependencies, permissions, inputs, and artifacts. Do not make the pipeline portable by copying two divergent implementations of the same deployment logic. - GitHub Actions must pin third-party actions to the repository-approved version or immutable SHA. GitLab CI must pin external includes, components, container images, and templates to an immutable version, SHA, or image digest. Never consume a mutable remote pipeline definition on a deployment path. - In GitLab, use `rules` for event routing, `needs` for explicit dependency and artifact flow, protected environments/refs for CD, and `id_tokens` rather than deprecated job JWT variables for OIDC. In GitHub, use explicit event filters, `needs`, environments, `permissions`, and `concurrency` with the same intent. - GitLab `secrets:vault` writes fetched secrets to a temporary file by default. That default conflicts with a zero-secret-to-disk policy: use OIDC to obtain only the short-lived value needed by the job, and use a reviewed non-file delivery mode only when the process can avoid logging or persisting it. - Deployment pipelines MUST NOT depend on volatile or unverified daily-build release assets from external repositories during automated execution. Release artifacts consumed during deployment MUST be immutable, verified in preflight, or hosted in internal artifact repositories. ### 1.1 Jenkins migration policy Jenkinsfile is a legacy compatibility route, not a recommended primary platform for new CI/CD work. Do not add a new Jenkinsfile or expand an existing one unless it is a time-bounded migration prerequisite. Migrate Jenkins pipelines incrementally: 1. Inventory triggers, branch rules, credentials, agents, shared libraries, artifacts, environment mutations, approvals, and rollback behavior. 2. Extract business logic from Groovy and inline shell into versioned, provider-neutral scripts with explicit inputs, exit codes, and tests. 3. Recreate CI first in GitHub Actions or GitLab CI: validation, scan, build, and immutable artifact publication. Recreate CD separately with OIDC → Vault, protected environment controls, concurrency/resource locking, and health checks. 4. Run a non-production parity period. Compare artifact digest, target set, rendered configuration, and health result; never let both systems apply to the same mutable environment concurrently. 5. Cut over one protected environment at a time, retain a documented rollback window, then disable Jenkins deployment triggers and revoke its long-lived credentials, agent permissions, and unused shared-library access. Do not translate Jenkins Groovy line-for-line into YAML. The target design must separate CI from CD, use immutable artifacts, and remove Jenkins-only credential or controller assumptions. ## 2. No inline scripts Keep every provider command stanza to a single call. Put non-trivial shell/Python in an executable, repository-controlled script and validate it with `bash -n`. GitHub Actions uses `.github/scripts/`; GitLab CI uses `ci/scripts/` or the repository's established equivalent. GitHub Actions adapter: ```yaml - name: Install dependencies run: ${{ github.workspace }}/.github/scripts/<workflow>_<step>_install-deps.sh ``` GitLab CI adapter: ```yaml validate: stage: validate script: - ./ci/scripts/validate.sh ``` Pass Vault secrets and other values into scripts as job environment variables, never inline in the command. ## 3. Pin external dependencies Reuse the target repository's already-approved dependency version unless the task is an explicit upgrade. Verify the exact tag, SHA, or digest against its publisher; never invent a version. New security-sensitive workflows should prefer immutable full-SHA or digest pins where that repository already uses them. | Action | Tag | |---|---| | `actions/checkout` | target-repository approved version/SHA | | `actions/setup-python` | target-repository approved version/SHA | | `actions/upload-artifact` / `download-artifact` | target-repository approved version/SHA | | `hashicorp/vault-action` | target-repository approved version/SHA | | `hashicorp/setup-terraform` | target-repository approved version/SHA | ## 3. Shared scripts Reuse cross-workflow logic via `common_*.sh` scripts under `.github/scripts/` (e.g. `common_terraform_init_backend.sh`, `common_run_ansible_playbook.sh`, `common_configure_ssh_key.sh`). Step scripts delegate rather than copy: ```bash #!/usr/bin/env bash set -e DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" exec "${DIR}/common_terraform_init_backend.sh" ``` ## 4. Shared scripts Reuse cross-workflow logic via `common_*.sh` scripts under the provider adapter directory (`.github/scripts/` for GitHub Actions; `ci/scripts/` or equivalent for GitLab CI), for example `common_terraform_init_backend.sh`, `common_run_ansible_playbook.sh`, and `common_configure_ssh_key.sh`. Step scripts delegate rather than copy: ### 4.1 Generalize by shape, not by step One script per workflow step is the default drift, and it scales badly: a directory named `<workflow>_<job>_<step>.sh` reads as organised while being N copies of the same three commands. Group by *invocation shape* instead, and let the caller supply what differs. Most CI scripts collapse into a handful of shapes — an `ansible-playbook` run, a `terraform` subcommand, an `ssh` action against a host resolved from the CMDB, a render/validate step. Audit by shape before adding another file: ```bash for f in .github/scripts/*.sh; do grep -q 'ansible-playbook' "$f" && echo "playbook $f" && continue grep -q 'terraform ' "$f" && echo "terraform $f" && continue grep -q 'ssh_opts=' "$f" && echo "ssh $f" done | sort | uniq -c ``` If a shape has more than two or three members, it wants one parameterised entry point. A set of playbook wrappers usually differs only in the playbook filename, how the target host is passed, and an optional extra-vars file — everything else, including the inventory path, is identical and should live in one place. **The cost of not doing this is measured in repeated fixes.** When the same defect has to be patched in three sibling scripts, and a wrong inventory path in four, the duplication is not stylistic — each copy is a place the next guard will be forgotten. A shared entry point means the reachability assert (§9) and the required-variable check are written once and inherited by every caller. **Standardise how the target is selected.** Passing the host as `-e "<role>_hosts=${HOST}"` and passing it as `--limit "${HOST}"` are not interchangeable: they fail differently when the value is empty. An empty `-e` var against a playbook whose `hosts:` reads from it aborts with "Keyword 'hosts' cannot have empty values", while an empty `--limit` against a playbook with a static `hosts:` line does not narrow anything. Pick one mechanism per repository so the empty case has one known behaviour, and assert the variable regardless (§9). ### 4.2 No hardcoded values in scripts Anything that varies by environment, target, or release is a parameter. Domains, hostnames, IPs, zone names, registry namespaces and image tags belong in the caller's `env:` block or a single named constant — never inline in a script, and never as a per-file fallback. - **A default that differs between files is worse than no default.** Divergent fallbacks produce output that is valid, plausible, and wrong in some files only, which is far harder to spot than a uniform failure. If several files each carry their own `X or 'some-domain'`, they will drift, and at least one will end up pointing at production. - **Reject the "sensible default" for anything that selects a target.** A default target is a silent choice made on the operator's behalf at the exact moment they forgot to make one. Require it and fail (§9). - **A literal that appears once is still a parameter if it varies by environment.** Judge by whether the value *could* differ per environment, not by how many times it currently appears. - Values that genuinely never vary — a fixed image tag the pipeline itself builds, a path the pipeline itself creates — are implementation detail and may stay inline. Parameterising them adds indirection for no reuse. ## 5. Concurrency & matrix safety - **Scope concurrency by workflow + environment** so environments never block each other: ```yaml concurrency: group: <workflow-name>-${{ github.event.inputs.vault_env_path || 'uat' }} cancel-in-progress: false ``` - GitLab CD jobs use an equivalent `resource_group` keyed by the mutable environment/resource and must not be `interruptible` while a stateful migration or Terraform operation is active. - **Default matrix outputs** to `[]` / `0` when their source file is missing or during `destroy`: ```bash if [ -f cmdb.json ]; then hosts="$(jq -c 'keys' cmdb.json)"; count="$(jq 'length' cmdb.json)" else hosts="[]"; count="0" fi ``` - **Guard matrix jobs** on non-empty *and* non-zero counts: ```yaml if: ${{ needs.provision.outputs.count != '' && needs.provision.outputs.count != '0' && github.event.inputs.terraform_action == 'apply' }} ``` - Set `fail-fast: false` on deployment matrices so one bad host doesn't cancel the rest. ## 6. Permissions & log hygiene - Declare least-privilege permissions at the top level: ```yaml permissions: contents: read id-token: write # OIDC → Vault ``` - Never `set -x` around secrets. Mask any secret parsed by hand: `echo "::add-mask::$SECRET"`. - In GitLab, protect deployment refs/environments and use job `id_tokens` for OIDC. Do not grant deploy authority through broadly available project variables or retired `CI_JOB_JWT` variables. ## 7. Non-interactive tooling CI steps must never block on a prompt: - Terraform: `terraform init -input=false`, `terraform apply -auto-approve -input=false` - Ansible: `ANSIBLE_HOST_KEY_CHECKING=False`, `-o IdentityFile=~/.ssh/id_deploy -o StrictHostKeyChecking=no` Upload `cmdb.json` and `inventory.ini` as auditable artifacts. In GitHub Actions, use the repository-approved artifact action; in GitLab, declare `artifacts` and an explicit `needs:artifacts` relationship for the consuming job. ## 8. Workflow roles ### 8.0 Cross-repository snapshot orchestration When one release spans multiple repositories, treat the snapshot as a first-class coordination artifact rather than as a collection of unrelated successful jobs: - Resolve the source ref to an expected commit SHA per repository before tag creation. - Create one immutable snapshot tag per repository using the same daily series (`uat-daily-build-YYYY.MM.DD-r1` … `-rN`) across the matrix. Allocate `r1` for the first UTC-date snapshot and the next available suffix for every same-date retry or later snapshot. Never move, delete, or force-update an existing tag. - Keep the tag fan-out matrix separate from the required artifact-build matrix. A tag existing in a repository does not prove that its image, package, chart, or release manifest was built. - Match build runs by repository, snapshot tag, expected SHA, intended event or workflow, and successful conclusion. Matching only a branch/tag name can reuse an old retry run. - Aggregate an auditable per-repository result with distinct states such as `tag_ready`, `unchanged`, `build_succeeded`, `build_failed`, `manifest_missing`, `build_timeout`, and `build_lookup_failed`. Pending or unknown states are not successful. - Resolve and reserve the suffix once before matrix fan-out. Independent jobs must not calculate different retry tags; if any participating repository already has the candidate tag, abort allocation and advance the whole cross-repository series together. The workflow summary should include the snapshot tag, source ref, expected SHA, resolved SHA, build URL, artifact/manifest result, and retry reason where applicable. A snapshot is deployable only when every required repository passes all checks; a successful tag job alone is not sufficient. ### 8.0.1 Tagging and release preflight contract Stable and daily builds may call the same tagging helper, but the helper MUST validate an explicit tag kind and reject an environment/tag mismatch before creating or updating any ref. Production entry points MUST accept only `refs/tags/v*` or `refs/heads/release/v*`; they MUST reject `main`, pull-request refs, arbitrary dispatch refs, and `daily-build-*` / `uat-daily-build-*` tags. The stable path and daily path differ by tag semantics, not by a second mutable tagging implementation: | Kind | Required semantics | Retry | |---|---|---| | Stable `vMAJOR.MINOR.PATCH` | reviewed release point, immutable, production-eligible | new SemVer tag | | `daily-build-*` / `uat-daily-build-*` | non-production snapshot, never production-eligible | new `-rN` suffix | The release preflight MUST check, as one consistent matrix, the triggering event and ref, resolved source SHA, artifact build/digest/provenance, required test conclusions, GitOps desired tag, environment route, Vault role/KV path, and rollback evidence. The same computed environment must feed routing, credentials, and test selection; if one resolver says UAT while another says production, fail before credentials or mutation. A tag-exists or workflow-dispatch success is not an artifact, test, or deployment success. A multi-cloud IaC repo typically splits responsibilities across these five patterns. The file names are examples — rename to match your repo: | Pattern | Example file | Role | |---|---|---| | Orchestrator | `pipeline-master.yaml` | Calls child workflows via `workflow_call` + `secrets: inherit` | | Multi-stage delivery | `deploy.yaml` | Job graph (`provision → deploy_* → migrate → switch_dns`); state passed via artifacts | | Component matrix | `resources-matrix.yaml` | `strategy.matrix` over `fromJSON(inputs.components_json)` | | PR quality gate | `validate-pr.yaml` | `pull_request` + `checkout` `fetch-depth: 0` + secret scan (e.g. `gitleaks`) | | Readiness checker | `check-ready.yaml` | `workflow_dispatch`; inspects prior run status via the Actions API | ### 8.1 Dual-plane telemetry and billing delivery When a release carries usage accounting and operational observability together, model them as two consumers of the same exporter output. The billing plane and the real-time monitoring plane MUST be independently observable and MUST NOT make each other a runtime prerequisite: ```text Xray -> exporter -> Vector -> authenticated Billing ingest -> shared PostgreSQL -> Accounts API -> Portal +--> Prometheus remote_write / observability -> Grafana ``` - The exporter MUST emit a canonical snapshot identity containing the account key (email and/or UUID), source/node identity, collection time, and byte counters. When one account appears on multiple nodes or inbounds, normalize and aggregate by the canonical UUID; do not create a separate billable account for each inbound tag. - Vector SHOULD be the fan-out boundary. Billing consumes an authenticated push/ingest endpoint by default; Billing MUST NOT directly pull the exporter as the normal path. A direct-pull mode, if retained, is an explicitly named rollback/compatibility mode and MUST be disabled by default. - The Vector-to-Billing sink MUST use an environment-scoped service credential, bounded retry, a durable buffer where supported, and a payload format that preserves the snapshot identity. Redact credentials from config dumps and logs. Prometheus remote write remains a separate sink and must keep its own health signal. - Billing ingestion MUST be idempotent. Use a deterministic event/checkpoint identity, reject or safely ignore duplicate snapshots, and persist the resulting ledger/quota state in the shared PostgreSQL database. Schema initialization and migrations MUST be idempotent and deployed before the reader path is enabled. - Accounts is the aggregation/read API over that shared database. Portal MUST read usage through Accounts rather than querying PostgreSQL or Billing directly. A Portal value of zero is not proof of zero traffic until the upstream ingest, ledger, and Accounts query checkpoints have been verified. The release evidence MUST distinguish these checkpoints: | Checkpoint | Evidence | Failure meaning | |---|---|---| | exporter collection | snapshot counter/log or test endpoint | Xray identity/collection issue | | Vector billing sink | accepted event, retry/buffer status | routing/auth/backpressure issue | | Billing ingest | HTTP status and idempotency result | endpoint/token/payload issue | | PostgreSQL write | ledger/quota row and migration state | schema/transaction issue | | Accounts read | authenticated usage summary | API/query/identity issue | | Portal display | browser/API response | frontend mapping/cache issue | | Grafana path | remote-write arrival and dashboard series | observability path issue; not a billing verdict | Do not close a release from a green deployment job alone. A service being `active` proves process health, not data flow. Record the release tag, expected source SHA, deployment run, target environment, and each checkpoint's timestamp in the repository's task/progress record. ### 8.2 CI and CD must be separate CI establishes whether a revision is safe to promote; CD changes a managed environment. They MAY share reusable scripts or callable workflows, but MUST remain separate workflow boundaries, permissions, and success criteria. | Concern | CI workflow | CD workflow | | --- | --- | --- | | Trigger | Pull requests and ordinary branch pushes | Only the repository's approved promotion event or an explicitly authorized dispatch | | Purpose | Lint, test, render, validate, scan, and build | Consume a promoted release, deploy, migrate, verify, and report rollout state | | Credentials | Read-only by default; no cloud mutation or production Vault role | Least-privilege OIDC/Vault role for the selected environment | | Infrastructure | `terraform fmt`/`validate`/`plan` only | Explicitly confirmed `apply`, replacement, migration, or DNS action only | | Artifact | Build once; publish immutable digest/version plus provenance | Download and verify the exact promoted digest/version; never rebuild from a moving ref |