ci-cd-workflow-spec

name

ci-cd-workflow-spec

description

General CI/CD workflow standards for AI Workspace Infra pipelines and external scripts. Use when creating, refactoring, or auditing CI, CD, promotion, deployment, or GitHub Actions workflows in platform-ops-toolkit, artifacts, gitops, playbooks, iac_modules, observability.svc.plus, or their shell scripts. Covers CI/CD separation, reusable scripts, immutable artifacts, least-privilege OIDC, Terraform/Ansible safety, CMDB artifacts, and false-green prevention without blindly modernizing legacy workflows.

CI/CD Workflow Specification

General rules for clean, secure, reproducible CI/CD workflows. The supported first-class adapters are GitHub Actions YAML and GitLab CI YAML; their syntax, file paths, and keywords implement this policy but do not replace it. For policy that spans tools, defer to the sibling standards rather than restating it here: General rules for clean, secure, reproducible CI/CD workflows. GitHub Actions is the current implementation context, so its file paths and syntax appear in examples; treat them as adapters, not as the policy itself. Keep the rules, but rename every example workflow, script, job, and role for the target repository. For policy that spans tools, defer to the sibling standards rather than restating it here:

Read AI Workspace Infra Repository Map first. Apply the target repository's current workflow conventions; do not copy a legacy workflow's inline scripts, secrets, or broad trigger scope into a new delivery path.

• Environment routing (SIT/UAT/Prod) & Vault OIDC role names → multi-environment-delivery-and-release
• No-inline-scripts / code purity for HCL & playbooks → infrastructure-as-code-spec, config-as-code-spec
• Branching, PR targets, and committed-secret response → project-development-standard

1. Supported pipeline adapters

Use GitHub Actions or GitLab CI for new primary pipelines. Both are YAML-based, work with hosted or self-managed/open-source-compatible installations, and can share portable scripts, OCI artifacts, Terraform plans, CMDB files, and Vault OIDC contracts.

Concern	GitHub Actions	GitLab CI
Pipeline file	.github/workflows/*.yml	.gitlab-ci.yml and reviewed local includes
External logic	.github/scripts/	repository-local ci/scripts/ or equivalent
Job dependency	needs	stages + needs
Reuse	workflow_call	local/reviewed include, child pipeline, or component
OIDC to Vault	permissions: id-token: write	job id_tokens with a narrowly scoped aud claim
Serialization	concurrency	resource_group
Artifact handoff	upload-artifact / download-artifact	artifacts plus explicit needs:artifacts

• Keep provider-neutral deployment logic in repository-controlled scripts; YAML selects stages, dependencies, permissions, inputs, and artifacts. Do not make the pipeline portable by copying two divergent implementations of the same deployment logic.
• GitHub Actions must pin third-party actions to the repository-approved version or immutable SHA. GitLab CI must pin external includes, components, container images, and templates to an immutable version, SHA, or image digest. Never consume a mutable remote pipeline definition on a deployment path.
• In GitLab, use rules for event routing, needs for explicit dependency and artifact flow, protected environments/refs for CD, and id_tokens rather than deprecated job JWT variables for OIDC. In GitHub, use explicit event filters, needs, environments, permissions, and concurrency with the same intent.
• GitLab secrets:vault writes fetched secrets to a temporary file by default. That default conflicts with a zero-secret-to-disk policy: use OIDC to obtain only the short-lived value needed by the job, and use a reviewed non-file delivery mode only when the process can avoid logging or persisting it.
• Deployment pipelines MUST NOT depend on volatile or unverified daily-build release assets from external repositories during automated execution. Release artifacts consumed during deployment MUST be immutable, verified in preflight, or hosted in internal artifact repositories.

1.1 Jenkins migration policy

Jenkinsfile is a legacy compatibility route, not a recommended primary platform for new CI/CD work. Do not add a new Jenkinsfile or expand an existing one unless it is a time-bounded migration prerequisite.

Migrate Jenkins pipelines incrementally:

1. Inventory triggers, branch rules, credentials, agents, shared libraries, artifacts, environment mutations, approvals, and rollback behavior.
2. Extract business logic from Groovy and inline shell into versioned, provider-neutral scripts with explicit inputs, exit codes, and tests.
3. Recreate CI first in GitHub Actions or GitLab CI: validation, scan, build, and immutable artifact publication. Recreate CD separately with OIDC → Vault, protected environment controls, concurrency/resource locking, and health checks.
4. Run a non-production parity period. Compare artifact digest, target set, rendered configuration, and health result; never let both systems apply to the same mutable environment concurrently.
5. Cut over one protected environment at a time, retain a documented rollback window, then disable Jenkins deployment triggers and revoke its long-lived credentials, agent permissions, and unused shared-library access.

Do not translate Jenkins Groovy line-for-line into YAML. The target design must separate CI from CD, use immutable artifacts, and remove Jenkins-only credential or controller assumptions.

2. No inline scripts

Keep every provider command stanza to a single call. Put non-trivial shell/Python in an executable, repository-controlled script and validate it with bash -n. GitHub Actions uses .github/scripts/; GitLab CI uses ci/scripts/ or the repository's established equivalent.

GitHub Actions adapter:

- name: Install dependencies
  run: ${{ github.workspace }}/.github/scripts/<workflow>_<step>_install-deps.sh

GitLab CI adapter:

validate:
  stage: validate
  script:
    - ./ci/scripts/validate.sh

Pass Vault secrets and other values into scripts as job environment variables, never inline in the command.

3. Pin external dependencies

Reuse the target repository's already-approved dependency version unless the task is an explicit upgrade. Verify the exact tag, SHA, or digest against its publisher; never invent a version. New security-sensitive workflows should prefer immutable full-SHA or digest pins where that repository already uses them.

Action	Tag
actions/checkout	target-repository approved version/SHA
actions/setup-python	target-repository approved version/SHA
actions/upload-artifact / download-artifact	target-repository approved version/SHA
hashicorp/vault-action	target-repository approved version/SHA
hashicorp/setup-terraform	target-repository approved version/SHA

3. Shared scripts

Reuse cross-workflow logic via common_*.sh scripts under .github/scripts/ (e.g. common_terraform_init_backend.sh, common_run_ansible_playbook.sh, common_configure_ssh_key.sh). Step scripts delegate rather than copy:

#!/usr/bin/env bash
set -e
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
exec "${DIR}/common_terraform_init_backend.sh"

4. Shared scripts

Reuse cross-workflow logic via common_*.sh scripts under the provider adapter directory (.github/scripts/ for GitHub Actions; ci/scripts/ or equivalent for GitLab CI), for example common_terraform_init_backend.sh, common_run_ansible_playbook.sh, and common_configure_ssh_key.sh. Step scripts delegate rather than copy:

4.1 Generalize by shape, not by step

One script per workflow step is the default drift, and it scales badly: a directory named <workflow>_<job>_<step>.sh reads as organised while being N copies of the same three commands. Group by invocation shape instead, and let the caller supply what differs.

Most CI scripts collapse into a handful of shapes — an ansible-playbook run, a terraform subcommand, an ssh action against a host resolved from the CMDB, a render/validate step. Audit by shape before adding another file:

for f in .github/scripts/*.sh; do
  grep -q 'ansible-playbook' "$f" && echo "playbook  $f" && continue
  grep -q 'terraform '       "$f" && echo "terraform $f" && continue
  grep -q 'ssh_opts='        "$f" && echo "ssh       $f"
done | sort | uniq -c

If a shape has more than two or three members, it wants one parameterised entry point. A set of playbook wrappers usually differs only in the playbook filename, how the target host is passed, and an optional extra-vars file — everything else, including the inventory path, is identical and should live in one place.

The cost of not doing this is measured in repeated fixes. When the same defect has to be patched in three sibling scripts, and a wrong inventory path in four, the duplication is not stylistic — each copy is a place the next guard will be forgotten. A shared entry point means the reachability assert (§9) and the required-variable check are written once and inherited by every caller.

Standardise how the target is selected. Passing the host as -e "<role>_hosts=${HOST}" and passing it as --limit "${HOST}" are not interchangeable: they fail differently when the value is empty. An empty -e var against a playbook whose hosts: reads from it aborts with "Keyword 'hosts' cannot have empty values", while an empty --limit against a playbook with a static hosts: line does not narrow anything. Pick one mechanism per repository so the empty case has one known behaviour, and assert the variable regardless (§9).

4.2 No hardcoded values in scripts

Anything that varies by environment, target, or release is a parameter. Domains, hostnames, IPs, zone names, registry namespaces and image tags belong in the caller's env: block or a single named constant — never inline in a script, and never as a per-file fallback.

• A default that differs between files is worse than no default. Divergent fallbacks produce output that is valid, plausible, and wrong in some files only, which is far harder to spot than a uniform failure. If several files each carry their own X or 'some-domain', they will drift, and at least one will end up pointing at production.
• Reject the "sensible default" for anything that selects a target. A default target is a silent choice made on the operator's behalf at the exact moment they forgot to make one. Require it and fail (§9).
• A literal that appears once is still a parameter if it varies by environment. Judge by whether the value could differ per environment, not by how many times it currently appears.
• Values that genuinely never vary — a fixed image tag the pipeline itself builds, a path the pipeline itself creates — are implementation detail and may stay inline. Parameterising them adds indirection for no reuse.

5. Concurrency & matrix safety

• Scope concurrency by workflow + environment so environments never block each other:

  concurrency:
    group: <workflow-name>-${{ github.event.inputs.vault_env_path || 'uat' }}
    cancel-in-progress: false
  

• GitLab CD jobs use an equivalent resource_group keyed by the mutable environment/resource and must not be interruptible while a stateful migration or Terraform operation is active.
• Default matrix outputs to [] / 0 when their source file is missing or during destroy:

  if [ -f cmdb.json ]; then
    hosts="$(jq -c 'keys' cmdb.json)"; count="$(jq 'length' cmdb.json)"
  else
    hosts="[]"; count="0"
  fi
  

• Guard matrix jobs on non-empty and non-zero counts:

  if: ${{ needs.provision.outputs.count != '' && needs.provision.outputs.count != '0' && github.event.inputs.terraform_action == 'apply' }}
  

• Set fail-fast: false on deployment matrices so one bad host doesn't cancel the rest.

6. Permissions & log hygiene

• Declare least-privilege permissions at the top level:

  permissions:
    contents: read
    id-token: write   # OIDC → Vault
  

• Never set -x around secrets. Mask any secret parsed by hand: echo "::add-mask::$SECRET".
• In GitLab, protect deployment refs/environments and use job id_tokens for OIDC. Do not grant deploy authority through broadly available project variables or retired CI_JOB_JWT variables.

7. Non-interactive tooling

CI steps must never block on a prompt:

• Terraform: terraform init -input=false, terraform apply -auto-approve -input=false
• Ansible: ANSIBLE_HOST_KEY_CHECKING=False, -o IdentityFile=~/.ssh/id_deploy -o StrictHostKeyChecking=no

Upload cmdb.json and inventory.ini as auditable artifacts. In GitHub Actions, use the repository-approved artifact action; in GitLab, declare artifacts and an explicit needs:artifacts relationship for the consuming job.

8. Workflow roles

8.0 Cross-repository snapshot orchestration

When one release spans multiple repositories, treat the snapshot as a first-class coordination artifact rather than as a collection of unrelated successful jobs:

• Resolve the source ref to an expected commit SHA per repository before tag creation.
• Create one immutable snapshot tag per repository using the same daily series (uat-daily-build-YYYY.MM.DD-r1 … -rN) across the matrix. Allocate r1 for the first UTC-date snapshot and the next available suffix for every same-date retry or later snapshot. Never move, delete, or force-update an existing tag.
• Keep the tag fan-out matrix separate from the required artifact-build matrix. A tag existing in a repository does not prove that its image, package, chart, or release manifest was built.
• Match build runs by repository, snapshot tag, expected SHA, intended event or workflow, and successful conclusion. Matching only a branch/tag name can reuse an old retry run.
• Aggregate an auditable per-repository result with distinct states such as tag_ready, unchanged, build_succeeded, build_failed, manifest_missing, build_timeout, and build_lookup_failed. Pending or unknown states are not successful.
• Resolve and reserve the suffix once before matrix fan-out. Independent jobs must not calculate different retry tags; if any participating repository already has the candidate tag, abort allocation and advance the whole cross-repository series together.

The workflow summary should include the snapshot tag, source ref, expected SHA, resolved SHA, build URL, artifact/manifest result, and retry reason where applicable. A snapshot is deployable only when every required repository passes all checks; a successful tag job alone is not sufficient.

8.0.1 Tagging and release preflight contract

Stable and daily builds may call the same tagging helper, but the helper MUST validate an explicit tag kind and reject an environment/tag mismatch before creating or updating any ref. Production entry points MUST accept only refs/tags/v* or refs/heads/release/v*; they MUST reject main, pull-request refs, arbitrary dispatch refs, and daily-build-* / uat-daily-build-* tags.

The stable path and daily path differ by tag semantics, not by a second mutable tagging implementation:

Kind	Required semantics	Retry
Stable vMAJOR.MINOR.PATCH	reviewed release point, immutable, production-eligible	new SemVer tag
daily-build-* / uat-daily-build-*	non-production snapshot, never production-eligible	new -rN suffix

The release preflight MUST check, as one consistent matrix, the triggering event and ref, resolved source SHA, artifact build/digest/provenance, required test conclusions, GitOps desired tag, environment route, Vault role/KV path, and rollback evidence. The same computed environment must feed routing, credentials, and test selection; if one resolver says UAT while another says production, fail before credentials or mutation. A tag-exists or workflow-dispatch success is not an artifact, test, or deployment success.

A multi-cloud IaC repo typically splits responsibilities across these five patterns. The file names are examples — rename to match your repo:

Pattern	Example file	Role
Orchestrator	pipeline-master.yaml	Calls child workflows via workflow_call + secrets: inherit
Multi-stage delivery	deploy.yaml	Job graph (provision → deploy_* → migrate → switch_dns); state passed via artifacts
Component matrix	resources-matrix.yaml	strategy.matrix over fromJSON(inputs.components_json)
PR quality gate	validate-pr.yaml	pull_request + checkout fetch-depth: 0 + secret scan (e.g. gitleaks)
Readiness checker	check-ready.yaml	workflow_dispatch; inspects prior run status via the Actions API

8.1 Dual-plane telemetry and billing delivery

When a release carries usage accounting and operational observability together, model them as two consumers of the same exporter output. The billing plane and the real-time monitoring plane MUST be independently observable and MUST NOT make each other a runtime prerequisite:

Xray -> exporter -> Vector -> authenticated Billing ingest -> shared PostgreSQL
                                      -> Accounts API -> Portal
                         +--> Prometheus remote_write / observability -> Grafana

• The exporter MUST emit a canonical snapshot identity containing the account key (email and/or UUID), source/node identity, collection time, and byte counters. When one account appears on multiple nodes or inbounds, normalize and aggregate by the canonical UUID; do not create a separate billable account for each inbound tag.
• Vector SHOULD be the fan-out boundary. Billing consumes an authenticated push/ingest endpoint by default; Billing MUST NOT directly pull the exporter as the normal path. A direct-pull mode, if retained, is an explicitly named rollback/compatibility mode and MUST be disabled by default.
• The Vector-to-Billing sink MUST use an environment-scoped service credential, bounded retry, a durable buffer where supported, and a payload format that preserves the snapshot identity. Redact credentials from config dumps and logs. Prometheus remote write remains a separate sink and must keep its own health signal.
• Billing ingestion MUST be idempotent. Use a deterministic event/checkpoint identity, reject or safely ignore duplicate snapshots, and persist the resulting ledger/quota state in the shared PostgreSQL database. Schema initialization and migrations MUST be idempotent and deployed before the reader path is enabled.
• Accounts is the aggregation/read API over that shared database. Portal MUST read usage through Accounts rather than querying PostgreSQL or Billing directly. A Portal value of zero is not proof of zero traffic until the upstream ingest, ledger, and Accounts query checkpoints have been verified.

The release evidence MUST distinguish these checkpoints:

Checkpoint	Evidence	Failure meaning
exporter collection	snapshot counter/log or test endpoint	Xray identity/collection issue
Vector billing sink	accepted event, retry/buffer status	routing/auth/backpressure issue
Billing ingest	HTTP status and idempotency result	endpoint/token/payload issue
PostgreSQL write	ledger/quota row and migration state	schema/transaction issue
Accounts read	authenticated usage summary	API/query/identity issue
Portal display	browser/API response	frontend mapping/cache issue
Grafana path	remote-write arrival and dashboard series	observability path issue; not a billing verdict

Do not close a release from a green deployment job alone. A service being active proves process health, not data flow. Record the release tag, expected source SHA, deployment run, target environment, and each checkpoint's timestamp in the repository's task/progress record.

8.2 CI and CD must be separate

CI establishes whether a revision is safe to promote; CD changes a managed environment. They MAY share reusable scripts or callable workflows, but MUST remain separate workflow boundaries, permissions, and success criteria.

Concern	CI workflow	CD workflow
Trigger	Pull requests and ordinary branch pushes	Only the repository's approved promotion event or an explicitly authorized dispatch
Purpose	Lint, test, render, validate, scan, and build	Consume a promoted release, deploy, migrate, verify, and report rollout state
Credentials	Read-only by default; no cloud mutation or production Vault role	Least-privilege OIDC/Vault role for the selected environment
Infrastructure	terraform fmt/validate/plan only	Explicitly confirmed apply, replacement, migration, or DNS action only
Artifact	Build once; publish immutable digest/version plus provenance	Download and verify the exact promoted digest/version; never rebuild from a moving ref