argo-workflows

name	argo-workflows
description	Authoring, linting, and submitting Argo Workflows and WorkflowTemplates in the lab. Use when writing or editing any .yaml file under argo/workflow-templates/, argo/bootstrap/, or argo/*.yaml, or when debugging a failed workflow run.
metadata	{"context7-sources":["/argoproj/argo-workflows","/kubernetes/website","/oras-project/oras","/websites/github_en_rest"]}

Argo Workflows — lab Skill

When to Use

• Editing any argo/workflow-templates/*.yaml or argo/bootstrap/*.yaml
• Writing a new pipeline (provision, test, teardown)
• Adding a new argo/*.yaml submit-time Workflow
• Debugging a stuck or failed workflow
• Adding a CronWorkflow to manifests/

When NOT to Use

• ArgoCD Application changes → gitops-argocd.md
• KubeVirt VM manifest design → kubevirt-vms.md
• behave/dogtail test authoring → test-authoring.md

Core Process

The workflow authoring guidance is split by topic:

• Authoring rules — template structure, parameters, outputs, hooks, linting, ArgoCD ownership.
• Common patterns — topic-grouped reference: image sync and pollers, concurrency and semaphores, publishing results to GitHub, when traps, CronWorkflows, storage and node safety, desktop QA probes, BuildStream sizing.

Common Rationalizations

| "The sub-template will see workflow.parameters directly." | It will not. Argo Workflows scopes parameters per-template. Always pass explicitly. | | "I applied the template with kubectl — it's fine." | ArgoCD selfHeal will overwrite it within minutes. Use git. | | "The lint passed locally, I'll skip CI." | CI runs against the same offline linter. If it passed locally, it passes in CI. | | "The template is DEPRECATED, I'll clean it up later." | It will never get cleaned up. Delete it now — prune: true handles the rest. | | "I need each step in the chain to have its own when guard." | Use a dag with depends: "prior.Succeeded" — downstream tasks cascade-omit automatically. |

Red Flags

• Host Namespace Crash Risk: Running containers with hostPID: true or hostIPC: true (which was previously thought to be needed by loopback installation, but is an anti-pattern). If the container fails or is terminated/deleted, argoexec will tear down and SIGTERM all processes in the host namespace, crashing k3s, sshd, and the node. Avoid hostPID: true entirely. Ensure containers-storage and loopback visibility are achieved via volume mounts without host PID exposure.
• Permission Denied Risk: Forgetting to mount an emptyDir at /tmp for containers running as non-root (1000) that need to write results, scripts, or temporary configs to /tmp. Inside bootc images, /tmp is root-owned and restricted, leading to immediate execution failures.
• Adding a separate log aggregation stack (Loki, Promtail, Vector, etc.) alongside Argo — Argo Server already retains pod logs for the workflow TTL. A separate stack duplicates storage, adds pods/PVCs, and creates a Helm-outside-ArgoCD installation with GitOps debt. argo logs covers the same use case.
• Outage Risk: Leaving nodes cordoned (SchedulingDisabled) after k3s upgrades or manual interventions. This completely blocks system pods (including CoreDNS!) from scheduling, causing cluster-wide DNS timeouts (read udp i/o timeout) and a silent, complete cluster outage. Always ensure nodes are uncordoned (kubectl uncordon) and Ready.
• Outage Risk: Setting low memory limits (under 2Gi) for any runner/script step that performs large file transfers (e.g. copying 400MB+ Flatcar update payloads over SCP/kubectl cp). File caching and transfer buffers will instantly trigger the container OOM-killer (exit code 137). Always set memory limits to at least 2Gi–4Gi for transfer-heavy steps.
• synchronization.semaphore: (singular) in any pipeline — deprecated, rejected by ArgoCD schema. Use synchronization.semaphores: (list with - configMapKeyRef: item)
• spec.schedule: (singular) on a CronWorkflow — field does not exist in CRD schema; use spec.schedules: (array)
• A pipeline with VMs and no spec.activeDeadlineSeconds — a stuck VM holds its semaphore slot forever
• A VM pipeline that adds a semaphore without a documented cross-workflow capacity need — ordinary VM concurrency is handled by virt-launcher memory requests and does not need a lock

Verification

Before marking any WorkflowTemplate change done:

• Nested GNOME QA targets force gnome-shell --headless and enable loginctl enable-linger for the test user before starting GDM
• Nested GNOME QA targets make the qecore-headless dconf ownership check tolerate a vanished /run/user/<uid>/dconf/user, and the patch is asserted (fail the provisioning step if it does not apply) rather than best-effort
• Nested GNOME QA targets grant the test user /dev/uinput by mode, not by group membership alone, and install setuptools alongside qecore
• All VM-running pipelines have spec.activeDeadlineSeconds set
• All queueable templates/dynamic workflows (e.g. build-containerdisk and digest-watch submit payloads) have a generous workflow-level activeDeadlineSeconds (e.g., 14400s / 4h) to avoid queue starvation
• Any new CronWorkflow uses spec.schedules: (array), not spec.schedule: (singular)
• Any scheduled workflow sharing a VM namespace or scarce runner uses a documented ConfigMap-backed template semaphore; clock separation alone is not treated as serialization
• All sub-template calls include explicit arguments: blocks
• Pipeline has onExit: cleanup handler
• All pod-running templates have resources: requests and limits
• Every node-pinned, high-memory shared template has a ConfigMap-backed template-level semaphore sized to the node's allocatable capacity
• Any semaphore intended for one DAG task is attached to a wrapper template, not directly to dag.tasks[]; the wrapper's steps call the external templateRef
• VM pipelines are scheduler-managed by default; if cross-workflow memory contention requires serialization, use a documented template-level ConfigMap semaphore with an explicit capacity