ops

name	ops
version	1.0
last_updated	2026-07-29
id	ops
one_line_purpose	Diagnose lab and CI failure signatures for testsuite runs.
entry_point	docs/skills/ci-ops/ops/SKILL.md
category	ci-ops
mcp_compliance_level	partial
status	active
dependencies	[]
tags	["ops","lab","debugging","gdm","argo"]
description	Operational gotchas and failure signatures for the testsuite lab/CI setup. Load when debugging e2e failures, GDM, oomd, Argo, or runner issues.
metadata	{"type":"pattern","audience":"agents","maturity":"stable"}

Operational Gotchas

# → success
# /etc/gdm/custom.conf
# WRONG — tests the container's DNS, not the VM's
# CORRECT — tests DNS inside the VM
# must return nothing
# Listener running?
# Jobs being picked up? (empty arc-runners is healthy — minRunners=0)
# Runner logs (catch before pod completes)
# WRONG — creates scratch/smoke.png/ directory, cp fails silently
# CORRECT — pull to dir, then find the PNG inside
# edit /tmp/ruleset.json, then PUT
# .github/actions/my-action/action.yml

When to Use

• Container-QA lanes fail in wait_for_shell.py with bus-unavailable
• VM boots to GDM greeter instead of a GNOME session
• Debugging infra-layer CI failures (runner container, D-Bus, AT-SPI)
• Adding new packages or patches to container/Containerfile.runner
• SSH assertion failures from unexpected output
• common-suite service health scenarios fail with unexpected ActiveState values
• Polkit rules presence check returns zero results
• A scheduled or push-only workflow is red and you need to find which merge broke it
• Session readiness fails with ServiceUnknown then Could not connect: No such file or directory

When NOT to Use

• Writing behave step logic → docs/skills/test-authoring/behave/SKILL.md
• GNOME AT-SPI/dogtail patterns → docs/skills/test-authoring/gnome/SKILL.md
• bootc lifecycle steps → docs/skills/test-authoring/bootc/SKILL.md
• Workflow inputs, migration runs, manual.yml → docs/skills/ci-ops/e2e-workflow/SKILL.md

---

Red Flags

• Using _run(cmd) in smoke suite for DNS or network checks (runs on container, not VM)
• Raising the wait_for_shell.py readiness timeout to fix bus-unavailable (the socket is absent indefinitely, not late — find why no session registers)
• Assuming a bus-unavailable readiness failure is a testsuite bug (the client re-resolves the bus every attempt; a permanently missing /run/user/1000 is lab-side)
• Setting sys.exit(1) inside before_scenario (kills all subsequent scenarios silently)
• Lowering the SSH timeout below 60 seconds (hardware commands are slow in QEMU)
• Adding a second -monitor flag to QEMU (breaks )