| name | investigate-aether |
| description | Interactively investigate Aether outages, alerts, regressions, performance issues, failed deployments, missing telemetry, backup failures, authentication problems, and network/security questions using live read-only evidence, including human follow-up on an Inquest incident. Use Grafana as the primary correlation surface across Prometheus metrics, Loki logs, Tempo traces, ClickHouse Zeek/Suricata data, Kubernetes state, Fleet, Talos, SSH, and service APIs. This is not the unattended Grafana-to-Kestra-to-Holmes-to-GitLab pipeline owned by sibling ../inquest; consume its incident record when present, independently verify its RCA, and stop before remediation or live mutation. |
Investigate Aether
Investigate from current evidence, not repository speculation. Start narrow, correlate independent sources, and remain read-only.
Read First
- Read the root
AGENTS.md for shell, authentication, Kubernetes-context, and live-patching rules.
- Use
$navigate-aether if runtime, placement, or ownership is unclear.
- Read only the source reference needed for the next question:
- Grafana: dashboards, alerts, datasource discovery, API access.
- Prometheus: metrics, targets, labels, retention, known metric families.
- Loki and Tempo: logs, traces, indexed labels, coverage limits.
- Kubernetes and hosts: cluster events/logs, Talos, Fleet, SSH.
- Network security: ClickHouse, Zeek, Suricata, Hubble, Tetragon.
- Inquest: automated alert intake, incident correlation, and pipeline-failure follow-up.
- Investigation playbooks: common symptom-to-evidence paths.
Inquest Boundary
- Sibling
../inquest owns unattended alert intake, fan-out,
fingerprint dedupe, Kestra flows, the GitLab so/aether/incidents lifecycle,
and the Holmes call. Aether owns the platform and integration those flows use.
- This skill is the human-invoked read-only path. For an Inquest alert or issue,
start with its fingerprint and issue timeline, treat the Holmes comment as an
untrusted hypothesis, then corroborate it with live evidence.
- Do not create, update, or close Inquest issues; invoke or retry Holmes; or
rerun/test Kestra flows unless the user explicitly asks to operate or debug
Inquest. A pipeline failure itself may still be investigated read-only.
Investigation Workflow
1. Frame the question
Record:
- affected component and user-visible impact
- expected behavior
- first and last known timestamps with timezone
- relevant change/deployment window
- whether the request is diagnosis-only or includes remediation
Use absolute timestamps in findings. Begin with a short range around the event and expand only when evidence requires it.
2. Verify access and target
nix develop -c task login:status
nix develop -c kubectl config current-context
Check Kubernetes context only for cluster work; it must be admin@aether-k8s. Do not refresh otherwise-working credentials preemptively. Resolve host aliases and addresses from inventory/shared config, never memory.
task login is the unified AWS, Google WIF, OpenBao, Ceph RGW, and SSH flow. Use task login -- --ssh when only the SSH certificate is missing. Use task k8s:auth only to repair wrong/stale Kubernetes or Talos configuration because it overwrites local kubeconfig and talosconfig.
3. Start at Grafana
- Check Grafana health.
- Search dashboards and inspect the closest existing dashboard.
- Summarize active alerts; identify the intentional
DeadMansSwitch separately.
- Discover datasource UIDs dynamically.
- Query one backend selected by the symptom.
Use the tested read-only helper from the repo root:
nix develop -c .agents/skills/investigate-aether/scripts/grafana-read.bb health
nix develop -c .agents/skills/investigate-aether/scripts/grafana-read.bb dashboards
nix develop -c .agents/skills/investigate-aether/scripts/grafana-read.bb alerts
Do not fan out across every datasource before forming a hypothesis.
4. Select evidence
| Question | First evidence | Corroborate with |
|---|
| Availability or saturation | Prometheus/Grafana dashboard | Loki, Kubernetes, SSH/service API |
| Application error | Loki | Prometheus, Tempo, Kubernetes state |
| Cross-service latency | Tempo when instrumented | Prometheus latency plus Loki |
| Failed Kubernetes rollout | Workload status and targeted events | Loki/pod logs, Prometheus restarts |
| VM/LXC/host failure | Prometheus/PVE metrics | SSH journal, systemd, Podman, Fleet |
| Network or IDS question | ClickHouse Zeek/Suricata | Hubble/Tetragon, DNS/proxy logs |
| Backup failure | Operation/API and backup metrics | job logs, storage/lock state |
| Missing data | target and collector health | exporter/agent logs and live endpoint |
| Inquest incident follow-up | GitLab incident issue and fingerprint | Grafana alert plus independent live telemetry |
Missing telemetry is an evidence gap, not proof of recovery or inactivity.
5. Correlate a timeline
For every claimed cause, capture at least two of:
- a state transition or alert timestamp
- a metric change over the same window
- a matching log/event/trace
- a failed dependency or endpoint
- a deployment/configuration change
Treat old failed pods, high-volume logs, repeated image-pull-secret warnings, and persistent security review alerts as possible background noise until they match impact and timing.
6. Stop before mutation
Investigation commands are read-only. Do not patch, restart, roll out, apply, delete, mute, acknowledge, or change dashboards/alerts. If remediation is requested, present the IaC path and follow root AGENTS.md; any live patch still requires explicit approval with command, impact, rollback, and verification.
Output Contract
Lead with the current state and root cause:
## Current State
## Impact
## Timeline
## Evidence
## Root Cause
## Secondary Symptoms
## Confidence and Gaps
## Recommended IaC Remediation
## Verification Plan
For a health/status request, give the concise status first. State which live surfaces were queried and which were unavailable. Never include tokens, decrypted secrets, credentials, or unnecessary sensitive log payloads.
