query-kubernetes-logs

name	query-kubernetes-logs
description	Query and troubleshoot Anton Kubernetes logs through the OTel DaemonSet to Loki pipeline. Use for incident investigation, workload debugging, recent warning or error searches, Grafana Explore or LogQL queries, checking log collection, or explaining why a pod's logs are missing.

Query Kubernetes Logs

Use this skill to turn a log question into a bounded, evidence-backed Loki query. The normal result is a short report containing the exact time range, LogQL selector, stream count, sample count, relevant labels, and redacted examples or a clear explanation of why no logs were found.

Read first

1. Read the repository AGENTS.md and use the repository's mise exec -- wrapper for cluster commands.
2. Read the Kubernetes logs runbook when the question involves retention, storage, Grafana access, or a missing-log investigation.
3. Read the query catalog for starting queries.
4. Run python3 scripts/validate-log-contract.py --show for the canonical vocabulary, indexed labels, and retention policy.
5. Read the Loki HelmRelease or OTel HelmRelease when a query result conflicts with the configured pipeline.

Treat live command output as current evidence and repository files as the source of truth for intended configuration. Do not claim that a log was collected merely because the manifest looks correct.

Safety and privacy

• Verify the Kubernetes context before querying: mise exec -- kubectl config current-context.
• Read-only commands are the default: kubectl get, describe, logs, flux get, and HTTP GETs to an already-approved local endpoint.
• Treat port-forward as a live networking mutation. Obtain explicit operator approval, bind it to localhost, and close it before finishing the turn.
• Do not run apply, delete, reconcile, suspend, rollout restart, scale, or storage changes from this skill. Hand proposed mutations to the operator for approval.
• When approved, keep the port-forward local. Never expose Loki publicly or print S3, Grafana, or Kubernetes Secret values.
• Logs can contain credentials, cookies, tokens, personal data, and request bodies. Redact those fields in the report and keep samples short.
• Start with a narrow time range and limit; widen only when the evidence justifies it.

Pipeline facts

The production path is:

node /var/log/pods
  -> OTel Collector DaemonSet (observability)
  -> Loki monolithic StatefulSet :3100 (observability)
  -> SeaweedFS S3 bucket loki (storage)
  -> existing Grafana Loki datasource

Important names and behavior:

• Loki service: loki.observability.svc.cluster.local:3100.
• Grafana datasource: Loki, UID loki.
• OTel DaemonSet: otel-collector-opentelemetry-collector-agent.
• The general receiver tails pod files outside airflow and lakehouse. It starts at file end.
• The workflow receiver tails airflow and lakehouse pod files. It starts unseen files at the beginning and stores checkpoints on the node.
• Indexed labels and normalized severities come from scripts/validate-log-contract.py --show; use that output rather than a copied list.
• Pod name, UID, node, image, container ID, and file path are structured metadata, not indexed labels. Filter them with a structured metadata expression such as | k8s_pod_name="loki-0", or use a line filter.
• Retention and missing-severity behavior come from the executable contract; do not infer them from a remembered value or query window.
• Loki has one 20 GiB Longhorn PVC. Chunks and ruler data use the existing SeaweedFS S3 service and loki bucket. SeaweedFS's 10,000 MB volume limit creates more logical volume slots; it is not a Loki retention quota.
• Spark Application History is not a Loki stream. For applications, jobs, stages, executors, or SQL history, use the runbook's History Server path.

Standard workflow

1. Check the path

Run a small health pulse before interpreting an empty query:

mise exec -- kubectl config current-context
mise exec -- flux get ks -A | rg 'cluster-apps|loki|otel-collector'
mise exec -- flux get hr -A | rg 'loki|otel-collector|kube-prometheus-stack'
mise exec -- kubectl get pods -n observability -o wide | rg 'loki|otel-collector|grafana|NAME'

For a missing-log question, also inspect the collector and Loki's recent errors without dumping an unbounded log:

mise exec -- kubectl get ds otel-collector-opentelemetry-collector-agent -n observability
mise exec -- kubectl logs -n observability ds/otel-collector-opentelemetry-collector-agent --since=10m
mise exec -- kubectl logs -n observability loki-0 -c loki --since=10m

2. Query through a local port-forward

After explicit operator approval, start a localhost-only port-forward and keep its process identity so it can be stopped during cleanup.

In one terminal:

mise exec -- kubectl -n observability port-forward --address 127.0.0.1 svc/loki 3100:3100

In another:

curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3100/loki/api/v1/labels | jq

Use Grafana Explore when a human needs to browse context. Use Loki's HTTP API when an agent needs deterministic, scriptable evidence:

curl -fsS --get http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={severity=~"fatal|error"}' \
  --data-urlencode 'since=24h' \
  --data-urlencode 'limit=100' | jq

The response contains streams with a stream label map and values as [nanosecond_timestamp, line] pairs. Count streams and samples before selecting a small number of redacted examples.

3. Build the query in two stages

First select indexed labels. Then narrow with structured metadata or content:

{k8s_namespace_name="observability", severity=~"fatal|error"}
| k8s_pod_name="loki-0"
|~ "(?i)(timeout|failed|panic)"

Do not begin with an unbounded regex over every stream. Do not assume a pod name is an indexed label. Use | json only when the log body is JSON; parse fields for filtering or formatting instead of promoting high-cardinality values to labels.

Useful starting queries:

{severity=~"fatal|error"}
{severity=~"fatal|error|warn", k8s_namespace_name="observability"}
{k8s_namespace_name="observability", k8s_deployment_name="loki"}
{k8s_namespace_name="default", k8s_container_name="api"}
  |~ "(?i)(error|panic|exception|timeout)"
{k8s_namespace_name="flux-system", k8s_deployment_name="helm-controller", severity="error"}
  | json
{k8s_namespace_name="observability", severity="warn"} |= "failed"

Always include since=... or explicit start/end in API requests. Report the query, range, limits, stream/sample counts, and whether the result came from labels, structured metadata, or content matching.

Missing-log diagnosis

If a query is empty, follow this order:

1. Confirm Loki is Ready, its Service has endpoints, and the query time range includes the expected event.
2. Confirm the OTel DaemonSet has one Ready pod per eligible node.
3. Inspect recent collector logs for file discovery, parser failures, checkpoint, queue, retry, export, or HTTP 5xx errors.
4. Inspect the rendered collector ConfigMap for two non-overlapping file_log receivers, a read-only pod-log mount, checkpoint storage, and severity grouping before batching.
5. Query indexed labels first. Then use structured metadata for pod name or a line filter for content.
6. Apply start_at: end only to the general receiver. Airflow and lakehouse files start at the beginning and resume from persistent checkpoints.
7. If Loki reports object-store or writable-volume errors, inspect SeaweedFS S3 health and free capacity without printing credentials:

mise exec -- kubectl logs -n storage -l app.kubernetes.io/component=s3 --since=10m
mise exec -- kubectl get pvc storage-loki-0 -n observability

Do not “fix” a missing-log result by increasing retention or storage before establishing whether ingestion, querying, or object storage is the failing stage.

Agent report format

Return this compact structure:

Status: found | no matching logs | pipeline issue | blocked
Context: <verified Kubernetes context and workload/time window>
Query: <exact LogQL selector and filters>
Range/limit: <time range and limit>
Evidence: <stream count, sample count, key labels, health checks>
Examples: <short, redacted lines with timestamps>
Next step: <one safe follow-up, or the explicitly approved mutation needed>

If no logs match, distinguish “no matching records in this range” from “the pipeline is unhealthy” and from “the collector never replayed the old file.”