debug-container-service

name

debug-container-service

description

Debug, troubleshoot, and fix container-based services (Docker, Podman) configured in this homelab infrastructure repo. Use when a service is broken, crashing, returning errors, or behaving unexpectedly. Triggers on: "X is not working", "X gives an error", "debug service X", "fix container X", "troubleshoot X", service crashes, 500/400 errors from a running service, unexpected behavior after an update, config changes that broke something. Also use when asked to investigate why a service stopped or started failing.

Container Service Debugging

Work through the steps below in order. Stop as soon as the problem is resolved — do not continue to further steps unnecessarily.

Before You Begin

Make sure you have:

• The error message or symptom (ask the user if not provided)
• The service name in category/service-name format (e.g. ai/open-webui, security/traefik, media/video/jellyfin)

The compose file is at docker/<category>/<service>/<service>.yaml. If the user mentions an error but not the service name, ask. If the service is obvious from context, proceed.

If you know the service name but not its category, look it up instead of guessing: list categories with the mcp__infra-mcp__get-container-categories MCP tool, or run find docker -name '<service>.yaml'.

Scope. This skill covers container/compose-level problems. If the root cause turns out to be host-level — host firewall, DNS, storage/permissions on a mount, Proxmox networking, or a down dependency host (e.g. a remote NAS) — stop patching the container, state that clearly, and hand the host-level fix back to the user.

---

Step 0 — Confirm the Service Exists and Is Running

Before analysing logs, confirm the container is actually deployed and running — an empty log stream from a container that never started is a different problem.

docker ps -a --filter "name=<container_name>" --format '{{.Names}}\t{{.Status}}'

• Not listed at all → the service was never started. Check it is registered in config/docker/<hostname>/services.yaml with state: up, then bring it up (scripts/labctl.py service up category/service-name).
• Restarting / Exited → it is crash-looping. The logs (Step 2) will contain the startup error; this is expected, proceed.
• Up → running but misbehaving; proceed normally.

---

Step 1 — Get Compose Config and Identify Services

Use the MCP tool to read the resolved compose configuration:

mcp__infra-mcp__container-service-config(service_name="category/service-name")

From the config, identify:

• Container name (needed for docker logs — usually set explicitly as container_name:)
• Volumes (paths to check for file-based logs)
• Environment variables (may reveal misconfiguration)
• Image tag (needed for source code lookups in Steps 4–7)

Environment variables are loaded from .env files in this precedence order (later overrides earlier; canonical reference: docker/guidelines.md → "Environment Variables Management"):

1. config/docker/.env — common to all hosts
2. config/docker/<hostname>/.env — host-specific
3. config/docker/.env.<service> — service-specific (all hosts)
4. config/docker/<hostname>/.env.<service> — host + service specific

Check indirect dependencies too. Many errors originate upstream of the failing service:

• Traefik — reverse proxy; routing or TLS misconfiguration shows up as connection errors in the app
• Authelia — a 401/403 may be Authelia rejecting the session, not the app itself
• Cloudflared — tunnel connectivity issues affect all externally-exposed services
• Ollama / LiteLLM — if an AI frontend errors, check the model provider first

---

Step 2 — Check Docker Logs (Always Do This First)

Never make config changes before reading the actual logs. The log output reveals:

• The exact error message and exception type
• The file and line number where the error occurs — this identifies which code path is failing, which determines which fix applies
• Whether the problem is in this service or in a dependency it's calling

docker logs <container_name> --tail 100 2>&1
docker logs <container_name> --tail 200 2>&1 | grep -i "error\|exception\|warn\|fail\|traceback"

Or use the labctl wrapper (handles env var substitution):

scripts/labctl.py service logs category/service-name --tail 200
scripts/labctl.py service logs category/service-name --follow
scripts/labctl.py service logs category/service-name --since 30m
scripts/labctl.py service logs category/service-name --timestamps

Key insight from logs: Pay attention to the source location (e.g. module.file:function:line). An error in process_chat vs images_endpoint are different code paths that may require completely different fixes, even if the symptom looks the same.

If docker logs return empty or near-empty output: some services (notably Traefik) write all logs to files rather than stdout. Skip immediately to Step 3 — do not waste tool calls on alternative retrieval methods (e.g. querying an internal API).

If dependent services are suspected, check their logs too.

---

Step 3 — Check File-Based Logs

Some services write additional structured logs to their data volume. Check:

1. Get the host-side volume paths for the container:

   docker inspect <container_name> --format '{{range .Mounts}}{{.Source}}{{"\n"}}{{end}}'
   

2. Look for log files in those paths:

   find <source_path> -name "*.log" -type f | head -20
   find <source_path> -path "*/logs/*" -type f | head -20
   

3. Read recent entries of any log files found.

---

Step 4 — Search Documentation

The GitHub repository URL is usually in a comment at the top of the compose file. Use it to find the <org>/<repo> for the commands below.

Use Context7 to search the service's official documentation for the specific error or feature:

mcp__context7__resolve-library-id(libraryName="<Service Name>", query="<error or feature>")
mcp__context7__query-docs(libraryId="...", query="<specific error message or config option>")

Critical rule: Before adding any environment variable or config option to a service YAML, verify the exact name exists in the source code or official docs. Do not guess or derive variable names from patterns — look them up.

Use the GitHub API to read the actual source for the version in use:

gh api repos/<org>/<repo>/contents/<path>?ref=<version-tag> --jq '.content' | base64 -d | grep -i "<variable>"

The gh-based steps (4–7) need an authenticated CLI. If a gh call fails with an auth or rate-limit error, run gh auth status — if unauthenticated, ask the user to run gh auth login (do not attempt it yourself), and meanwhile fall back to Context7 docs and web search.

---

Step 5 — Search GitHub Issues

If the error message is specific (a stack trace fragment, a quoted error string, an unusual behavior), search the project's issue tracker early. Many errors in popular open-source services are already documented with working fixes.

gh issue list -R <org>/<repo> --search "<key error words>" --state all --limit 10

Read the comments on relevant issues — the fix is often in the comments, not the issue body. Look for:

• Confirmed fixes (env vars, config changes)
• Workarounds
• Which version introduced or fixed the problem

gh issue view <number> -R <org>/<repo> --comments

---

Step 6 — Check Version / Release Notes

If the issue appeared after an update, or if a fix exists in a newer version:

# List available image tags
mcp__infra-mcp__list-container-tags(image="<registry/image>")

# Browse recent release notes
gh api repos/<org>/<repo>/releases --jq '.[0:5] | .[] | {tag: .tag_name, body: .body[:400]}'

Upgrading: If a newer version contains the fix, update the image tag in the service YAML file (docker/category/service/service.yaml) and recreate the container.

Downgrading: If a recent update introduced the regression, note the last known-good version, set the image tag in the service YAML back to it, and recreate the container:

scripts/labctl.py service recreate category/service-name

Then re-check the logs (Step 2) and tell the user the version was pinned, so the regression can be reported upstream.

---

Step 7 — Source Code Inspection and Patch (Last Resort)

Only reach this step after Steps 1–6 have not resolved the issue.

Before making any change: Explain to the user exactly what you intend to change in the source, why it fixes the issue, and what the trade-offs are. Ask for confirmation.

Read the relevant source for the exact deployed version:

gh api repos/<org>/<repo>/contents/<path>?ref=<version-tag> --jq '.content' | base64 -d

# List directory contents
gh api repos/<org>/<repo>/contents/<path>?ref=<version-tag> --jq '.[].path'

Choose the least invasive fix, in this order:

1. Environment variable or config change — no files to mount, most maintainable
2. Mount a patched file via Docker volume — add a bind mount in the service YAML pointing to a patched copy stored in the repo (e.g. ./patched_file.py:/app/path/file.py:ro); document clearly why it exists
3. Custom Docker image — only if a volume mount is not feasible; significantly increases maintenance burden

When adding a service to the proxy network: remove any host port binding for the port Traefik proxies (e.g. 8080:8080). Traefik reaches the container via Docker networking — the host binding is redundant and will conflict if another service uses the same port. Ports needed for direct external access (e.g. BitTorrent, UDP services) should remain.

Always revert wrong fixes before applying the correct one. Accumulating incorrect changes obscures the actual state and makes future debugging harder.

After applying any fix, recreate the container to pick up changes:

scripts/labctl.py service recreate category/service-name
# or
mcp__infra-mcp__container-service-recreate(service_name="category/service-name")

Then re-check the logs (Step 2) to confirm the fix worked.