docker-debugging

# Docker Container Debugging Implements comprehensive Docker container debugging workflows for diagnosing crashes, OOM errors, network issues, volume mount problems, resource contention, and optimization issues using Docker CLI commands, inspection tools, and log analysis. ## TL;DR Checklist - [ ] Check container logs with `docker logs --tail 100 <container>` - [ ] Inspect container configuration with `docker inspect <container>` - [ ] Monitor resource usage with `docker stats --no-stream` - [ ] Verify network connectivity with `docker network inspect <network>` - [ ] Check volume mounts with `docker inspect --format='{{.Mounts}}' <container>` - [ ] Review container exit codes and crash patterns - [ ] Use `docker system df` to identify disk space issues - [ ] Check for OOM kills with `dmesg | grep -i 'killed process'` --- ## When to Use Use Docker debugging when: - Container crashes immediately on startup with exit code 137 (OOM) or other non-zero codes - Network connectivity issues between containers or to external services - Volume mounts appear empty or fail silently - Container runs but exhibits unexpected behavior or performance issues - Docker Compose services fail to start or communicate - Resource contention detected (high CPU, memory, or I/O) - Layer caching problems during image builds - Health checks fail intermittently - Container logs are missing or incomplete --- ## When NOT to Use Avoid this skill for: - Application-level bugs inside containers (use code-debugging skills instead) - Infrastructure provisioning issues (use CNCF infrastructure skills) - Kubernetes-specific debugging (use kubernetes-debugging skill) - Cloud provider API issues (use cloud-provider debugging skills) - Network infrastructure outside Docker (use agent-network-troubleshooting for Docker-native issues) - Container orchestration failures (use agent-orchestration-debugging skills) - Performance issues requiring application profiling (use code-profiling skills) --- ## Core Workflow Docker debugging follows a systematic approach to identify and resolve container issues: 1. **Assess Container State** — Check container status, logs, and recent events. **Checkpoint:** Determine if container is running, crashed, or stuck in restart loop. 2. **Inspect Configuration** — Analyze container configuration, environment variables, and mounted volumes. **Checkpoint:** Verify configuration matches expectations and requirements. 3. **Check Network Connectivity** — Test container network connectivity and service discovery. **Checkpoint:** Confirm network policies and DNS resolution are working. 4. **Monitor Resources** — Analyze CPU, memory, and I/O usage patterns. **Checkpoint:** Identify resource constraints or bottlenecks. 5. **Review Logs and Events** — Examine application and system logs for errors. **Checkpoint:** Correlate container events with application errors. 6. **Optimize and Retry** — Apply fixes and restart container with corrected configuration. **Checkpoint:** Container starts successfully and performs as expected. --- ## Implementation Patterns ### Pattern 1: Container Crash Diagnosis (Exit Codes and OOM Errors) Diagnose why containers crash on startup or during execution using exit codes and system logs. ```bash # ✅ GOOD — Check container exit code and recent logs # Check if container is running or crashed docker ps -a --filter "name=myapp" --format "table {{.Names}}\t{{.Status}}\t{{.ExitCode}}" # Get recent logs with timestamps docker logs --tail 100 --timestamps myapp-container 2>&1 | tail -50 # Check for OOM kill in system logs dmesg | grep -i 'killed process' | tail -20 # Or check docker logs for OOM indicator docker logs myapp-container | grep -i 'oom' # Get container details including exit code docker inspect --format='{{.State.Status}} - ExitCode: {{.State.ExitCode}}' myapp-container ``` ```bash # ❌ BAD — Generic container check, missing context docker ps -a # ❌ No filter, shows all containers including historical # ❌ Missing exit code in output # ❌ No log inspection ``` ```bash # ✅ GOOD — OOM diagnosis script #!/bin/bash # Script to diagnose OOM kills CONTAINER_NAME=$1 if [ -z "$CONTAINER_NAME" ]; then echo "Usage: $0 <container-name>" exit 1 fi # Check container exit code EXIT_CODE=$(docker inspect --format='{{.State.ExitCode}}' "$CONTAINER_NAME" 2>/dev/null) if [ "$EXIT_CODE" = "137" ]; then echo "Container $CONTAINER_NAME exited with code 137 (SIGKILL - likely OOM)" # Check system OOM logs echo "=== System OOM Logs ===" dmesg | grep -i 'killed process' | grep "$CONTAINER_NAME" | tail -10 # Check container memory usage echo "=== Container Memory Stats ===" docker stats --no-stream "$CONTAINER_NAME" 2>/dev/null || echo "Container not running" # Check container memory limit echo "=== Container Memory Limit ===" docker inspect --format='{{.HostConfig.Memory}}' "$CONTAINER_NAME" else echo "Container $CONTAINER_NAME exited with code $EXIT_CODE" echo "Check logs: docker logs --tail 100 $CONTAINER_NAME" fi ``` ```bash # ✅ GOOD — Generic crash diagnosis with common exit codes # Exit code 125: Container command could not be invoked # Exit code 126: Command found but not executable # Exit code 127: Command not found # Exit code 134: SIGABRT # Exit code 137: SIGKILL (OOM) # Exit code 143: SIGTERM #!/bin/bash # Crash diagnosis helper CRASH_EXIT_CODES=( "125:Container command could not be invoked" "126:Command found but not executable" "127:Command not found" "134:SIGABRT" "137:SIGKILL (OOM)" "143:SIGTERM" ) for entry in "${CRASH_EXIT_CODES[@]}"; do IFS=':' read -r code description <<< "$entry" echo "Exit code $code: $description" done ``` ```bash # ✅ GOOD — Get comprehensive container crash info docker inspect \ --format=' Status: {{.State.Status}} ExitCode: {{.State.ExitCode}} Error: {{.State.Error}} StartedAt: {{.State.StartedAt}} FinishedAt: {{.State.FinishedAt}} OOMKilled: {{.State.OOMKilled}} Death: {{.State.Dead}} Paused: {{.State.Paused}} Running: {{.State.Running}} ' myapp-container ``` --- ### Pattern 2: Network Troubleshooting (Container Connectivity and DNS) Diagnose network issues between containers, external services, and DNS resolution. ```bash # ✅ GOOD — Check container network connectivity # Check if container is connected to expected networks docker network ls docker network inspect bridge --format='{{range .Containers}}{{.Name}} {{end}}' # Test DNS resolution from inside container docker exec myapp-container nslookup api.external-service.com docker exec myapp-container host api.external-service.com # Test connectivity to other containers docker exec myapp-container ping -c 3 db-container docker exec myapp-container curl -v http://api-container:8080/health # Check firewall rules docker exec myapp-container iptables -L -n ``` ```bash # ❌ BAD — Basic connectivity test without context docker exec myapp-container ping google.com # ❌ No timeout, may hang indefinitely # ❌ No context about what should be reachable # ❌ No DNS check ``` ```bash # ✅ GOOD — Comprehensive network diagnostic script #!/bin/bash # Network diagnostic script for Docker containers CONTAINER=$1 EXPECTED_DNS=$2 TARGET_HOST=$3 if [ -z "$CONTAINER" ]; then echo "Usage: $0 <container> [expected-dns] [target-host]" exit 1 fi echo "=== Container Network Diagnostics ===" echo "Container: $CONTAINER" # Check container networks echo "=== Connected Networks ===" docker inspect --format='{{range $key, $value := .NetworkSettings.Networks}}{{$key}} ({{(index $value.IPCache).IPAddress}}){{end}}' "$CONTAINER" # Test DNS resolution if [ -n "$EXPECTED_DNS" ]; then echo "=== DNS Resolution Test ===" docker exec "$CONTAINER" nslookup "$EXPECTED_DNS" 2>&1 fi # Test host connectivity if [ -n "$TARGET_HOST" ]; then echo "=== Host Connectivity Test ===" docker exec "$CONTAINER" nc -zv "$TARGET_HOST" 80 2>&1 || \ docker exec "$CONTAINER" curl -s --connect-timeout 5 "http://$TARGET_HOST" 2>&1 || \ echo "Connection failed" fi # Check /etc/hosts echo "=== /etc/hosts Contents ===" docker exec "$CONTAINER" cat /etc/hosts # Check /etc/resolv.conf echo "=== DNS Configuration ===" docker exec "$CONTAINER" cat /etc/resolv.conf ``` ```bash # ✅ GOOD — Test Docker Compose service connectivity #!/bin/bash # Test connectivity between Docker Compose services COMPOSE_PROJECT=${1:-default} # Get service names SERVICES=$(docker ps --filter "label=com.docker.compose.project=$COMPOSE_PROJECT" --format '{{.Names}}' | tr '\n' ' ') echo "Services in project $COMPOSE_PROJECT: $SERVICES" # Test each service can reach every other service for src in $SERVICES; do for dst in $SERVICES; do if [ "$src" != "$dst" ]; then echo "Testing $src -> $dst" docker exec "$src" nc -zv "$dst" 80 2>&1 | grep -v "^$" || true fi done done ``` --- ### Pattern 3: Volume Mount Issues (Empty Volumes and Permission Problems) Diagnose volume mount problems including empty directories, permission errors, and path issues. ```bash # ✅ GOOD — Inspect volume mounts and permissions # Check container mounts docker inspect --format='{{range .Mounts}}{{.Name}}: {{.Destination}} ({{.Type}}) {{end}}' myapp-container # Verify host path exists and is accessible docker run --rm -v /host/path:/data alpine ls -la /data # Check volume driver and options docker volume inspect myapp-data --format='{{json .}}' # Test volume permissions from inside container docker exec myapp-container ls -la /app/data docker exec myapp-container touch /app/data/test.txt 2>&1 ``` ```bash # ❌ BAD — Basic mount check without verification docker inspect --format='{{.Mounts}}' myapp-container # ❌ Output is hard to read # ❌ No permission test # ❌ No host path verification ``` ```bash # ✅ GOOD — Volume mount diagnostic script #!/bin/bash # Volume mount diagnostic script CONTAINER=$1 EXPECTED_MOUNT=$2 if [ -z "$CONTAINER" ]; then echo "Usage: $0 <container> [expected-mount-path]" exit 1 fi echo "=== Volume Mount Diagnostics ===" echo "Container: $CONTAINER" # List all mounts echo "=== Container Mounts ===" docker inspect --format='{{range .Mounts}} Name: {{.Name}} Destination: {{.Destination}} Source: {{.Source}} Type: {{.Type}} Readonly: {{.RW}} Driver: {{.Driver}} Options: {{.Options}} {{end}}' "$CONTAINER" # Check if mount exists in container if [ -n "$EXPECTED_MOUNT" ]; then echo "=== Checking $EXPECTED_MOUNT ===" docker exec "$CONTAINER" ls -la "$EXPECTED_MOUNT" 2>&1 || \ echo "Path $EXPECTED_MOUNT not found or not accessible" fi # Check for empty directory (common issue) echo "=== Checking for Empty Mounts ===" docker exec "$CONTAINER" sh -c 'find /app/data -mindepth 1 -maxdepth 1 2>/dev/null | wc -l' | \ while read count; do if [ "$count" -eq 0 ]; then echo "WARNING: /app/data appears empty" else echo "INFO: /app/data contains $count items" fi done # Check permissions echo "=== Volume Permissions ===" docker exec "$CONTAINER" stat /app/data 2>&1 | head -10 || \ echo "Cannot stat /app/data" ``` ```bash # ✅ GOOD — Test bind mount from host #!/bin/bash # Test bind mount from host to container HOST_PATH=$1 CONTAINER_PATH=${2:-/test} if [ -z "$HOST_PATH" ]; then echo "Usage: $0 <host-path> [container-path]" exit 1 fi echo "=== Bind Mount Test ===" echo "Host path: $HOST_PATH" echo "Container path: $CONTAINER_PATH" # Check host path exists if [ ! -d "$HOST_PATH" ] && [ ! -f "$HOST_PATH" ]; then echo "ERROR: Host path does not exist: $HOST_PATH" exit 1 fi # Test mount with temp container CONTAINER_ID=$(docker run -d -v "$HOST_PATH:$CONTAINER_PATH" alpine sleep 300) echo "=== Mount Contents ===" docker exec "$CONTAINER_ID" ls -la "$CONTAINER_PATH" 2>&1 || \ echo "Mount failed or path inaccessible" echo "=== File Count ===" docker exec "$CONTAINER_ID" find "$CONTAINER_PATH" -type f 2>/dev/null | wc -l # Cleanup docker stop "$CONTAINER_ID" >/dev/null 2>&1 docker rm "$CONTAINER_ID" >/dev/null 2>&1 ``` ```bash # ✅ GOOD — Check Docker volume driver # List volumes with driver info docker volume ls --format '{{.Name}}\t{{.Driver}}' # Inspect specific volume docker volume inspect myapp-data --format='Driver: {{.Driver}}\nMountpoint: {{.Mountpoint}}\nOptions: {{.Options}}' # Check if volume is local or remote (nfs, cifs, etc.) docker inspect --format='{{range .Mounts}}{{.Driver}} {{end}}' myapp-container ``` ---