docker-compose-patterns

# Docker Compose Patterns ## When to Use **Use this skill when:** - User is designing or refactoring a `docker-compose.yml` file for local development, CI/CD, or staging environments - User asks how to structure multi-service applications with Docker Compose (e.g., app + database + cache + reverse proxy) - User wants to manage multiple environments (dev/staging/prod) using Compose overrides or profiles - User needs guidance on secrets management, health checks, networking, volume strategies, or dependency ordering in Compose - User asks about service scaling, rolling restarts, or blue-green-style patterns achievable with Compose - User wants to optimize Compose for CI pipeline speed (layer caching, build contexts, parallel startup) - User is troubleshooting a broken Compose setup (services not starting, networking issues, volume permission errors, dependency cycles) **Do NOT use this skill when:** - User needs Kubernetes orchestration patterns -- use a Kubernetes-specific skill instead (Compose is not a replacement for Kubernetes in production at scale) - User is asking about standalone `docker build` or `docker run` commands without a Compose context - User needs infrastructure-as-code for cloud resources (Terraform, Pulumi, CDK) -- Compose only manages containers, not cloud infrastructure - User is asking about Docker Swarm mode (Compose v3 YAML is reused but the deployment model is entirely different -- use a Swarm-specific skill) - User needs service mesh, mTLS, or advanced traffic management -- Compose lacks sidecar proxy support; recommend a Kubernetes + Istio/Linkerd path - User is working in a fully managed container platform like AWS ECS with Fargate -- while Compose-to-ECS adapters exist, the production patterns diverge significantly - User needs multi-region active-active deployment -- Compose has no concept of availability zones or regional failover --- ## Process ### 1. Establish the Compose Topology Before writing a single line of YAML, map the services and their relationships: - **Identify all services** and categorize them: application services (stateless), data services (stateful), infrastructure services (reverse proxy, queue, cache), and tooling services (migration runners, seed scripts, test runners) - **Draw the dependency graph** explicitly -- which services must be healthy before others start? A typical web app has: `db` → `migrations` → `api` → `frontend`. Circular dependencies are impossible; if you perceive one, a service boundary is wrong - **Classify network exposure**: which services need to be reachable from the host (published ports), which only need to communicate service-to-service (internal networks only), and which must be completely isolated - **Identify volume needs**: which services write persistent state (databases, uploaded files, logs), which need shared read-only assets (static files mounted into multiple services), and which should use ephemeral storage (caches that should reset on restart) - **Count environment-specific differences**: list every value that changes between dev/staging/prod -- these become variables in `.env` files or Compose override files, never hardcoded in the base `docker-compose.yml` ### 2. Choose the File Structure Pattern Select the right file organization strategy before writing configuration: - **Single-file pattern** (appropriate for: ≤4 services, single environment, personal projects): one `docker-compose.yml` with all configuration. Keep it simple; resist premature abstraction - **Base + override pattern** (appropriate for: 2+ environments, team projects): `docker-compose.yml` as the canonical service definition, `docker-compose.override.yml` automatically applied in dev (mounts source code, exposes debug ports), `docker-compose.prod.yml` applied explicitly for production-like runs. Use `docker compose -f docker-compose.yml -f docker-compose.prod.yml up` - **Profile-based pattern** (appropriate for: optional services that vary by role): use the `profiles` key to group services. Core services have no profile (always start). Optional services like `mailhog`, `pgadmin`, or `load-testing` are in named profiles and started only with `--profile tooling` - **Include-based pattern** (Compose v2.20+ / Docker Compose plugin): use `include:` to compose multiple separate Compose files into one logical application. Useful when sub-teams own service definitions independently. Not widely supported in older CI environments -- check the Docker Compose version available Decide on `.env` file strategy: - Use `.env` for non-sensitive defaults (image tags, port numbers, service names) - Never commit secrets to `.env` -- use `.env.example` with placeholder values checked into version control - In CI, inject secrets via environment variables or a secrets manager, not from files ### 3. Design the Network Topology Docker Compose networking determines both security posture and service discoverability: - **Default network**: every `docker compose up` creates one default bridge network. All services join it automatically. Service names become DNS hostnames. This is sufficient for simple projects - **Segmented networks** (recommended for production-like environments): create explicit named networks and assign services to only the networks they need - `frontend_net`: reverse proxy ↔ application containers - `backend_net`: application containers ↔ databases, caches, queues - Only the reverse proxy is on both networks (it bridges them) - Databases must NOT be on `frontend_net`; they should never be reachable from the proxy layer - **`internal: true` flag**: mark backend networks as internal (no internet access) for database and cache networks. This prevents a compromised application container from making outbound calls on the database network - **Published ports**: only publish ports to the host when human or external-tool access is needed (e.g., port 5432 published for a developer's database GUI). Never publish database ports in staging/prod -- use `docker compose exec` or a bastion approach instead - **IPv6**: Compose supports IPv6 but requires explicit `enable_ipv6: true` on network definitions and an address pool. Only configure this if your application explicitly needs it ### 4. Configure Health Checks and Dependency Ordering The `depends_on` key alone does not wait for a service to be ready -- it only waits for the container to start. Use health checks to enforce true readiness: - **Define `healthcheck` on every data service**: at minimum, databases, caches, and message brokers should declare health checks - PostgreSQL: `test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]`, `interval: 5s`, `timeout: 5s`, `retries: 10`, `start_period: 30s` - MySQL/MariaDB: `test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]` - Redis: `test: ["CMD", "redis-cli", "ping"]`, `interval: 3s`, `timeout: 3s`, `retries: 5` - RabbitMQ: `test: ["CMD", "rabbitmq-diagnostics", "check_port_connectivity"]` - Generic HTTP: `test: ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]` - **Wire `depends_on` with `condition: service_healthy`**: this forces Compose to pause the dependent service startup until the dependency reports healthy ```yaml depends_on: db: condition: service_healthy redis: condition: service_healthy ``` - **Use `condition: service_completed_successfully`** for one-shot services like database migration runners -- the API service should not start until migrations have finished - **Set `start_period`** to accommodate slow-starting services (JVM warmup, database initialization). Failures during `start_period` do not count toward `retries`. For PostgreSQL initializing a large schema, use `start_period: 60s` - **`restart` policies**: use `restart: unless-stopped` for long-running services in development, `restart: on-failure:5` in staging to catch crash loops early without infinite restart loops ### 5. Manage Secrets and Configuration Never put credentials in `docker-compose.yml` directly. Apply a layered configuration strategy: - **Layer 1 -- `.env` file** (non-sensitive, environment-specific defaults): image tags (`APP_IMAGE_TAG=main-abc1234`), port mappings (`POSTGRES_EXPOSED_PORT=5433`), resource limits, feature flags - **Layer 2 -- Environment variable passthrough**: for sensitive values in CI, define the variable in the Compose file as `ENV_VAR:` (no value) -- Docker Compose will read it from the shell environment. This is how CI secrets should flow in - **Layer 3 -- Docker secrets (Compose v3.1+)**: use the `secrets:` top-level key to mount files into containers at `/run/secrets/<secret_name>`. Applications must be written to read from files, not environment variables. This is the most secure approach available in Compose ```yaml secrets: db_password: file: ./secrets/db_password.txt services: db: secrets: - db_password environment: POSTGRES_PASSWORD_FILE: /run/secrets/db_password ``` - **Never do**: `environment: DATABASE_URL: postgres://user:hardcoded_password@db/mydb` -- this leaks credentials into `docker inspect` output, process environment dumps, and version control if accidentally committed - **Config objects** (Compose v3.3+): use `configs:` for non-sensitive configuration files (nginx configs, application config files) that should be mounted read-only into containers without being baked into the image ### 6. Set Resource Limits and Isolation Compose files without resource limits allow a single misbehaving service to starve all others: - **`mem_limit` and `cpus`** (Compose v2 syntax, still works with Compose plugin): ```yaml mem_limit: 512m cpus: "0.5" ``` - **`deploy.resources`** (Compose v3 syntax, preferred for compatibility with Swarm): ```yaml deploy: resources: limits: cpus: "0.5" memory: 512M reservations: cpus: "0.25" memory: 256M ``` - **Typical sizing guidelines for development stacks**: - PostgreSQL: 512M--1G memory, 0.5--1 CPU - Redis: 128M--256M memory, 0.25 CPU - Node.js/Python API: 256M--512M memory, 0.5 CPU - Java/JVM service: 512M--2G memory, 1 CPU (set `-Xmx` to 80% of container limit) - Nginx/Traefik reverse proxy: 64M--128M memory, 0.1 CPU - **OOM kill behavior**: when a container exceeds `mem_limit`, it receives SIGKILL. Set `mem_swappiness: 0` to disable swap for containers that must not degrade gracefully - **Log limits**: always configure log rotation to prevent disk exhaustion ```yaml logging: driver: json-file options: max-size: "10m" max-file: "3" ``` ### 7. Optimize Builds for Development Speed and CI Build performance directly impacts developer productivity and CI costs: - **Use BuildKit**: set `DOCKER_BUILDKIT=1` or use `docker compose build` with the Compose plugin (BuildKit is enabled by default in Docker Desktop 23.0+). BuildKit enables parallel layer builds, better caching, and `--mount=type=cache` in Dockerfiles - **Separate dev and prod Dockerfiles**: `Dockerfile` for production (minimal, multi-stage, no dev dependencies), `Dockerfile.dev` for development (includes hot-reload tools, debuggers, test runners). Reference with `build.dockerfile: Dockerfile.dev` in the override file - **Multi-stage builds in production Dockerfiles**: builder stage (install all deps, compile, run tests) → runtime stage (copy only artifacts, minimal base image). This pattern reduces final image size by 60--80% for compiled languages - **Layer ordering for cache efficiency**: order Dockerfile instructions from least-frequently-changed to most-frequently-changed: OS packages → language runtime → dependency manifests (package.json, requirements.txt) → dependency install → application code copy - **Build context optimization**: always include a `.dockerignore` file. Exclude: `node_modules/`, `__pycache__/`, `.git/`, `*.log`, `.env`, test output directories. A bloated build context is one of the top causes of slow builds - **`cache_from` in CI**: specify previously built images as cache sources to avoid rebuilding unchanged layers ```yaml build: context: . cache_from: - myapp:${CI_COMMIT_BRANCH:-main} ``` - **Bind mounts for development hot-reload**: in the dev override file, mount source code into the container so the process inside can detect file changes. Combine with `inotify`-based file watchers (nodemon, watchexec, air for Go). Use `delegated` consistency mode on macOS to improve I/O performance: `- type: bind, source: ./src, target: /app/src, consistency: delegated` ### 8. Write the Operational Runbook Inline Production-ready Compose configurations document their own operation: - **Add a `# Usage` comment block at the top of every Compose file** with the four most common commands: start, stop, logs, exec into a specific service - **Document all required environment variables** in `.env.example` with descriptions and example values for every variable. A new developer should be able to run the stack in under 5 minutes using only the README and `.env.example` - **Include a `Makefile` or `justfile`** with named targets for common workflows: `make up`, `make down`, `make logs`, `make migrate`, `make test`, `make shell SERVICE=api`. This abstracts the verbose `docker compose -f ... -f ... --env-file ...` commands - **Version-pin base images**: use digests or specific version tags (`postgres:15.4-alpine3.18`), never `latest`. Unpinned images cause silent breakage when upstream updates land - **Label all services** with metadata for tooling and observability: ```yaml labels: com.mycompany.service: "api" com.mycompany.version: "${APP_VERSION}" com.mycompany.team: "platform" ``` --- ## Output Format When producing a Docker Compose configuration recommendation, structure the response as follows: ### Service Topology Diagram ``` [host:80/443] │ [reverse-proxy] ── frontend_net ──> [api:3000] │ backend_net ──┼──> [postgres:5432] └──> [redis:6379] ``` ### Environment File Template (`.env.example`) ```bash # Application APP_IMAGE_TAG=latest APP_PORT=3000 NODE_ENV=development # Database POSTGRES_HOST=db POSTGRES_PORT=5432 POSTGRES_DB=myapp_dev POSTGRES_USER=myapp POSTGRES_PASSWORD=changeme_in_production # Cache REDIS_HOST=redis REDIS_PORT=6379 REDIS_PASSWORD= # Reverse Proxy NGINX_HOST_PORT=80 ``` ### Base Compose File (`docker-compose.yml`) ```yaml # docker-compose.yml # Base service definitions -- environment-agnostic configuration. # Run with: docker compose up # For production: docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d version: "3.9" # ───────────────────────────────────────────────────────────────────────────── # Named volumes for persistent data # ───────────────────────────────────────────────────────────────────────────── volumes: postgres_data: driver: local redis_data: driver: local # ───────────────────────────────────────────────────────────────────────────── # Networks: segmented to limit blast radius of a compromised container # ───────────────────────────────────────────────────────────────────────────── networks: frontend_net: driver: bridge backend_net: driver: bridge internal: true # No internet access -- data services only # ───────────────────────────────────────────────────────────────────────────── # Reusable configuration fragments (YAML anchors) # ───────────────────────────────────────────────────────────────────────────── x-logging: &default-logging driver: json-file options: max-size: "10m" max-file: "3" x-restart-policy: &default-restart restart: unless-stopped services: # ─── Reverse Proxy ────────────────────────────────────────────────────────── nginx: image: nginx:1.25.3-alpine <<: *default-restart ports: - "${NGINX_HOST_PORT:-80}:80" volumes: - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro networks: - frontend_net depends_on: api: condition: service_healthy logging: *default-logging deploy: resources: limits: cpus: "0.2" memory: 64M # ─── Application API ──────────────────────────────────────────────────────── api: image: myapp-api:${APP_IMAGE_TAG:-latest} build: context: . dockerfile: Dockerfile target: production <<: *default-restart environment: NODE_ENV: production PORT: 3000 DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB} REDIS_URL: redis://:${REDIS_PASSWORD}@redis:6379 networks: - frontend_net - backend_net depends_on: db: condition: service_healthy redis: condition: service_healthy migrations: condition: service_completed_successfully healthcheck: test: ["CMD-SHELL", "curl -sf http://localhost:3000/health || exit 1"] interval: 10s timeout: 5s retries: 5 start_period: 30s logging: *default-logging deploy: resources: limits: cpus: "0.5" memory: 512M reservations: cpus: "0.25" memory: 256M # ─── Database Migration Runner (one-shot service) ─────────────────────────── migrations: image: myapp-api:${APP_IMAGE_TAG:-latest} build: context: . dockerfile: Dockerfile target: production