| name | docker-compose-patterns |
| description | Guides expert-level docker compose patterns implementation: cloud and automation decision frameworks, production-ready patterns, and concrete templates for docker compose patterns workflows.
Use when the user asks about docker compose patterns, docker compose patterns configuration, or devops best practices for docker projects.
Do NOT use when the user needs a different devops cloud capability -- check sibling skills in the devops cloud subcategory.
|
| license | Apache-2.0 |
| metadata | {"author":"foundry-skills","version":"1.0.0","tags":"devops cloud automation","category":"devops-cloud","subcategory":"devops-cloud","depends":"","disclaimer":"none","difficulty":"intermediate"} |
Docker Compose Patterns
When to Use
Use this skill when:
- User is designing or refactoring a
docker-compose.yml file for local development, CI/CD, or staging environments
- User asks how to structure multi-service applications with Docker Compose (e.g., app + database + cache + reverse proxy)
- User wants to manage multiple environments (dev/staging/prod) using Compose overrides or profiles
- User needs guidance on secrets management, health checks, networking, volume strategies, or dependency ordering in Compose
- User asks about service scaling, rolling restarts, or blue-green-style patterns achievable with Compose
- User wants to optimize Compose for CI pipeline speed (layer caching, build contexts, parallel startup)
- User is troubleshooting a broken Compose setup (services not starting, networking issues, volume permission errors, dependency cycles)
Do NOT use this skill when:
- User needs Kubernetes orchestration patterns -- use a Kubernetes-specific skill instead (Compose is not a replacement for Kubernetes in production at scale)
- User is asking about standalone
docker build or docker run commands without a Compose context
- User needs infrastructure-as-code for cloud resources (Terraform, Pulumi, CDK) -- Compose only manages containers, not cloud infrastructure
- User is asking about Docker Swarm mode (Compose v3 YAML is reused but the deployment model is entirely different -- use a Swarm-specific skill)
- User needs service mesh, mTLS, or advanced traffic management -- Compose lacks sidecar proxy support; recommend a Kubernetes + Istio/Linkerd path
- User is working in a fully managed container platform like AWS ECS with Fargate -- while Compose-to-ECS adapters exist, the production patterns diverge significantly
- User needs multi-region active-active deployment -- Compose has no concept of availability zones or regional failover
Process
1. Establish the Compose Topology
Before writing a single line of YAML, map the services and their relationships:
- Identify all services and categorize them: application services (stateless), data services (stateful), infrastructure services (reverse proxy, queue, cache), and tooling services (migration runners, seed scripts, test runners)
- Draw the dependency graph explicitly -- which services must be healthy before others start? A typical web app has:
db → migrations → api → frontend. Circular dependencies are impossible; if you perceive one, a service boundary is wrong
- Classify network exposure: which services need to be reachable from the host (published ports), which only need to communicate service-to-service (internal networks only), and which must be completely isolated
- Identify volume needs: which services write persistent state (databases, uploaded files, logs), which need shared read-only assets (static files mounted into multiple services), and which should use ephemeral storage (caches that should reset on restart)
- Count environment-specific differences: list every value that changes between dev/staging/prod -- these become variables in
.env files or Compose override files, never hardcoded in the base docker-compose.yml
2. Choose the File Structure Pattern
Select the right file organization strategy before writing configuration:
- Single-file pattern (appropriate for: ≤4 services, single environment, personal projects): one
docker-compose.yml with all configuration. Keep it simple; resist premature abstraction
- Base + override pattern (appropriate for: 2+ environments, team projects):
docker-compose.yml as the canonical service definition, docker-compose.override.yml automatically applied in dev (mounts source code, exposes debug ports), docker-compose.prod.yml applied explicitly for production-like runs. Use docker compose -f docker-compose.yml -f docker-compose.prod.yml up
- Profile-based pattern (appropriate for: optional services that vary by role): use the
profiles key to group services. Core services have no profile (always start). Optional services like mailhog, pgadmin, or load-testing are in named profiles and started only with --profile tooling
- Include-based pattern (Compose v2.20+ / Docker Compose plugin): use
include: to compose multiple separate Compose files into one logical application. Useful when sub-teams own service definitions independently. Not widely supported in older CI environments -- check the Docker Compose version available
Decide on .env file strategy:
- Use
.env for non-sensitive defaults (image tags, port numbers, service names)
- Never commit secrets to
.env -- use .env.example with placeholder values checked into version control
- In CI, inject secrets via environment variables or a secrets manager, not from files
3. Design the Network Topology
Docker Compose networking determines both security posture and service discoverability:
- Default network: every
docker compose up creates one default bridge network. All services join it automatically. Service names become DNS hostnames. This is sufficient for simple projects
- Segmented networks (recommended for production-like environments): create explicit named networks and assign services to only the networks they need
frontend_net: reverse proxy ↔ application containersbackend_net: application containers ↔ databases, caches, queues- Only the reverse proxy is on both networks (it bridges them)
- Databases must NOT be on
frontend_net; they should never be reachable from the proxy layer
internal: true flag: mark backend networks as internal (no internet access) for database and cache networks. This prevents a compromised application container from making outbound calls on the database network- Published ports: only publish ports to the host when human or external-tool access is needed (e.g., port 5432 published for a developer's database GUI). Never publish database ports in staging/prod -- use
docker compose exec or a bastion approach instead
- IPv6: Compose supports IPv6 but requires explicit
enable_ipv6: true on network definitions and an address pool. Only configure this if your application explicitly needs it
4. Configure Health Checks and Dependency Ordering
The depends_on key alone does not wait for a service to be ready -- it only waits for the container to start. Use health checks to enforce true readiness:
5. Manage Secrets and Configuration
Never put credentials in docker-compose.yml directly. Apply a layered configuration strategy:
6. Set Resource Limits and Isolation
Compose files without resource limits allow a single misbehaving service to starve all others:
mem_limit and cpus (Compose v2 syntax, still works with Compose plugin):
mem_limit: 512m
cpus: "0.5"
deploy.resources (Compose v3 syntax, preferred for compatibility with Swarm):
deploy:
resources:
limits:
cpus: "0.5"
memory: 512M
reservations:
cpus: "0.25"
memory: 256M
- Typical sizing guidelines for development stacks:
- PostgreSQL: 512M--1G memory, 0.5--1 CPU
- Redis: 128M--256M memory, 0.25 CPU
- Node.js/Python API: 256M--512M memory, 0.5 CPU
- Java/JVM service: 512M--2G memory, 1 CPU (set
-Xmx to 80% of container limit)
- Nginx/Traefik reverse proxy: 64M--128M memory, 0.1 CPU
- OOM kill behavior: when a container exceeds
mem_limit, it receives SIGKILL. Set mem_swappiness: 0 to disable swap for containers that must not degrade gracefully
- Log limits: always configure log rotation to prevent disk exhaustion
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
7. Optimize Builds for Development Speed and CI
Build performance directly impacts developer productivity and CI costs:
8. Write the Operational Runbook Inline
Production-ready Compose configurations document their own operation:
Output Format
When producing a Docker Compose configuration recommendation, structure the response as follows:
Service Topology Diagram
[host:80/443]
│
[reverse-proxy] ── frontend_net ──> [api:3000]
│
backend_net ──┼──> [postgres:5432]
└──> [redis:6379]
Environment File Template (.env.example)
APP_IMAGE_TAG=latest
APP_PORT=3000
NODE_ENV=development
POSTGRES_HOST=db
POSTGRES_PORT=5432
POSTGRES_DB=myapp_dev
POSTGRES_USER=myapp
POSTGRES_PASSWORD=changeme_in_production
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_PASSWORD=
NGINX_HOST_PORT=80
Base Compose File (docker-compose.yml)
version: "3.9"
volumes:
postgres_data:
driver: local
redis_data:
driver: local
networks:
frontend_net:
driver: bridge
backend_net:
driver: bridge
internal: true
x-logging: &default-logging
driver: json-file
options:
max-size: "10m"
max-file: "3"
x-restart-policy: &default-restart
[, ]
[, , ]
[, ]
[, , , , ]
Development Override File (docker-compose.override.yml)
services:
api:
build:
dockerfile: Dockerfile.dev
target: development
command: ["npm", "run", "dev"]
environment:
NODE_ENV: development
LOG_LEVEL: debug
volumes:
- ./src:/app/src:cached
- /app/node_modules
ports:
- "9229:9229"
db:
ports:
- "${POSTGRES_EXPOSED_PORT:-5432}:5432"
redis:
ports:
- "6379:6379"
pgadmin:
image: dpage/pgadmin4:7.8
Decision Matrix
| Pattern | Use When | Avoid When | Complexity |
|---|
| Single file | ≤4 services, 1 environment | Multiple environments, team projects | Low |
| Base + override | Team project, 2+ envs | Very simple local tooling | Medium |
| Profile-based | Optional services (tools, test DBs) | All services required always | Low-Medium |
| Include-based | Sub-team service ownership, monorepo | Older CI (check Compose version) | Medium-High |
| Secrets file mount | Any credential in staging/prod | Dev secrets (overkill) | Low |
| YAML anchors | Repeated config blocks (3+ services) | 1-2 repetitions (just copy) | Low |
Rules
-
NEVER use condition: service_started for database dependencies. This is the default behavior and it does not wait for the database to accept connections. Always define a healthcheck on the database service and use condition: service_healthy. Failure to do this causes race-condition startup failures that are intermittent and hard to debug.
-
NEVER hardcode credentials in docker-compose.yml. Use ${VARIABLE:?error message} syntax for required secrets -- the :? syntax causes Compose to fail immediately with a clear error if the variable is unset, rather than starting with an empty password.
-
ALWAYS set restart: "no" explicitly on one-shot services (migration runners, seed scripts, test runners). If omitted, Compose defaults may cause the service to restart in a loop after it exits, consuming resources and polluting logs.
-
NEVER mount the entire project root into a container. volumes: - .:/app mounts .git/, .env, credentials, and IDE files into the container. Mount only the directories the service needs (./src:/app/src). Use explicit anonymous volumes to protect container-specific directories like node_modules from being overwritten by host mounts.
-
ALWAYS specify image versions with a specific tag, never latest. latest is not a version -- it is an alias that changes when the upstream maintainer decides to update it. Unpinned images cause environments to silently diverge. Pin to a version + variant combination like postgres:15.4-alpine3.18.
-
NEVER put both image: and build: in the same service definition without understanding the semantics. When both are present, image: specifies the name to tag the built image with. docker compose up will not automatically pull -- it will use the local image. In CI, explicitly run docker compose build before docker compose up to ensure the image is current.
-
ALWAYS configure log rotation. Without logging.options.max-size and max-file, a verbose service can fill a disk within hours. Every service that might produce high log volume (any application container) must have log limits configured.
Edge Cases
1. Volume Permission Errors on Linux Hosts
On Linux, container processes run as specific UIDs, but named volumes are created with root ownership by default. A service running as uid=1000 (common for Node.js or Python images) cannot write to a volume owned by uid=0.
Resolution approach:
- Option A (entrypoint fix): add an entrypoint script that runs
chown -R appuser:appgroup /data before launching the main process. This requires the entrypoint to briefly run as root (user: root at container start), then exec su-exec appuser <main_command>
- Option B (init container): add a companion one-shot service with
user: root that runs chown on the volume, then use depends_on: condition: service_completed_successfully to gate the main service
- Option C (tmpfs for ephemeral needs): if the data is not persistent, use
tmpfs: /data instead of a volume -- tmpfs is always writable by the container user
- Avoid: changing volume permissions by making the container run as root. This removes a security boundary and is a vulnerability in shared environments.
2. Networking Conflicts with Host or VPN
When Docker creates bridge networks, it assigns subnets from the 172.16.0.0/12 range by default. Many corporate VPNs and office networks also use this range, causing routing conflicts where Docker traffic is sent through the VPN tunnel instead of the local bridge.
Resolution approach:
- Explicitly configure network subnets outside the conflicting range:
networks:
backend_net:
driver: bridge
ipam:
config:
- subnet: 192.168.200.0/24
- For a team-wide issue, configure Docker's default address pool in
/etc/docker/daemon.json:
{ "default-address-pools": [{"base":"10.200.0.0/16","size":24}] }
- Document the chosen subnets in the project README so future developers don't reassign conflicting ranges
3. Services With Slow Initialization (JVM, .NET Runtime, Large Models)
JVM services may take 30--90 seconds to start and bind their health endpoint. The default healthcheck.start_period of 0s means the first few health check failures count against retries, potentially killing the service before it has a chance to start.
Resolution approach:
- Set
start_period to 110% of the measured cold-start time. For a JVM service that takes 60 seconds to start in CI: start_period: 75s
- Use a lightweight health endpoint (
/actuator/health for Spring, /health for most frameworks) that does not do deep dependency checks on startup -- save dependency checks for readiness probes
- In development, set
interval: 10s to reduce CPU overhead from repeated health checks during startup; in CI, use interval: 5s for faster feedback
- Consider adding a
JAVA_TOOL_OPTIONS: "-XX:TieredStopAtLevel=1" environment variable in the dev override to disable JIT compilation during development, reducing startup time by 50% at the cost of throughput
4. Running Compose in CI/CD Pipelines
CI environments introduce constraints not present in development: no persistent layer cache between runs (unless configured), no Docker Desktop, potentially rootless Docker, and the need for deterministic teardown.
Resolution approach:
- Explicitly pull base images before building:
docker compose pull --ignore-pull-failures caches image layers, then docker compose build uses the cached layers
- For GitHub Actions: use
actions/cache with the Docker layer cache exporter: type=gha,mode=max
- Always run cleanup:
docker compose down -v --remove-orphans as a CI teardown step, even on failure (use if: always() in GitHub Actions). The -v flag removes anonymous volumes that would otherwise accumulate and fill the CI runner disk
- Use
docker compose run --rm service_name test_command to run tests in the Compose network, then exit without leaving a stopped container behind
- Set
COMPOSE_PROJECT_NAME to a CI-run-specific value (e.g., the PR number) when running multiple simultaneous CI jobs on the same host to prevent project-name collisions
5. macOS Performance Problems with Bind Mounts
Docker Desktop on macOS runs containers in a Linux VM. Bind mounts that cross the macOS/VM boundary are notoriously slow for I/O-intensive workloads. A Node.js project with 50,000 files in node_modules performing module resolution on every hot-reload can be 10x slower than the same workload on native Linux.
Resolution approach:
6. Database Schema Migrations as Compose Services
Running migrations as a Compose service is powerful but introduces failure modes if not handled carefully.
Resolution approach:
- The migration service must be idempotent -- running it twice must produce the same result as running it once. All major migration frameworks (Flyway, Liquibase, Alembic, Sequelize) are idempotent by default; verify your tool maintains this guarantee
- If the migration fails (exits non-zero), the dependent API service must not start. With
condition: service_completed_successfully, Compose enforces this correctly
- During development, migrations run on every
docker compose up. In staging/prod, trigger migrations as a separate deploy step before rolling out new application images
- Never run migrations with the same database user as the application. Create a
migrator user with DDL privileges and a appuser with only DML (SELECT, INSERT, UPDATE, DELETE). This limits the blast radius of SQL injection vulnerabilities
7. Multi-Architecture Images (ARM64 / Apple Silicon)
Developers on Apple Silicon (M1/M2/M3) Macs run an ARM64 Linux VM. Many legacy or enterprise images are only built for linux/amd64, causing significant performance degradation through QEMU emulation (commonly 5--10x slower CPU performance).
Resolution approach:
- Always specify the platform explicitly for images that don't have ARM64 variants to make the performance tradeoff visible and intentional:
platform: linux/amd64
- For in-house application images, build multi-arch images using
docker buildx build --platform linux/amd64,linux/arm64 --push. This eliminates the emulation penalty
- Use the
DOCKER_DEFAULT_PLATFORM=linux/amd64 environment variable as a developer-machine workaround when many services lack ARM64 images, so you don't need to annotate each service
- When a service is slow in development on Apple Silicon and fast on Linux CI, suspect platform emulation before optimizing application code
Example
Input: "I'm building a SaaS web application with a React frontend, a Node.js/Express API, PostgreSQL, and Redis. I have a team of 5 developers. We need dev, staging, and prod configurations. Some developers use macOS M2, some use Linux. We've been struggling with database race conditions on startup and slow hot-reload on macOS."
Output:
Diagnosis and Pattern Selection
This project maps to the base + override pattern with YAML anchors for shared configuration. The two specific problems reported -- database race conditions and macOS hot-reload performance -- both have known, specific solutions.
Root cause of race conditions: the previous Compose file almost certainly used depends_on without condition: service_healthy. The fix is to add a proper PostgreSQL health check and wire the API's depends_on to condition: service_healthy.
Root cause of macOS hot-reload slowness: the node_modules directory was likely included in the bind mount. Shielding it with an anonymous volume will resolve this.
File Layout
myapp/
├── docker-compose.yml # Base: service definitions, networks, volumes
├── docker-compose.override.yml # Dev: auto-applied, hot-reload, exposed ports
├── docker-compose.prod.yml # Prod-like: immutable images, no debug ports
├── docker-compose.ci.yml # CI: parallel-safe, deterministic teardown
├── .env.example # Template for all required variables
├── Makefile # Developer convenience commands
├── Dockerfile # Production multi-stage build
├── Dockerfile.dev # Development build with nodemon
└── nginx/
└── nginx.conf # Reverse proxy configuration
.env.example
APP_IMAGE_TAG=latest
API_PORT=3000
NODE_ENV=development
POSTGRES_DB=myapp_dev
POSTGRES_USER=myapp
POSTGRES_PASSWORD=changeme_for_local_dev
POSTGRES_EXPOSED_PORT=5433
REDIS_PASSWORD=
NGINX_HOST_PORT=80
docker-compose.yml (Base)
version: "3.9"
x-logging: &logging
driver: json-file
options:
max-size: "10m"
max-file: "3"
x-api-base: &api-base
image: myapp-api:${APP_IMAGE_TAG:-latest}
build:
context: .
dockerfile: Dockerfile
environment:
PORT: ${API_PORT:-3000}
DATABASE_URL: postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}
REDIS_URL: redis://:${REDIS_PASSWORD:-}@redis:6379
networks:
- frontend_net
- backend_net
logging: *logging
volumes:
postgres_data:
driver: local
redis_data:
driver:
{ , }
[, ]
{ , }
{ , }
[, , ]
[, ]
{ , }
{ , }
[, ]
{ , }
docker-compose.override.yml (Development -- auto-applied)
services:
api:
build:
dockerfile: Dockerfile.dev
command: ["npm", "run", "dev"]
environment:
NODE_ENV: development
LOG_LEVEL: debug
volumes:
- ./src:/app/src:cached
- ./package.json:/app/package.json:ro
- /app/node_modules
ports:
- "9229:9229"
db:
ports:
- "${POSTGRES_EXPOSED_PORT:-5433}:5432"
redis:
ports:
- "6379:6379"
pgadmin:
image: dpage/pgadmin4:7.8
profiles: []
Makefile
.PHONY: up down logs restart migrate shell test lint validate
up:
docker compose up --build
up-d:
docker compose up --build -d
down:
docker compose down --remove-orphans
reset:
docker compose down -v --remove-orphans
logs:
docker compose logs -f $(SERVICE)
migrate:
docker compose run --rm migrations
shell:
docker compose exec $(or $(SERVICE),api) sh
test:
docker compose run --rm api npm test
validate:
docker compose config --quiet && echo "Compose config is valid"
lint:
docker compose -f docker-compose.yml -f docker-compose.prod.yml config --quiet
Verification Checklist
After implementing these configurations, verify the following:
| Check | Command | Expected |
|---|
| Startup race condition resolved | docker compose up 3 times | API starts only after DB is healthy every time |
| macOS hot-reload works | Edit a .js file in src/ | Service reloads in <2 seconds |
| Volumes are named | docker volume ls | myapp_postgres_data, myapp_redis_data visible |
| No credential leakage | docker inspect myapp-api | No raw passwords in Env array |
| Log limits active | docker inspect myapp-api | grep LogConfig | max-size: 10m present |
| Backend network is internal | docker network inspect myapp_backend_net | "Internal": true |
| Config validates | make validate | No errors, no warnings |
