dify-docker-architecture

Layout and service names below follow Dify 1.17.x (checked against the `1.17.1` tag). Older releases have fewer services and a monolithic `.env.example`; read the target release's own files when in doubt. ## Directory Layout Dify is cloned from `https://github.com/langgenius/dify`. The Docker deployment lives in the `docker/` subdirectory: ``` dify/ # Git repo root (DIFY_ROOT) ├── api/ # Backend API source ├── web/ # Frontend source ├── docker/ # Docker deployment (DOCKER_DIR) │ ├── docker-compose.yaml # GENERATED by generate_docker_compose — do not edit │ ├── docker-compose-template.yaml # Source the compose file is generated from │ ├── generate_docker_compose # Regenerates docker-compose.yaml │ ├── docker-compose.override.yaml # Optional, yours: Compose reads it, upstream git ignores it │ ├── docker-compose.middleware.yaml # Middleware only, for developing Dify from source │ ├── docker-compose.e2b.yaml # E2B sandbox variant (since 1.17.0) │ ├── .env.example # Essential startup settings (about 240 keys) │ ├── .env # User's configuration (gitignored), read last — wins │ ├── envs/ # Optional settings, grouped (see below) │ ├── dify-env-sync.sh # Official env sync script (also dify-env-sync.py) │ ├── env-backup/ # Timestamped .env backups (gitignored) │ ├── nginx/ # nginx.conf.template, proxy.conf.template, https.conf.template, │ │ └── conf.d/ # conf.d/default.conf.template (the server config is generated from it at start) │ ├── ssrf_proxy/ # squid.conf.template, squid-agent.conf.template, squid-common.conf.template │ ├── certbot/ # Certbot helper files │ └── volumes/ # Persistent data (created at runtime) │ ├── db/data/ # postgres │ ├── redis/data/ │ ├── app/storage/ # uploaded files, generated SECRET_KEY │ ├── plugin_daemon/ │ ├── weaviate/ # default vector store │ ├── sandbox/ │ └── certbot/ ├── .github/ └── ... ``` **Working directory detection:** Before any operation, detect whether the user is in: - `dify/` root — has `docker/` subdirectory with `docker-compose.yaml` - `dify/docker/` — has `docker-compose.yaml` and `.env.example` directly Git operations run from `dify/` root. Docker and .env operations run from `dify/docker/`. Customizations belong in `.env`, `envs/*.env` and `docker-compose.override.yaml`, never in the generated `docker-compose.yaml`. ## Docker Compose Services Requires **Docker Compose >= 2.24.0** (`docker compose version`); the compose files use `env_file … required: false`. | Service | Image | Port | Purpose | |---------|-------|------|---------| | `init_permissions` | `busybox` | — | One-shot: sets storage permissions, then exits. `Exited (0)` in `docker compose ps` is normal | | `api` | `langgenius/dify-api` | 5001 (internal) | Backend REST API, webhooks | | `api_websocket` | `langgenius/dify-api` | — | Workflow collaboration WebSocket; profile `collaboration` (since 1.14.1) | | `worker` | `langgenius/dify-api` | — | Celery async task worker | | `worker_beat` | `langgenius/dify-api` | — | Celery Beat scheduler | | `web` | `langgenius/dify-web` | 3000 (internal) | Next.js frontend UI | | `plugin_daemon` | `langgenius/dify-plugin-daemon` (`-local` tag) | 5002 (internal) | Plugin execution service | | `agent_backend` | `langgenius/dify-agent-backend` | internal | Agent runs (since 1.16.0) | | `local_sandbox` | `langgenius/dify-agent-local-sandbox` | 5004 (internal) | Agent shell sandbox (since 1.16.0) | | `agent_ssrf_proxy` | `ubuntu/squid` | 3128 (internal) | Forward proxy for the agent sandbox (since 1.16.1) | | `sandbox` | `langgenius/dify-sandbox` | 8194 (internal) | Code-node execution sandbox | | `ssrf_proxy` | `ubuntu/squid` | 3128 (internal) | SSRF protection proxy | | `nginx` | `nginx` | `EXPOSE_NGINX_PORT` (80), `EXPOSE_NGINX_SSL_PORT` (443) | Reverse proxy, SSL | | `redis` | `redis:6-alpine` | 6379 (internal) | Cache + Celery broker | | `db_postgres` | `postgres:15-alpine` | 5432 (internal) | PostgreSQL (profile `postgresql`, default) | | `db_mysql` | `mysql:8.0` | 3306 (internal) | MySQL (profile `mysql`, when `DB_TYPE=mysql`) | | `weaviate` | `cr.weaviate.io/semitechnologies/weaviate:1.39.2` | 8080 (internal) | Default vector database, selected by `VECTOR_STORE`; one service per supported store | | `certbot` | `certbot/certbot` | — | Certificate renewal (profile `certbot`) | **Notes:** - That is 15–16 services in a default start. Count against `docker compose ps`, not against a fixed number. - `api`, `api_websocket` and `worker` use the **same image** — different startup mode - The plugin daemon service listens on 5002 (`PLUGIN_DAEMON_PORT`, `PLUGIN_DAEMON_URL`) - Port 5003 is the plugin remote-debugging port, published on the host through `EXPOSE_PLUGIN_DEBUGGING_PORT` - `COMPOSE_PROFILES` controls which database, vector store and optional services start - Weaviate: before 1.17.1 the image was `semitechnologies/weaviate:1.27.0`. The 1.17.1 move to 1.39.2 needs a staged volume upgrade — see the `update-workflow` skill, "Weaviate Migration Path" - `db_postgres` has a health check enabled ## Container Naming Convention Docker Compose derives the project name from the directory that holds the compose file (there is no `name:` in the file): - If compose file is in `/opt/dify/docker/`, project name = `docker` - Containers are named `{project}-{service}-1` (e.g., `docker-api-1`, `docker-web-1`) - If using `-p custom` or `COMPOSE_PROJECT_NAME`, then `custom-api-1` - Network: `{project}_default` Read the project name from the labels, not by parsing container names: ```bash docker ps --filter "label=com.docker.compose.service=api" \ --format '{{.Label "com.docker.compose.project"}}' docker compose ls # every project and its compose file ``` Add `--filter "label=com.docker.compose.project.working_dir=<docker-dir>"` when several Dify stacks share a host. ## .env.example and envs/ Structure Since 1.14.1 the configuration is split. The root `.env.example` carries only what a default deployment needs to start (about 240 keys). Optional, provider-specific and service-specific settings live in templates under `envs/`: | Path | Holds | |------|-------| | `envs/core-services/` | `shared.env.example` (the largest, 500+ keys), `api`, `web`, `worker`, `worker-beat`, `sandbox`, `plugin-daemon`, `dify-agent`, `local-sandbox` | | `envs/databases/` | `db-postgres`, `db-mysql`, `redis` | | `envs/infrastructure/` | `nginx`, `ssrf-proxy`, `certbot`, `etcd`, `minio`, `milvus-standalone` | | `envs/vectorstores/` | One file per vector store (`weaviate`, `qdrant`, `milvus`, `pgvector`, …) | | `envs/security.env.example` | Credentials and tokens of optional integrations | | `envs/middleware.env.example` | For `docker-compose.middleware.yaml` | To use one, copy it without the `.example` suffix (`cp envs/vectorstores/milvus.env.example envs/vectorstores/milvus.env`). Compose loads `envs/**/*.env` when present (`required: false`) and reads `.env` **last**, so a value in `.env` beats the same key in `envs/`. `*.env` files are gitignored. Main sections of the root file: | Section | Key Variables | Notes | |---------|--------------|-------| | **Security** | `SECRET_KEY`, `INIT_PASSWORD` | Empty `SECRET_KEY` (since 1.14.1): the API generates a key and stores it in `volumes/app/storage`; back that directory up, never change the key after start | | **Server** | `LOG_LEVEL`, `MIGRATION_ENABLED`, `DEPLOY_ENV` | MIGRATION_ENABLED=true by default | | **Database** | `DB_TYPE`, `DB_USERNAME`, `DB_PASSWORD`, `DB_HOST`, `DB_PORT`, `DB_DATABASE` | Default: postgres/difyai123456 — change for production | | **Redis** | `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD` | DB 0: cache, DB 1: Celery | | **Storage** | `STORAGE_TYPE`, `OPENDAL_SCHEME`, `OPENDAL_FS_ROOT` | Default: `STORAGE_TYPE=opendal`, `OPENDAL_SCHEME=fs`. S3/Azure and others are in `envs/` | | **Vector DB** | `VECTOR_STORE`, `WEAVIATE_ENDPOINT`, `WEAVIATE_API_KEY` | Default: weaviate. Other stores in `envs/vectorstores/` | | **URLs** | `CONSOLE_API_URL`, `CONSOLE_WEB_URL`, `SERVICE_API_URL` | Leave empty for same-domain | | **Plugin Daemon** | `PLUGIN_DAEMON_PORT`, `PLUGIN_DAEMON_URL`, `EXPOSE_PLUGIN_DEBUGGING_PORT` | Daemon 5002, debugging 5003 | | **Agent** | `DIFY_AGENT_API_TOKEN`, `DIFY_AGENT_SERVER_SECRET_KEY` | Development defaults — change them in production | | **Nginx** | `EXPOSE_NGINX_PORT`, `EXPOSE_NGINX_SSL_PORT`, `NGINX_PORT` | Host ports are the `EXPOSE_*` pair; `NGINX_PORT` is the container-internal port | | **Compose** | `COMPOSE_PROFILES` | Default `${VECTOR_STORE:-weaviate},${DB_TYPE:-postgresql},collaboration` | `COMPOSE_PROFILES` is not derived automatically: an `.env` written before 1.14.1 has the value without `collaboration`, and then `api_websocket` never starts. ## Database Migrations - `MIGRATION_ENABLED=true` (default) — the API entrypoint runs `flask upgrade-db` on every start - Manual: `docker compose exec -T api flask upgrade-db`; other Dify CLI commands run the same way (`docker compose exec -T api flask <cmd>`) - **No downgrade support** — once upgraded, rolling back requires restoring the database from a backup taken with the stack down (see the `update-workflow` skill) - A logical dump needs `pg_dumpall`, not `pg_dump` of one database, because plugins keep their data in a second database (`dify_plugin`): `docker compose exec -T db_postgres pg_dumpall -U "$DB_USERNAME" > <backup-dir>/postgres.sql`. With `DB_TYPE=mysql` the commands differ (`mysqldump` in `db_mysql`) ## Critical Facts 1. **Forward-only updates** — Dify does not support version downgrades due to migration incompatibilities 2. **Volume mounts** — `api`, `api_websocket` and `worker` share `volumes/app/storage` for file uploads 3. **COMPOSE_PROFILES** — controls which services start; a missing profile silently skips its services (`collaboration`, the vector store, the database) 4. **Weaviate** — Python client v4 needs server 1.27+. The bundled server image is `cr.weaviate.io/semitechnologies/weaviate:1.39.2` from Dify 1.17.1; always stop it with `docker compose stop -t -1 weaviate`, never `docker kill` or `docker rm -f` 5. **Nginx** — the server config is generated from `default.conf.template` when the container starts; change the template or the `NGINX_*` variables, not the generated file 6. **Generated compose file** — `docker-compose.yaml` is overwritten by every release; keep customizations in `.env`, `envs/` and `docker-compose.override.yaml`