host-lan-overlay-rebuild

name	host-lan-overlay-rebuild
description	Safely rebuild or recreate IfcPipeline control-plane containers (redis, postgres, seaweedfs, api-gateway, workers) on the primary VM without breaking remote workers. Use when rebuilding, recreating, or running docker compose up/build for the ifcpipeline stack, or after a recreate left remote workers with Connection refused.

IfcPipeline control-plane rebuilds with the host-lan overlay

IfcPipeline is a multi-host stack. The control plane (redis, postgres, seaweedfs, api-gateway) runs on the primary VM at PIPELINE_LAN_IP=192.168.101.195. Remote workers run on bimbotw1 (192.168.109.54) and connect to redis/postgres/seaweedfs over the LAN. (Object storage is SeaweedFS on port 8333; MinIO was decommissioned 2026-06.)

docker-compose.host-lan.yml adds the PIPELINE_LAN_IP port bindings; docker-compose.yml does not. Running docker compose up -d without the overlay makes Compose recreate redis/seaweedfs/postgres back to loopback-only, stripping the LAN bindings and breaking every remote worker with Error 111 ... Connection refused. This happens even when you only target an unrelated service like api-gateway, because Compose reconciles dependencies.

Golden rule

On the primary host, ALWAYS pass both compose files:

cd /home/bimbot-ubuntu/apps/ifcpipeline
docker compose -f docker-compose.yml -f docker-compose.host-lan.yml <build|up -d> <service>

Rebuild workflow

- [ ] 1. cd /home/bimbot-ubuntu/apps/ifcpipeline
- [ ] 2. build (if image/code changed) WITH overlay
- [ ] 3. up -d WITH overlay
- [ ] 4. verify LAN bindings present
- [ ] 5. verify remote workers still connected

Step 2 + 3 — build and recreate (example: api-gateway):

docker compose -f docker-compose.yml -f docker-compose.host-lan.yml build api-gateway
docker compose -f docker-compose.yml -f docker-compose.host-lan.yml up -d api-gateway

Step 4 — verify LAN bindings (must show 192.168.101.195):

docker ps --format '{{.Names}}: {{.Ports}}' | grep -E 'ifcpipeline-(redis|seaweedfs)|postgres_objectstorage'

Step 5 — verify remote worker queues are registered:

for q in ifcclash ifcdiff ifcpatch ifctester; do
  echo -n "$q: "; docker exec ifcpipeline-redis-1 redis-cli --raw SCARD "rq:workers:$q"; done

Recovery: LAN bindings were stripped

cd /home/bimbot-ubuntu/apps/ifcpipeline
docker compose -f docker-compose.yml -f docker-compose.host-lan.yml up -d redis postgres seaweedfs

Then on bimbotw1, restart any worker stuck in reconnect backoff (it will not self-heal from deep backoff):

cd /home/bimbot-w1/apps/ifcpipeline
docker compose -f docker-compose.remote-workers.yml --env-file .env.remote restart <worker>

DOCKER-USER iptables allow-rules (worker VM allowlist on 6379/5432/8333) persist across recreates — only the published ports vanish, so re-applying the overlay restores access.

Firewall: restrict LAN ports to the worker VM (don't break internal traffic)

The control-plane LAN ports must be locked to the worker VM, but the rules must be scoped to the external interface. With net.bridge.bridge-nf-call-iptables=1 (default), container-to-container traffic on the docker bridge ALSO traverses DOCKER-USER, so an interface-agnostic rule like iptables -A DOCKER-USER -p tcp --dport 5432 -j DROP silently drops api-gateway->postgres, n8n->postgres, workers->redis, etc. Symptom: internal connections time out (not "refused"); n8n crash-loops with "Could not establish database connection".

Apply / repair with the idempotent script (scopes every rule to the NIC holding PIPELINE_LAN_IP, e.g. eth0):

cd /home/bimbot-ubuntu/apps/ifcpipeline
sudo ./scripts/apply-host-lan-firewall.sh

Diagnose a suspected drop (timeout = dropped, refused = path OK):

docker exec ifcpipeline-api-gateway-1 python3 -c \
 'import socket;s=socket.socket();s.settimeout(4);s.connect(("postgres",5432));print("OK")'

Worker-VM rebuilds (bimbotw1)

Remote-worker images bake shared/ + tasks.py. After changing them, sync to the worker repo and rebuild WITH the env file:

cd /home/bimbot-w1/apps/ifcpipeline
docker compose -f docker-compose.remote-workers.yml --env-file .env.remote build <worker>
docker compose -f docker-compose.remote-workers.yml --env-file .env.remote up -d <worker>