ci-ubuntu-docker-test

name	ci-ubuntu-docker-test
title	Reproduce CI failures locally via ubuntu Docker
description	Use Docker to reproduce GitHub Actions ubuntu-latest failures on a macOS dev machine, including Node version mismatches, shell-specific signal behavior (dash vs bash), native module mismatches, and pnpm strict install behavior. Validates fixes locally before pushing to save CI round-trips.
version	1.0.0
author	Hermes Agent
tags	["docker","ci","ubuntu","node","pnpm","spawn","signals","troubleshooting"]

Reproduce CI failures locally via ubuntu Docker

Trigger: When a GitHub Actions run on ubuntu-latest fails but the same test passes locally on macOS, OR when you want to verify a fix without burning CI minutes on each iteration.

Why macOS ≠ ubuntu (the gap that costs CI round-trips)

GitHub Actions uses ubuntu-latest runners. The dev machine is almost always macOS. The differences that bite:

Dimension	macOS dev	ubuntu CI
Default /bin/sh	bash (3.2)	dash (0.5)
sleep, yes, cat behavior	BSD	GNU
Node version	often 22+ via brew	pinned via setup-node (the workflow)
pnpm install	hoists deps to ~/node_modules for compat	strict isolation (no hoisting) unless configured
Native modules (rollup, esbuild, better-sqlite3)	macOS ARM64 binaries	rebuilt from source for linux-x64
spawn + kill behavior	bash forwards signals to children	dash ignores SIGTERM at the process level

The signal-handling gap is the one that bit us on 2026-06-01 — see references/case-study-execute-command-timeout.md.

Quick start

# 1. Pull a small ubuntu image (cached after first run)
docker pull node:20-bullseye-slim  # match your CI's setup-node version

# 2. Run a quick isolated repro of a Node behavior
cat > /tmp/test.mjs << 'EOF'
import { spawn } from "node:child_process";
const child = spawn("sh", ["-c", "sleep 60"], { detached: true });
setTimeout(() => { try { process.kill(-child.pid, "SIGTERM"); } catch {} }, 1000);
child.on("close", (code, signal) => {
  console.log(`close at ${Date.now() - start}ms, code: ${code}, signal: ${signal}`);
  process.exit(0);
});
const start = Date.now();
setTimeout(() => { console.log("FAIL: not closed in 10s"); process.exit(2); }, 10000);
EOF
docker run --rm -v /tmp/test.mjs:/tmp/test.mjs node:20-bullseye-slim node /tmp/test.mjs

# 3. For a full repo test, mount the workspace
docker run --rm -v "$PWD:/repo" -w /repo node:20-bullseye-slim sh -c '
  which pnpm || npm install -g pnpm@10.11.0
  pnpm install --frozen-lockfile
  pnpm test
'

Pinned pitfalls (Ubuntu-specific)

1. dash ignores SIGTERM at the process level

$ sh -c "sleep 60"   # default /bin/sh on ubuntu
$ kill -TERM $!      # sends SIGTERM to the sh process
                      # sh (dash) DOES NOT EXIT — sleep 60 keeps running

Reproduce with the quick-start test above. Fix: send to the process group:

// ❌ Does NOT work on ubuntu
const child = spawn("sh", ["-c", "sleep 60"]);
setTimeout(() => child.kill("SIGTERM"), 1000);

// ✅ Works on macOS AND ubuntu
const child = spawn("sh", ["-c", "sleep 60"], { detached: true });
setTimeout(() => process.kill(-child.pid, "SIGTERM"), 1000);

detached: true makes the child a process group leader, so pid === pgid and process.kill(-pid, ...) kills the whole tree (sh + sleep grandchild).

1b. "File missing on CI but passes locally" — reproduce by moving the file aside

This is the other common CI-only failure pattern (untracked files referenced by committed imports — also covered as pattern #1 in monorepo-ci-debugging). Unlike pattern #1a (signal handling, ubuntu-specific), this one reproduces on macOS too — you just need to simulate the CI clean checkout:

# 1. Pick the file mentioned in the "Cannot find module" error
# 2. Move it aside (simulates CI not having it)
mv packages/<pkg>/src/<file>.ts /tmp/<file>.ts.bak

# 3. Nuke build artifacts + cache
rm -rf packages/*/dist .turbo packages/dashboard/.next

# 4. Run the failing CI step
pnpm lint --force
# Expected: same "Cannot find module '<file>.js'" error as CI

# 5. Restore
mv /tmp/<file>.ts.bak packages/<pkg>/src/<file>.ts

Why this works: locally, tsc --noEmit reads the working tree, including untracked files. CI does git checkout first, which leaves the working tree clean — untracked files disappear. Moving the file aside reproduces that state. git ls-files --error-unmatch only tells you it's untracked; it doesn't prove CI will fail.

After reproducing, the fix is almost always git add + commit + push — no code changes needed. Hit this twice in the 2026-06-01 promptqueue session (event-bus.ts, then og-client.ts); both were the same "committer forgot to git add" mistake. Pattern recognition: when you see a recent commit message like feat: add X but the actual x.ts is untracked, that's the pattern.

2. Always escalate to SIGKILL after a grace period

Even with process group kill, dash sometimes needs ~200ms to propagate. Use a 2-stage timer:

setTimeout(() => {
  try { process.kill(-child.pid, "SIGTERM"); } catch {}
  setTimeout(() => {
    try { process.kill(-child.pid, "SIGKILL"); } catch {}
  }, 200).unref();
}, timeoutMs);

.unref() prevents the second timer from keeping the event loop alive after close.

3. spawn({ timeout: ms }) and manual setTimeout race

Node's spawn timeout option sends SIGKILL (not SIGTERM) at exactly the timeout. If your code also has a manual setTimeout that sends SIGTERM at the same instant, you can get:

• One signal wins, the other is dropped
• The child receives only SIGKILL → close event fires with no graceful shutdown
• Or the manual kill fires first and gets swallowed by the spawn-timeout

Fix: pick one mechanism. Use only the manual setTimeout (with the detached + process group kill pattern above). Drop the timeout option from spawn.

4. pnpm strict isolation hides @types/node

Locally, pnpm may hoist @types/node from transitive deps to ~/node_modules so every package can see it. CI's pnpm install --frozen-lockfile does not hoist by default — each package must declare @types/node explicitly in its own devDependencies.

Symptom in CI:

src/providers/cli-provider.ts(1,29): error TS2307: Cannot find module 'node:child_process' or its corresponding type declarations.

Fix: add to the affected package's devDependencies:

"@types/node": "^20.0.0"  // match the CI runner's node version

5. Native module ABI mismatch

If you run pnpm test inside a docker container that was pnpm install-ed on the host, you'll see:

Error: Cannot find module '...rollup/dist/native.js'
code: 'MODULE_NOT_FOUND'

Cause: rollup's native binary is built for the host OS (macOS ARM64) but the container is linux-x64. Fix: pnpm install inside the container too, or use --ignore-scripts and rebuild only what you need:

docker run --rm -v "$PWD:/repo" -w /repo node:20-bullseye-slim sh -c '
  npm install -g pnpm@10.11.0
  pnpm install --frozen-lockfile
  pnpm test
'

Verification checklist before pushing

After fixing a CI failure, run this locally:

# 1. Nuke all build artifacts and cache
rm -rf packages/*/dist packages/*/.next .turbo packages/dashboard/tsconfig.tsbuildinfo

# 2. Force-fresh run of each CI job
pnpm lint --force   # turbo bypasses cache, all packages truly re-run
pnpm test --force
pnpm build --force

# 3. Grep for any error text in any output
pnpm lint --force 2>&1 | grep -iE "error|fail|✖" | head -5
pnpm test --force 2>&1 | grep -iE "error|fail|✖" | head -5
pnpm build --force 2>&1 | grep -iE "error|fail|✖" | head -5

# 4. For signal-handling fixes specifically, validate in docker:
docker run --rm -v /tmp/test.mjs:/tmp/test.mjs node:20-bullseye-slim node /tmp/test.mjs

When NOT to use

• The CI failure is in a domain-specific dependency (e.g. better-sqlite3 not building on ubuntu) — you can still reproduce, but fixing may need a Dockerfile change
• The failure is in a third-party action (e.g. pnpm/action-setup@v4 cache issues) — reproduce in a clean ubuntu VM, not docker
• The failure is a network/registry issue (e.g. npm install hitting a private registry) — docker doesn't help

Real example: the 2026-06-01 promptqueue debug session

This skill exists because the user's promptqueue project hit a string of CI failures over ~5 rounds, where each round cost 5-10 minutes of GitHub Actions time. The breakthrough came from running the failing test in a node:20-bullseye-slim container and observing:

$ docker run --rm -v /tmp/test.mjs:/tmp/test.mjs node:20-bullseye-slim node /tmp/test.mjs
Testing 'sh -c sleep 60' with manual SIGTERM at 1s...
  Timer fired at 1005ms, kill returned: true, pid: 18
  TIMEOUT: not closed in 8s, killing -9

The "kill returned: true" was the critical clue — it meant the kill syscall succeeded but the process ignored the signal. That's a shell-specific behavior, not a Node bug. Only by running the same script on ubuntu (where /bin/sh is dash) did this become visible.

For the full timeline and the other 5 CI fixes from that session, see references/case-연구-promptqueue-ci-debug.md.

A follow-up debug session on 2026-06-02 hit the same project with the same shape of error (untracked file referenced by committed imports) — see references/case-연구-og-client-untracked.md. Different root cause class, but same diagnostic workflow applies.

Related skills

• we-mp-rss-troubleshooting — Docker container debugging (for the wechat-mp-rss service)
• cron-job-prompt-recovery — Recovering broken cron jobs (one of the 6 fixes in the case study)
• hermes-tool-corruption-pitfalls — execute_code + read_file cache issues