multi-env-deploy-and-e2e-testing

name	multi-env-deploy-and-e2e-testing
description	Multi-environment deployment isolation + end-to-end testing discipline. Use when deploying to multiple targets (Docker/Cloudflare/GitHub), fixing cross-environment issues, or verifying web UI changes. Prevents: breaking other environments while fixing one, false-positive testing, repeated failures.

Multi-Environment Deploy & E2E Testing

Lessons from a session where fixing Docker broke Cloudflare, and telling the user "fixed" 5 times without actually verifying.

When to Load This Skill

• Deploying to 2+ environments (Docker, Cloudflare, GitHub Pages, VPS, etc.)
• Fixing web UI issues (links, navigation, rendering)
• User reports "still broken" after you said "fixed"
• Modifying files that affect multiple build targets

---

Part 1: Environment Isolation

The Problem

Source files (docs/) feed multiple environments. Changing them affects all. Build artifacts (site/) are shared. Config files (Dockerfile, nginx.conf, wrangler.toml) are environment-specific but live in the same repo.

The Rule: Know Your Blast Radius

Before modifying ANY file, classify it:

SOURCE (affects ALL environments):
  docs/           Markdown source files
  mkdocs.yml      Build config
  requirements.txt
  scripts/        Build scripts

ARTIFACT (temporary, rebuilt from source):
  site/           MkDocs build output
  out/            Next.js build output
  public/         Static assets

DOCKER-ONLY:
  Dockerfile
  nginx.conf
  docker-compose.yml

CLOUDFLARE-ONLY:
  wrangler.toml
  _headers
  _redirects

GITHUB-ONLY:
  .github/workflows/

SHARED CONFIG (affects multiple):
  package.json    May affect build
  tsconfig.json   May affect build

Before Modifying: Impact Checklist

□ Which environments will this change affect?
□ Do I need to rebuild/redeploy all environments or just one?
□ Will this break the other environments' build or runtime?
□ Do I need to test in all environments after the change?

Deployment Isolation Pattern

deploy/
├── docker/
│   ├── Dockerfile
│   ├── nginx.conf
│   └── docker-compose.yml
├── cloudflare/
│   └── wrangler.toml
└── github/
    └── workflow.yml

scripts/
├── build.sh        # Shared build
├── deploy.sh       # Master deploy
└── deploy-env.sh   # Per-env deploy

The site/ Directory Trap

Never directly edit build artifacts. They get overwritten on next build.

❌ sed -i '' 's|old|new|g' site/*.html     # Lost on rebuild
✅ sed -i '' 's|old|new|g' docs/*.md       # Persists
✅ Then rebuild: mkdocs build

The Split-Brain Build Trap (Docker vs Cloudflare)

Docker compose builds INSIDE the container — it does NOT touch the local site/ directory. Cloudflare Pages (wrangler pages deploy site) deploys FROM the local site/. These are independent build paths.

docs/*.md  ──→  Docker build (inside container)  ──→  localhost:8002 ✅
docs/*.md  ──→  local mkdocs build                ──→  site/          (must run manually!)
site/      ──→  wrangler pages deploy              ──→  CF Pages      (deploys whatever is in site/)

Anti-pattern: Edit .md → docker compose up -d --build → wrangler pages deploy site → CF Pages shows OLD content.

Correct workflow:

# 1. Edit source
vim docs/some-file.md

# 2. Rebuild LOCAL site/ (NOT just Docker!)
docker run --rm -v "$(pwd):/build" -w /build <builder-image> mkdocs build
# OR: .venv/bin/python -m mkdocs build

# 3. Verify locally
docker compose up -d --build
# Playwright: http://localhost:8002

# 4. Deploy Cloudflare (from fresh local site/)
npx wrangler pages deploy site --project-name=<name>

# 5. Playwright verify CF preview URL BEFORE telling user it's done

Multi-Env Deploy Workflow

# 1. Make source changes
vim docs/some-file.md

# 2. Rebuild
./scripts/build.sh

# 3. Test locally first
./scripts/deploy.sh docker --build
# → verify at localhost:8002

# 4. Deploy to other environments
./scripts/deploy.sh cloudflare
./scripts/deploy.sh github

# 5. Verify ALL environments
curl -sI http://localhost:8002/page       # Docker
curl -sI https://cf.pages.dev/page        # Cloudflare
curl -sI https://user.github.io/repo/page # GitHub

---

Part 2: End-to-End Testing Discipline

The Problem

Using curl or browser_navigate(direct_url) to verify fixes, when the actual bug is triggered by clicking a link on a page. This gives false positives — you report "fixed" when it's not.

The Rule: Test What The User Tests

If the user says "I click X and it goes to Y", your test must be:

1. Open the page containing X
2. Click X
3. Verify it goes to Y

NOT:

• curl http://target-url (bypasses all client-side logic)
• browser_navigate(target-url) (bypasses link resolution)
• Check if file exists on disk (irrelevant to user experience)

Playwright E2E Pattern

# ✅ CORRECT: Simulate user action
browser_navigate("http://localhost:8002/ch01-ai-basics")
browser_click(link_ref)    # Click "001. Article Title"
browser_console("window.location.href")  # Verify URL changed
browser_snapshot()         # Verify content loaded

# ❌ WRONG: Direct URL access
browser_navigate("http://localhost:8002/ch01-001-article")
# This bypasses the link click — doesn't test the actual bug

Port Stability Check

When testing in Docker (localhost:8002), verify links don't change the port:

// After clicking a link, check:
JSON.stringify({
  url: window.location.href,
  port: window.location.port,  // Must still be "8002"
  title: document.title,
  h1: document.querySelector('article h1')?.innerText,
  hasContent: document.querySelector('article')?.innerText?.length > 100
})

If port changes (e.g., to 8080), the link is using an absolute URL that bypasses Docker's port mapping.

When curl Is Insufficient

curl tests:     HTTP layer only
                No JavaScript execution
                No link resolution
                No redirects from JS
                No SPA navigation

Playwright tests: Full browser context
                  JavaScript execution
                  Link click behavior
                  Client-side redirects
                  Framework-specific navigation

Verification Checklist

Before saying "fixed":

□ Reproduced the exact user-reported steps
□ Tested with Playwright (not curl)
□ Clicked the link (not direct URL)
□ Verified destination URL matches expected
□ Verified destination page content loads
□ Tested both with and without trailing slash (if relevant)
□ Checked that fix doesn't break other pages
□ Rebuilt/redeployed the affected environment

---

Part 3: The "Fixed" Reporting Protocol

The Problem

Saying "已修复" or "Playwright 验证通过" before actually verifying, causing the user to waste time testing broken fixes repeatedly.

The Rule: Three Levels of Confidence

Level 1: "I made a change" (not verified)
  → "I modified X. Please verify at your end."

Level 2: "I tested it myself" (partially verified)
  → "I tested [specific scenario] and it works. 
     Please verify from your browser."

Level 3: "Fully verified" (end-to-end)
  → "I followed your exact steps: opened page, clicked link, 
     confirmed redirect to correct URL with correct content."

Never claim Level 3 without actually doing Level 3.

The Anti-Pattern

❌ "已修复" (after changing a file)
❌ "Playwright 验证通过" (after direct URL access)
❌ "✅ 正常" (after curl returns 200)

✅ "我改了 nginx.conf，请你清浏览器缓存后访问 
    http://localhost:8002/ch01-ai-basics 点击第一个链接验证"

---

Part 4: Framework-Specific Gotchas

MkDocs Material

• Instant Loading: JavaScript intercepts link clicks for SPA navigation. Direct URL access works differently than clicking links.
• use_directory_urls: Affects link format. false = .html files, true = dir/index.html.
• Build output: site/ directory structure depends on nav config and use_directory_urls.
• CSS base href — DO NOT USE with GH Pages subpath: <base href="/"> in overrides/main.html BREAKS GitHub Pages when served from a subpath (e.g. /wiki-book/). All relative links resolve to root instead of the subpath, causing 404s. With use_directory_urls: false, MkDocs generates relative .html links that work correctly WITHOUT any <base> tag across all environments (root path AND subpath). Fix: Remove <base> from overrides/main.html entirely. If CSS breaks in subdirectory pages, the real fix is using relative .html links in chapter indexes, not <base>.
• Directory flattening: MkDocs generates both flat (ch01-001-xxx.html) and nested (ch01/001-xxx/index.html) outputs. The nested ones have stale links. Fix: remove ch01/...ch20/ directories after build in Dockerfile.
• Link format in nav: Chapter index files link to articles. With use_directory_urls: false, links must be relative .html format: ](actual-filename.html), not ](/ch01-001-slug/) (absolute directory URL). The slug in the link MUST match the actual source filename — if source file is ch01-001-2026年最值得关注的15款开发者工具深度解读.md, the link must be ](ch01-001-2026年最值得关注的15款开发者工具深度解读.html), not ](ch01-001-2026-15.html). See references/link-slug-mismatch-fix.md.
• CSS path resolution: When use_directory_urls: false, relative CSS paths work correctly without <base> tag. Do NOT add <base href="/"> — it breaks GitHub Pages subpath deployment.
• Nav sections create directories: Even with , nav sections like "第一篇 · 入门篇" cause MkDocs to create subdirectories with their own . These duplicate the flat files and have wrong relative links. Fix: delete , etc. after build.

Next.js

• Static export: output: 'export' generates flat files. Links must use absolute paths.
• params is a Promise: In Next.js 16+, params must be awaited.
• Build artifacts: out/ or .next/standalone/ — never edit directly.
• Static export = NO API routes: output: 'export' in next.config.mjs means the project is a static site. API routes (app/api/*/route.ts) will FAIL at build time with: export const dynamic = "force-static"/export const revalidate not configured on route. If you need server-side logic (e.g., serving audio files from disk), either: (1) remove output: 'export' and use a Node.js server, (2) use an external API (e.g., the OpenMAIC Docker container's /api/classroom-media/ endpoint), or (3) copy files to public/ at build time.
• Props threading for toolbar features: When adding a UI control to a deeply nested component (e.g., CanvasToolbar inside CanvasArea inside PlaybackChromeRoot), ALL intermediate components must: (1) add prop to interface, (2) destructure in function params, (3) pass to child. Missing any step causes silent failure — the button simply won't render with no error. Chain: PlaybackChromeRoot → CanvasArea → CanvasToolbar. Each extends the next one's props interface (CanvasAreaProps extends CanvasToolbarProps), so adding to the leaf interface auto-exposes it in ancestors, but each still needs explicit destructuring and pass-through.
• Docker build cache hides code changes: docker compose build may use cached layers even after source files change. The COPY . . layer cache is based on file checksums, which may not update if files are modified in-place. Verification: docker exec <container> grep -r 'your-change' /app/.next/static/chunks/ — if empty, the build used stale cache. Fix (fast): echo "# $(date +%s)" >> .dockerignore to invalidate the cache layer without --no-cache (which rebuilds everything, 10+ min). The .dockerignore change forces Docker to re-evaluate the build context, invalidating the COPY . . layer. Fix (thorough): docker compose build --no-cache. After restart, verify with: — must be >0.

Docker + nginx

• Port mapping: Internal port (8080) ≠ external port (8002). Rewrites can change ports.
• try_files: Order matters. $uri.html before $uri/index.html.
• Build cache: docker compose build uses cache. Use --no-cache for clean builds.

Cloudflare Pages

• Production domain: jinguo.tech (NOT wiki.jinguo.tech). The user corrected this — always use jinguo.tech for production URLs.
• File size limit: 25MB per file. Search indexes often exceed this.
• Search index compression: Cloudflare auto-compresses served files (gzip/brotli). A 21MB search_index.json serves as ~8MB over the wire. But the upload limit is 25MB raw.
• Search index trimming: To fit under 25MB, trim text content to ~80 chars per doc:

  for doc in data['docs']:
      if len(doc.get('text','')) > 80:
          doc['text'] = doc['text'][:80]
  

• No server-side logic: Static files only. No Node.js, no Python.
• Deploy = upload: wrangler pages deploy site/ uploads all files.
• Caching: Cloudflare handles caching automatically. Custom headers via site/_headers file:

  /search/search_index.json
    Cache-Control: public, max-age=86400, immutable
  

MkDocs + Docker/nginx

• use_directory_urls: false: Generates flat .html files instead of dir/index.html. Changes link format.
• <base href="/"> override — REMOVED (was harmful): Previously added to fix CSS loading, but it broke GitHub Pages subpath navigation entirely. All relative links resolved to root / instead of /wiki-book/, causing 404 on every chapter/article link. Root cause: <base href="/"> in overrides/main.html overrides the browser's relative URL resolution. With use_directory_urls: false, MkDocs generates relative .html links that work correctly WITHOUT any <base> tag across all environments (root path AND subpath). Fix applied: Removed <base> entirely from overrides/main.html. The file should contain only {% extends "base.html" %}{% block site_meta %}{{ super() }}{% endblock %} — no <base> tag. If CSS breaks in subdirectory pages, the real fix is using relative .html links in chapter indexes, not <base>.
• Chapter index links: Must use /ch01-001-slug/ format (absolute, no .md), not ch01/001-slug.md (relative with extension).
• custom_dir: overrides: Required in theme: config to use template overrides.
• Dockerfile must COPY overrides/: COPY overrides/ overrides/ before COPY docs/ docs/.

---

Part 5: Numeric Data Propagation (Wiki-Book & MkDocs Sites)

When changing a number that appears in prose (article count, entity count, domain count, etc.), it's almost always in MORE places than you think. The default pattern is: change 2-3 obvious spots, miss 3-4 hidden ones.

The Rule: Grep Before AND After

# BEFORE changing — find ALL occurrences
grep -rn "OLD_NUMBER" docs/ mkdocs.yml README.md

# AFTER changing — verify zero remain (ignore URLs)
grep -rn "OLD_NUMBER" docs/ mkdocs.yml README.md | grep -v "http"

Common Hidden Locations in MkDocs Material

Location	Why it's missed
mkdocs.yml → site_description	Becomes <meta name="description"> — not visible in page body
mkdocs.yml → copyright	Rendered in footer — separate from page content
docs/index.md → footer line	"内容来源：XXX 篇" at bottom, below main table
docs/references.md → intro paragraph	Separate page, separate copy of the number
docs/index.md → <blockquote>	Hero section at top
README.md → body paragraphs	Not just the header table — check ALL prose

Anti-Pattern: Partial Fix

❌ Changed README + index.md → deployed → references still shows old number
✅ grep → found 6 locations → changed all → grep again → 0 matches → deploy

Docker Rebuild Gotcha

docker compose up -d --build may NOT pick up source file changes if the builder stage has cached layers (especially when using a pre-built builder image). For numeric/text content changes:

# Reliable rebuild
docker compose down
docker compose build --no-cache
docker compose up -d

Then verify with Playwright (not curl) that the new content rendered.

Critical: Even after Docker rebuild succeeds, the local site/ is STALE. If deploying to Cloudflare Pages, you MUST rebuild site/ separately (see Split-Brain Build Trap above).

---

Part 6: Debugging Flowchart

User reports issue
  │
  ├─ "Can you reproduce it?" 
  │   ├─ Yes → Test with Playwright (click, not navigate)
  │   └─ No  → Ask for exact steps
  │
  ├─ Identify which environment(s) affected
  │   ├─ Docker only → Modify Dockerfile/nginx.conf
  │   ├─ Cloudflare only → Modify wrangler.toml
  │   ├─ All environments → Modify docs/ source
  │   └─ Check: will this change break others?
  │
  ├─ Make the change
  │   ├─ Source change? → Rebuild all affected environments
  │   └─ Config change? → Rebuild only that environment
  │
  ├─ Verify (LEVEL 3)
  │   ├─ Reproduce user's exact steps
  │   ├─ Use Playwright (click, not navigate)
  │   └─ Check all affected environments
  │
  └─ Report
      ├─ "I tested [steps] and it works. Please verify."
      └─ Never: "已修复" without Level 3 verification

---

Part 7: Minimal-Change Feature Addition

When adding a feature to an existing component (e.g., subtitle overlay to a player), follow the surgical approach:

1. Create new component in isolation — Don't modify existing components until you have the new piece working standalone.
2. Hook into existing state — Don't create new state if existing state already tracks what you need (e.g., lectureSpeech already has narration text).
3. Add 2 lines to the parent — Import + render. No interface changes, no prop threading, no layout restructuring.
4. Don't change the layout — User explicitly said "不要改变课堂的模式" (don't change the classroom mode). The feature should be additive, not restructuring.

Anti-pattern: Rewriting client.tsx from scratch to add a subtitle, which changes the entire layout and breaks the user's expectations.

Pattern: Create subtitle-overlay.tsx standalone → add import + <SubtitleOverlay /> to PlaybackChromeRoot.tsx → done.

Overlay Positioning Pitfall: Bottom Toolbar Occlusion

When adding a fixed-position overlay to a page with a bottom toolbar (e.g., OpenMAIC's playback controls), bottom: 24px places the overlay BEHIND the toolbar. The toolbar is typically 60-80px tall.

Fix: Use bottom: 80px (or whatever clears the toolbar height). Verify with Playwright that the overlay is visible:

const el = document.querySelector('[style*="position: fixed"][style*="bottom"]');
const rect = el.getBoundingClientRect();
// rect.bottom must be ABOVE the toolbar top
// toolbar top ≈ viewport.height - 80px

Detection: If console.log shows the element exists with correct text but browser_vision says "no subtitle visible", the overlay is likely occluded by another element.

User Frustration Signal: "不要改变 X 的模式"

When the user says "don't change the mode/structure of X", they mean:

• Keep the existing layout EXACTLY as-is
• Only add the new feature as an overlay or supplementary element
• Don't restructure, rename, or reorganize existing components
• Don't add new props to intermediate components unless absolutely necessary

This happened 3 times in one session — the user rolled back changes repeatedly because I kept restructuring the component. The lesson: additive only, no restructuring.

---

Quick Reference

See also:

• references/client-side-rag-patterns.md — Hybrid client-server RAG for static sites, multi-env fallback, E2E test patterns
• references/mkdocs-material-deployment-pitfalls.md — MkDocs + Docker + nginx gotchas
• references/book-compilation-pitfalls.md — Duplicate files, slug variants, directory flattening
• references/video-export-patterns.md — OpenMAIC video export (HTML vs app-as-renderer)

Quick Reference

# Test link click with Playwright
browser_navigate("http://localhost:8002/page-with-link")
browser_click(link_ref)
assert browser_console("window.location.href") == "expected-url"

# Check blast radius before editing
# docs/ → all envs | site/ → rebuild only | deploy/ → one env

# Deploy all environments
./scripts/deploy.sh all

# Verify all environments
for url in "http://localhost:8002/page" "https://cf.pages.dev/page"; do
  curl -sI "$url" | head -1
done

# Find ALL occurrences of a number before changing it
grep -rn "OLD_NUMBER" docs/ mkdocs.yml README.md | grep -v "http"
# After changing — verify zero remain
grep -rn "OLD_NUMBER" docs/ mkdocs.yml README.md | grep -v "http"

# Reliable Docker rebuild (picks up source changes)
docker compose down && docker compose build --no-cache && docker compose up -d

Reference Files

• references/mkdocs-material-deploy-pitfalls.md — CSS loading, duplicate directories, Cloudflare 25MB limit, theme override block names
• references/video-export-via-playwright.md — Using running app as renderer, cropping UI, ffmpeg commands, SRT generation
• references/duplicate-file-detection.md — Detecting and removing slug variants (Chinese/English) in compiled outputs
• references/link-slug-mismatch-fix.md — Fixing 3447 broken chapter→article links caused by truncated English slugs vs Chinese filenames
• references/batch-text-replace-safety.md — Rules for safe bulk find-replace (exclude metadata, word boundaries, verify sample first)
• references/nextjs-static-export-audio-pattern.md — Serving 21GB audio from external API when static export can't have API routes
• references/nextjs-docker-build-caching.md — Docker image caching hides Next.js code changes; verify with grep + Playwright
• references/openmaic-subtitle-pattern.md — Minimal-change subtitle overlay for OpenMAIC player (hook into existing lectureSpeech state)