deploy-nextjs-standalone

name	deploy-nextjs-standalone
description	Deployment guide for Next.js standalone output mode — build, static files, PM2, nginx proxy. If user reports `/_next/static/*` 404 + MIME type errors after a deploy, jump straight to "Symptom: ChunkLoadError" below.

Deploy: Next.js Standalone Mode

This project uses output: 'standalone' in next.config.ts. The build output is at .next/standalone/server.js.

Build Checklist

# 1. Build
npm run build

# 2. Static files MUST be copied manually
cp -r .next/static .next/standalone/.next/static

# 3. Copy .env
cp .env .next/standalone/.env

# 4. Start
pm2 start .next/standalone/server.js --name cet4-web

Never forget step 2. Without it, /_next/static/ URLs return 404, causing the browser to reject CSS/JS with MIME type errors.

Why step 2 is needed

Next.js standalone mode uses @vercel/nft (Node File Trace) to trace require() calls and copy only the files needed to run the server. This means:

• copy-traced-files only copies JS modules that are require()'d by the server
• static/chunks/ contains per-page lazy-load chunks generated by the bundler — these are not require()'d on startup, so they are never traced and never copied
• Result: .next/standalone/.next/static/chunks/ will always be missing files unless you manually sync

Symptom: ChunkLoadError

After deploying without syncing static files, the browser shows:

GET /_next/static/chunks/0_mnfdu5xtgcn.js 404 (Not Found)
ChunkLoadError: Failed to load chunk /_next/static/chunks/0_mnfdu5xtgcn.js
Refused to execute script because its MIME type ('text/html') is not executable

The 404 returns an HTML error page instead of JS, hence the MIME type refusal.

The nginx dual-path trap

Nginx typically serves /_next/static/ directly from the project root (not from standalone):

# nginx config (typical BT-Panel setup)
location /_next/static/ {
    alias /www/wwwroot/site/.next/static/;
    expires 365d;
}

But PM2 runs the server from .next/standalone/server.js, which serves /_next/static/ from .next/standalone/.next/static/. This creates two paths that both need static files:

Path	Used by
.next/static/	nginx direct serving
.next/standalone/.next/static/	Next.js server fallback

If nginx has a location /_next/static/ block, it takes priority and the first path is your only path. If not, requests fall through to the Node server, which needs the second path. Both must be kept in sync.

Reliable sync command

# After every build:
cp -r .next/static .next/standalone/.next/static

# Or if deploying via tarball, sync server-side:
rsync -avz .next/static/ root@server:/path/to/.next/static/
rsync -avz .next/static/ /path/to/.next/standalone/.next/static/

macOS tar gotchas

When building on macOS and deploying to Linux, tar carries extra baggage by default:

• LIBARCHIVE.xattr.* extended attributes (Finder info, ACLs, quarantine flag)
• AppleDouble ._* sidecar files (created for every file with extended attrs)

A typical macOS build tarball can contain 3,000+ ._* sidecars that:

• Get written to the server's filesystem on extract
• Are invisible in normal listings but waste inodes
• Cause tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr...' spam in logs (cosmetic but noisy)

Fix: build the tarball with xattrs stripped:

tar -cf deploy.tar \
  --no-xattrs \
  --exclude='._*' \
  --exclude='.env*' \
  -C .next/standalone .

Defense in depth: clean up on server after extract (in case the tarball was created elsewhere with macOS):

find /path/to/standalone -name '._*' -type f -delete 2>/dev/null
find /path/to/standalone -name '._*' -type d -empty -delete 2>/dev/null

Verify: find /path/to/standalone -name '._*' | wc -l should return 0.

PM2

pm2 status                    # check if running
pm2 logs cet4-web --lines 20  # view recent logs
pm2 restart cet4-web          # restart
pm2 stop cet4-web             # stop
pm2 delete cet4-web           # remove from list

Build Troubleshooting

TypeScript error in fix-word-userid.ts — This is a one-off script, not app code. Fix the error or exclude it from tsconfig.json:

# Fix: add updatedAt
sed -i 's|data: { name: mapping.group, userId: correctId }|data: { name: mapping.group, userId: correctId, updatedAt: new Date() }|' fix-word-userid.ts

# Or exclude from tsconfig
sed -i 's|"scripts"|"scripts", "fix-word-userid.ts"|' tsconfig.json

Clean rebuild — When in doubt, start fresh:

rm -rf .next && npm run build && cp -r .next/static .next/standalone/.next/static

Deploy Script Hygiene

A deploy script that swallows errors is worse than one that fails loudly. Common anti-patterns that caused real outages on this project:

Anti-pattern 1: 2>/dev/null || true

# ❌ BAD: tar errors are hidden, "✓ Extracted" is a lie
tar -xzf deploy.tar.gz -C standalone 2>/dev/null | grep -v "LIBARCHIVE.xattr" || true
echo "✓ Extracted"

# ❌ BAD: cp silently fails when source doesn't exist
cp -r .next/static .next/standalone/.next/static 2>/dev/null || true

These produce green checkmarks and zero extraction. Result: production 404 / MIME type errors that took 10+ minutes to debug.

Anti-pattern 2: running cp on the server for sync

# ❌ BAD: server's root-level .next/static may be from a different build
#         approach (pre-standalone), in a stale state, or wrong arch
cp -r .next/static .next/standalone/.next/static

The server's .next/static/ is a legacy pre-standalone build path. It may exist, be from a previous next build mode, or be from a different OS. If you cp from it on the server, you copy stale chunks over the freshly extracted ones.

Fix: The tarball should already contain .next/static/ (via step 2 of the build checklist). Don't re-copy from the server. The tarball is the source of truth.

Rules of thumb

1. set -euo pipefail is not enough — pipe stages still hide failures. Verify after each step.
2. Never || true on success-path commands. If you need to continue on failure, use explicit if ! cmd; then ...; fi.
3. Print expected counts. EXTRACTED_FILES=$(find . -type f | wc -l) after tar is more useful than "✓ Extracted".
4. The tarball is the source of truth. Don't cp from intermediate paths on the server — the server's paths may be stale or wrong-arch.

Reference: what deploy-full.sh looked like before/after this fix

- tar -xzf /tmp/eztor-deploy-*.tar.gz -C standalone 2>/dev/null | grep -v "LIBARCHIVE.xattr" || true
+ tar -xzf /tmp/eztor-deploy-*.tar.gz -C standalone
+ EXTRACTED_FILES=$(find standalone -type f | wc -l)
+ echo "  ✓ Extracted ($EXTRACTED_FILES files)"

- # Sync static files (nginx serves from .next/static)
- cp -r .next/static .next/standalone/.next/static 2>/dev/null || true
+ # Note: .next/static is already inside the tarball (from build checklist step 2),
+ # so no separate cp from server's stale .next/static is needed.

nginx Proxy

The server uses nginx (BT-Panel) as reverse proxy. The vhost config at /www/server/panel/vhost/nginx/114.55.58.90.conf includes proxy configs from /www/server/panel/vhost/nginx/proxy/114.55.58.90/*.conf.

Critical: Disable proxy_cache

BT-Panel default proxy.conf enables global proxy caching (proxy_cache cache_one). This caches ALL proxied responses including HTML pages, ignoring the backend's Cache-Control: no-store. Always add proxy_cache off; in the proxy location block:

location ^~ /
{
    proxy_pass http://127.0.0.1:3000/;
    proxy_cache off;   # ← REQUIRED for dynamic sites
    ...
}

After enabling/disabling cache, clear the old cache:

rm -rf /www/server/nginx/proxy_cache_dir/*
nginx -t && nginx -s reload

Debugging Nginx cache issues

If curl to localhost:3000 shows new content but browser shows old content, Nginx is caching:

# Compare: direct vs proxied
curl -sI http://localhost:3000/ | grep -i x-build-id
curl -sI https://eztor.dogeggcode.cyou/ | grep -i x-build-id

Reload nginx after changes

nginx -t && nginx -s reload

Watch for duplicate location blocks

BT-Panel proxy directory may have multiple .conf files. The main config includes all of them. If two files define the same location, nginx will fail:

ls /www/server/panel/vhost/nginx/proxy/114.55.58.90/
# Only one file should define "location ^~ /"

Pre-Deploy Extraction Verification

After tar -xzf on the server, before pm2 restart, verify the extract actually worked. The "Deployment Verification" section (below) catches this AFTER restart — better to catch BEFORE you take downtime.

Minimum checks

cd /www/wwwroot/<site>/.next

# 1. Total file count (sanity)
TOTAL=$(find standalone -type f | wc -l)
[ "$TOTAL" -gt 1000 ] || { echo "✗ Only $TOTAL files extracted, aborting"; exit 1; }

# 2. Critical files exist
test -f standalone/server.js            || { echo "✗ server.js missing"; exit 1; }
test -f standalone/BUILD_ID.txt         || { echo "✗ BUILD_ID.txt missing"; exit 1; }
test -d standalone/.next/static/chunks  || { echo "✗ static/chunks missing"; exit 1; }
test -d standalone/.next/server         || { echo "✗ server chunks missing"; exit 1; }

# 3. Static chunks count is reasonable
CHUNKS=$(find standalone/.next/static/chunks -type f | wc -l)
[ "$CHUNKS" -gt 50 ] || { echo "✗ Only $CHUNKS chunks, expected >50"; exit 1; }

# 4. No macOS xattr sidecars leaked through
SIDECARS=$(find standalone -name '._*' 2>/dev/null | wc -l)
[ "$SIDECARS" -eq 0 ] || { echo "⚠ $SIDECARS macOS sidecars, cleaning"; find standalone -name '._*' -delete 2>/dev/null; }

echo "✓ Pre-deploy extraction OK ($TOTAL files, $CHUNKS chunks)"

Why these checks

Check	Catches
find . -type f | wc -l	Empty / partial tarball extraction
server.js present	Wrong source dir tarred
BUILD_ID.txt present	Forgot to echo $TIMESTAMP > BUILD_ID.txt before tar
static/chunks/ non-empty	Forgot build checklist step 2
chunks count > 50	Truncated tarball (network error)
._* count == 0	macOS tar didn't strip xattrs

Bonus: a one-liner for the panic moment

# After tar -xzf, before pm2 restart:
test -f .next/standalone/server.js && \
test -d .next/standalone/.next/static/chunks && \
test "$(find .next/standalone/.next/static/chunks -type f | wc -l)" -gt 50 && \
echo "OK to restart" || echo "DO NOT RESTART — extraction incomplete"

If you see "DO NOT RESTART" and you've already stopped PM2, fix the extraction first, then restart. Don't pm2 restart on a half-installed standalone — you'll get ChunkLoadError.

Deployment Verification

Always verify deployment in this order. If any step fails, stop and fix before proceeding.

1. Server-side: BUILD_ID matches

cat /www/wwwroot/114.55.58.90/.next/standalone/BUILD_ID.txt

2. Server-side: Next.js responds correctly (bypass Nginx)

curl -sI http://localhost:3000/ | grep -i x-build-id
curl -s http://localhost:3000/ | grep -o 'data-build-id="[^"]*"'

Both must match the BUILD_ID file. If they don't, tarball extraction failed or PM2 didn't restart.

3. PM2 status

pm2 list                          # uptime should be recent
pm2 describe cet4-web             # script path, restarts, memory
pm2 logs cet4-web --lines 20      # check for startup errors

4. Browser verification (after clearing site data)

// DevTools Console
document.documentElement.dataset.buildId   // must match BUILD_ID
fetch('/').then(r => console.log(r.headers.get('X-Build-Id')))  // must match BUILD_ID

If step 2 passes but browser shows old version → Nginx cache. Redeploy alone won't fix it.

Build ID Infrastructure

The project embeds a build timestamp for verification:

Mechanism	File	Visible
data-build-id attribute	src/app/layout.tsx	<html> tag in DevTools
X-Build-Id HTTP header	next.config.ts headers()	Network tab response headers
BUILD_ID.txt file	.next/standalone/BUILD_ID.txt	Server filesystem

The BUILD_ID is set by deploy.sh via export NEXT_PUBLIC_BUILD_ID=$(date +%Y%m%d_%H%M%S) before building. It is hard-coded into the build output and does not depend on runtime environment variables.

Cross-Platform Build: Prisma OpenSSL Compatibility

Critical: If building on macOS and deploying to Linux, the Prisma client may be compiled for the wrong OpenSSL version.

Symptom

Prisma Client could not locate the Query Engine for runtime "debian-openssl-3.0.x".
This happened because Prisma Client was generated for "debian-openssl-1.1.x"

The app crashes with 502 on every request.

Root Cause

Prisma generates platform-specific query engine binaries. The local build picks up the macOS or older Linux engine, but the server runs a different OpenSSL version (e.g., after an OS upgrade or Alibaba Cloud instance restart).

Fix

Add binaryTargets to prisma/schema.prisma:

generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "debian-openssl-3.0.x"]
}

Then regenerate:

npx prisma generate

This must be done BEFORE building. The engine binaries are bundled into the standalone output at build time.

Server-side hotfix (if already deployed)

If you can't rebuild locally, regenerate on the server:

cd /www/wwwroot/114.55.58.90
# Update schema
sed -i '/provider      = "prisma-client-js"/a\  binaryTargets = ["native", "debian-openssl-3.0.x"]' prisma/schema.prisma
# Generate (uses project root node_modules, not standalone)
npx prisma generate
# Copy engines to standalone
cp -r node_modules/@prisma .next/standalone/node_modules/
cp -r node_modules/.prisma .next/standalone/node_modules/
# Verify both engines exist
ls .next/standalone/node_modules/.prisma/client/libquery_engine-*
# Restart
pm2 restart cet4-web

Runtime Dependencies (require vs import)

Standalone mode only bundles files traced via require() at startup. Dependencies loaded dynamically at runtime (e.g., ipa-dict for phonetic validation) are not included in the tarball.

How to identify

If a module is loaded like this, it won't be in standalone:

const mod = require('ipa-dict/lib/en_US')  // dynamic, not traced

Fix

Manually copy to server after extracting tarball:

scp -r node_modules/ipa-dict root@server:/path/.next/standalone/node_modules/

Or include it in the tarball creation:

tar -czf deploy.tar.gz .next/standalone prisma node_modules/ipa-dict

Server Restart Safety

Alibaba Cloud forced restart (or any hard reboot) may leave services in unexpected states:

nginx not auto-starting

BT-Panel nginx is often disabled in systemd:

systemctl is-enabled nginx  # might show "disabled"

After a server restart, always check:

ps aux | grep nginx
# If not running:
/www/server/nginx/sbin/nginx

Services to verify after restart

pm2 list                    # PM2 processes
ps aux | grep nginx         # nginx
docker ps                   # Docker containers
ss -tlnp | grep :3000       # App listening
ss -tlnp | grep :5432       # PostgreSQL

SSH Instability During Incidents

When the server is under heavy load (OOM, process crash loops), SSH may become unresponsive. Symptoms:

• Connection timed out during banner exchange
• Connection closed by remote port 22
• Verbose mode shows key exchange succeeds but authentication hangs

What to do

1. Wait — the server may recover on its own (PM2 auto-restart, OOM killer)
2. Alibaba Cloud console — use VNC remote connection as fallback
3. Ping test — ping -c 3 <ip> to verify network layer is alive
4. Don't panic — if PM2 auto-restarts the app, it may recover without SSH

Prevention

• Set max_memory_restart in PM2 to prevent OOM
• Monitor with pm2 monit during deployments
• Keep SSH sessions alive: ServerAliveInterval=30 in SSH config