Design backends that survive redeploys, server reboots, and modest scaling. Covers stateless application servers, state placement (object storage, managed databases, Redis), immutable deploy artifacts, health checks, graceful shutdown, database migrations that don't lock the world, and the twelve-factor baseline. Invoke when designing a new backend, when uploads disappear after a redeploy, or when productionizing a vibe-coded prototype.
Design backends that survive redeploys, server reboots, and modest scaling. Covers stateless application servers, state placement (object storage, managed databases, Redis), immutable deploy artifacts, health checks, graceful shutdown, database migrations that don't lock the world, and the twelve-factor baseline. Invoke when designing a new backend, when uploads disappear after a redeploy, or when productionizing a vibe-coded prototype.
Backend Architecture — the Solutions Architect Baseline
The most common mistake in vibe-coded backends is treating the application server as if it were a desktop computer: state lives where the code lives, the disk is permanent, restarts are exceptional, one instance runs forever. None of that survives contact with production.
This skill is the architectural baseline that makes a backend production-ready. It is not security-specific — but a backend that loses data, drops jobs, or rolls back to a half-state during deploys is also a backend whose incident-response surface looks much larger than it should. Pairs with the hardening skills, but operates at a different layer.
When to invoke
"User uploads disappeared after we redeployed"
"The site went down during deploy and we lost the in-flight orders"
"I need to scale to a second server but nothing works when there are two"
"We're moving from prototype to production"
"We're handing off the codebase to a team that has to run it"
Designing a new backend from scratch and you want the decisions correct on day one
Reviewing an inherited backend before scaling user count or revenue
The single most important rule
App servers are cattle. State lives elsewhere.
Any data that must survive a deploy, reboot, container restart, or instance replacement is state. State does not belong on the app server's local disk, in its memory, or anywhere bound to the instance's lifetime. It belongs in a system designed for durability.
This rule, followed consistently, eliminates 80% of "where did my data go?" incidents.
The state-placement decision matrix
For every piece of data your app handles, decide where it lives before writing the code:
Data type
Right place
Wrong place
User-uploaded files (images, PDFs, docs)
Object storage (S3, R2, B2, Spaces) behind a CDN
App server's uploads/ directory
Generated thumbnails / derivatives
Object storage, or regenerated on-demand from originals
App server's filesystem
Database rows (orders, users, posts)
Managed Postgres / MySQL, or a dedicated DB VM with backups
Every "Wrong place" row in the table is a real outage waiting to happen. Walk this list against your current codebase — anything misplaced is a refactor candidate.
The "images disappear on redeploy" trap — concretely
This one bites everyone once. Anatomy:
App is built as a Docker image. Image is immutable.
App writes uploaded images to ./uploads/ (relative to working directory inside the container).
Deploy: new image is pulled, old container stopped, new container started.
The new container has a fresh filesystem from the image. The uploads/ directory exists but is empty.
All previously-uploaded user images now return 404.
The same trap occurs without Docker: PaaS platforms (Heroku, Railway, Fly, Render) typically run on ephemeral filesystems. The "disk" is a feature of the container, not the platform.
Fix — use object storage
// Bad — writes to local disk that disappears on redeployimport fs from'node:fs/promises';
asyncfunctionsaveUpload(file: Buffer, key: string) {
await fs.writeFile(`./uploads/${key}`, file);
return`/uploads/${key}`; // served by app, lost on deploy
}
// Good — write to S3-compatible storage; URL is permanentimport { S3Client, PutObjectCommand } from'@aws-sdk/client-s3';
const s3 = newS3Client({
region: 'auto',
endpoint: process.env.S3_ENDPOINT, // R2: https://<acct>.r2.cloudflarestorage.comcredentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID!,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY!,
},
});
asyncfunctionsaveUpload(file: Buffer, key: string) {
await s3.send(newPutObjectCommand({
Bucket: process.env.S3_BUCKET,
Key: key,
Body: file,
ContentType: detectMime(file),
}));
return`${process.env.CDN_BASE_URL}/${key}`; // served by CDN, survives redeploy
}
Recommended stack (cheap and fast for small/medium projects):
Cloudflare R2 for storage — zero egress fees, S3-compatible API
Cloudflare custom domain in front of R2 for the public URLs
Pre-signed URLs for uploads if you want the browser to upload directly to R2 (bypasses your app server for big files)
Alternatives: AWS S3 + CloudFront, Backblaze B2, DigitalOcean Spaces, MinIO if you must self-host.
If your sessions live in-process (Express express-session with default MemoryStore, Next.js naive session-in-Map), every deploy logs out every user. Same problem if you scale to two instances — half of requests land on the instance that never saw the login.
Signed cookies (the session payload itself is in the cookie, signed by the server). Works for small sessions; the cookie grows with the data.
JWT with short TTL + refresh tokens. See auth-hardening for the tradeoffs.
Background jobs — setTimeout is not a queue
// Bad — process is killed on deploy, job never runs
app.post('/api/sign-up', async (req, res) => {
await db.users.create(req.body);
setTimeout(() =>sendWelcomeEmail(req.body.email), 60_000); // ☠️
res.json({ ok: true });
});
// Good — persistent queue, retries, observabilityimport { Queue, Worker } from'bullmq';
const emailQueue = newQueue('emails', { connection: { url: process.env.REDIS_URL }});
app.post('/api/sign-up', async (req, res) => {
await db.users.create(req.body);
await emailQueue.add('welcome', { email: req.body.email }, {
attempts: 5,
backoff: { type: 'exponential', delay: 1000 },
removeOnComplete: 1000,
});
res.json({ ok: true });
});
// Worker — separate process, can be on the same or different hostnewWorker('emails', async (job) => {
awaitsendWelcomeEmail(job.data.email);
}, { connection: { url: process.env.REDIS_URL }});
What you get:
Retries with backoff
Dead-letter queue for poison messages
Persistence — restarting the worker does not lose jobs
Observable — BullMQ has an admin UI
For cron-style "every hour" jobs, the right place is also the queue (BullMQ has repeat, or use a dedicated cron worker), or a managed scheduler (AWS EventBridge, Cloudflare Workers Cron). Not a setInterval in the web process.
Deploys — immutable artifacts, health checks, graceful shutdown
Three patterns that together make deploys boring:
Immutable artifacts
A deploy is "ship this exact image / artifact to production." Not "pull, install, configure." The same artifact that ran in staging runs in prod. If something breaks in prod, rolling back means pointing at the previous artifact — no re-install, no risk of a different state.
Docker image with explicit tag (myapp:1.2.3 or SHA-digest), not :latest
Or built bundle (zip / tarball) checked into a versioned artifact store
Build once, deploy that exact thing to every environment
Health checks
Two endpoints, two purposes:
// Liveness — is the process alive? Cheap, no dependencies.
app.get('/healthz', (req, res) => res.json({ ok: true }));
// Readiness — is the process actually ready to serve traffic?
app.get('/readyz', async (req, res) => {
try {
awaitPromise.all([
db.$queryRaw`SELECT 1`,
redis.ping(),
]);
res.json({ ok: true });
} catch (err) {
res.status(503).json({ ok: false, err: String(err) });
}
});
The orchestrator (Docker, Kubernetes, your reverse proxy) routes traffic only when /readyz returns 200. Without this, new containers get traffic before they have warmed connections to the DB and produce 500s for the first 30 seconds of every deploy.
Graceful shutdown
When the orchestrator sends SIGTERM, stop accepting new requests, finish in-flight ones, close connections, then exit. Most frameworks ship this; some make you wire it.
const server = app.listen(port);
constshutdown = async (signal: string) => {
console.log(`${signal} — draining`);
server.close(async () => {
await redis.quit();
await db.$disconnect();
process.exit(0);
});
// Hard cap — if drain takes too long, force exitsetTimeout(() => process.exit(1), 30_000).unref();
};
process.on('SIGTERM', () =>shutdown('SIGTERM'));
process.on('SIGINT', () =>shutdown('SIGINT'));
Combined: zero-downtime rolling deploys become trivial.
Database operations — migrations and connections
Migrations
Forward-compatible by default: never break the old code. New code can run against old schema, old code can run against new schema, until both sides have rolled out.
Expand-then-contract: add a new column / table first, deploy code that writes to both, backfill, then remove the old. No "rename column" in a single migration in production.
Lock-free for large tables: in Postgres, prefer ADD COLUMN ... DEFAULT NULL (instant) over ADD COLUMN ... DEFAULT 'x' NOT NULL (rewrites the table). Use CREATE INDEX CONCURRENTLY for indexes on large tables.
Run migrations as a separate step, not at app boot. Two app instances racing to run the same migration at startup is its own incident class.
Connection pooling
App processes open too many connections to Postgres by default. Postgres struggles past ~100 active connections. Use a pooler:
PgBouncer — battle-tested, transaction-mode for most apps
Supabase, Neon, RDS Proxy — managed pooler in front of managed Postgres
Prisma Accelerate / Drizzle's pool — ORM-level pooling
Set per-instance pool size = (maxConnections / numInstances) − headroom. For 4 instances and a 100-connection budget, that's ~20 connections per instance with 20 spare.
Order creation dedupes on a client-generated idempotency key (X-Idempotency-Key header), not on auto-incremented IDs.
Email sends dedupe on (template, recipient, trigger-id).
File uploads dedupe on content hash (compute SHA-256 of the file, use as the object key).
A retry storm should be boring, not catastrophic.
Observability — three pillars, minimum viable
For a small backend, "observability" is not a buzzword — it is three concrete things:
Centralized logs — see log-strategy. Cloud logging, Loki, hosted. Not "ssh in to read /var/log".
Per-request tracing — generate a request_id at the edge (Cloudflare adds one for free), thread it through every log line and every downstream service call. When something breaks, you can reconstruct the full path.
Error reporting — Sentry / Rollbar / Highlight / GlitchTip. Each unhandled exception lands with stack trace, breadcrumbs, user context. Without this you only see errors users actively complain about.
Metrics (Prometheus / Grafana / Datadog) are great when traffic grows; not day-one critical.
The cost-aware tier — when to add complexity
Most vibe-coded projects do not need Kubernetes, microservices, multi-region, or a service mesh. The cost of premature complexity is real: more failure modes, more "why is this not working" hours, more bills.
Reasonable trajectory:
Stage
Architecture
0–1k users
Single VPS, app + Postgres + Redis colocated, daily off-host backups, Cloudflare in front
1k–10k users
Same, plus object storage for files, separate worker process, managed Postgres or DB on a dedicated VM
10k–100k users
Multiple app instances behind a load balancer, managed Postgres with read replica, queue is its own service, CDN is essential
100k+ users
Dedicated DB nodes, autoscaling app tier, regional split if latency demands, real SRE practices
Skip stages only if you have a specific reason. "We might need Kubernetes someday" is not a reason for today.
Twelve-Factor — the canonical reference
Most of this skill is a practical application of The Twelve-Factor App, an excellent guide written in 2011 that has aged remarkably well. The twelve factors:
Codebase — one codebase tracked in revision control, many deploys
Dependencies — explicitly declared and isolated (lockfiles)
Config — stored in the environment, not the code
Backing services — treated as attached resources (DB, cache, queue are configurable)
Build, release, run — strictly separate stages
Processes — execute the app as one or more stateless processes
Port binding — export services via port binding (no external web server required at the app layer)
Concurrency — scale out via the process model
Disposability — fast startup and graceful shutdown
Dev/prod parity — keep dev, staging, and production as similar as possible
Logs — treat logs as event streams
Admin processes — run admin/management tasks as one-off processes
Read it once. Then read it again every six months. Most "production incidents" in small-team backends are violations of one of these twelve.
Quick architecture review checklist
Before declaring a backend production-ready, walk through:
App processes are stateless — restarting an instance loses nothing
User uploads go to object storage (S3 / R2 / equivalent), not local disk
Sessions are in Redis / DB / signed cookies, not in-process memory
Background jobs run via a persistent queue, not setTimeout / setInterval
Cron jobs run via a real scheduler, not the web process
Cache is explicit (Redis / KV / CDN), not implicit in-memory hashmaps
Database has off-host backups, restore-tested at least once
Database migrations are forward-compatible; never "rename + drop" in one step
Connection pooling is in place (PgBouncer or equivalent)
App has /healthz (liveness) and /readyz (readiness) endpoints
App handles SIGTERM gracefully — drains connections, finishes in-flight work
Deploys produce immutable artifacts (image tag or SHA digest), no in-place mutation
Rollback is a single command pointing at a previous artifact
Logs ship to a central destination, with request_id threading
Unhandled errors land in Sentry / equivalent
All state-changing operations are idempotent
Configuration is in env vars; no environment-specific values baked into the image
.env is not in the image, not in the repo
If anything is "no", you have an outage waiting. Fix before scaling, not after.
What this skill will not do
Tell you to use Kubernetes / microservices / multi-region for a project that does not need them
Endorse writing application data to the app server's local disk in production
Help build a backend whose deploys lose user data because "we'll fix it later"
Replace experience — architecture is judgment. This skill gives you the patterns; the calls are still yours
