toolforge-nodejs

name	toolforge-nodejs
description	Deploy and manage Node.js web services on Wikimedia Toolforge Kubernetes — zero-dependency server patterns, webservice commands, PORT configuration, static file serving with caching headers, npm on NFS, environment variables, logging, and common pitfalls
license	MIT
compatibility	opencode
depends_on	["wikimedia-toolforge","wikimedia-api-access"]
skill_discovery_hints	[{"keywords":["Toolforge Node","Node.js","Node webservice","node22","node18","npm","package.json"]},{"keywords":["Toolforge port","PORT env","webservice port","Toolforge Kubernetes","web service"]},{"keywords":["Toolforge static files","express","http server","zero dependency","built-in http"]},{"keywords":["Cache-Control","Varnish","Toolforge proxy","immutable cache","static asset"]},{"keywords":["Toolforge environment","env vars","secrets","toolforge env","config","environment"]},{"keywords":["Toolforge logs","kubectl logs","webservice logs","debug","tail logs","toolforge debug"]},{"keywords":["Node streaming","buffer","readFile","createReadStream","Pannellum","XHR","WebGL"]},{"keywords":["Toolforge cron","node job","scheduled task","cron node","batch job"]}]
last_verified	"2026-06-20T00:00:00.000Z"

⚠️ Prerequisites: This skill assumes you have a Toolforge account and can SSH in. See wikimedia-toolforge for account setup, tool creation, and basic SSH configuration.

💡 Zero dependencies by default: Toolforge's NFS filesystem has slow metadata operations — npm install can take minutes. The patterns in this skill use only Node.js built-in modules by default, which means zero npm install, zero package.json, and instant deployment. Add dependencies only when you must.

⚠️ Toolforge Rule #2 — Open Source License Required: All code in the Tools project must be published under an OSI-approved open source license. Add a LICENSE file to your repository before deploying. The absence of a license means default copyright laws apply, which is counter to the principles of the Wikimedia movement. See the full rules.

---

Quick Reference

Goal	Command / Pattern
Deploy a new Node.js service	See SOP 1
Choose a runtime	node22 (default), node18 (legacy) — see SOP 1.1
Start/restart a webservice	become <tool> webservice --backend=kubernetes node22 start
See logs	become <tool> kubectl logs -f deployment/<tool>
Set environment variables	become <tool> toolforge env set NAME value
Serve static files	Use readFile() with correct MIME types — see SOP 3
Add npm dependencies	Use a package.json and npm install on the tool's NFS — see SOP 5
Run a cron job	become <tool> job schedule node22 --command 'node script.mjs' "0 * * * *"

---

SOP 1: Bootstrap a Zero-Dependency Node.js Server

The fastest way to get a Node.js web service running on Toolforge — no npm install, no package.json, just a single .mjs file.

1.1 Choose a Runtime

Toolforge offers these Node.js runtime versions:

Runtime	Kubernetes Image	Status
node22	docker-registry.tools.wmflabs.org/toolforge-node22-sssd-web:latest	✅ Current default (Node 22.x)
node18	docker-registry.tools.wmflabs.org/toolforge-node18-sssd-web:latest	⚠️ Legacy (Node 18.x, available but not recommended for new tools)

The runtime image includes the operating system, Node.js binary, and basic tools. Your code lives on NFS at /data/project/<tool>/.

1.2 Create the Tool (One Time)

# From your local machine
ssh <user>@login.toolforge.org toolforge tools create my-tool

1.3 Write Your Server

Create a single-file server that uses only Node.js built-in modules. The key detail: Toolforge sets the PORT environment variable — your server must bind to it.

// server.mjs — Zero-dependency Node.js web service for Toolforge
import { createServer } from 'node:http';
import { readFile } from 'node:fs/promises';
import { existsSync } from 'node:fs';
import { join, extname, resolve as pathResolve, sep as pathSep } from 'node:path';
import { fileURLToPath } from 'node:url';

const __dirname = fileURLToPath(new URL('.', import.meta.url));
const PORT = process.env.PORT || 8765;

// MIME types for static files
const MIME = {
    '.html': 'text/html; charset=utf-8',
    '.css':  'text/css; charset=utf-8',
    '.js':   'application/javascript; charset=utf-8',
    '.mjs':  'application/javascript; charset=utf-8',
    '.json': 'application/json; charset=utf-8',
    '.png':  'image/png',
    '.jpg':  'image/jpeg',
    '.jpeg': 'image/jpeg',
    '.svg':  'image/svg+xml',
};

const server = createServer(async (req, res) => {
    const url = new URL(req.url, `http://localhost:${PORT}`);
    const pathname = url.pathname;

    // API routes
    if (pathname === '/api/hello' && req.method === 'GET') {
        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.end(JSON.stringify({ message: 'Hello from Toolforge!' }));
        return;
    }

    // Static files
    try {
        const filePath = join(__dirname, 'public', pathname === '/' ? 'index.html' : pathname);
        // Security: prevent directory traversal — resolve and verify containment
        // startsWith without trailing sep matches sibling dirs (e.g. my-tool2)
        const safeDir = __dirname + path.sep;
        const resolved = path.resolve(filePath);
        if (!resolved.startsWith(safeDir)) {
            res.writeHead(403); res.end('Forbidden');
            return;
        }
        const data = await readFile(filePath);
        const ext = extname(filePath);
        res.writeHead(200, {
            'Content-Type': MIME[ext] || 'application/octet-stream',
            'Content-Length': data.length,
            // One-week browser cache for static assets
            'Cache-Control': pathname.startsWith('/images/')
                ? 'public, max-age=604800, immutable'
                : 'public, max-age=3600',
        });
        res.end(data);
    } catch {
        res.writeHead(404, { 'Cache-Control': 'no-store' });
        res.end('Not found');
    }
});

server.listen(PORT, () => {
    console.log(`Server running on port ${PORT}`);
});

Naming convention: Use .mjs extension for ES module syntax. If you need require(), use .cjs. For mixed use, a package.json with "type": "module" makes .js files ES modules.

1.4 Deploy

# From your local machine
scp server.mjs <user>@login.toolforge.org:/data/project/my-tool/

# Create a public/ directory for static files
ssh <user>@login.toolforge.org "mkdir -p /data/project/my-tool/public"

# Copy your static files
scp index.html style.css <user>@login.toolforge.org:/data/project/my-tool/public/

# Start the webservice
ssh <user>@login.toolforge.org "become my-tool webservice --backend=kubernetes node22 start"

1.5 Verify

# Check status
ssh <user>@login.toolforge.org "become my-tool webservice --backend=kubernetes node22 status"

# View logs
ssh <user>@login.toolforge.org "become my-tool kubectl logs -f deployment/my-tool"

# Test via curl or browser
curl https://my-tool.toolforge.org/
curl https://my-tool.toolforge.org/api/hello

---

SOP 2: The PORT Environment Variable

Toolforge's Kubernetes proxy routes incoming requests to your container. It ignores any port you specify in code and expects your server to listen on the port given by the PORT environment variable.

Do This

const PORT = process.env.PORT || 8765;  // 8765 is a safe local dev default
server.listen(PORT, () => console.log(`Listening on ${PORT}`));

Don't Do This

server.listen(3000);        // ❌ Toolforge proxy won't find it
server.listen(80);          // ❌ Not running as root
server.listen('8080');      // ❌ PORT is a number, not a string

How It Works

Browser ─→ toolforge.org ─→ Kubernetes Ingress ─→ Pod:PORT
                                                      │
                                                process.env.PORT
                                               (e.g., "31245")

The PORT value is assigned per-pod and is not predictable — always read it from the environment.

---

SOP 3: Static File Serving with Correct Caching

Cache-Control Headers

Toolforge uses Varnish as an HTTP accelerator in front of Kubernetes. Setting proper cache headers is critical for performance:

Asset Type	Recommendation	Effect
Images (panoramas, thumbnails)	public, max-age=604800, immutable	1 week, never revalidates
JS/CSS (versioned)	public, max-age=31536000, immutable	1 year, never revalidates
HTML pages	public, max-age=3600	1 hour, revalidates
API responses	no-cache (or omit caching)	Always revalidates
Error pages (404, 500)	no-store	Never caches errors

function cacheHeaders(pathname) {
    if (pathname.startsWith('/images/')) {
        return 'public, max-age=604800, immutable';  // 1 week
    }
    if (pathname.match(/\.(js|css|mjs)$/)) {
        return 'public, max-age=31536000, immutable'; // 1 year
    }
    if (pathname.startsWith('/api/')) {
        return 'no-cache';  // API responses
    }
    return 'public, max-age=3600';  // HTML and everything else
}

Why immutable Matters

The immutable directive tells the browser it never needs to revalidate the resource before its max-age expires. For panorama images that are cache-busted with ?v= timestamps, this saves a round-trip on every page load. Without it, browsers send If-Modified-Since/If-None-Match headers even for cached assets.

Never Cache Error Responses

// ❌ Wrong — browser caches the 404 page forever
res.writeHead(404, { 'Cache-Control': 'public, max-age=604800' });
res.end('Not found');

// ✅ Correct — browser never caches error responses
res.writeHead(404, { 'Cache-Control': 'no-store' });
res.end('Not found');

MIME Types Must Be Explicit

Node.js http module doesn't set MIME types automatically. Missing or wrong MIME types cause browsers to refuse to load resources:

const MIME = {
    '.html': 'text/html; charset=utf-8',
    '.css':  'text/css; charset=utf-8',
    '.js':   'application/javascript; charset=utf-8',
    '.mjs':  'application/javascript; charset=utf-8',
    '.json': 'application/json; charset=utf-8',
    '.png':  'image/png',
    '.jpg':  'image/jpeg',
    '.jpeg': 'image/jpeg',
    '.svg':  'image/svg+xml',
};

// Always set Content-Type and Content-Length
res.writeHead(200, {
    'Content-Type': MIME[ext] || 'application/octet-stream',
    'Content-Length': data.length,
});

---

SOP 4: Serving Images for WebGL/Pannellum — Never Stream

Pannellum's 360° viewer loads panorama images via XMLHttpRequest and expects a complete, buffered response. Streaming responses (using createReadStream().pipe(res)) causes net::ERR_FAILED 200 (OK) — the browser sees HTTP 200 but can't read the response body as a Blob.

❌ Broken: Streaming

// DO NOT USE — Pannellum fails with ERR_FAILED
if (stat.size > 1024 * 1024) {
    const { createReadStream } = await import('node:fs');
    createReadStream(filePath).pipe(res);
    return;
}

✅ Correct: Buffering

// Always read the entire file into memory before responding
const data = await readFile(filePath);
res.writeHead(200, {
    'Content-Type': 'image/jpeg',
    'Content-Length': data.length,
    'Cache-Control': 'public, max-age=604800, immutable',
});
res.end(data);

Why buffering is safe on Toolforge: Your images are on the local NFS mount — reads are fast. The memory cost (a few MB per image) is negligible on Toolforge's Kubernetes pods with 2GB+ RAM limits.

When Streaming IS Safe

Use Case	Method	Why
<img> tags	Streaming works	Browser handles streamed images natively
fetch() + response.blob()	Streaming works	Different internal path than XHR
Audio/video file serving	Streaming preferred	Media elements handle buffered seeking
Large file downloads	Streaming preferred	Lower memory footprint

---

SOP 5: Adding npm Dependencies

For most Wikimedia tools, the built-in http module is sufficient. When you need external packages:

5.1 Create package.json

{
  "name": "my-tool",
  "version": "1.0.0",
  "type": "module",
  "dependencies": {
    "express": "^4.18.0"
  },
  "scripts": {
    "start": "node server.mjs",
    "dev": "node --watch server.mjs"
  }
}

The "type": "module" field makes all .js files use ES module syntax (import/export). Use .cjs extension for files that need CommonJS (require).

5.2 Install on Toolforge

NFS metadata operations are slow. A full npm install can take 2–5 minutes:

become my-tool npm install

5.3 Optimize Install Speed

• Use .npmrc to avoid audit/lockfile overhead:

  audit=false
  fund=false
  lockfile=false
  

• Pin exact versions in package.json to avoid resolution time on every deploy
• Install locally, copy binary: npm pack on your machine, upload the .tgz, extract on Toolforge
• Cache node_modules — don't delete it on every deploy. Only re-run npm install when dependencies change.

5.4 When to Use Dependencies vs. Built-In

Need	Recommendation
Simple REST API	Built-in http module — zero dependencies
Static file server	Built-in http + fs/promises
JSON body parsing	Built-in JSON.parse() + streaming body collection
URL parsing	Built-in URL class
Routing (simple)	if/else on url.pathname
Routing (complex, 10+ routes)	express or fastify
SQLite database	better-sqlite3
WebSocket	ws package
Templating	Built-in template literals or ejs

Rule of thumb: Start with zero dependencies. Add packages only when the built-in solution becomes unwieldy.

---

SOP 6: Environment Variables and Secrets

Setting Variables

# Set variables — these persist across restarts
become my-tool toolforge env set API_KEY "sk-abc123"
become my-tool toolforge env set NODE_ENV "production"
become my-tool toolforge env set LOG_LEVEL "info"

# List all variables
become my-tool toolforge env list

# Remove a variable
become my-tool toolforge env unset API_KEY

Reading Variables in Code

const apiKey = process.env.API_KEY;
if (!apiKey) {
    console.error('API_KEY not set — some features will be unavailable');
}

Caveats

• Variables are set per-tool, not per-webservice-instance. All pods for the same tool share the same environment.
• Restart the webservice after changing env vars — they're loaded at process start, not dynamically:

  become my-tool webservice --backend=kubernetes node22 restart
  

• Never hardcode secrets in source files committed to version control. Always use toolforge env set.

---

SOP 7: Logging and Debugging

Kubernetes Logs

# Tail logs from the current pod
become my-tool kubectl logs -f deployment/my-tool

# Last N lines
become my-tool kubectl logs --tail=100 deployment/my-tool

# Follow a specific pod (useful with multiple replicas)
become my-tool kubectl logs -f pod/my-tool-7d8f9e2a4-x3y7z

# Search for errors
become my-tool kubectl logs deployment/my-tool | grep -i error

Structured Logging (Zero-Dependency)

function log(level, message, data = {}) {
    const entry = {
        timestamp: new Date().toISOString(),
        level,
        message,
        ...data,
    };
    // Write to stdout — Kubernetes collects this
    process.stdout.write(JSON.stringify(entry) + '\n');
}

// Usage
log('info', 'Server starting', { port: PORT });
log('error', 'Failed to resolve file', { file: filename, error: e.message });

Common Issues

Symptom	Likely Cause	Fix
Connection refused	Server not listening on PORT	Check process.env.PORT in code
CrashLoopBackOff in status	Server crashes on startup	Run kubectl logs to see the error
ImagePullBackOff	Runtime image not found	Check runtime name (node22 not nodejs22)
Slow first request	Cold start + NFS metadata	Use a keepalive or health check endpoint
ERR_FAILED 200 (OK)	Streamed response to Pannellum	Use readFile() instead of streaming
404 for existing files	Path resolution wrong	Log filePath and check directory traversal

Health Check Endpoint

Toolforge's Kubernetes expects the server to be responsive. Add a lightweight health check:

server.on('listening', () => {
    // Signal to Kubernetes that the server is ready
    console.log('READY');
});

---

SOP 8: Cron Jobs

For batch processing, periodic syncs, or scheduled maintenance:

# Run a script every hour
become my-tool job schedule node22 \
    --command 'node /data/project/my-tool/cron/cleanup.mjs' \
    --no-filelog \
    "0 * * * *"

# Run once
become my-tool job submit node22 \
    --command 'node /data/project/my-tool/cron/backfill.mjs'

Cron Job Script Template

// cron/cleanup.mjs — Scheduled script (no HTTP server needed)
import { readdir, unlink, stat } from 'node:fs/promises';
import { join } from 'node:path';

const CACHE_DIR = '/data/project/my-tool/images';
const MAX_AGE_DAYS = 7;

async function cleanup() {
    const files = await readdir(CACHE_DIR);
    const now = Date.now();
    let removed = 0;

    for (const file of files) {
        const filePath = join(CACHE_DIR, file);
        const stats = await stat(filePath);
        const ageDays = (now - stats.atimeMs) / (1000 * 60 * 60 * 24);
        if (ageDays > MAX_AGE_DAYS) {
            await unlink(filePath);
            removed++;
        }
    }

    console.log(`Cleanup: removed ${removed} files (threshold: ${MAX_AGE_DAYS} days)`);
}

cleanup().catch(err => {
    console.error('Cleanup failed:', err);
    process.exit(1);
});

---

SOP 9: Restarting and Updating

Restart (zero-downtime)

become my-tool webservice --backend=kubernetes node22 restart

The Kubernetes deployment performs a rolling restart — no downtime as long as your server starts quickly.

Update Code

# Upload new version
scp server.mjs <user>@login.toolforge.org:/data/project/my-tool/

# Restart to pick up changes
ssh <user>@login.toolforge.org "become my-tool webservice --backend=kubernetes node22 restart"

Stop

become my-tool webservice --backend=kubernetes node22 stop

---

Reference: Quick Command Cheat Sheet

# === Lifecycle ===
become my-tool webservice --backend=kubernetes node22 start
become my-tool webservice --backend=kubernetes node22 stop
become my-tool webservice --backend=kubernetes node22 restart
become my-tool webservice --backend=kubernetes node22 status

# === Logs ===
become my-tool kubectl logs -f deployment/my-tool
become my-tool kubectl logs --tail=100 deployment/my-tool

# === File Transfer (from local machine) ===
scp server.mjs <user>@login.toolforge.org:/data/project/my-tool/
scp -r public/ <user>@login.toolforge.org:/data/project/my-tool/

# === Environment ===
become my-tool toolforge env set KEY value
become my-tool toolforge env list
become my-tool toolforge env unset KEY

# === Scheduled Jobs ===
become my-tool job schedule node22 --command 'node script.mjs' "0 * * * *"
become my-tool job submit node22 --command 'node script.mjs'

# === Direct Shell ===
become my-tool

---

Reference: Common Pitfalls

Pitfall	Why It Happens	Prevention
Missing PORT	Hardcoded port in code	Always use `process.env.PORT
Streaming to Pannellum	createReadStream().pipe(res) causes ERR_FAILED	Always use readFile() + res.end() for images
No Cache-Control	Missing headers = re-download on every page load	Always set cache headers for static assets
Cached 404 pages	404 served with Cache-Control: public	Use no-store for error responses
Slow npm install	NFS metadata operations are slow	Minimize dependencies; use .npmrc to skip audit
Missing MIME type	Content-Type not set, browser refuses to load	Explicit MIME map for all file extensions
Traversal vulnerability	join(dir, pathname) can escape the root	Always path.resolve() and check startsWith(dir + path.sep)
Crash on startup	Unhandled promise rejection in createServer	Wrap async handlers in try/catch
Stale image cache	Commons file overwritten, old URL still cached	Append ?v={timestamp} to cached image URLs
CORS errors	upload.wikimedia.org has no CORS headers	Proxy images through your own server

---

See Also

• wikimedia-toolforge — Account setup, tool creation, SSH, Kubernetes pods, cron jobs, database access
  • SOP 9: Frontend Assets (Privacy-Preserving CDN) — Must-load JS/CSS from tools-static.wmflabs.org/cdnjs/ instead of external CDNs
• commons-file-resolution — Resolving File: references to HTTP URLs, caching images for same-origin serving
• wikimedia-api-access — User-Agent, rate limiting, error handling
• wikimedia-commons-thumbnails — Thumbnail URL generation and format conversion
• wikimedia-security-and-privacy — CDN privacy risks, data minimization, and third-party script policies