wikimedia-toolforge

name	wikimedia-toolforge
description	Manage Toolforge accounts, web services, Kubernetes pods, cron jobs, and file deployment for Wikimedia tools
license	MIT
compatibility	opencode
depends_on	["wikimedia-api-access"]
skill_discovery_hints	[{"keywords":["Toolforge","tool hosting","Kubernetes","web service","cron job","deploy"]},{"keywords":["toolforge tools create","become","webservice","toolforge jobs"]},{"keywords":["CDN","cdnjs","tools-static","privacy-preserving CDN","content delivery","third-party script","external CDN tracking"]}]
last_verified	"2026-06-10T00:00:00.000Z"

Toolforge (formerly Wikimedia Tool Labs) is a cloud hosting platform for community-developed tools that interact with Wikimedia wikis and data. This skill covers account setup, service management, deployment, and debugging.

Prerequisites

• A Toolforge account approved via Wikimedia Foundation onboarding
• SSH public key added to your Toolforge account via the admin console
• ssh-agent running locally with your private key loaded (ssh-add ~/.ssh/id_ed25519)
• The following environment variable set (optional but recommended):
  • TOOLFORGE_USER — Your Toolforge shell/LDAP username

SOP 1: Account & Project Setup

1.1 Verify Account

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org hostname

If successful, you will see a hostname like tools-sgebastion-XX. If the connection hangs or fails, check that your SSH key is added to the admin console.

1.2 Create a New Tool

Each tool is a sub-account with its own directory, database access, and web service.

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org toolforge tools create my-tool-name

Naming rules:

• Lowercase letters, digits, and hyphens only
• Must be unique across all Toolforge
• Should be descriptive (e.g., pageview-analyzer, category-watchdog)

After creation, the tool's home directory is at /data/project/my-tool-name/.

⚠️ Toolforge Rule #2 — Open Source License Required: All code in the Tools project must be published under an OSI-approved open source license. Add a LICENSE file to your repository before deploying. The absence of a license means default copyright laws apply, which is counter to the principles of the Wikimedia movement. See the full rules.

1.3 Configure Tool Permissions

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org toolforge tools maintainers add my-tool-name ${TOOLFORGE_USER:-your-username}

This adds you as a maintainer so you can deploy files and manage services.

SOP 2: File Deployment

2.1 Deploy Files via SSH

# Deploy a single file
scp my-script.py ${TOOLFORGE_USER:-your-username}@login.toolforge.org:/data/project/my-tool-name/

# Deploy an entire directory
rsync -avz --exclude '.*' ./my-tool/ ${TOOLFORGE_USER:-your-username}@login.toolforge.org:/data/project/my-tool-name/

Always use rsync for repeated deployments — it only transfers changed files and preserves permissions.

2.2 Deploy via Git (Recommended for Maintained Tools)

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
cd /data/project/my-tool-name/
git clone https://github.com/your-org/my-tool.git .

Then update with git pull on subsequent deployments. This provides version history and rollback.

2.3 Set Executable Permissions

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org chmod +x /data/project/my-tool-name/my-script.py

2.4 Fix File Ownership (the take command)

Files copied via scp or rsync arrive owned by your shell user, not the tool user. Many tool commands require tool-user ownership:

# File landed as wrong user after scp/rsync
ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
take /data/project/my-tool-name/my-script.py

The take command changes ownership to the current tool user. It requires that the calling user is the current owner (prevents abuse).

SOP 3: Web Services

Toolforge supports several web service backends. Choose based on your needs:

Backend	Use Case	Start Command
webservice --backend=kubernetes python3.11	Python web apps (Flask, Django, FastAPI)	webservice --backend=kubernetes python3.11 start
webservice --backend=kubernetes node	Node.js web apps (Express)	webservice --backend=kubernetes node start
webservice --backend=kubernetes php8.2	PHP web apps	webservice --backend=kubernetes php8.2 start
webservice --backend=kubernetes static	Static file serving (HTML/JS)	webservice --backend=kubernetes static start
webservice --backend=buildservice	Build Service container (any language)	webservice --backend=buildservice start

3.1 Start a Web Service

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
cd /data/project/my-tool-name
webservice --backend=kubernetes python3.11 start

The become command switches to the tool's service account, which has the correct permissions and environment variables.

3.2 Check Service Status

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
webservice --backend=kubernetes python3.11 status

3.3 Stop a Web Service

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
webservice --backend=kubernetes python3.11 stop

3.4 Restart a Web Service

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
webservice --backend=kubernetes python3.11 restart

3.5 Web Service Entry Point

The web service looks for a server.py (Python), app.js (Node), or index.php (PHP) in the tool's home directory by default. To use a custom entry point, set the WEB_CONCURRENCY or create a launch.sh script:

#!/bin/bash
# launch.sh — custom web service entry point
cd /data/project/my-tool-name
gunicorn -w 4 -b 0.0.0.0:8000 my_app:app

Mark it executable: chmod +x launch.sh, then start with:

webservice --backend=kubernetes python3.11 start --launch launch.sh

SOP 4: Kubernetes Jobs (One-Off Tasks)

For one-off or batch processing tasks that don't need a persistent web service:

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
toolforge jobs run my-job-name --command "python3 /data/project/my-tool-name/my_script.py" --image python3.11 --wait

Common Job Options

Option	Description
--command "..."	Command to run inside the container
--image python3.11	Container image (choose based on language/runtime)
--wait	Wait for the job to finish before returning logs
--mem 2Gi	Memory limit (default: 1Gi, max: 4Gi)
--cpu 1	CPU cores (default: 1, max: 2)
--filelog	Stream logs to a file on NFS (for later inspection)
--timestamps	Add timestamps to log output

Check Job Status

become my-tool-name
toolforge jobs list
toolforge jobs logs my-job-name

Delete a Job

become my-tool-name
toolforge jobs delete my-job-name

NFS Writes from Inside Pods (Non-Root Caveats)

Kubernetes job containers run as the tool user, not root. The NFS home at /data/project/<tool>/ is writable, but system directories are not:

Can write to	Cannot write to
/data/project/<tool>/ (NFS home)	/usr/local/ (root owned)
/tmp/ (world-writable)	/var/lib/dpkg/ (root owned)
$HOME (on NFS)	/etc/ (root owned)
$TOOL_DATA_DIR	/data/ (root-owned)

Consequences:

• Cannot apt-get install anything
• Cannot npm install -g without setting NPM_CONFIG_PREFIX to a writable dir
• Cannot write to /data/ — use /data/project/<tool>/ or $TOOL_DATA_DIR instead
• kubectl cp may fail with permission errors on the kubeconfig file (use kubectl exec -i ... -- sh -c 'cat > /path' < localfile instead)

SOP 5: Cron Jobs

For scheduled, recurring tasks:

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
crontab -e

Add a line in standard cron format:

# Run daily at 2:00 AM UTC
0 2 * * * /usr/bin/python3 /data/project/my-tool-name/daily_report.py >> /data/project/my-tool-name/logs/cron.log 2>&1

Cron job guardrails:

• Always redirect output to a log file (otherwise cron emails it to you and can fill your mailbox)
• Use absolute paths for all commands and file references
• Test the script manually before scheduling it via cron
• Set a reasonable schedule — do not run on sub-minute intervals
• For long-running jobs (over 10 minutes), use toolforge jobs (SOP 4) triggered by cron instead of running directly

List Cron Jobs

become my-tool-name
crontab -l

SOP 6: Environment Variables & Secrets

6.1 Set Environment Variables

Toolforge provides a set-webservice-env command for web services:

become my-tool-name
toolforge env set MY_VARIABLE my_value

6.2 View Environment Variables

become my-tool-name
toolforge env list

6.3 Remove an Environment Variable

become my-tool-name
toolforge env unset MY_VARIABLE

6.4 Secrets (Database Passwords, API Keys)

Store sensitive values as environment variables via toolforge env set. These are stored securely and not shown in env list output. Do not hardcode secrets in source files.

6.5 MediaWiki API Authentication (Bot Passwords)

If your Toolforge tool authenticates against the MediaWiki API to make edits (e.g., via bot passwords or Pywikibot), be aware that the API's lgname parameter requires underscores where usernames have spaces:

# ✗ Fails silently with "Unknown error"
lgname = "AL Wiki MIT@mybot"

# ✓ Works
lgname = "AL_Wiki_MIT@mybot"

See the pywikibot skill ("Login fails" troubleshooting section) and the wikimedia-api-access skill ("Login Username Quirk" section) for full details.

SOP 7: Logs & Debugging

7.1 Web Service Logs

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
webservice --backend=kubernetes python3.11 logs

Add --tail to see only the last N lines:

webservice --backend=kubernetes python3.11 logs --tail=50

7.2 Job Logs

become my-tool-name
toolforge jobs logs my-job-name

7.3 Check Disk Usage

become my-tool-name
du -sh /data/project/my-tool-name/

7.4 Interactive Debugging via Shell

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name
# Now run commands interactively from the tool's home directory
python3 -c "import requests; print(requests.get('https://en.wikipedia.org/api/rest_v1/page/summary/Python_(programming_language)').json())"

SOP 8: Build Service (Container Images)

The Toolforge Build Service allows you to build custom container images from a public Git repository using Cloud Native Buildpacks. This is the modern, recommended way to deploy tools — it frees you from per-language base images and gives you control over the runtime.

8.1 How It Works

Instead of deploying files to NFS and using a language-specific webservice backend, you:

1. Host your code in a public Git repository (GitLab, GitHub, Gerrit)
2. Add a Procfile at the root of your repo defining how to start your app
3. Run toolforge build start to build a container image from your repository
4. Run the built image as a web service or a job

The Build Service supports: Python, Node.js, PHP, Ruby, Go, Java/JVM, .NET, and Rust. It can also install OS-level Apt packages and compile frontend assets with Node.js at build time.

8.2 Prerequisites

• A public Git repository containing your tool's code
• A Procfile at the repository root
• become access to your tool account

8.3 Procfile Format

The Procfile is a plain text file named Procfile (no extension) at the root of your repo:

web: gunicorn --bind=0.0.0.0 --workers=4 --forwarded-allow-ips=* app:app
migrate: python -m app.django migrate

• The web: entry defines what runs when you start a web service
• Other entries (e.g., migrate:) define what runs as jobs
• Every process type in the Procfile becomes an executable command in the container

Important: Do not name a process type after a real command (e.g., do not use celery: as a process type — use run-celery: instead).

8.4 Building an Image

ssh ${TOOLFORGE_USER:-your-username}@login.toolforge.org
become my-tool-name

# Build from the default branch (HEAD)
toolforge build start https://gitlab.wikimedia.org/toolforge-repos/my-tool

# Build from a specific branch, tag, or commit
toolforge build start --ref v1.2.0 https://gitlab.wikimedia.org/toolforge-repos/my-tool

# Use the latest buildpacks (newer language versions)
toolforge build start -L https://gitlab.wikimedia.org/toolforge-repos/my-tool

# Pass build-time environment variables
toolforge build start --envvar NODE_ENV=production https://gitlab.wikimedia.org/toolforge-repos/my-tool

8.5 Starting a Web Service

become my-tool-name

# Start the web service from the built image
toolforge webservice buildservice start --mount=none

# Alternatively, create a service.template so 'webservice start' works directly:
echo -e 'type: buildservice\nmount: none' > service.template

# Check status
toolforge webservice buildservice status

# View logs
toolforge webservice buildservice logs -f

# Restart after a new build
toolforge webservice buildservice restart

# Stop
toolforge webservice buildservice stop

# Get an interactive shell inside the running container
toolforge webservice buildservice shell

Note on NFS mounts: By default, --mount=none is recommended. If your tool needs to read/write files in /data/project/, use --mount=all and reference the path via the $TOOL_DATA_DIR environment variable instead of relying on $HOME.

8.6 Running a Job

become my-tool-name

# Run the 'migrate' process from your Procfile
toolforge jobs run --wait --image my-tool/my-tool:latest --command "migrate" some-job

# Run with arguments
toolforge jobs run --wait --image my-tool/my-tool:latest --command "migrate --production" migrate-job

# Pass composite commands via shell wrapper
toolforge jobs run --wait --image my-tool/my-tool:latest --command "sh -c 'env; nodejs --version'" debug-job

8.7 Checking Build Logs

become my-tool-name

# View the most recent build log
toolforge build logs

# Check build quota
toolforge build quota

8.8 Updating Code

To deploy a new version:

become my-tool-name

# 1. Push new code to your Git repo
# 2. Trigger a new build
toolforge build start https://gitlab.wikimedia.org/toolforge-repos/my-tool
# 3. Restart the web service to use the new image
toolforge webservice buildservice restart

8.9 Installing Apt Packages

Create a project.toml at the root of your repository:

[_]
schema-version = "0.2"

[com.heroku.buildpacks.deb-packages]
install = [
    "imagemagick",
    "php",
]

Packages from Ubuntu 24.04 (Noble) can be looked up at https://packages.ubuntu.com/noble/.

8.10 Known Limitations

• Limited to a single primary language runtime per image (Node.js can be a secondary runtime for asset compilation)
• No LDAP connection inside containers — commands like id <user> will not work
• $HOME does not point to /data/project/<tool>/ — use $TOOL_DATA_DIR instead
• NFS is not mounted by default — use --mount=all explicitly when needed
• Build images have a storage quota — check with toolforge build quota and request increases via Phabricator if needed

Guardrails & Common Pitfalls

1. Use become for interactive sessions, sudo -niu for SSH one-liners. Running become my-tool-name interactively (SSH in, then become, then commands) works correctly. But become uses exec internally, which replaces the shell — so command chaining over SSH fails:

   # ❌ BROKEN — second command runs as YOUR user, not the tool
   ssh [e-mail] "become my-tool-name; webservice restart"
   
   # ✅ CORRECT — use sudo -niu for chained SSH commands
   ssh [e-mail] "sudo -niu tools.my-tool-name webservice restart"
   

2. SSH key expiry — Toolforge SSH keys expire after a period. If you get permission denied, regenerate your key in the admin console and re-add it to ssh-agent.

3. NFS latency — /data/project/ is on NFS. File operations can be slow. Avoid frequent small writes. Use local /tmp/ for temporary files and move results to NFS only when needed.

4. Resource limits — Kubernetes pods have 1 CPU and 1Gi RAM by default. Use toolforge jobs with --mem and --cpu flags for larger tasks. Do not run resource-intensive tasks on bastion or login nodes.

5. Do not run long processes on login — The login shell is for administration only. Long-running processes should be jobs or web services. Processes running for more than 30 minutes on login may be killed without warning.

6. Database connections — For replica database access, see the wikimedia-database skill. For tool-owned databases (MariaDB), use become my-tool-name and run sql my-tool-name to access the tool's database. If you need the actual MySQL username and password (e.g., for an external client or connection string), find them in the tool's home directory after become:

   become my-tool-name
   cat replica.my.cnf
   

   This prints [client] with user and password fields. The sql command is still the recommended way to connect interactively, but replica.my.cnf is useful when configuring ORM connection strings or database drivers in application code.

7. Static file caching — Static web services (--backend=kubernetes static) serve from /data/project/my-tool-name/. Files are cached; wait a few minutes after deployment or use a versioned URL pattern (style.v2.css).

8. Test locally first — Deploying broken code to Toolforge wastes time. Test scripts locally with representative data before deploying.

9. Clean up old jobs — Kubernetes job history accumulates. Delete completed jobs that are no longer needed using toolforge jobs delete.

10. Build Service: Git repo required — The Build Service requires a public Git repository. Private repos are not supported. The Procfile must be at the repository root and named exactly Procfile (no extension). After a build, you must restart the web service to pick up the new image — toolforge build start alone does not restart running services.

11. Build Service: NFS and $HOME — Build Service containers do not have NFS mounted by default. Use --mount=all to mount it. Inside the container, $HOME does not point to /data/project/<tool>/ — use the $TOOL_DATA_DIR environment variable instead for tool home directory paths.

12. Wait ~1 minute after tool creation before become works. Use sudo -u as an immediate workaround.

13. Refs vary between browser sessions — When using Playwright or browser automation for toolsadmin, always snapshot to read the current accessibility tree instead of hardcoding element refs.

Common Toolforge Mistakes

Mistake	Symptom	Fix
become <tool>; cmd1; cmd2	Subsequent commands run as original user	Use sudo -niu tools.<tool> cmd1; cmd2 instead
become <tool>; webservice restart	TLS cert errors or commands run as wrong user	Use sudo -niu tools.<tool> webservice restart
--kubeconfig=PATH	stat .kubeconfig: no such file	Use --kubeconfig PATH (space, not =)
toolforge env set	No such command 'env'	Use toolforge envvars create
Nested " inside " over SSH	unexpected EOF	Use heredoc or alternate quote layers
scp to /data/project/	Permission denied (lands as wrong user)	Pipe through sudo; use take
kubectl exec -it in non-tty	the input device is not a TTY	Use -i without -t, or add -t to the outer SSH command
kubectl cp to a running pod	Permission errors reading kubeconfig	Use kubectl exec -i ... -- sh -c 'cat > /path' < localfile instead
Writing to /data/ from inside a pod	Permission denied	Use /data/project/<tool>/ or $TOOL_DATA_DIR instead
printf '\\e' for escape sequences	\e becomes literal escape char (0x1B)	Use printf '%s' '\\e' with %s format to output literally
become <tool> sh -c "\$VAR"	Variable empty on bastion	Escaped $ passes through to the tool shell; unescaped expands on bastion

Example Workflows

Full Web App Deployment

# 1. Create tool
ssh [e-mail] toolforge tools create my-web-tool

# 2. Deploy code
rsync -avz ./my-web-app/ [e-mail]:/data/project/my-web-tool/

# 3. Set environment variables
ssh [e-mail] "become my-web-tool; toolforge env set API_KEY your-api-key-here"

# 4. Start web service
ssh [e-mail] "become my-web-tool; webservice --backend=kubernetes python3.11 start"

# 5. Verify it's running
ssh [e-mail] "become my-web-tool; webservice --backend=kubernetes python3.11 status"

Daily Data Collection (Cron + Job)

# 1. Deploy the script
rsync -avz collect_data.py [e-mail]:/data/project/my-tool/

# 2. Set cron to trigger a Kubernetes job
ssh [e-mail] "become my-tool; crontab -l | { cat; echo '0 3 * * * toolforge jobs run daily-collect --command \"python3 /data/project/my-tool/collect_data.py\" --image python3.11 --wait --filelog >> /data/project/my-tool/logs/cron_trigger.log 2>&1'; } | crontab -"

Build Service Web App (Modern Workflow)

# 1. Create tool
ssh [e-mail] toolforge tools create my-build-tool

# 2. Push code to a public Git repository (e.g., GitLab)
#    Ensure a Procfile exists at the root:
#      web: gunicorn --bind=0.0.0.0 --workers=4 app:app

# 3. Build the container image from the Git repo
ssh [e-mail] "become my-build-tool; toolforge build start https://gitlab.wikimedia.org/toolforge-repos/my-build-tool"

# 4. Start as a build service web service
ssh [e-mail] "become my-build-tool; toolforge webservice buildservice start --mount=none"

# 5. Verify it's running
ssh [e-mail] "become my-build-tool; toolforge webservice buildservice status"

# 6. To update: push new code to Git, rebuild, restart
#    ssh login.toolforge.org "become my-build-tool; toolforge build start <repo-url>"
#    ssh login.toolforge.org "become my-build-tool; toolforge webservice buildservice restart"

---

Tooling

This skill includes helper scripts, reference docs, and templates:

🔧 Deploy Tool (scripts/deploy.sh)

Deploy files to a Toolforge tool via rsync with dry-run preview.

./scripts/deploy.sh ./my-web-app my-tool-name

Features dry-run confirmation, permission setting, and post-deploy steps.

Note: For the Build Service (SOP 8), you do not use rsync deployment. Instead, push code to a public Git repository and use toolforge build start. See the Build Service workflow example above.

🔧 Status Check (scripts/status.sh)

Check web service status, Kubernetes jobs, disk usage, and active processes.

./scripts/status.sh my-tool-name

🔧 Kubernetes Job Manager (scripts/manage-k8s.sh)

Manage Kubernetes jobs: run, list, logs, delete, and status.

./scripts/manage-k8s.sh my-tool-name run my-job "python3 /data/project/my-tool/script.py"
./scripts/manage-k8s.sh my-tool-name list
./scripts/manage-k8s.sh my-tool-name logs my-job

🔧 Cron Job Manager (scripts/manage-cron.sh)

Manage cron jobs: list, add, remove by pattern, or clear all.

./scripts/manage-cron.sh my-tool-name list
./scripts/manage-cron.sh my-tool-name add '0 2 * * *' 'python3 /data/project/my-tool/daily.py >> /data/project/my-tool/logs/cron.log 2>&1'
./scripts/manage-cron.sh my-tool-name remove daily.py

📚 CLI Reference (references/toolforge-cli.md)

Quick reference of all Toolforge CLI commands organized by category:

• Account & tools, web services (including buildservice), Kubernetes jobs, environment variables, cron, file operations, database

🧩 Deploy Config (assets/deploy-config.sh)

Environment variable template for deployment scripts:

cp assets/deploy-config.sh my-config.sh
# Edit my-config.sh with your Toolforge username and tool name
source my-config.sh
./scripts/deploy.sh ./my-app my-tool-name

🐍 Flask App Template (assets/app-template.py)

Ready-to-deploy Flask app with:

• Home page with status
• Health check endpoint (/api/status)
• Wikipedia API proxy (/api/summary/<title>, /api/search?q=...)
• Proper User-Agent for Wikimedia API requests
• WSGI entry point for gunicorn

cp assets/app-template.py server.py
# Edit, test locally with `python3 server.py`, then deploy

For Build Service deployment: Add a Procfile at your repo root containing:

web: gunicorn --bind=0.0.0.0 --workers=4 --forwarded-allow-ips=* app:app

Then push to a public Git repo and run toolforge build start.

---

SOP 9: Frontend Assets (Privacy-Preserving CDN)

Toolforge tools must load JavaScript, CSS, and fonts from Wikimedia's internal cdnjs mirror — never from external CDNs that track users.

Rationale

External CDNs (cdnjs.cloudflare.com, unpkg.com, Google Fonts) can log IPs and track users. Wikimedia's Toolforge Web Hosting Policy requires the internal mirror for privacy compliance.

CDN URLs

Only one hostname works reliably:

https://tools-static.wmflabs.org/cdnjs/ajax/libs/<library>/<version>/<file>

Examples:

<script src="https://tools-static.wmflabs.org/cdnjs/ajax/libs/d3/7.9.0/d3.min.js"></script>
<script src="https://tools-static.wmflabs.org/cdnjs/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
<link rel="stylesheet" href="https://tools-static.wmflabs.org/cdnjs/ajax/libs/twitter-bootstrap/5.3.0/css/bootstrap.min.css">

Finding Libraries

Search the cdnjs API to find versions, then construct the mirror URL:

# Search for a library
curl -s "https://api.cdnjs.com/libraries?search=d3&fields=version,latest"
# → {"name":"d3","version":"7.9.0",...}

# Construct mirror URL by replacing cdnjs.cloudflare.com/ajax/libs
# with tools-static.wmflabs.org/cdnjs/ajax/libs

Guardrails

Do	Don't
Use tools-static.wmflabs.org/cdnjs/	Use cdnjs.cloudflare.com, unpkg.com, jsdelivr.net
Pin exact versions (7.9.0)	Use latest or version-agnostic URLs
Verify availability before deploying	Assume all cdnjs libraries are mirrored

The full CDN mirror guide with troubleshooting is at references/cdn-mirror-guide.md.

Tooling

File	Purpose
scripts/check-cdn.sh	Verify a library is on the mirror (by name or full URL)
scripts/list-available.sh	Search available libraries
assets/load-template.html	HTML page loading jQuery, Bootstrap, Font Awesome from CDN
assets/load-template.js	Dynamic JS loader for programmatic use

---

Scripting Guide: Multi-Layer SSH Commands

Running commands on Toolforge involves nested shells: local → SSH → bastion → become/sudo → command. Each layer adds quoting complexity.

The Shell Stack

Local machine ──▶ SSH ──▶ Bastion shell ──▶ sudo -niu tools.<tool> ──▶ Command

Every ", $, \\, `, and ' must survive all layers.

Pattern 1: Simple Commands

# No special characters — works directly
ssh [e-mail] "sudo -niu tools.mytool toolforge jobs list"

# Single-quoted arguments inside the remote string
ssh [e-mail] "sudo -niu tools.mytool toolforge jobs run myjob --command 'python3 script.py'"

Pattern 2: The take Command Alternative for File Transfer

You can't scp directly to /data/project/<tool>/ as your shell user (it's owned by tools.<toolname>). Two approaches:

A) Staging area + take:

scp file.txt [e-mail]:/home/user/
ssh [e-mail] \
  "sudo -niu tools.mytool sh -c 'cp /home/user/file.txt /data/project/mytool/ && take /data/project/mytool/file.txt'"

B) Pipe through SSH:

cat localfile | ssh [e-mail] \
  "sudo -niu tools.mytool sh -c 'cat > /data/project/mytool/localfile'"

Pattern 3: Passing stdin to a Remote Command

# Pipe a local file into a remote command as the tool user
cat my-script.sh | ssh [e-mail] \
  "sudo -niu tools.mytool sh -c 'cat > /data/project/mytool/my-script.sh'"

# Heredoc piped through SSH
cat << 'EOF' | ssh [e-mail] \
  "sudo -niu tools.mytool sh -c 'cat > /data/project/mytool/config.json'"
{
  "key": "value"
}
EOF

Key insight for heredocs: If you quote the delimiter (<< 'EOF'), the content is treated as a literal string — no variable expansion happens locally. This is usually what you want when writing to remote files.

Pattern 4: Writing Files Inside a Pod (kubectl exec)

More reliable than kubectl cp (avoids kubeconfig permission issues):

# Pipe local content into a file inside a pod
cat config.json | kubectl exec -i my-pod -n tool-mytool -- sh -c 'cat > /tmp/config.json'

# Heredoc works the same way
cat << 'EOF' | kubectl exec -i my-pod -n tool-mytool -- sh -c 'cat > /tmp/.env'
API_KEY=sk-...
EOF

Pattern 5: Commands with Internal Quotes

# Outer single quotes let you use double quotes inside for the remote command
ssh [e-mail] \
  'sudo -niu tools.mytool kubectl exec pod -n tool-mytool -- sh -c "echo hello"'

The Bash Quoting Idiom Trap

The idiom 'single'\''quote' embeds a single quote inside a single-quoted string:

# IDIOM: 'text1'\''text2' produces: text1'text2
#         ^^^^^^  ^^ ^^^^^^
#         quoted  \'  quoted
#                 (escaped single quote between two quoted parts)

Common mistake — space after \':

# WRONG: 'text1'\'' text2'
# The space after \' STARTS a new argument
# Same for 'text1\' 'text2'

# RIGHT: 'text1'\''text2' — no space, all one argument

Safer alternative: build strings in multiple steps:

printf '%s' 'PS1=' >> /tmp/file      # PS1=
printf '%s' '\\[\\e...' >> /tmp/file  # the actual PS1 value
printf '%s' "'" >> /tmp/file          # closing single quote
echo "" >> /tmp/file                  # newline

Writing Scripts That Run on the Bastion (No Local SSH Layer)

When you write a script that runs on the bastion itself (not from your local machine), there's one fewer shell layer, which is much simpler:

# /data/project/mytool/myscript.sh (runs as tools.mytool)

# Simple: direct kubectl with --kubeconfig (use SPACE, not =)
kubectl --kubeconfig /data/project/mytool/.kube/config get pods -n tool-mytool

# Multi-line script inside pod (escape $ for variables you want expanded IN the pod)
kubectl exec my-pod -n tool-mytool -- sh -c "
    export PATH=/tmp/npm/bin:\$PATH
    node --version
"

Critical rule for bastion scripts:

• Variables you want expanded ON THE BASTION use ${VAR} (no escaping)
• Variables you want expanded INSIDE THE POD use \${VAR} or \$VAR (escaped $)

---

Cross-References

Related Skill	Why
wikimedia-database	SQL replicas — SSH tunnel setup shared with Toolforge
toolforge-nodejs	Node.js web services on Toolforge Kubernetes
toolforge-python	Python web services on Toolforge Kubernetes (Flask + gunicorn)
wikimedia-eventstreams	Real-time streams for monitoring tool events
wikimedia-i18n-l10n-for-tools	Multilingual design for Toolforge tools
wikimedia-ml-services	ML inference (Lift Wing) — often deployed as Toolforge services
wikimedia-phabricator	Report Toolforge bugs and track feature requests
wikimedia-security-and-privacy	Secrets management and data minimization for Toolforge tools
pywikibot	Bot framework — often deployed on Toolforge