domino-jobs

name	domino-jobs
description	Create, run, and manage Domino Jobs - batch executions for scripts, training, and data processing. Covers job configuration, hardware tiers, scheduled jobs (cron), monitoring status, viewing logs, and API-driven execution. Use when running batch workloads, scheduling recurring tasks, or automating training pipelines.

Domino Jobs Skill

Description

This skill helps users create, run, and manage Domino Jobs - batch executions for running scripts, training models, and processing data.

Activation

Activate this skill when users want to:

• Run a script or notebook as a batch job
• Schedule recurring jobs
• Configure job settings (hardware, environment)
• Monitor job status and results
• Run jobs via API or CLI

What is a Domino Job?

A Job is a batch execution that runs a script or command in Domino. Unlike workspaces, jobs:

• Run to completion without user interaction
• Are fully reproducible with tracked inputs/outputs
• Can be scheduled to run automatically
• Scale to any hardware tier

Creating and Running Jobs

Via Domino UI

1. Navigate to your project
2. Click Jobs in the navigation
3. Click Run
4. Configure:
   • File to Run: Script path (e.g., train.py)
   • Arguments: Command-line arguments (optional)
   • Hardware Tier: Select resources
   • Compute Environment: Select environment
5. Click Start

Via Python (requests)

import requests, os

TOKEN = requests.get("http://localhost:8899/access-token").text.strip()
BASE = os.environ["DOMINO_API_HOST"]
PROJECT_ID = os.environ["DOMINO_PROJECT_ID"]
headers = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}

# Start a job
response = requests.post(
    f"{BASE}/api/jobs/v1/jobs",
    headers=headers,
    json={
        "projectId": PROJECT_ID,
        "runCommand": "python train.py --epochs 100",
        "title": "Training run",
    }
)
job = response.json()
print(f"Job ID: {job['id']}")

Via Domino CLI

# Start a job with script
domino run train.py

# Run with arguments
domino run train.py arg1 arg2 arg3

# Wait for job to complete
domino run --wait train.py arg1 arg2

# Run direct command (not a script)
domino run --direct "pip freeze | grep pandas"

Via REST API

TOKEN=$(curl -s http://localhost:8899/access-token)
curl -X POST "$DOMINO_API_HOST/api/jobs/v1/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"projectId\": \"$DOMINO_PROJECT_ID\",
    \"runCommand\": \"python train.py\",
    \"hardwareTierId\": \"tier-id\",
    \"environmentId\": \"env-id\"
  }"

Job Commands

Run Python Script

python train.py

Run with Arguments

python train.py --data /mnt/data/train.csv --output /mnt/artifacts/model.pkl

Run Jupyter Notebook

jupyter nbconvert --to notebook --execute notebook.ipynb

Run R Script

Rscript analysis.R

Run Shell Script

bash pipeline.sh

Scheduled Jobs

Creating a Scheduled Job

1. Go to Scheduled Jobs in your project
2. Click New Scheduled Job
3. Configure:
   • Name: Descriptive name
   • Command: Script to execute
   • Schedule: Cron expression or preset
   • Hardware Tier: Resources to use
   • Environment: Compute environment

Schedule Examples

Schedule	Cron Expression
Every hour	0 0 * * * ?
Daily at midnight	0 0 0 * * ?
Every Monday 9 AM	0 0 9 ? * MON
First of month	0 0 0 1 * ?

Cron Expression Format

┌───────────── second (0-59)
│ ┌───────────── minute (0-59)
│ │ ┌───────────── hour (0-23)
│ │ │ ┌───────────── day of month (1-31)
│ │ │ │ ┌───────────── month (1-12)
│ │ │ │ │ ┌───────────── day of week (0-7, SUN-SAT)
│ │ │ │ │ │
* * * * * *

Run Modes

Sequential: Wait for previous job to complete before starting next

# Good for jobs that depend on previous output
# Example: Daily model retrain that uses previous day's data

Concurrent: Allow multiple jobs to run simultaneously

# Good for independent jobs
# Example: Hourly data refresh that doesn't depend on previous runs

Job Notifications

Email Notifications

Configure in job settings:

• Success notifications
• Failure notifications
• Specific email addresses

Model API Updates

Trigger Model API republish after job completes:

1. In scheduled job settings
2. Select Update Model API option
3. Choose the Model API to update

Accessing Job Results

Output Files

Files written to /mnt/ directories are available after job completion:

• /mnt/results/ - Custom outputs
• /mnt/artifacts/ - Model artifacts

Job Logs

View logs in Domino UI or via API:

# Get job logs
logs = domino.runs_get_logs(run_id)
print(logs)

Stdout/Stderr

All print statements and errors are captured in job logs.

Environment Variables in Jobs

import os

# Domino-provided
run_id = os.environ.get('DOMINO_RUN_ID')
project_name = os.environ.get('DOMINO_PROJECT_NAME')
username = os.environ.get('DOMINO_USER_NAME')

# Custom (set in project or job settings)
api_key = os.environ.get('MY_API_KEY')

Job Best Practices

1. Parameterize Scripts

import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--data-path', required=True)
parser.add_argument('--model-output', required=True)
parser.add_argument('--epochs', type=int, default=100)
args = parser.parse_args()

2. Log Progress

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("Starting training...")
logger.info(f"Epoch {epoch}/{total_epochs}")
logger.info("Training complete!")

3. Handle Failures Gracefully

try:
    train_model(data)
except Exception as e:
    logger.error(f"Training failed: {e}")
    # Save checkpoint
    save_checkpoint(model, "checkpoint.pt")
    raise

4. Save Artifacts

import joblib

# Save model
joblib.dump(model, "/mnt/artifacts/model.joblib")

# Save metrics
with open("/mnt/artifacts/metrics.json", "w") as f:
    json.dump(metrics, f)

Monitoring Jobs

Job Status

• Pending: Waiting for resources
• Running: Currently executing
• Succeeded: Completed successfully
• Failed: Exited with error
• Stopped: Manually cancelled

Check Status via API

status = domino.runs_status(run_id)
print(f"Status: {status['status']}")
print(f"Started: {status['startedAt']}")

Stop a Running Job

domino.runs_stop(run_id)

Troubleshooting

Job Fails Immediately

• Check script syntax
• Verify file paths exist
• Check environment has required packages

Job Times Out

• Increase hardware tier resources
• Optimize code performance
• Check for infinite loops

Out of Memory

• Use larger hardware tier
• Optimize data loading (chunking, generators)
• Clear variables when no longer needed

API Reference

Before writing or verifying any API call, use the cluster swagger to confirm current endpoint paths and field names. Use public docs for workflow context and field explanations.

Get the cluster base URL: $DOMINO_API_HOST (injected by Domino into every workspace, job, and app).

Fetch the swagger spec:

# No authentication required for the public API spec
curl "$DOMINO_API_HOST/assets/public-api.json"
# Browser UI: $DOMINO_API_HOST/assets/lib/swagger-ui/index.html?url=/assets/public-api.json#/

Public docs (workflow context and field explanations):

• API Guide
• Create and run Jobs
• Scheduled Jobs
• Work with Jobs