nomad

# Nomad ## Purpose Deploy and operate HashiCorp Nomad for workload scheduling, service orchestration, batch processing, and canary deployments with Consul and Vault integration. ## Agent Protocol ### Trigger Exact user phrases: "nomad", "nomad job", "nomad cluster", "nomad server", "nomad client", "nomad ACL", "nomad autoscaler", "nomad pack", "nomad CSI", "nomad batch job", "nomad periodic job", "nomad scaling", "nomad canary", "nomad drain", "nomad template", "nomad service discovery", "nomad connect", "consul connect nomad". ### Input Context - Nomad cluster size and topology. - Integration with Consul and Vault. - Job types: service, batch, system, periodic, parameterized. - Networking: host, bridge, Consul Connect. - Storage: host volumes or CSI. ### Output Artifact Nomad job specification (HCL) or cluster configuration. ### Response Format Nomad HCL job specification. No preamble. ### Completion Criteria - [ ] Job specification written with correct type, task groups, tasks. - [ ] Networking configured (host, bridge, or Consul Connect sidecar). - [ ] Secrets from Vault or template stanza configured. - [ ] Update strategy (canary, rolling, blue-green) defined. - [ ] Scaling policy defined. - [ ] ACL policies and namespace configuration set. - [ ] Monitoring with Nomad autoscaler and Consul health checks. ### Max Response Length 400 lines. ## Quick Start Write `job.hcl` → `nomad job plan job.hcl` (dry run) → `nomad job run job.hcl` → `nomad job status my-job` → `nomad alloc status <alloc-id>` → `nomad job promote my-job` for canary. ## Decision Tree: Job Types | Job Type | Use Case | Lifecycle | Restart Policy | |----------|----------|-----------|----------------| | **service** | Web apps, API servers, long-running | Runs continuously | Always restart on failure | | **batch** | Data processing, migrations | Runs to completion | Retry on failure, then fail | | **system** | Log shippers, node-exporter, CNI | Runs on every client | Always restart, runs everywhere | | **periodic** | Cron jobs, scheduled tasks | Batch job run on cron | Each run is a batch job instance | | **parameterized** | CI/CD jobs, ad-hoc tasks | Dispatched with payload | Each dispatch is a batch job | ## Core Workflow ### Step 1: Nomad Cluster Configuration ```hcl # server.hcl server { enabled = true bootstrap_expect = 3 data_dir = "/opt/nomad/data" } # Client configuration client { enabled = true node_class = "general" servers = ["10.0.1.10:4647", "10.0.1.11:4647", "10.0.1.12:4647"] host_volume "docker-data" { path = "/data/docker" read_only = false } host_volume "prometheus-data" { path = "/data/prometheus" read_only = false } } # Consul integration consul { address = "10.0.1.20:8500" server_service_name = "nomad" client_service_name = "nomad-client" auto_advertise = true } # Vault integration vault { enabled = true address = "https://vault.service.consul:8200" create_from_role = "nomad-cluster" } # ACL configuration acl { enabled = true } # Audit logging audit { enabled = true } ``` ### Step 2: Service Job with Canary ```hcl # webapp.hcl job "webapp" { datacenters = ["dc1"] namespace = "production" type = "service" group "web" { count = 3 network { mode = "bridge" port "http" { to = 8080 } } service { name = "webapp" port = "http" provider = "consul" tags = ["production", "web"] check { type = "http" path = "/healthz" interval = "10s" timeout = "2s" } check_restart { limit = 3 grace = "30s" ignore_warnings = false } } task "web" { driver = "docker" config { image = "myorg/webapp:${NOMAD_ALLOC_INDEX}" ports = ["http"] volumes = [ "local/config.yaml:/etc/webapp/config.yaml", ] } template { data = <<EOH server: port: {{ env "NOMAD_PORT_http" }} environment: {{ key "environments/production/name" }} database: url: {{ with secret "secret/data/webapp" }}{{ .Data.data.database_url }}{{ end }} EOH destination = "local/config.yaml" } resources { cpu = 500 memory = 256 } env { LOG_LEVEL = "info" NODE_NAME = "${node.unique.name}" } } update { max_parallel = 1 canary = 1 min_healthy_time = "30s" healthy_deadline = "5m" progress_deadline = "10m" auto_revert = true auto_promote = false stagger = "30s" } restart { attempts = 3 delay = "15s" interval = "30m" mode = "fail" } reschedule { attempts = 10 interval = "168h" delay = "30s" delay_function = "exponential" max_delay = "1h" } } } ``` ### Step 3: Batch Job (Data Processing) ```hcl job "data-pipeline" { datacenters = ["dc1"] type = "batch" group "process" { count = 1 task "etl" { driver = "docker" config { image = "myorg/etl:latest" args = ["--input", "${NOMARDATA_DIR}/input", "--output", "${NOMARDATA_DIR}/output"] } artifact { source = "https://data.example.com/input/${NOMAD_JOB_NAME}.csv" destination = "local/input" options { checksum = "md5:c2713d50..." } } resources { cpu = 2000 memory = 4096 } logs { max_files = 3 max_file_size = 10 } } } } ``` ### Step 4: System Job (Node-Level) ```hcl job "node-exporter" { datacenters = ["dc1"] type = "system" group "exporter" { network { mode = "host" port "metrics" { static = 9100 } } service { name = "prometheus-node-exporter" port = "metrics" provider = "consul" } task "exporter" { driver = "docker" config { image = "prom/node-exporter:latest" args = [ "--path.rootfs=/host", "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)", ] volumes = ["/:/host:ro,rslave"] } resources { cpu = 100 memory = 64 } } } } ``` ### Step 5: Periodic Job (Scheduled) ```hcl job "db-backup" { datacenters = ["dc1"] type = "batch" periodic { cron = "0 2 * * *" time_zone = "UTC" prohibit_overlap = true } group "backup" { task "pg_dump" { driver = "docker" config { image = "myorg/pg-backup:latest" args = [ "--host", "postgres.service.consul", "--db", "production", "--bucket", "s3://backups/db/" ] } template { data = <<EOH PGPASSWORD={{ with secret "secret/data/postgres" }}{{ .Data.data.password }}{{ end }} AWS_ACCESS_KEY_ID={{ with secret "secret/data/aws" }}{{ .Data.data.access_key }}{{ end }} AWS_SECRET_ACCESS_KEY={{ with secret "secret/data/aws" }}{{ .Data.data.secret_key }}{{ end }} EOH destination = "secrets/env" env = true } resources { cpu = 500 memory = 512 } } } } ``` ### Step 6: Parameterized Job (CI/CD Dispatch) ```hcl job "ci-build" { datacenters = ["dc1"] type = "batch" parameterized { payload = "required" payload_opts = ["repository", "branch", "commit_sha"] } group "build" { task "build" { driver = "docker" config { image = "myorg/ci-runner:latest" args = [ "build", "--repo", "${NOMAD_META_repository}", "--branch", "${NOMAD_META_branch}", "--sha", "${NOMAD_META_commit_sha}", ] payload = "local/payload.tar.gz" } resources { cpu = 2000 memory = 4096 } } } } ``` ```bash # Dispatch a parameterized job nomad job dispatch -meta repository=myorg/app \ -meta branch=main \ -meta commit_sha=abc123 \ ci-build # Check status of specific dispatch nomad job status ci-build/dispatch-abc123 ``` ### Step 7: Consul Connect (Service Mesh) ```hcl job "api" { datacenters = ["dc1"] type = "service" group "api" { count = 3 network { mode = "bridge" } service { name = "api" port = "http" provider = "nomad" connect { sidecar_service { proxy { upstreams { destination_name = "postgres" local_bind_port = 5432 } upstreams { destination_name = "redis" local_bind_port = 6379 } } } } } task "api" { driver = "docker" config { image = "myorg/api:latest" } env { DATABASE_URL = "postgres://user:pass@${NOMAD_UPSTREAM_ADDR_postgres}/db" REDIS_URL = "redis://${NOMAD_UPSTREAM_ADDR_redis}/0" } resources { cpu = 500 memory = 256 } } } } ``` ### Step 8: CSI Volume (PostgreSQL on Nomad) ```hcl # Register volume first: # nomad volume create volume.hcl # volume.hcl type = "csi" id = "postgres-data" name = "postgres-data" capability { access_mode = "single-node-writer" attachment_mode = "file-system" } plugin_id = "org.democratic-csi.nfs" secrets { server = "nfs.example.com" share = "/exports/postgres" } # Job using CSI volume job "postgres" { datacenters = ["dc1"] type = "service" group "db" { volume "data" { type = "csi" source = "postgres-data" access_mode = "single-node-writer" attachment_mode = "file-system" } network { mode = "bridge" port "postgres" { to = 5432 } } service { name = "postgres" port = "postgres" provider = "consul" check { type = "tcp" interval = "10s" timeout = "2s" } } task "postgres" { driver = "docker" config { image = "postgres:16" } volume_mount { volume = "data" destination = "/var/lib/postgresql/data" } env { POSTGRES_PASSWORD = "{{ with secret \"secret/data/postgres\" }}{{ .Data.data.password }}{{ end }}" POSTGRES_DB = "app"