background-job-designer

name

background-job-designer

description

Designs and scaffolds background job, queue worker, and task scheduling patterns: job definitions, queue configuration, retry policies, dead-letter handling, and monitoring hooks. Use this skill whenever a user says "add a background job for...", "process this asynchronously", "schedule this task", "write a queue worker", "set up a job queue", "add retry logic for this operation", "process emails in the background", "implement a job system", or "how do I run this without blocking the request". Also activate when a user has a slow or unreliable operation that shouldn't block an API response. Supports BullMQ/Node, Celery/Python, and Go goroutine patterns. Do NOT use for real-time event streaming (use Kafka/event-sourcing patterns) or scheduled cron infrastructure setup.

Background Job Designer

Design and scaffold background job queues, workers, retry policies, and monitoring — for operations that shouldn't block the request/response cycle.

When to Use

• Sending emails, notifications, or webhooks asynchronously
• Processing file uploads in the background (resize, convert, analyze)
• Running heavy computations or reports without blocking an API
• Implementing retry-with-backoff for unreliable external API calls
• Scheduling recurring tasks (daily digest, cleanup jobs, billing)

When NOT to Use

• Real-time event streaming between services (Kafka, NATS)
• Simple cron jobs (use a system cron or a hosted scheduler)
• Synchronous workflows where the user needs an immediate result

---

Workflow

Step 1 — Analyze the Job Requirements

Ask for:

1. What the job does: Description and estimated duration
2. Triggers: On-demand (API call) / Scheduled (cron) / Event-driven
3. Reliability needs: At-least-once vs. exactly-once delivery
4. Failure handling: How many retries? Exponential backoff? Dead-letter queue?
5. Stack: Node.js (BullMQ), Python (Celery), Go (async workers)
6. Infrastructure: Redis available for BullMQ/Celery? Or use a hosted queue (SQS, Cloud Tasks)?

Step 2 — Design the Job Architecture

API Request → Enqueue Job → Queue (Redis/SQS) → Worker → Process → Success/Failure
                                                          ↓
                                                    Retry (exponential backoff)
                                                          ↓ (after maxRetries)
                                                    Dead-Letter Queue → Alert

Step 3 — Scaffold with BullMQ (Node.js)

Job definitions:

// jobs/types.ts
export interface SendEmailJobData {
  to:       string;
  subject:  string;
  template: string;
  variables:Record<string, string>;
}

export interface ProcessUploadJobData {
  uploadId: string;
  fileKey:  string;
  userId:   string;
}

Queue configuration:

// jobs/queues.ts
import { Queue, QueueOptions } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis(process.env.REDIS_URL!, { maxRetriesPerRequest: null });

const defaultQueueOpts: QueueOptions = { connection };

export const emailQueue = new Queue<SendEmailJobData>('email', {
  ...defaultQueueOpts,
  defaultJobOptions: {
    removeOnComplete: { count: 100 },
    removeOnFail:     { count: 200 },
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 },
  },
});

export const uploadQueue = new Queue<ProcessUploadJobData>('upload-processing', {
  ...defaultQueueOpts,
  defaultJobOptions: {
    attempts: 5,
    backoff:  { type: 'exponential', delay: 5000 },
  },
});

Worker:

// jobs/workers/emailWorker.ts
import { Worker, Job } from 'bullmq';
import IORedis from 'ioredis';
import { sendEmail } from '../../services/email';
import type { SendEmailJobData } from '../types';

const connection = new IORedis(process.env.REDIS_URL!, { maxRetriesPerRequest: null });

const emailWorker = new Worker<SendEmailJobData>(
  'email',
  async (job: Job<SendEmailJobData>) => {
    const { to, subject, template, variables } = job.data;
    console.log(`[email-worker] Processing job ${job.id} — sending to ${to}`);

    await job.updateProgress(10);
    await sendEmail({ to, subject, template, variables });
    await job.updateProgress(100);
  },
  {
    connection,
    concurrency: 5,  // process 5 jobs concurrently
  }
);

emailWorker.on('completed', job => {
  console.log(`[email-worker] Job ${job.id} completed`);
});

emailWorker.on('failed', (job, err) => {
  console.error(`[email-worker] Job ${job?.id} failed:`, err.message);
});

Enqueue from API route:

// routes/users.ts
import { emailQueue } from '../jobs/queues';

router.post('/users', asyncHandler(async (req, res) => {
  const user = await userService.create(req.body);

  // Enqueue welcome email asynchronously — don't await
  await emailQueue.add('welcome-email', {
    to:       user.email,
    subject:  'Welcome to the platform',
    template: 'welcome',
    variables: { name: user.name },
  });

  res.status(201).json(user);
}));

Dead-letter queue handling:

// jobs/deadLetterHandler.ts
import { QueueEvents } from 'bullmq';

const emailQueueEvents = new QueueEvents('email', { connection });

emailQueueEvents.on('failed', async ({ jobId, failedReason, prev }) => {
  // After all retries exhausted
  const job = await emailQueue.getJob(jobId);
  if (job && job.attemptsMade >= job.opts.attempts!) {
    await alertingService.notify({
      type: 'job_dead_lettered',
      queue: 'email',
      jobId,
      data: job.data,
      error: failedReason,
    });
  }
});

Step 4 — Celery Pattern (Python)

# tasks/email.py
from celery import Celery
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)
app = Celery('tasks', broker=os.environ['REDIS_URL'])
app.conf.task_default_retry_delay = 60  # seconds

@app.task(
    bind=True,
    max_retries=3,
    autoretry_for=(Exception,),
    retry_backoff=True,         # exponential backoff
    retry_jitter=True,
)
def send_welcome_email(self, user_id: str):
    logger.info(f'Sending welcome email for user {user_id}')
    try:
        user = User.objects.get(pk=user_id)
        email_service.send_welcome(user)
    except TemporaryEmailError as exc:
        raise self.retry(exc=exc)
    except Exception as exc:
        logger.error(f'Failed permanently: {exc}')
        raise  # goes to dead-letter / FAILURE state

Step 5 — Monitoring Checklist

Configure:

• Queue depth monitoring (alert if > N jobs pending for > X minutes)
• Failed job alerts (dead-letter queue drain)
• Worker health checks (heartbeat endpoint)
• Job duration p50/p99 metrics

---

Output Format

1. Architecture diagram (text) — request → queue → worker flow
2. Job type definitions — TypeScript interfaces or Python dataclasses
3. Queue configuration — with default retry policy and backoff
4. Worker file — complete with error handling and logging
5. Enqueue example — how to trigger the job from an API route
6. Dead-letter handler — alerting when jobs exhaust retries

---

Safety & Confirmation

• Never enqueue jobs that modify data without idempotency — use a job ID to prevent double-processing.
• Use exponential backoff by default (2s, 4s, 8s) — not fixed retry intervals.
• For exactly-once requirements, note that most queues guarantee at-least-once; idempotency must be handled in the job itself.
• Never put large payloads in a job — put the ID and fetch the data inside the worker.