| name | orchestrator |
| description | Runs the full AI content pipeline as a single coordinated workflow using BullMQ and Redis. Use when the user asks to "run the pipeline", "start the campaign", "kick off the full workflow", "orchestrate the agents", "run all agents", or provides a job payload with task_name and task_date. Manages agent dependency ordering, validates skip conditions, enqueues BullMQ jobs, and tracks completion status. Always use this skill when multiple agents need to run in sequence or parallel for a single campaign. Do NOT use for running a single agent in isolation.
|
Orchestrator
Receives a Job Payload and runs the full AI content pipeline as a coordinated BullMQ workflow. Validates inputs, respects skip flags, enforces dependency ordering, and tracks every agent's job status through to the Distribution Agent.
When to Use This Skill
- User says "run the pipeline", "start the campaign", "orchestrate all agents"
- User provides a task_name and task_date and wants all agents to run
- User wants to skip specific agents (research, image, or video) but still run the full pipeline
- User wants to track which jobs are queued, running, complete, or failed
CRITICAL: Dependency Order
The pipeline must always respect this execution order:
Research Agent
│
├──► Ad Creative Designer ─┐
├──► Video Ad Specialist ─┼──► Distribution Agent
└──► Copywriter Agent ─┘
- Research Agent runs first (unless skipped with a validated source folder)
- Ad Creative Designer, Video Ad Specialist, and Copywriter Agent run in parallel after research completes
- Distribution Agent runs last, only after all three middle agents complete (or are skipped)
Step 1: Receive and Validate Job Payload
Accept a Job Payload as JSON. Required and optional fields:
{
"task_name": "coldbrew_campaign",
"task_date": "2026-03-15",
"project_dir": "prj/coldbrew-coffee-co",
"platform_targets": ["instagram", "youtube"],
"skip_research": false,
"skip_image": false,
"skip_video": false,
"source_folder": null
}
Validation Rules
| Field | Required | Rule |
|---|
task_name | Yes | Non-empty string, used in folder names and job IDs |
task_date | Yes | Format YYYY-MM-DD |
platform_targets | Yes | Array containing at least one of: instagram, youtube |
skip_research | No | Default false. If true, source_folder or <project_dir>/assets/<task_name>/ must exist |
skip_image | No | Default false. If true, Ad Creative Designer is marked complete without running |
skip_video | No | Default false. If true, Video Ad Specialist is marked complete without running |
source_folder | No | Path to pre-existing assets folder if skipping research |
Skip Research Validation
If skip_research is true:
Step 2: Enqueue Jobs via BullMQ
Run the orchestrator script to enqueue all jobs:
node pipeline/orchestrator.js --payload '{"task_name":"coldbrew_campaign","task_date":"2026-03-15","platform_targets":["instagram","youtube"]}'
Or using a saved payload file:
node pipeline/orchestrator.js --file pipeline/payloads/coldbrew_demo.json
Or via npm:
npm run pipeline:run
The orchestrator will:
- Validate the payload
- Determine which jobs to skip
- Enqueue each agent as a BullMQ job on the
ai-content-pipeline queue
- Print a job summary table
Expected Output Per Job
{
"job_name": "video_ad_specialist",
"job_id": "42",
"status": "queued",
"dependencies": ["research_agent"],
"notes": "Waiting on: research_agent"
}
Job statuses: queued · running · complete · failed
Skipped jobs return immediately as complete with a note:
{
"job_name": "ad_creative_designer",
"status": "complete",
"dependencies": ["research_agent"],
"notes": "Skipped per user flag: skip_image"
}
Step 3: Start the Worker
In a separate terminal, start the BullMQ worker to process queued jobs:
node pipeline/worker.js
Or via npm:
npm run pipeline:worker
The worker processes up to 3 jobs concurrently (configurable in pipeline/worker.js). Video Ad Specialist and Ad Creative Designer can run in parallel after Research Agent completes.
Step 4: Track Job Status
Monitor job progress by reading log files generated per agent:
<project_dir>/outputs/<task_name>_<date>/logs/
├── research_agent.log
├── ad_creative_designer.log
├── video_ad_specialist.log
├── copywriter_agent.log
├── distribution_agent.log
└── <agent>_error.log ← only on failure
Each log entry includes a timestamp and status message. Report to the user:
- Which agents are queued, running, complete, or failed
- The output path of each completed agent
- Any error messages from failed agents
Step 5: Handle Failures
If a job fails:
- BullMQ retries automatically up to 3 times with exponential backoff (5s base)
- After 3 failures, the job is marked
failed and an error log is written
- Report the failure to the user with the error message and log file path
- Suggest remediation: fix the issue, then re-enqueue the specific failed job by re-running the orchestrator with the same payload
Step 6: Report Pipeline Completion
Once all jobs reach complete (or complete/skipped), generate a final pipeline summary:
── Pipeline Complete: coldbrew_campaign (2026-03-15) ──────────────────
research_agent ✅ complete → <project_dir>/outputs/coldbrew_campaign_2026-03-15/research_results.json
ad_creative_designer ✅ complete → <project_dir>/outputs/coldbrew_campaign_2026-03-15/ads/layout.json
video_ad_specialist ⏭ skipped → skip_video flag
copywriter_agent ✅ complete → <project_dir>/outputs/coldbrew_campaign_2026-03-15/copy/
distribution_agent ✅ complete → <project_dir>/outputs/coldbrew_campaign_2026-03-15/Publish coldbrew_campaign 2026-03-15.md
Next step: Reference "Publish coldbrew_campaign 2026-03-15.md" to trigger publishing.
───────────────────────────────────────────────────────────────────────
Pipeline Output Folder Structure
<project_dir>/outputs/<task_name>_<date>/
├── research_results.json
├── research_brief.md
├── interactive_report.html
├── media_urls.json
├── ads/
│ └── layout.json
├── video/
│ └── ad.mp4
├── copy/
│ ├── instagram_caption.txt
│ ├── threads_post.txt
│ └── youtube_metadata.json
├── logs/
│ ├── research_agent.log
│ ├── copywriter_agent.log
│ └── distribution_agent.log
└── Publish <task_name> <date>.md
Skip Scenarios Reference
| Scenario | Flags | What Happens |
|---|
| Full pipeline | all false | All 5 agents run in order |
| Skip research, have assets | skip_research: true | Verify <project_dir>/assets/<task_name>/, then run remaining agents |
| Image-only campaign | skip_video: true | Video job marked complete, pipeline continues |
| Video-only campaign | skip_image: true | Image job marked complete, pipeline continues |
| Copy + distribute only | skip_research, skip_image, skip_video all true | Runs Copywriter + Distribution only |
Troubleshooting
Redis connection refused
Check that UPSTASH_REDIS_ENDPOINT and UPSTASH_REDIS_PASSWORD are set correctly in .env. Get the values from the Upstash dashboard → your Redis database → Connect → ioredis tab.
Job stuck in queued state
The worker is not running. Start it in a separate terminal: npm run pipeline:worker.
Research skip fails — source folder missing
Upload media assets to <project_dir>/assets/<task_name>/ before re-running. The pipeline will not proceed until this folder exists.
Distribution Agent runs before copy is ready
BullMQ concurrency may have advanced it too quickly. Check that copywriter_agent shows complete in logs before Distribution starts. Adjust worker concurrency in pipeline/worker.js if needed.
Quality Checklist
Before reporting pipeline complete, verify:
