daemon-development

license	Apache-2.0
name	daemon-development
description	Build daemon/background processes that start on boot, run continuously, and manage their own lifecycle. Covers macOS launchd (plist files, agents vs daemons), Linux systemd (unit files), Windows services, process supervision, logging, health checks, graceful shutdown, auto-restart, and AI-powered daemons that manage LLM API connections and rate limits. Activate on: "daemon", "background process", "launchd", "systemd", "service file", "plist", "launch agent", "launch daemon", "auto-start", "always running", "process supervisor", "pm2", "background service", "boot service", "AI daemon", "long-running process". NOT for: container orchestration (use devops-automator), cron jobs that run and exit (use task-scheduler), web server deployment (use backend-architect).
allowed-tools	Read,Write,Edit,Bash,Glob,Grep,WebSearch,WebFetch
metadata	{"category":"Infrastructure & DevOps","tags":["daemon","launchd","systemd","background-process","process-supervision","lifecycle","devops","ai-daemon"],"pairs-with":[{"skill":"always-on-agent-architecture","reason":"Always-on agents are daemons with AI-specific lifecycle needs"},{"skill":"devops-automator","reason":"Deployment and service management overlap"},{"skill":"background-job-orchestrator","reason":"Background jobs run inside daemon processes"}]}
category	Backend & Infrastructure
tags	["daemon","background-process","service","system-programming","linux"]

Daemon Development

Build long-running background processes that start reliably, run continuously, recover from failures, and shut down gracefully. Expert-level daemon architecture across macOS launchd, Linux systemd, and AI-powered services.

Decision Points

1. Platform-Specific Init System Choice

Platform Detected?
├─ macOS
│  ├─ Must run at boot (no user login): LaunchDaemon → /Library/LaunchDaemons/
│  ├─ User session required (GUI/files): LaunchAgent → ~/Library/LaunchAgents/
│  └─ System-wide user service: LaunchAgent → /Library/LaunchAgents/
├─ Linux
│  ├─ systemd available: systemd unit file → /etc/systemd/system/
│  ├─ Legacy SysV: init.d script (rare, avoid if possible)
│  └─ Container: s6 or built-in supervision
└─ Cross-platform dev
   ├─ Node.js app: pm2 for development, systemd/launchd for production
   └─ Other languages: Direct systemd/launchd implementation

2. Service Type Configuration

Daemon Startup Behavior?
├─ Simple process (doesn't fork)
│  ├─ systemd: Type=simple
│  └─ launchd: Standard plist (no special keys)
├─ Signals readiness when ready
│  ├─ systemd: Type=notify + sd_notify("READY=1")
│  └─ launchd: N/A (use health check instead)
├─ Forks child process (legacy)
│  ├─ systemd: Type=forking + PIDFile (avoid)
│  └─ launchd: Not supported (rewrite to not fork)
└─ Socket-activated
   ├─ systemd: Type=simple + [Socket] section
   └─ launchd: Sockets dict in plist

3. Restart Policy Design

Failure Recovery Strategy?
├─ Critical service (must always run)
│  ├─ systemd: Restart=always, RestartSec=5
│  └─ launchd: KeepAlive=true, ThrottleInterval=10
├─ Crash recovery only
│  ├─ systemd: Restart=on-failure
│  └─ launchd: KeepAlive={SuccessfulExit=false}
├─ Manual restart preferred
│  ├─ systemd: Restart=no
│  └─ launchd: KeepAlive=false
└─ Rate-limited restart
   ├─ systemd: StartLimitBurst=5, StartLimitIntervalSec=60
   └─ launchd: ThrottleInterval=30 (built-in)

4. AI Daemon Rate Limiting Strategy

LLM API Connection Pattern?
├─ Single provider, token bucket
│  ├─ Token estimation: prompt_tokens + max_completion_tokens
│  ├─ Bucket refill: tokens_per_minute from provider limits
│  └─ Overflow: Queue requests with priority
├─ Multi-provider failover
│  ├─ Circuit breaker per provider (3 failures = 30s timeout)
│  ├─ Rate limit per provider independently
│  └─ Failover order: primary → secondary → queue
├─ Streaming responses
│  ├─ Reserve tokens optimistically
│  ├─ Adjust on actual_tokens in real-time
│  └─ Handle mid-stream rate limits gracefully
└─ Batch processing
   ├─ Group similar requests to maximize throughput
   └─ Split large batches if they hit rate limits

5. Graceful Shutdown Handling

SIGTERM Received?
├─ Web server daemon
│  ├─ 1. server.close() - stop accepting new connections
│  ├─ 2. Wait for active requests (timeout: TimeoutStopSec-5s)
│  ├─ 3. Close database connections
│  └─ 4. exit(0)
├─ Queue worker daemon
│  ├─ 1. Stop polling for new jobs
│  ├─ 2. Finish current job (timeout protection)
│  ├─ 3. Flush any pending state
│  └─ 4. exit(0)
├─ AI daemon
│  ├─ 1. Stop accepting new LLM requests
│  ├─ 2. Drain in-flight requests (respect provider timeouts)
│  ├─ 3. Save rate limit state to disk
│  └─ 4. Close provider connections, exit(0)
└─ Database/stateful daemon
   ├─ 1. Checkpoint/flush transactions
   ├─ 2. Close client connections gracefully
   ├─ 3. Release file locks
   └─ 4. exit(0)