| name | workflow-debugging |
| description | Debug AEM Workflow issues on AEM as a Cloud Service including stuck workflows, failed steps, missing Inbox tasks, launcher failures, stale instances, thread pool exhaustion, queue backlogs, purge failures, and permissions errors. Use when the user reports workflow problems on Cloud Service, asks why a workflow is stuck or failed, needs step-by-step troubleshooting, or provides thread dumps, configuration output, or Sling Job console output for analysis. |
| license | Apache-2.0 |
AEM Workflow Debugging — Cloud Service
Production-grade debugging for AEM Granite Workflow engine, launcher, Inbox, Sling Jobs, thread pools, and purge on AEM as a Cloud Service.
Variant Scope
- This skill is cloud-service-only.
- No JMX access — use Developer Console, Cloud Manager logs, custom HTTP APIs.
- Config changes require code in Git and Cloud Manager pipeline deploy.
When to use this skill
- Workflow stuck, not progressing, failed, not starting, task not in Inbox, purge/repository bloat, permissions, queue backlog, thread pool exhaustion, auto-advancement not working.
- User provides thread dumps, Sling Job console output, or error.log excerpts from Cloud Manager.
- Environment: AEM as a Cloud Service (no JMX; use Developer Console, Cloud Manager logs).
Step 1: Map symptom to runbook
| Symptom | symptom_id | Runbook | First action |
|---|
| Workflow stuck (not advancing) | workflow_stuck_not_progressing | runbook-workflow-stuck.md | Open instance; note current step type. No work item → stale. |
| Task not in Inbox | task_not_in_inbox | runbook-task-not-in-inbox.md | Confirm Participant step; assignee = logged-in user; Inbox filters. |
| Workflow not starting (launcher) | workflow_not_starting_launcher | runbook-launcher-not-starting.md | Launcher enabled; path/event match payload. |
| Workflow fails or shows error | workflow_fails_or_shows_error | runbook-workflow-fails-or-shows-error.md | Instance history; error.log for instance ID; payload and process. |
| Step failed, retries exhausted | step_failed_retries_exhausted | runbook-failed-work-items.md | Logs → process.label → Inbox retry. |
| Stale (no current work item) | stale_workflow_no_work_item | runbook-stale-workflows.md | Custom API/script to detect and restart stale workflows. |
| Repository bloat / too many instances | repository_bloat_too_many_instances | runbook-purge-and-cleanup.md | Purge Scheduler configuration. |
| User cannot see or complete item | user_cannot_see_or_complete_item | runbook-inbox-and-permissions.md | Assignee/initiator/superuser; enforce flags. |
| Cannot delete model | cannot_delete_model | runbook-model-delete-and-update.md | Count running workflows → terminate → delete. |
| Slow throughput / queue backlog | slow_throughput_queue_backlog | runbook-job-throughput-and-concurrency.md | Sling Job statistics; max.procs; Sling thread pool. |
| Auto-advancement not working | workflow_auto_advance_failure | runbook-job-throughput-and-concurrency.md | Check default thread pool saturation; Sling Scheduler; timeout jobs. |
| New workflow not working | workflow_setup_validation | runbook-validate-workflow-setup.md | Model sync, launcher, process registration, permissions. |
Step 2: Decision tree (workflow stuck)
- No current work item? → Stale. Use custom API/script to detect and restart stale workflows.
- Participant step → Assignee exists? Inbox visible? Payload accessible? Dynamic participant resolver returning correct user?
- Process step → Search error.log for instance ID. Check:
process.label registered, payload path exists, bundle active, no exception in execute().
- OR/AND Split → Condition evaluates correctly? Routes exist? No dead-end branches? Model synced?
Step 3: Thread dump & thread pool analysis
Thread dumps on Cloud Service are obtained via Developer Console or by requesting from support. Configuration output may also need to be requested from support.
3a. Sling default thread pool (critical path)
The Sling Scheduler ApacheSlingdefault uses ThreadPool: default. This pool fires:
com/adobe/granite/workflow/timeout/job (auto-advancement)- Oak observation events
- All Quartz-scheduled jobs
Check in thread pool output:
| Field | Healthy | Problem |
|---|
| active count | < max pool size | = max pool size (saturated) |
| block policy | RUN | ABORT (rejects tasks when full) |
| max pool size | ≥ 20 | Low values starve schedulers |
If active count = max pool size AND block policy = ABORT:
- New scheduled tasks (including workflow timeout/auto-advance jobs) are silently rejected
- This is the #1 cause of auto-advancement failure
Check in thread dump:
- Search for
sling-default- threads
- If all threads show same stack (e.g. stuck on HTTP call, database, or external service), that's the blocking culprit
- Note
elapsed time — threads stuck for hours indicate a hung external call without timeout
3b. Sling Job thread pool
Check Apache Sling Job Thread Pool:
- active count vs max pool size
- If saturated, Sling Jobs cannot execute (workflow jobs stall)
3c. Granite Workflow Queue
Check Sling Jobs output:
| Field | Healthy | Problem |
|---|
| Queued Jobs (overall) | 0 | > 0 (jobs waiting) |
| Failed Jobs | 0 | > 0 (step failures) |
| Active Jobs | 0-N | 0 when Queued > 0 (jobs not picked up) |
Check topic statistics for workflow model:
- Topic:
com/adobe/granite/workflow/job/var/workflow/models/<modelName>
- High
Failed Jobs / low Finished Jobs ratio → process step throwing exceptions
Check Granite Workflow Queue configuration:
- Type: Topic Round Robin
- Max Parallel: 1 (default; consider increasing for throughput)
- Max Retries: 10
3d. Sling Scheduler
Check Sling Scheduler output:
- Verify
com/adobe/granite/workflow/timeout/job scheduled jobs exist
nextFireTime: null → job already fired or deregistered- Verify which ThreadPool the scheduler uses (should be
default)
Step 4: Error log patterns
Download error.log from Cloud Manager → Environments → Logs, or use log streaming.
| Pattern | Cause | Action |
|---|
Error executing workflow step | Process step exception | Check stack; fix process code or payload |
getProcess for '<name>' failed | No WorkflowProcess registered | Deploy bundle; match process.label |
Cannot archive workitem | Archive failure → stale risk | Detect and restart stale workflows |
refreshing the session since we had to wait for a lock | Lock contention | Increase cq.workflow.job.max.procs via config in Git |
Terminate failed / Resume failed / Suspend failed | Permissions (not initiator/superuser) | Check enforceWorkflowInitiatorPermissions; add to superusers |
PathNotFoundException (workflow/payload) | Payload/launcher path missing | Verify payload exists; check launcher config path |
Error adding launcher config | Launcher config path not created | Create /conf/global/settings/workflow/launcher/config |
retrys exceeded - remove isTransient | Transient workflow failed after retries | Fix process code; instance persisted for admin handling |
RejectedExecutionException | Thread pool full with ABORT policy | Increase pool size or change policy to RUN via config; fix stuck threads |
Workflow is already finished | Terminate on completed/aborted instance | Check logic calling terminate |
Workflow purge '<name>' : repository exception | Purge JCR error | Check permissions; repo health |
Step 5: Configuration checklist
Config changes on Cloud Service require OSGi config files in the Git repository, deployed via Cloud Manager pipeline.
| Config | Property | Check |
|---|
| WorkflowSessionFactory | cq.workflow.job.retry | Default 3; increase for flaky steps |
| WorkflowSessionFactory | cq.workflow.job.max.procs | -1 = CPU cores; increase for throughput |
| WorkflowSessionFactory | granite.workflow.enforceWorkitemAssigneePermissions | true = only assignee sees items |
| WorkflowSessionFactory | granite.workflow.enforceWorkflowInitiatorPermissions | true = only initiator can terminate |
| WorkflowSessionFactory | cq.workflow.superuser | Must include admin users/groups |
| DefaultThreadPool (default) | block policy | ABORT can reject timeout jobs; prefer RUN |
| DefaultThreadPool (default) | max pool size | 20 default; increase if many schedulers |
| Granite Workflow Queue | Max Parallel | 1 default; increase for throughput |
| Purge Scheduler | scheduledpurge.daysold | 30 default; tune per environment |
Step 6: Remediation quick reference
| Action | Cloud Service approach |
|---|
| Retry failed work item | Inbox Retry; no JMX available |
| Restart stale workflows | Custom API/script; no JMX available |
| Purge completed | Purge Scheduler (OSGi config in Git) |
| Increase parallelism | Config in Git: cq.workflow.job.max.procs; deploy via pipeline |
| Fix thread pool exhaustion | Restart instance (immediate); fix stuck scheduler code; change block policy to RUN via config |
| Fix process not found | Deploy bundle; process.label must match; Sync model |
| Fix auto-advancement | Verify default pool not saturated; timeout jobs scheduled; block policy = RUN |
Step 7: Common root cause patterns (from real incidents)
Pattern A: Thread pool starvation → auto-advance failure
Symptom: Workflow auto-advancement stops; timeout jobs not firing; workflows stuck at participant step despite timeout configured.
Root cause chain:
- Custom scheduler makes blocking HTTP call without timeout
concurrent = true allows overlapping executions on each cron trigger- Each stuck execution consumes a
default pool thread indefinitely
- All 20 threads consumed → pool saturated
- Block policy = ABORT → new Quartz jobs rejected silently
- Workflow timeout jobs (
com/adobe/granite/workflow/timeout/job) cannot fire
- Auto-advancement never happens
Diagnosis checklist:
Fix: Restart instance (immediate); fix scheduler code (add HTTP timeout, set concurrent=false); change pool policy to RUN; increase pool size. All config changes via Git + pipeline.
Pattern B: High workflow job failure rate
Symptom: numberOfFailedJobs >> numberOfFinishedJobs for a workflow topic.
Root cause: Process step exception, payload deleted, or process not registered.
Diagnosis: Search error.log (from Cloud Manager) for Error executing workflow step + model name. Check process.label in OSGi components.
Pattern C: Stale workflows accumulating
Symptom: Workflows in RUNNING state but no work items; Inbox empty despite running instances.
Root cause: Cannot archive workitem during transition; JCR session crash during step completion.
Diagnosis: Search for Cannot archive workitem in Cloud Manager logs; use custom API to count stale workflows.
References
