Workflow Debugging

AEM Workflow Debugging — Cloud Service

Production-grade debugging for the AEM Granite Workflow engine, launcher, Inbox, Sling Jobs, thread pools, and purge on AEM as a Cloud Service (AEMaaCS).

Audience

AEMaaCS developers and operators (and the IDE LLM acting on their behalf) diagnosing stuck or failed workflows on a local AEMaaCS SDK or a cloud environment — Developer Console, Sling Job Console, and Cloud Manager Logs available; no production Felix Console JMX or filesystem access.

Variant Scope

• AEM as a Cloud Service only.
• Not for AEM 6.5 LTS / AMS. If the target is 6.5 LTS, stop and load the 6.5-lts variant of this skill — JMX-based remediation, Felix Console runtime config, AMS log filesystem access, and jstack thread dumps documented there do not apply on AEMaaCS.
• No JMX access on AEMaaCS production. Diagnosis is read-only via Developer Console, Sling Job Console, and Cloud Manager Logs. Never recommend JMX-based remediation (restartStaleWorkflows, purgeCompleted, terminate, retryFailedWorkItems) — those are 6.5-LTS-only mechanisms. Use Inbox Retry, Purge Scheduler (OSGi config in Git), custom servlets like StaleWorkflowServlet, and Cloud Manager pipeline-driven config changes instead.
• All remediation lands via Git + Cloud Manager pipeline: OSGi configs in ui.config, custom servlets in core, ACLs in ui.apps/.../repoinit. There is no Felix Console write access on cloud environments.
• See reference.md for runbook locations and diagnostic tool pointers.

Dependencies

This skill is largely self-contained but routes back into the dev skills when the root cause is a code or model defect:

• workflow-development — when the diagnosis is "process step throws / not registered / leaks resources"
• workflow-model-design — when the diagnosis is "model has wrong split rule / missing transition / wrong step type"
• workflow-launchers — when the diagnosis is "launcher not firing / re-trigger loop"
• workflow-triaging — load instead of this skill if the user is mining Cloud Manager Logs across multiple environments rather than diagnosing one

When to use this skill

• Workflow stuck, not progressing, failed, not starting, task not in Inbox, purge/repository bloat, permissions, queue backlog, thread pool exhaustion, auto-advancement not working.
• User provides thread dumps, Sling Job console output, or error.log excerpts from Cloud Manager.
• Environment: AEM as a Cloud Service (no JMX; use Developer Console, Cloud Manager logs).

Step 1: Map symptom to first action

Step 2: Decision tree (workflow stuck)

1. No current work item? → Stale. Deploy a custom StaleWorkflowServlet to your core bundle — this is a privileged endpoint; secure it per the checklist in Step 6 before deploying. Call GET /bin/support/workflow/stale?dryRun=true to enumerate, then POST ...?dryRun=false to restart.
2. Participant step → Assignee exists? Inbox visible? Payload accessible? Dynamic participant resolver returning correct user?
3. Process step → Search error.log for instance ID. Check: process.label registered, payload path exists, bundle active, no exception in execute().
4. OR/AND Split → Condition evaluates correctly? Routes exist? No dead-end branches? Model synced?

Step 3: Thread dump & thread pool analysis

Thread dumps and status-producer output on AEMaaCS are obtained via Developer Console → Status → Thread Dump / OSGi / Sling Jobs / Sling Scheduler. For anything not exposed in Developer Console, open an Adobe Support ticket — never attempt to SSH into an AEMaaCS pod.

3a. Sling default thread pool (critical path)

The Sling Scheduler ApacheSlingdefault uses ThreadPool: default. This pool runs:

• Oak observation events
• All Quartz-scheduled jobs — including the workflow timeout-detection scheduler that emits com/adobe/granite/workflow/timeout/job events to the Sling Job system (the job itself then runs on the Granite Workflow Queue, see Step 3c)

Check in thread pool output (Developer Console → Status → Sling Thread Pools, /system/console/status-slingthreadpools):

If active count = max pool size AND block policy = ABORT:

• New scheduled tasks (including workflow timeout/auto-advance jobs) are silently rejected
• This is the #1 cause of auto-advancement failure

Check in thread dump:

• Search for sling-default- threads
• If all threads show same stack (e.g. stuck on HTTP call, database, or external service), that's the blocking culprit
• Note elapsed time — threads stuck for hours indicate a hung external call without timeout

3b. Sling Job thread pool

Check Apache Sling Job Thread Pool:

• active count vs max pool size
• If saturated, Sling Jobs cannot execute (workflow jobs stall)

3c. Granite Workflow Queue

Check the Sling Jobs page (/system/console/slingevent):

Check topic statistics for workflow model:

• Topic: com/adobe/granite/workflow/job/var/workflow/models/<modelName>
• High Failed Jobs / low Finished Jobs ratio → process step throwing exceptions

Check Granite Workflow Queue configuration:

• Type: Topic Round Robin
• Max Parallel: 0.5 OOTB on the AEMaaCS SDK (50% of available CPU cores). Adobe's Workflows Best Practices recommends between half and three-quarters of processor cores. Verify the running value at Developer Console → /system/console/slingevent before assuming. Override via org.apache.sling.event.jobs.QueueConfiguration-<alias>.cfg.json in Git if you need to raise it.
• Max Retries: 10

3d. Sling Scheduler

Check the Sling Scheduler page (Developer Console → Status → Sling Scheduler, /system/console/status-slingscheduler):

• This page lists Quartz-style schedulers, not Sling Job topics. The workflow-related entry visible here is the periodic WorkflowStatsMBean collector (used by the Statistics MBean). nextFireTime: null means the trigger was deregistered.
• The com/adobe/granite/workflow/timeout/job topic itself is a Sling Job, not a Quartz job — check it on the Sling Jobs page (/system/console/slingevent), not here.
• Confirm ApacheSlingdefault uses ThreadPool: default — that's how the periodic timeout-detection scheduler reaches the workflow engine.

Step 4: Error log patterns

Download error.log from Cloud Manager → Environments → Logs, or use log streaming.

Step 5: Configuration checklist (Cloud Service — all via Git + pipeline)

Every config below is an OSGi JSON file under ui.config/src/main/content/jcr_root/apps/<project>/osgiconfig/config.author/ (or config.author.prod/ / config.author.stage/ for run-mode scoping).

AEMaaCS caveat — Sling DefaultThreadPool config may be platform-reserved

Some Sling core configs are filtered or overridden by the AEMaaCS platform layer. The DefaultThreadPool config may land via pipeline and be silently ignored. Always verify after deploy:

• Open /system/console/status-Threads (Developer Console → Status → Threads).
• Find the default pool row.
• Confirm maxPoolSize and block policy reflect your config values.

If the numbers don't change, the PID is Adobe-managed on your environment — do not try to work around it. Open an Adobe Support ticket, attach a thread dump and the thread-pool status, and request Engineering lift the pool size or change block policy for that environment.

Permission and identity gotchas specific to AEMaaCS

• Never list IMS user IDs in cq.workflow.superuser. IMS principals change on re-invite. Reference a JCR group that repoinit creates. Correct repoinit syntax:

  create group workflow-administrators
  add admin to group workflow-administrators
  

  The group keyword after to is required — without it the repoinit parser fails and the entire script aborts on startup.
• Custom process steps that need elevated access must use a service user + ServiceUserMapperImpl.amended-*.cfg.json, not resolver.adaptTo(Session.class) + admin.
• enforceWorkflowInitiatorPermissions=true + an initiator who is a rotated IMS user leaves workflows unterminable except by superuser. On Cloud Service, prefer superuser-group membership for anyone expected to recover workflows.

Step 6: Remediation quick reference (Cloud Service)

Securing a custom support servlet (e.g. StaleWorkflowServlet, bulk-retry servlet): These endpoints restart, replay, or purge workflows and must be treated as privileged. A servlet that does any of this with no access control is a remote-code-style hazard — do not ship one without all of the following:

• Author-only, non-public path. Bind it under an author-only path such as /bin/support/... and confirm your Dispatcher/CDN config does not route /bin/support on publish. Where practical, bind by resource type behind an access-controlled resource rather than an open path.
• Authorize every request in code. Resolve the request user and verify membership in an operations/admin group (created via repoinit) before doing any work; return 403 otherwise. Never rely on the path alone for protection.
• Service user for the elevated work — never an admin session. Map a dedicated sub-service via org.apache.sling.serviceusermapping.impl.ServiceUserMapperImpl.amended-* and grant it only the workflow ACLs it needs (see workflow-development). Never call loginAdministrative().
• Default to dry run. Treat a missing or invalid dryRun as true; require an explicit dryRun=false on POST for the destructive path. Log the caller, scope (model, instance count), and outcome for audit.
• Scope and cap. Always support &model=<modelId> so an operator can limit blast radius, and cap the number of instances acted on per call.

Pod-restart reality on AEMaaCS: Cloud Manager does not expose a customer-facing pod-restart or env-restart action. The only way a customer can trigger a restart is an Adobe Support ticket. A restart bounces the running author/publish node — in-flight authoring sessions are lost, active jobs are requeued, there is no hot-swap. Treat it as last-resort mitigation, not a fix, and always file the long-term code/config fix in the same support conversation.

Step 7: Common root cause patterns (from real incidents)

Pattern A: Thread pool starvation → auto-advance failure

Symptom: Workflow auto-advancement stops; timeout jobs not firing; workflows stuck at participant step despite timeout configured.

Root cause chain:

1. Custom scheduler makes blocking HTTP call without timeout
2. concurrent = true allows overlapping executions on each cron trigger
3. Each stuck execution consumes a default pool thread indefinitely
4. All pool threads consumed (OOTB on the AEMaaCS SDK that's 10; production environments may differ) → pool saturated
5. If block policy has been changed to ABORT (OOTB is RUN), new Quartz triggers are rejected silently; on RUN they instead pile up on the caller thread and back-pressure the dispatch
6. The workflow timeout-detection scheduler cannot dispatch new com/adobe/granite/workflow/timeout/job events
7. Auto-advancement never happens

Diagnosis checklist:

• Thread pool output: Pool default → active count = max pool size?
• Thread pool output: Pool default → block policy = ABORT?
• Thread dump: All sling-default-* threads stuck on same stack?
• Sling Jobs output: Workflow job topic has high Failed Jobs?
• Sling Scheduler output: ThreadPool = default for ApacheSlingdefault?

Fix: Request pod restart (immediate mitigation, coordinate with customer — see Step 6 caveat); commit scheduler fix (HTTP timeout, scheduler.concurrent=false) to Git; land org.apache.sling.commons.threads.impl.DefaultThreadPool-default.cfg.json with blockPolicy=RUN and maxPoolSize=50; deploy via Cloud Manager pipeline.

Pattern B: High workflow job failure rate

Symptom: numberOfFailedJobs >> numberOfFinishedJobs for a workflow topic.

Root cause: Process step exception, payload deleted, or process not registered.

Diagnosis: Download or stream error.log from Cloud Manager → Environments → Logs; grep for Error executing workflow step + model name. Cross-check the process.label in Developer Console → OSGi → Components against the model's Process step.

Pattern C: Stale workflows accumulating

Symptom: Workflows in RUNNING state but no work items; Inbox empty despite running instances.

Root cause: Cannot archive workitem during transition; JCR session crash during step completion.

Diagnosis: grep Cloud Manager logs for Cannot archive workitem. For live count, deploy a custom StaleWorkflowServlet (secure it per the Step 6 checklist) and invoke GET /bin/support/workflow/stale?dryRun=true — it returns a JSON report without side effects.

Routing back to dev skills

Once diagnosis identifies a code or model defect (not an operational issue on a healthy implementation), route back into the development skills:

References

• For runbook locations and diagnostic tool pointers: see reference.md