AEM Workflow Triaging — 6.5 LTS / AMS
Classify workflow issues, determine what logs and data to gather, and map to the correct runbook or log search. Optimized for production support on AEM 6.5 LTS and Adobe Managed Services (AMS).
Audience
AEM 6.5 LTS / AMS operators and developers (and the IDE LLM acting on their behalf) classifying workflow incidents across one or more hosts — using host + time-range + logs (direct filesystem, AMS log access, or Splunk) and read-only JMX metrics, before drilling into a single instance. Use this skill for cross-host log mining and symptom classification; switch to workflow-debugging once the instance and root cause are identified.
Variant Scope
- This skill is 6.5-lts-only (includes AMS).
- Log access via direct filesystem (
crx-quickstart/logs/error.log), AMS log access, or Splunk (if indexed).
- JMX available via Felix Console (
/system/console/jmx) or a JMX client for workflow counts, queue metrics, and remediation.
- Not for AEM as a Cloud Service. If the target is AEMaaCS, stop and use the cloud-service variant of this skill — JMX is not available on cloud production, logs are accessed via Cloud Manager (not the filesystem), and remediation lands through Git + pipeline rather than Felix Console. Several signatures and diagnostic surfaces here do not apply as written on AEMaaCS.
Dependencies
workflow-debugging — once a symptom is classified and a host/instance is identified, route here for the step-by-step runbook and remediation.workflow-debugging/reference.md — diagnostic tool pointers, JMX/config locations, log patterns, and external doc links for 6.5 LTS / AMS.
When to use this skill
- User asks: "Workflow errors on <host> for the past X hours", "Workflow activity on <host>", "Why did workflow X fail?", "What should I collect to debug this workflow ticket?"
- User needs: Symptom classification, log patterns to search, Splunk queries, or required inputs for a runbook.
- Context: AEM 6.5 LTS / AMS (author/publish hostname format).
Step 1: Classify symptom (symptom_id)
Map the user's description to a symptom_id and runbook.
| User says / observes |
symptom_id |
Runbook |
| Workflow not moving to next step; stuck in Running |
workflow_stuck_not_progressing |
runbook-workflow-stuck.md |
| Task should be in Inbox but is not visible |
task_not_in_inbox |
runbook-task-not-in-inbox.md |
| Workflow should start automatically but no instance created |
workflow_not_starting_launcher |
runbook-launcher-not-starting.md |
| Workflow in Failed state or step shows error |
workflow_fails_or_shows_error |
runbook-workflow-fails-or-shows-error.md |
| Step failed after retries; failure item in Inbox |
step_failed_retries_exhausted |
runbook-failed-work-items.md |
| Instance Running but no current work item (inconsistent) |
stale_workflow_no_work_item |
runbook-stale-workflows.md |
| Too many instances; slow queries; disk/repo bloat |
repository_bloat_too_many_instances |
runbook-purge-and-cleanup.md |
| User cannot see work item or complete/delegate/return |
user_cannot_see_or_complete_item |
runbook-inbox-and-permissions.md |
| Cannot delete workflow model (running instances) |
cannot_delete_model |
runbook-model-delete-and-update.md |
| Jobs queued a long time; slow completion; queue depth high |
slow_throughput_queue_backlog |
runbook-job-throughput-and-concurrency.md |
| Auto-advance / timeout jobs not firing; participant step stuck past its configured timeout |
workflow_auto_advance_failure |
runbook-job-throughput-and-concurrency.md |
| New or changed workflow not starting or step not executing |
workflow_setup_validation |
runbook-validate-workflow-setup.md |
Each runbook-*.md above is a symptom section in the workflow-debugging skill, not a separate file to open. That skill's Step 1 maps every symptom_id to a first action, and its numbered steps are the runbook body. Classify here, then hand off there.
WorkItem vs. TaskManager task — do not confuse these. Most workflow Inbox items are workflow work items (WorkItem), created by Participant steps and managed by the workflow engine; they are stored under /var/workflow/instances, not in TaskManager. TaskManager (/var/taskmanagement/tasks) only holds tasks created explicitly via the Task API — used by Projects, Assets tasks, and custom integrations. Both paths are browsable in CRXDE Lite. For task_not_in_inbox and user_cannot_see_or_complete_item symptoms on a workflow: investigate the Participant step assignee configuration, Inbox filters, and workflow permissions — not TaskManager storage. Diagnosing the wrong backend wastes significant time.
Step 2: Required inputs for triage
Before suggesting a runbook or Splunk search, try to obtain:
| Input |
Purpose |
| Host / instance |
Author/publish hostname (e.g. an AMS author/publish host, or on-prem hostname). |
| Time range |
e.g. "past 4 hours", "past 10 hours" – for log/Splunk scope. |
| Workflow model or step name |
e.g. "Dynamic Media Reupload", "DAM Update Asset", "testmodel". |
| Instance ID (if known) |
From Workflow Console URL or payload; ties logs to one instance. |
| Payload path (if known) |
e.g. /content/dam/...; for path-related errors. |
| Log source |
Splunk index/sourcetype, direct filesystem error.log, or AMS log request. |
If the user only provides host + time, respond with the generic workflow error searches and note that narrowing by model/instance ID will improve accuracy.
Step 3: Log patterns and Splunk (what to search)
Logs on 6.5 / AMS are accessible via direct filesystem (crx-quickstart/logs/error.log), AMS log access, or Splunk (if indexed).
| Scenario |
Primary log pattern(s) |
Splunk hint |
| Step failed |
Error executing workflow step |
Add instance ID or model name to narrow. |
| Process not found |
getProcess for '*' failed |
Extract process name for OSGi check in Felix Console (process.label). |
| Stuck at Process step |
Same as step failed + getProcess |
Combine with payload path. |
| Stale workflow |
Cannot archive workitem |
Correlate time with instance ID. |
| Lock / throughput |
refreshing the session since we had to wait for a lock |
Timechart by host. Real lever on 6.5 is queue.maxparallel on the Granite Workflow Queue — see workflow-debugging; reduce concurrent writes to the same path. |
| Permission |
Terminate failed / Resume failed / Suspend failed + verifyAccess |
Or AccessControlException. Check enforceWorkflowInitiatorPermissions. |
| Payload path |
PathNotFoundException + workflow/payload |
Payload deleted, or launcher config path missing. |
| Launcher not starting |
Error adding launcher config / Error retrieving launcher config entries |
Path: /conf/global/settings/workflow/launcher/config. |
| Purge failure |
Workflow purge '*' : |
Filter by repository exception / invalid state. |
| Transient workflow retries exhausted |
retrys exceeded - remove isTransient |
Process step kept throwing after cq.workflow.job.retry retries; instance persisted for admin handling. |
| Thread pool full |
RejectedExecutionException |
default pool saturated with block policy ABORT → timeout/auto-advance jobs dropped. |
| Operation on finished instance |
Workflow is already finished |
Check logic that calls terminate/resume on a completed or aborted instance. |
Example Splunk searches — <your_aem_index> is a placeholder; substitute your organization's AEM log index:
- All workflow step errors (last 24h):
index=<your_aem_index> "Error executing workflow step" | table _time host message | sort - _time
- Process not registered:
index=<your_aem_index> "getProcess for" "failed" | table _time host message
- By workflow model or instance:
index=<your_aem_index> ("Error executing workflow step" OR WorkflowException) (message=*<modelName>* OR message=*<instanceId>*) | sort - _time
- Lock contention:
index=<your_aem_index> "refreshing the session since we had to wait for a lock" | table _time host message
- Thread pool exhaustion (auto-advance impact):
index=<your_aem_index> "RejectedExecutionException" | table _time host message
Note: Index names, sourcetypes, and field names are environment-specific — they depend on how your AEM logs are onboarded to Splunk. There is no fixed index; confirm the correct one with whoever owns your Splunk onboarding before running these. Narrow by host, time range, model, and instance ID — both for accuracy and to avoid over-broad data exposure.
Step 4: JMX-based diagnostics
On 6.5 / AMS, JMX exposes metrics not available from logs alone, plus remediation operations. All workflow maintenance and diagnostic operations live on one MBean, com.adobe.granite.workflow:type=Maintenance (via /system/console/jmx); a second MBean, com.adobe.granite.workflow:type=Statistics, exposes time-series execution metrics for trend analysis.
Targeting the right MBean: match the object name com.adobe.granite.workflow:type=Maintenance exactly (addressed URL-encoded on the JMX console as .../jmx/com.adobe.granite.workflow:type%3DMaintenance). A "workflow" filter over the JMX registry also returns several com.adobe.granite.workflow.core.* event-listener MBeans — those are not the maintenance surface, so select by the full type=Maintenance object name, not a name substring. The entries in the tables below are operations (methods to invoke, each taking an optional model string), not readable attributes.
Triage is diagnostic-first — classify with read-only operations:
| What to check |
JMX operation (read-only) |
Purpose |
| Stale workflow count |
countStaleWorkflows(model) |
Detect stale instances without a current work item |
| Running workflow count |
countRunningWorkflows(model) |
Count active instances for a model |
| Failed workflow count |
returnFailedWorkflowCount(model) |
Count failed instances (overall or per model) |
| Queue depth |
returnWorkflowQueueInfo |
Check Granite Workflow Queue backlog |
| Job statistics |
returnSystemJobInfo |
Sling Job overall stats |
| Job topic info |
returnWorkflowJobTopicInfo |
Per-topic queued/failed/finished counts |
| Purge preview |
purgeCompleted(model, days, dryRun=true) |
Count purgeable instances without deleting |
Destructive operations — do NOT run as part of triage. The operations below change or delete production workflow data. They belong in a deliberate remediation step (see workflow-debugging), not in classification:
| Operation |
Effect |
retryFailedWorkItems(dryRun, model) |
Replays failed work items |
restartStaleWorkflows(model, dryRun) |
Restarts stale instances |
purgeCompleted / purgeActive(model, days, dryRun) |
Deletes completed/active instances |
terminateFailedInstances(restart, dryRun, model) |
Terminates (optionally restarts) failed instances |
Always run dryRun=true first, confirm the count and scope, scope by model where possible, and only execute after the root cause is fixed and with change-control approval.
Step 5: Example triage prompts and responses
| User prompt |
Triage response |
| "Workflow errors on <host> for the past X hours" |
Classify as workflow_fails_or_shows_error / step_failed_retries_exhausted. Search error.log or Splunk for "Error executing workflow step", "Error processing workflow job", "getProcess for … failed". Also check JMX returnSystemJobInfo / returnFailedWorkflowCount for failed counts. Route to workflow-debugging (symptom: workflow_fails_or_shows_error). |
| "Workflow activity on <host> for the past X hours" |
Clarify: "activity" = counts or errors? For counts, use JMX countRunningWorkflows, returnSystemJobInfo (read-only). For errors, use log searches. |
| "Why did <workflow-or-step> fail? Show failure details." |
Need: host, time range, and if possible instance ID. Search error.log for "Error executing workflow step" + model/step name or instance ID. Also check Felix Console → OSGi Components for process.label. Route to workflow-debugging (workflow_fails_or_shows_error). |
| "Task not in Inbox" |
symptom_id: task_not_in_inbox. Confirm it is a workflow work item (/var/workflow/instances), not a TaskManager task. Gather: instance ID, assignee; check Inbox filters and granite.workflow.enforceWorkitemAssigneePermissions in Felix Console. Route to workflow-debugging. |
| "Workflow not starting" |
symptom_id: workflow_not_starting_launcher. Search logs for launcher errors; check launcher config under /conf/global/settings/workflow/launcher/config. Route to workflow-debugging. |
| "Workflow stuck / not progressing" |
symptom_id: workflow_stuck_not_progressing. Use JMX countStaleWorkflows to check for stale instances. If not stale, follow the decision tree by step type. Route to workflow-debugging. |
| "Auto-advance / timeout jobs not firing" |
symptom_id: workflow_auto_advance_failure. Check the Sling Thread Pools page for default pool saturation and block policy ABORT; check the Sling Jobs page for the com/adobe/granite/workflow/timeout/job topic; search error.log for RejectedExecutionException. Route to workflow-debugging (job-throughput-and-concurrency). |
Step 6: What logs and JMX can and cannot answer
Can answer (logs + JMX on 6.5 / AMS):
- Step failures: exception type, message, stack (by host, time, model, step).
- Process not registered: which
process.label is missing (logs + Felix Console).
- Stuck: step errors, getProcess failures, lock wait, payload/path errors.
- Stale: JMX
countStaleWorkflows + "Cannot archive workitem" in logs.
- Queue metrics: JMX
returnWorkflowQueueInfo, returnSystemJobInfo, returnWorkflowJobTopicInfo.
- Running / failed instance counts: JMX
countRunningWorkflows, returnFailedWorkflowCount.
- Throughput: lock wait, session refresh, JobHandler volume.
- Permission: Terminate/Resume/Suspend failed, AccessControlException.
- Payload/launcher: PathNotFoundException, launcher config errors.
- Purge preview: JMX
purgeCompleted(dryRun=true) + "Workflow purge …" in logs.
- Thread pool state: Configuration status ZIP →
039_Sling_Thread_Pools.txt, or the Sling Thread Pools status page.
- Config state: Felix Console or config status ZIP →
003_Configurations.txt.
Cannot answer directly:
- Console state (e.g. "is there a current work item?"). Use the Workflow Console UI (
/libs/cq/workflow/admin/console/content/instances.html).
- Runtime process step code behavior. Requires code review + log correlation.
Always pair log-based triage with read-only JMX diagnostics and the appropriate runbook in workflow-debugging.
Safety & security guardrails
Triage runs against production. Keep it safe:
- Diagnose read-only; never mutate during triage. Classification and data-gathering use read-only JMX operations, log reads, and console status pages. The mutating operations in Step 4 (
retryFailedWorkItems, restartStaleWorkflows, purgeCompleted, purgeActive, terminateFailedInstances) change or delete production workflow data — run them only as a deliberate remediation step (see workflow-debugging): dryRun=true first, scoped by model, after the root cause is fixed, and with change-control approval.
- Least privilege. Felix Console and JMX are admin-level surfaces. Restrict them to operators. Keep
/system/console off public networks — on AMS/on-prem it must sit behind the Dispatcher deny rules and admin authentication, never reachable from publish or the internet.
- No secrets in queries or shared logs. Do not embed credentials, tokens, or API keys in Splunk searches or in anything pasted into a ticket; redact them if present.
- Treat logs and thread dumps as sensitive.
error.log, configuration status ZIPs, and thread dumps can contain payload paths, user IDs, and application data. Share only with authorized parties and redact customer PII before attaching to a ticket.
- Scope every search. Narrow Splunk/log searches by host, time range, model, and instance ID — for accuracy and to minimize data exposure.
References (in repo)
1---2name: workflow-triaging-23description: Triage AEM Workflow issues on AEM 6.5 LTS and AMS by classifying symptoms, gathering the right logs and metrics, and mapping to runbooks or Splunk searches. Use when the user asks for workflow activity/errors on a 6.5 host, needs to classify a Jira ticket, or wants to know what to collect for workflow debugging.4license: Apache-2.05---67# AEM Workflow Triaging — 6.5 LTS / AMS89Classify workflow issues, determine what logs and data to gather, and map to the correct runbook or log search. Optimized for **production support** on **AEM 6.5 LTS** and **Adobe Managed Services (AMS)**.1011## Audience1213AEM 6.5 LTS / AMS operators and developers (and the IDE LLM acting on their behalf) classifying workflow incidents across one or more hosts — using host + time-range + logs (direct filesystem, AMS log access, or Splunk) and read-only JMX metrics, before drilling into a single instance. Use this skill for cross-host log mining and symptom classification; switch to `workflow-debugging` once the instance and root cause are identified.1415## Variant Scope1617- This skill is **6.5-lts-only** (includes AMS).18- Log access via direct filesystem (`crx-quickstart/logs/error.log`), AMS log access, or Splunk (if indexed).19- JMX available via Felix Console (`/system/console/jmx`) or a JMX client for workflow counts, queue metrics, and remediation.20- **Not for AEM as a Cloud Service.** If the target is AEMaaCS, stop and use the cloud-service variant of this skill — JMX is not available on cloud production, logs are accessed via Cloud Manager (not the filesystem), and remediation lands through Git + pipeline rather than Felix Console. Several signatures and diagnostic surfaces here do not apply as written on AEMaaCS.2122## Dependencies2324- `workflow-debugging` — once a symptom is classified and a host/instance is identified, route here for the step-by-step runbook and remediation.25- `workflow-debugging/reference.md` — diagnostic tool pointers, JMX/config locations, log patterns, and external doc links for 6.5 LTS / AMS.2627---2829## When to use this skill3031- User asks: "Workflow errors on <host> for the past X hours", "Workflow activity on <host>", "Why did workflow X fail?", "What should I collect to debug this workflow ticket?"32- User needs: Symptom classification, log patterns to search, Splunk queries, or required inputs for a runbook.33- Context: AEM 6.5 LTS / AMS (author/publish hostname format).3435---3637## Step 1: Classify symptom (symptom_id)3839Map the user's description to a **symptom_id** and runbook.4041| User says / observes | symptom_id | Runbook |42|----------------------|------------|---------|43| Workflow not moving to next step; stuck in Running | workflow_stuck_not_progressing | runbook-workflow-stuck.md |44| Task should be in Inbox but is not visible | task_not_in_inbox | runbook-task-not-in-inbox.md |45| Workflow should start automatically but no instance created | workflow_not_starting_launcher | runbook-launcher-not-starting.md |46| Workflow in Failed state or step shows error | workflow_fails_or_shows_error | runbook-workflow-fails-or-shows-error.md |47| Step failed after retries; failure item in Inbox | step_failed_retries_exhausted | runbook-failed-work-items.md |48| Instance Running but no current work item (inconsistent) | stale_workflow_no_work_item | runbook-stale-workflows.md |49| Too many instances; slow queries; disk/repo bloat | repository_bloat_too_many_instances | runbook-purge-and-cleanup.md |50| User cannot see work item or complete/delegate/return | user_cannot_see_or_complete_item | runbook-inbox-and-permissions.md |51| Cannot delete workflow model (running instances) | cannot_delete_model | runbook-model-delete-and-update.md |52| Jobs queued a long time; slow completion; queue depth high | slow_throughput_queue_backlog | runbook-job-throughput-and-concurrency.md |53| Auto-advance / timeout jobs not firing; participant step stuck past its configured timeout | workflow_auto_advance_failure | runbook-job-throughput-and-concurrency.md |54| New or changed workflow not starting or step not executing | workflow_setup_validation | runbook-validate-workflow-setup.md |5556> Each `runbook-*.md` above is a symptom section in the [`workflow-debugging`](../workflow-debugging/SKILL.md) skill, not a separate file to open. That skill's Step 1 maps every `symptom_id` to a first action, and its numbered steps are the runbook body. Classify here, then hand off there.5758> **WorkItem vs. TaskManager task — do not confuse these.** Most workflow Inbox items are workflow work items (`WorkItem`), created by Participant steps and managed by the workflow engine; they are stored under `/var/workflow/instances`, not in TaskManager. TaskManager (`/var/taskmanagement/tasks`) only holds tasks created explicitly via the Task API — used by Projects, Assets tasks, and custom integrations. Both paths are browsable in CRXDE Lite. For `task_not_in_inbox` and `user_cannot_see_or_complete_item` symptoms on a workflow: investigate the Participant step assignee configuration, Inbox filters, and workflow permissions — not TaskManager storage. Diagnosing the wrong backend wastes significant time.5960---6162## Step 2: Required inputs for triage6364Before suggesting a runbook or Splunk search, try to obtain:6566| Input | Purpose |67|-------|---------|68| **Host / instance** | Author/publish hostname (e.g. an AMS `author`/`publish` host, or on-prem hostname). |69| **Time range** | e.g. "past 4 hours", "past 10 hours" – for log/Splunk scope. |70| **Workflow model or step name** | e.g. "Dynamic Media Reupload", "DAM Update Asset", "testmodel". |71| **Instance ID** (if known) | From Workflow Console URL or payload; ties logs to one instance. |72| **Payload path** (if known) | e.g. `/content/dam/...`; for path-related errors. |73| **Log source** | Splunk index/sourcetype, direct filesystem `error.log`, or AMS log request. |7475If the user only provides host + time, respond with the **generic** workflow error searches and note that narrowing by model/instance ID will improve accuracy.7677---7879## Step 3: Log patterns and Splunk (what to search)8081Logs on 6.5 / AMS are accessible via **direct filesystem** (`crx-quickstart/logs/error.log`), AMS log access, or **Splunk** (if indexed).8283| Scenario | Primary log pattern(s) | Splunk hint |84|----------|------------------------|-------------|85| Step failed | `Error executing workflow step` | Add instance ID or model name to narrow. |86| Process not found | `getProcess for '*' failed` | Extract process name for OSGi check in Felix Console (`process.label`). |87| Stuck at Process step | Same as step failed + `getProcess` | Combine with payload path. |88| Stale workflow | `Cannot archive workitem` | Correlate time with instance ID. |89| Lock / throughput | `refreshing the session since we had to wait for a lock` | Timechart by host. Real lever on 6.5 is `queue.maxparallel` on the Granite Workflow Queue — see `workflow-debugging`; reduce concurrent writes to the same path. |90| Permission | `Terminate failed` / `Resume failed` / `Suspend failed` + verifyAccess | Or `AccessControlException`. Check `enforceWorkflowInitiatorPermissions`. |91| Payload path | `PathNotFoundException` + workflow/payload | Payload deleted, or launcher config path missing. |92| Launcher not starting | `Error adding launcher config` / `Error retrieving launcher config entries` | Path: `/conf/global/settings/workflow/launcher/config`. |93| Purge failure | `Workflow purge '*' :` | Filter by repository exception / invalid state. |94| Transient workflow retries exhausted | `retrys exceeded - remove isTransient` | Process step kept throwing after `cq.workflow.job.retry` retries; instance persisted for admin handling. |95| Thread pool full | `RejectedExecutionException` | `default` pool saturated with block policy `ABORT` → timeout/auto-advance jobs dropped. |96| Operation on finished instance | `Workflow is already finished` | Check logic that calls terminate/resume on a completed or aborted instance. |9798**Example Splunk searches** — `<your_aem_index>` is a placeholder; substitute your organization's AEM log index:99100- All workflow step errors (last 24h):101 `index=<your_aem_index> "Error executing workflow step" | table _time host message | sort - _time`102- Process not registered:103 `index=<your_aem_index> "getProcess for" "failed" | table _time host message`104- By workflow model or instance:105 `index=<your_aem_index> ("Error executing workflow step" OR WorkflowException) (message=*<modelName>* OR message=*<instanceId>*) | sort - _time`106- Lock contention:107 `index=<your_aem_index> "refreshing the session since we had to wait for a lock" | table _time host message`108- Thread pool exhaustion (auto-advance impact):109 `index=<your_aem_index> "RejectedExecutionException" | table _time host message`110111> **Note:** Index names, sourcetypes, and field names are environment-specific — they depend on how your AEM logs are onboarded to Splunk. There is no fixed index; confirm the correct one with whoever owns your Splunk onboarding before running these. Narrow by host, time range, model, and instance ID — both for accuracy and to avoid over-broad data exposure.112113---114115## Step 4: JMX-based diagnostics116117On 6.5 / AMS, JMX exposes metrics not available from logs alone, plus remediation operations. All workflow maintenance and diagnostic operations live on one MBean, `com.adobe.granite.workflow:type=Maintenance` (via `/system/console/jmx`); a second MBean, `com.adobe.granite.workflow:type=Statistics`, exposes time-series execution metrics for trend analysis.118119> **Targeting the right MBean:** match the object name `com.adobe.granite.workflow:type=Maintenance` exactly (addressed URL-encoded on the JMX console as `.../jmx/com.adobe.granite.workflow:type%3DMaintenance`). A "workflow" filter over the JMX registry also returns several `com.adobe.granite.workflow.core.*` event-listener MBeans — those are not the maintenance surface, so select by the full `type=Maintenance` object name, not a name substring. The entries in the tables below are **operations** (methods to invoke, each taking an optional `model` string), not readable attributes.120121**Triage is diagnostic-first — classify with read-only operations:**122123| What to check | JMX operation (read-only) | Purpose |124|---------------|---------------------------|---------|125| Stale workflow count | `countStaleWorkflows(model)` | Detect stale instances without a current work item |126| Running workflow count | `countRunningWorkflows(model)` | Count active instances for a model |127| Failed workflow count | `returnFailedWorkflowCount(model)` | Count failed instances (overall or per model) |128| Queue depth | `returnWorkflowQueueInfo` | Check Granite Workflow Queue backlog |129| Job statistics | `returnSystemJobInfo` | Sling Job overall stats |130| Job topic info | `returnWorkflowJobTopicInfo` | Per-topic queued/failed/finished counts |131| Purge preview | `purgeCompleted(model, days, dryRun=true)` | Count purgeable instances **without deleting** |132133**Destructive operations — do NOT run as part of triage.** The operations below change or delete production workflow data. They belong in a deliberate remediation step (see `workflow-debugging`), not in classification:134135| Operation | Effect |136|-----------|--------|137| `retryFailedWorkItems(dryRun, model)` | Replays failed work items |138| `restartStaleWorkflows(model, dryRun)` | Restarts stale instances |139| `purgeCompleted` / `purgeActive(model, days, dryRun)` | Deletes completed/active instances |140| `terminateFailedInstances(restart, dryRun, model)` | Terminates (optionally restarts) failed instances |141142**Always run `dryRun=true` first, confirm the count and scope, scope by model where possible, and only execute after the root cause is fixed and with change-control approval.**143144---145146## Step 5: Example triage prompts and responses147148| User prompt | Triage response |149|-------------|------------------|150| "Workflow errors on <host> for the past X hours" | Classify as workflow_fails_or_shows_error / step_failed_retries_exhausted. Search error.log or Splunk for "Error executing workflow step", "Error processing workflow job", "getProcess for … failed". Also check JMX `returnSystemJobInfo` / `returnFailedWorkflowCount` for failed counts. Route to `workflow-debugging` (symptom: workflow_fails_or_shows_error). |151| "Workflow activity on <host> for the past X hours" | Clarify: "activity" = counts or errors? For counts, use JMX `countRunningWorkflows`, `returnSystemJobInfo` (read-only). For errors, use log searches. |152| "Why did <workflow-or-step> fail? Show failure details." | Need: host, time range, and if possible instance ID. Search error.log for "Error executing workflow step" + model/step name or instance ID. Also check Felix Console → OSGi Components for `process.label`. Route to `workflow-debugging` (workflow_fails_or_shows_error). |153| "Task not in Inbox" | symptom_id: task_not_in_inbox. Confirm it is a workflow work item (`/var/workflow/instances`), not a TaskManager task. Gather: instance ID, assignee; check Inbox filters and `granite.workflow.enforceWorkitemAssigneePermissions` in Felix Console. Route to `workflow-debugging`. |154| "Workflow not starting" | symptom_id: workflow_not_starting_launcher. Search logs for launcher errors; check launcher config under `/conf/global/settings/workflow/launcher/config`. Route to `workflow-debugging`. |155| "Workflow stuck / not progressing" | symptom_id: workflow_stuck_not_progressing. Use JMX `countStaleWorkflows` to check for stale instances. If not stale, follow the decision tree by step type. Route to `workflow-debugging`. |156| "Auto-advance / timeout jobs not firing" | symptom_id: workflow_auto_advance_failure. Check the Sling Thread Pools page for `default` pool saturation and block policy `ABORT`; check the Sling Jobs page for the `com/adobe/granite/workflow/timeout/job` topic; search error.log for `RejectedExecutionException`. Route to `workflow-debugging` (job-throughput-and-concurrency). |157158---159160## Step 6: What logs and JMX can and cannot answer161162**Can answer (logs + JMX on 6.5 / AMS):**163164- Step failures: exception type, message, stack (by host, time, model, step).165- Process not registered: which `process.label` is missing (logs + Felix Console).166- Stuck: step errors, getProcess failures, lock wait, payload/path errors.167- Stale: JMX `countStaleWorkflows` + "Cannot archive workitem" in logs.168- Queue metrics: JMX `returnWorkflowQueueInfo`, `returnSystemJobInfo`, `returnWorkflowJobTopicInfo`.169- Running / failed instance counts: JMX `countRunningWorkflows`, `returnFailedWorkflowCount`.170- Throughput: lock wait, session refresh, JobHandler volume.171- Permission: Terminate/Resume/Suspend failed, AccessControlException.172- Payload/launcher: PathNotFoundException, launcher config errors.173- Purge preview: JMX `purgeCompleted(dryRun=true)` + "Workflow purge …" in logs.174- Thread pool state: Configuration status ZIP → `039_Sling_Thread_Pools.txt`, or the Sling Thread Pools status page.175- Config state: Felix Console or config status ZIP → `003_Configurations.txt`.176177**Cannot answer directly:**178179- Console state (e.g. "is there a current work item?"). Use the Workflow Console UI (`/libs/cq/workflow/admin/console/content/instances.html`).180- Runtime process step code behavior. Requires code review + log correlation.181182Always pair log-based triage with read-only JMX diagnostics and the appropriate runbook in `workflow-debugging`.183184---185186## Safety & security guardrails187188Triage runs against production. Keep it safe:189190- **Diagnose read-only; never mutate during triage.** Classification and data-gathering use read-only JMX operations, log reads, and console status pages. The mutating operations in Step 4 (`retryFailedWorkItems`, `restartStaleWorkflows`, `purgeCompleted`, `purgeActive`, `terminateFailedInstances`) change or delete production workflow data — run them only as a deliberate remediation step (see `workflow-debugging`): `dryRun=true` first, scoped by model, after the root cause is fixed, and with change-control approval.191- **Least privilege.** Felix Console and JMX are admin-level surfaces. Restrict them to operators. Keep `/system/console` off public networks — on AMS/on-prem it must sit behind the Dispatcher deny rules and admin authentication, never reachable from publish or the internet.192- **No secrets in queries or shared logs.** Do not embed credentials, tokens, or API keys in Splunk searches or in anything pasted into a ticket; redact them if present.193- **Treat logs and thread dumps as sensitive.** `error.log`, configuration status ZIPs, and thread dumps can contain payload paths, user IDs, and application data. Share only with authorized parties and redact customer PII before attaching to a ticket.194- **Scope every search.** Narrow Splunk/log searches by host, time range, model, and instance ID — for accuracy and to minimize data exposure.195196---197198## References (in repo)199200- **Step-by-step runbook (per symptom):** [`../workflow-debugging/SKILL.md`](../workflow-debugging/SKILL.md)201- **Diagnostic tool pointers, JMX/config locations, and log patterns:** [`../workflow-debugging/reference.md`](../workflow-debugging/reference.md)202- **6.5 LTS guardrails (paths, service users, OSGi annotations):** [`../workflow-development/references/workflow-foundation/65-lts-guardrails.md`](../workflow-development/references/workflow-foundation/65-lts-guardrails.md)203- **Intent router / 6.5 LTS capabilities:** [`../SKILL.md`](../SKILL.md)
Adobe