Mule Troubleshooting

MuleSoft Troubleshooting and RCA

Trace symptoms across the complete execution path, distinguish the reporting component from the originating component, and require a discriminating check before calling a hypothesis the root cause.

Privacy and reuse boundary

• Treat repositories, logs, payloads, application names, endpoints, organization metadata, correlation identifiers, and audit users as sensitive.
• Use role labels such as Entry API, Orchestrator, System-facing API, Queue, and Target System in reusable examples.
• Never transfer names, identifiers, sample payloads, exact schedules, traffic volumes, error counts, or topology from a prior project into this skill or another project.
• Use actual project identity only when necessary inside the authorized investigation, and keep it out of reusable findings.

Investigation contract

Before analysis, establish:

• symptom, user impact, first-known time, environment, and timezone
• affected request, event, scheduler, batch job, or queue path
• participating applications and external dependencies
• recent deployments or configuration changes
• whether the user wants diagnosis only, a fix plan, or implementation

If the user cannot provide a fact, continue with available evidence and label the gap. Do not ask for information that repository or authorized runtime evidence can answer.

Use these confidence states:

Phase 1: Build the evidence set

1. Map the real execution path

Read the relevant Mule XML, referenced flows, DataWeave, connector configurations, property keys, error handlers, API contract, MUnit tests, deployment files, and current documentation. Follow:

• synchronous calls and their response path
• asynchronous publishes, acknowledgements, and consumers
• retry, reconnection, and fallback scopes
• scatter-gather, parallel-for-each, batch, and queue boundaries
• local and global error handlers

Do not infer API-led layers or dependency ownership from application names alone.

When an isolated reproduction or failing MUnit suite is material, use mule-testing to inventory and classify the test evidence. A correct failing test can support a product hypothesis; a mock mismatch, unfaithful event, or build failure cannot. Keep diagnosis read-only unless the user also requests implementation.

2. Collect aligned telemetry

Confirm Anypoint access before the first connector call, following <skills-root>/mule-ops/references/anypoint-readiness.md. When access is not Ready, offer setup, supplied exports, or a source-only diagnosis, then continue on the chosen path rather than retrying collection tools.

Use the mule-ops collection workflow when available. Otherwise obtain logs and metrics for every participating Mule application and the narrowest useful time window. Record actual coverage before comparing sources.

Prefer enough history to include baseline behavior and multiple occurrences, but do not silently default to a fixed duration when retention or incident timing suggests another window.

When the user supplies the evidence instead, request the artifacts the readiness reference lists — log export, error grouping, latency or memory charts, deployment history — with the application, environment, window, timezone, log level, and any truncation stated. Mark them as user-provided. They reach Correlated only when window and completeness are known, and absence of an entry in an export is not evidence of absence, so it cannot confirm a cause.

3. Identify primary signatures

Group by unique transaction and distinguish:

• originating connector or business error
• timeout or connection exception observed by a caller
• retry or reconnection summary
• error-handler and default exception-listener duplicates
• final response, acknowledgement, retry, dead-letter, or continuation outcome

Use correlation IDs where propagation is verified. If they are absent, use narrow time, route, and operation matching and reduce confidence.

4. Build a timeline

For each affected path, order:

1. trigger received
2. key processing stages
3. dependency request started
4. dependency response or failure
5. retry or fallback
6. error mapping
7. caller response or asynchronous disposition
8. deployment or infrastructure events nearby

This timeline prevents a later summary log or upstream timeout from being mistaken for the first failure.

Phase 2: Test hypotheses

Timeout and connection failures

Inventory each distinct timer instead of comparing similarly named properties blindly:

For a synchronous chain, verify that inner work, retries, and error mapping can finish with margin before the upstream deadline. A connection idle timeout is not automatically the request-processing deadline, so do not apply a universal idleTimeout >= responseTimeout rule across unrelated components.

Discriminate among:

• dependency slowness
• stale pooled connection or remote close
• exhausted connection pool
• caller deadline shorter than valid processing time
• proxy or load-balancer timeout
• CPU, thread, or event-loop starvation
• retry duration exceeding the caller's budget

Concurrency and rate limits

Estimate effective concurrency across the whole path:

• source consumer count
• flow maxConcurrency and connector-specific back pressure
• batch-job maxConcurrency and block size
• parallel scopes and asynchronous handoffs
• replica count and queue distribution
• connection-pool limits
• dependency quotas and rate limits

Treat batch block size as a memory and scheduling parameter, not a direct statement that every record in a block is sent concurrently. Mule processes record blocks in parallel according to batch concurrency and runtime capacity. Confirm behavior with metrics or a controlled load test before choosing a limit.

Evaluate multiple remedies where appropriate:

Do not recommend a numeric value without observed workload, replica count, and dependency capacity.

Error propagation and proxy behavior

A caller-side 5xx or timeout does not identify the origin. Inspect the intermediate application's operation result and error handler. No ERROR-level entry in an intermediate app can mean:

• insufficient log coverage
• failure logged at another level or only in a structured response
• handled or continued error
• transparent proxy behavior
• missing correlation propagation
• failure before or after that application

Keep the cause unresolved until evidence distinguishes these cases.

Recurring mechanism signatures

Use these signatures only as hypothesis starters. Confirm with code and aligned telemetry. Mechanism names match mule-development Classes A–E; also apply its security/configuration, capacity/lifecycle, delivery/transactions, privacy/observability, and validation cross-cutting gates.

When the signature matches Class D disposition gaps, fix classification first: permanent/poison must leave the eligible loop through terminal, quarantine, error-store, or manual-recovery disposition. Intentional indefinite retryable recovery needs backoff, monitoring, ownership, recovery criteria, and replay/manual recovery—not only higher dependency capacity.

Deploy-related bursts

Compare error windows with deployment and replica events. Lifecycle or event-after-stop signatures close to a rolling transition can support a deployment-related hypothesis, but timestamps alone do not prove it.

Before proposing suppression or on-error-continue, verify acknowledgement, persistence, redelivery, idempotency, and message-loss behavior. Never claim that a message will be reprocessed after swallowing an error unless the queue and source semantics prove it.

Memory pressure

Compare heap baseline after GC, allocation slope, full-GC activity, payload size, concurrent work, batch settings, queue depth, and replica divergence. A repeated sawtooth is normally collection activity; a rising post-GC baseline is only a leak hypothesis until retention evidence supports it.

Check for:

• full-payload logging or repeated serialization
• large repeatable in-memory streams
• oversized variables or queue messages
• high batch or parallel concurrency
• unbounded caches or object-store usage
• connector or custom-code resource retention

Phase 3: Produce the RCA or plan

Lead with the impact and highest-confidence conclusion. Include:

1. symptom and affected path
2. evidence coverage and gaps
3. timeline of the failure chain
4. hypotheses considered and discriminating evidence
5. confirmed or best-supported cause with confidence
6. contributing conditions
7. immediate containment options
8. durable fix options with tradeoffs
9. rollback and verification plan

For a fix plan, cite exact repository-relative flows, configuration keys, and tests to change. Keep recommendations separate from current behavior. Do not implement the plan unless requested.

Use Mermaid only when the multi-component sequence or before/after concurrency design is materially clearer than prose. Keep node labels role-based and free of project identity.

Phase 4: Verify a fix

Verify against the failure mechanism, not only deployment success:

1. Confirm the intended version and configuration are running.
2. Exercise the affected path under a representative workload.
3. Observe at least one relevant scheduler, batch, queue, or retry cycle when applicable.
4. Compare equivalent pre-change and post-change windows.
5. Confirm the primary signature is absent or reduced at the unique-transaction level.
6. Confirm latency, throughput, memory, and dependency pressure did not regress.
7. Scan for new errors, back pressure, duplicates, loss, or retry amplification.

If the event has not recurred or telemetry coverage is inadequate, report the result as provisional.

Completion checklist

• Trace all participating components and error handlers.
• Record actual telemetry coverage, timezone, and the Anypoint access state when it limited scope.
• Deduplicate multiple entries from the same transaction.
• Distinguish the first failure from downstream symptoms and retry summaries.
• Calculate the full timeout and concurrency budget before recommending values.
• Treat deploy overlap, missing logs, and sawtooth memory as signals rather than proof.
• Preserve client confidentiality and keep prior-project identity out of reusable examples and current-project output.
• State unresolved alternatives and the next discriminating check.