# Eventloop Audit

> Systematically audits any Armeria-based Java project for event loop blocking issues — discovers project patterns, scans for blocking operations, traces call chains, and produces a fix plan without modifying code. Use when experiencing latency spikes, connection timeouts, deadlocks, or thread starvation in Armeria services, or as a pre-release audit. Invoked as `/eventloop-audit` for the full project or `/eventloop-audit <path>` for a specific directory.

- Skill: `line/eventloop-audit` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add line/eventloop-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/line/eventloop-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: line (https://skillmd.com/u/line)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/line/eventloop-audit

---


# Event Loop Blocking Audit for Armeria Projects

## Overview

A systematic, multi-phase audit that discovers how a project uses Armeria, scans for all categories of event loop blocking, traces call chains to determine thread context, and produces a structured report with fix plans. **No code is modified** — only analysis and planning.

This skill is **framework-generic** — it works on any Armeria-based Java project by discovering the project's patterns dynamically, not assuming specific class names or directory structures.

## When to Use

- After setting up a new Armeria-based project and want to verify thread safety
- Before a release to audit for latent event loop blocking issues
- When experiencing unexplained latency spikes, connection timeouts, or deadlocks in an Armeria server
- After integrating a new library that may contain blocking calls
- As a periodic health check on the codebase's non-blocking discipline

## Thread Model Reference

For a comprehensive guide on Armeria's thread pools, executor selection, and programmatic usage patterns, see [armeria-thread-pool-guide.md](references/armeria-thread-pool-guide.md).

**The Golden Rule**: If `Thread.currentThread() instanceof NonBlocking` — you **MUST NOT** block. Armeria's event loop threads implement the `NonBlocking` marker interface. Any blocking operation on these threads stalls ALL connections sharing that event loop, causing cascading latency and potential deadlocks.

---

## Phase 1: Discovery — Understand the Project's Armeria Usage

Before searching for problems, discover HOW the project uses Armeria. Launch an **Explore agent** (or do it yourself) with these specific searches. Record findings as they'll inform all subsequent phases.

### 1.1 Find Armeria Server Configuration

```text
Grep: Server\.builder\(\)|ServerBuilder
Scope: **/*.java
```

Read each match and document:
- Worker group configuration (`workerGroup(n)`, `serviceWorkerGroup(n)`)
- Blocking executor configuration (`blockingTaskExecutor(...)`)
- Decorator chain (`.decorate(...)` calls)
- Request/idle timeout settings
- Any custom `EventLoopGroup` creation

### 1.2 Find HttpService Implementations

```text
Grep: implements HttpService|extends AbstractHttpService|extends SimpleDecoratingHttpService
Scope: **/*.java
```

These classes have `serve()` methods that run on the event loop. List all of them — they are the primary audit targets.

### 1.3 Find Annotated Services

```text
Grep: @Get\(|@Post\(|@Put\(|@Delete\(|@Head\(|@Patch\(|@Options\(
Scope: **/*.java
```

For each, check if the method or class has `@Blocking`. Methods without `@Blocking` run on the event loop.

### 1.4 Find gRPC Services

```text
Grep: extends \w+ImplBase|GrpcService\.builder\(\)
Scope: **/*.java
```

Check whether `useBlockingTaskExecutor(true)` is configured in the `GrpcService.builder()`. If not, all gRPC handler methods run on the event loop.

### 1.5 Find Thrift Services

```text
Grep: THttpService\.builder\(\)|THttpService\.of\(
Scope: **/*.java
```

Same check — look for `useBlockingTaskExecutor(true)`.

### 1.6 Find Decorators

```text
Grep: SimpleDecoratingHttpService|DecoratingHttpServiceFunction|\.decorate\(
Scope: **/*.java
```

Every decorator's `serve()` method runs on the event loop. List them all.

### 1.7 Find Custom Executors and Thread Pools

```text
Grep: blockingTaskExecutor\(\)|ctx\.eventLoop\(\)|Executors\.new|new ThreadPoolExecutor|new ForkJoinPool|ExecutorService
Scope: **/*.java
```

Document all custom thread pools and how they're used. These are the "safe zones" where blocking is OK.

### 1.8 Find Offloading Patterns

```text
Grep: supplyAsync|thenApplyAsync|thenComposeAsync|thenAcceptAsync|ctx\.blockingTaskExecutor|makeContextAware
Scope: **/*.java
```

Understand the project's conventions for offloading work from the event loop. Look for:
- **Base class methods** that submit work to a blocking executor (e.g., a `runBlocking(Supplier)` method in a base handler)
- **Utility classes** that wrap `ctx.blockingTaskExecutor()` with additional logic (cancellation, metrics, tracing)
- **Project-specific annotations** that trigger blocking executor dispatch (beyond Armeria's `@Blocking`)

A wrapper is "safe" if it provably submits the lambda/callable to a non-event-loop executor before the lambda body executes. Read the wrapper's implementation to verify — don't trust method names alone.

### 1.9 Build Architecture Map

Synthesize findings into a concise map:
- **Event loop entry points**: List of classes/methods where the event loop calls into application code
- **Safe execution wrappers**: Project-specific methods that offload to blocking executors
- **Thread pool inventory**: All thread pools, their sizing, and purpose
- **Decorator chain**: The order in which decorators wrap services

---

## Phase 2: Systematic Detection — 7 Categories

Search the entire project (or scoped path) for each category of blocking pattern. Exclude test files (`**/test/**`, `**/tests/**`) and generated code (`**/generated/**`, `**/target/**`, `**/build/**`).

See [detection-patterns.md](references/detection-patterns.md) for the complete search patterns and triage guidance for all 7 categories:

1. **Direct Blocking Calls** — `.join()`, `.get()`, `Thread.sleep()`, `.await()`, `.lock()`, etc.
2. **Blocking I/O Operations** — File I/O, network, JDBC, serialization, encryption
3. **Synchronization** — `synchronized`, `ReentrantLock`, `StampedLock` on request paths
4. **CompletableFuture Continuations on Wrong Executor** — `.thenApply()` without `Async` where completing thread is event loop
5. **Thread Pool Misconfiguration** — Unbounded pools, missing shutdown, undersized blocking executor
6. **Missing Armeria Idioms** — Missing `@Blocking`, missing `useBlockingTaskExecutor(true)`, blocking cancel handlers, missing context propagation
7. **Transitive / Hidden Blocking** — DNS resolution, class loading, synchronous logging appenders

---

## Phase 3: Call Chain Analysis

For each finding from Phase 2, perform call chain analysis to determine the thread context.

### Analysis Steps

1. **Identify enclosing method** — What method contains the potentially blocking call?

2. **Find all callers** — Grep for the method name across the project:
   ```text
   Grep: methodName\(
   Scope: **/*.java
   ```
   When handling overloaded methods (same name, different signatures), read each call site to confirm it calls the blocking variant. Also check superclass/interface hierarchies — a `serve()` override in a subclass is still called from the event loop if the parent is an `HttpService`.

3. **Trace to entry point** — Follow the call chain upward until you reach one of:
   - An Armeria entry point (`serve()`, `@Get`/`@Post` handler, gRPC handler, decorator)
   - A thread pool submission (`blockingTaskExecutor().execute(...)`, `supplyAsync(...)`, etc.)
   - A framework callback (`whenRequestCancelling`, `setOnCancelHandler`)
   - A lambda/method reference passed to an executor (`.submit(this::methodName)`)

   **Handling ambiguity**: If a method is called from BOTH event loop paths AND thread pool paths, classify based on the worst case (event loop path). Note both paths in the finding.

4. **Determine thread context** — At the entry point:
   - Is there a `@Blocking` annotation? → Safe
   - Is there `useBlockingTaskExecutor(true)`? → Safe
   - Is it inside a `supplyAsync(..., blockingExecutor)` call? → Safe
   - Is it inside a project-specific safe wrapper (discovered in Phase 1.8)? → Safe
   - None of the above? → **Event loop**

5. **Classify severity using this decision tree:**

```text
Is the blocking call provably inside a blocking executor or thread pool?
├── YES → SAFE (exclude from report or list separately)
├── NO, it's provably on the event loop:
│   ├── Is it an I/O operation, .join(), .get() without timeout, or Thread.sleep()?
│   │   └── YES → CRITICAL
│   ├── Is it .get(timeout, unit) or .await(timeout, unit)?
│   │   └── YES → HIGH (blocks event loop for the timeout duration)
│   ├── Is it a synchronized block or lock?
│   │   ├── Can it contend with a thread that holds the lock for >1ms?
│   │   │   └── YES → HIGH
│   │   └── Low contention, very fast critical section?
│   │       └── LOW
│   └── Is it a very fast operation (System.getProperty, in-memory map lookup)?
│       └── LOW
└── UNDETERMINED (e.g., CF continuation where completing thread varies):
    ├── Could the continuation body block?
    │   ├── YES → MEDIUM (must verify at runtime or refactor to be safe)
    │   └── NO (continuation is pure computation) → LOW
    └── Is the source future always completed by a thread pool?
        └── YES → SAFE
```

**Key distinction**: MEDIUM means "we cannot statically prove the thread context" — it requires either runtime verification (add logging/assertion) or defensive refactoring (use `*Async` variant with explicit executor).

---

## Phase 4: Fix Plan Templates

For each non-SAFE finding, recommend one of the fix approaches documented in [fix-templates.md](references/fix-templates.md):

| Template | When to use |
|----------|-------------|
| **FP-1** | Replace `.join()`/`.get()` with async chaining |
| **FP-2** | Offload blocking work to `ctx.blockingTaskExecutor()` |
| **FP-3** | Add `@Blocking` annotation to annotated service methods |
| **FP-4** | Enable `useBlockingTaskExecutor(true)` for gRPC/Thrift services |
| **FP-5** | Use `*Async` CF variant with explicit executor |
| **FP-6** | Add timeout to unavoidable blocking |
| **FP-7** | Make thread pool initialization thread-safe |
| **FP-8** | Wire context propagation with `makeContextAware` |

**Document the recommendation — do not apply it.**

---

## Phase 5: Produce the Report

### Per-Finding Format

```markdown
## Finding [ID]: [Short descriptive title]

**Severity**: CRITICAL / HIGH / MEDIUM / LOW
**File**: [relative path]:[line number]
**Thread context**: Event loop / Blocking executor / Undetermined
**Detection category**: [1-7 with category name]

### Description
[What the code does and why it's a blocking risk on the event loop]

### Call chain
[Armeria entry point (event loop)] → [Method A] → [Method B] → [blocking call at line X]

### Recommended fix
[Reference FP-1 through FP-8, with specific application to this case:
 which method to change, what the new code should look like, which executor to use]

### Risk assessment
[What could break when applying the fix, edge cases to consider, tests to run]
```

### Summary Table

At the end of all findings, produce a summary:

```markdown
| # | Severity | File | Line | Category | Blocking Pattern | Fix |
|---|----------|------|------|----------|-----------------|-----|
| 1 | CRITICAL | FooService.java | 42 | 1 - Direct blocking | `.join()` | FP-1 |
| 2 | HIGH | BarHandler.java | 88 | 2 - I/O | FileInputStream | FP-2 |
| ... | ... | ... | ... | ... | ... | ... |
```

### Statistics

```markdown
- Total findings: [N]
- CRITICAL: [N] | HIGH: [N] | MEDIUM: [N] | LOW: [N] | SAFE (excluded): [N]
- By category: Cat.1: [N] | Cat.2: [N] | Cat.3: [N] | Cat.4: [N] | Cat.5: [N] | Cat.6: [N] | Cat.7: [N]
```

### Follow-Up Recommendations

Always include these recommendations regardless of findings:

1. **Runtime detection**: Consider adding [BlockHound](https://github.com/reactor/BlockHound) to the test suite — it detects blocking calls on non-blocking threads at runtime
2. **Async logging**: Verify that the logging framework uses async appenders in production to avoid blocking on `log.*()` calls
3. **Event loop monitoring**: Enable Armeria's built-in event loop blocking detection:
   ```text
   -Dcom.linecorp.armeria.reportBlockedEventLoop=true
   ```
4. **Code review checklist**: Add "Does this handler offload blocking work?" to the team's code review checklist for Armeria services

---

## Execution Checklist

Follow this checklist to ensure nothing is skipped. Check off each item as you complete it.

### Phase 1: Discovery
- [ ] 1.1 — Found and documented Armeria server configuration(s)
- [ ] 1.2 — Listed all `HttpService` implementations
- [ ] 1.3 — Listed all annotated services, noted `@Blocking` presence/absence
- [ ] 1.4 — Listed all gRPC services, noted `useBlockingTaskExecutor` presence/absence
- [ ] 1.5 — Listed all Thrift services (if any)
- [ ] 1.6 — Listed all decorators
- [ ] 1.7 — Documented all custom executors and thread pools
- [ ] 1.8 — Identified project-specific offloading patterns
- [ ] 1.9 — Built architecture map (entry points, safe wrappers, thread pools)

### Phase 2: Detection
- [ ] Category 1 — Searched for all direct blocking calls
- [ ] Category 2 — Searched for all blocking I/O operations
- [ ] Category 3 — Searched for synchronization on request paths
- [ ] Category 4 — Audited CompletableFuture continuation executors
- [ ] Category 5 — Audited thread pool configurations
- [ ] Category 6 — Checked for missing Armeria idioms (@Blocking, useBlockingTaskExecutor, cancel handlers, context propagation)
- [ ] Category 7 — Searched for transitive/hidden blocking (DNS, class loading, logging config)

### Phase 3: Analysis
- [ ] Traced call chains for ALL non-obvious findings
- [ ] Classified every finding by severity
- [ ] Excluded SAFE findings from the report (or listed them separately)

### Phase 4: Fix Planning
- [ ] Assigned a fix template (FP-1 through FP-8) to each non-SAFE finding
- [ ] Documented specific fix details for each finding

### Phase 5: Reporting
- [ ] Produced per-finding reports
- [ ] Produced summary table
- [ ] Produced statistics
- [ ] Included follow-up recommendations

---

## Common Mistakes to Avoid

- **Stopping at grep hits** — A grep hit is not a finding. You must trace the call chain to determine thread context. Many `.join()` calls are inside thread pool code and are perfectly safe.
- **Ignoring non-async CF continuations** — `.thenApply()` without `Async` is the most subtle source of event loop blocking. The continuation runs on whatever thread completes the source future.
- **Missing cancel handlers** — Cancel handler lambdas always run on the event loop. They're easy to overlook but are a common source of blocking.
- **Assuming `synchronized` is always bad** — It's only a problem if (a) it's on a request path reachable from the event loop AND (b) the lock can be contended by another thread holding it for a long time.
- **Forgetting transitive blocking** — A clean `serve()` method that calls `helperMethod()` which calls `utilMethod()` which calls `.join()` is still blocking the event loop.
- **Ignoring logging** — If the logging framework uses synchronous file appenders, every `log.info()` on the event loop is a blocking I/O operation. This is a configuration issue, not a code issue, but still critical.