# Trace Delegation

> Trace Astra's current delegation engine: sub-run hierarchy, fan_out/pipeline/sequential/adversarial_review/fork patterns, journal events, verification retries, pause/resume, cancellation, and aggregation. Use when debugging delegated agents, child runs, team orchestration, or delegation progress.

- Skill: `matrixorigin/trace-delegation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add matrixorigin/trace-delegation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/matrixorigin/trace-delegation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: matrixorigin (https://skillmd.com/u/matrixorigin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/matrixorigin/trace-delegation

---


# Trace Delegation

Use this skill to reconstruct what the delegation engine actually did. Prefer durable evidence
from journal events and tracker-visible records over inferred architecture diagrams.

## Current Implementation Map

| Concern | Source |
| --- | --- |
| Engine, tracker, state transitions, verification/checkpoint gates | `crates/runtime/src/server/delegation/engine.rs` |
| HTTP handlers for delegate/list/pause/resume | `crates/runtime/src/server/delegation/handlers.rs` |
| Patterns, tiers, request/result types, aggregation | `crates/services/src/coordination.rs` |
| Journal event builders and metadata fields | `crates/services/src/session_journal.rs` |
| Parent/child messaging and broadcast groups | `crates/astra-messaging/src/` |
| Runtime lifecycle delegation hints | `crates/runtime/src/server/server_loop_host.rs` |

Do not use the old path `runtime/src/server/delegation_engine.rs`; the engine lives under
`runtime/src/server/delegation/`.

## Journal Events

These are the stable delegation events to group by `metadata.delegation_id`:

| Event | Key Metadata |
| --- | --- |
| `DelegationStarted` | `delegation_id`, `parent_run_id`, `pattern`, `agent_ids`, `agent_count` |
| `DelegationSubRunStarted` | `delegation_id`, `sub_run_id`, `parent_run_id`, `agent_id`, `status`, `depth`, `retry_of` |
| `DelegationRetry` | `delegation_id`, `original_run_id`, `retry_run_id`, `agent_id`, `attempt`, `reason` |
| `DelegationSubRunCompleted` | `delegation_id`, `sub_run_id`, `agent_id`, `status`, `error`, `output_preview` |
| `DelegationCompleted` | `delegation_id`, `pattern`, `total_sub_runs`, `succeeded`, `failed`, `aggregated_status`, `aggregated_output_preview` |

Important limitation: journal sub-run completion events do not include per-sub-run token counts.
`DelegationResult` and HTTP `DelegationResponse` carry token totals, but a journal-only trace
cannot reliably reconstruct token distribution.

## Locate Delegation Data

```bash
python3 - <<'PY'
import glob, json, os

events = []
names = {
    "DelegationStarted",
    "DelegationSubRunStarted",
    "DelegationRetry",
    "DelegationSubRunCompleted",
    "DelegationCompleted",
}
for path in glob.glob(os.path.expanduser("~/.astra/sessions/*.jsonl")):
    with open(path, "r", encoding="utf-8", errors="replace") as fh:
        for line_no, line in enumerate(fh, 1):
            try:
                event = json.loads(line)
            except Exception:
                continue
            if event.get("type") in names:
                meta = event.get("metadata") or {}
                events.append((event.get("ts", ""), path, line_no, event.get("type"), meta))

for ts, path, line_no, typ, meta in events[-80:]:
    did = str(meta.get("delegation_id", ""))[:16]
    rid = meta.get("sub_run_id") or meta.get("parent_run_id") or meta.get("retry_run_id") or ""
    print(f"{ts[:19]} | {typ:27s} | {did:16s} | {rid} | {path}:{line_no}")
PY
```

Resolve the target:

| Target | Action |
| --- | --- |
| Delegation ID | Filter events where `metadata.delegation_id` matches |
| Parent run ID | Find `DelegationStarted.metadata.parent_run_id`, then collect its delegation IDs |
| Session ID | Read that session journal only |
| `last` or omitted | Use the most recent `DelegationStarted` |

## Patterns And Aggregation

Pattern names emitted by the engine:

| Pattern | Journal string | Notes |
| --- | --- | --- |
| `CoordinationPattern::FanOut` | `fan_out` | Parallel agents, aggregated after all complete |
| `Pipeline` | `pipeline` | Current engine executes through the sequential path using stage agent IDs |
| `Sequential` | `sequential` | Ordered agents, optional stop-on-success |
| `AdversarialReview` | `adversarial_review` | Producer/reviewer rounds with max rounds |
| `Fork` | `fork` | N tasks sharing parent context, one user-tier agent, constrained recursion |

Aggregation strategies in current code are `FirstSuccess`, `AllResults`, `Consensus`, and
`LlmGuided`. Do not report `Merge` or `VoteOnBest`; those are stale names.

## Trace Workflow

1. Build a timeline from journal events sorted by timestamp and line number.
2. For each `DelegationStarted`, verify the expected pattern and agent set.
3. For each expected child, confirm a `DelegationSubRunStarted` event. Missing starts mean the
   engine registered a delegation but the sub-run did not enter `Running`.
4. For each started child, confirm terminal `DelegationSubRunCompleted`. Missing completion means
   pause, cancellation, crash, timeout, or unfinished state.
5. Fold `DelegationRetry` into the original child. A retry should mark the original as
   `verification_failed` and create a `retry_of` relationship on the retry run.
6. Confirm `DelegationCompleted` totals match the completed child set. A mismatch is a journal or
   aggregation bug.
7. If pause/resume is involved, inspect `pause_delegations_handler`, `resume_delegations_handler`,
   `pause_flags`, `cancel_tokens`, and `cleanup_delegation` in `engine.rs`.

## Failure Classification

| Symptom | Check |
| --- | --- |
| Delegation never starts | Engine configured in `AppState`, handler ownership check, request validation |
| No child runs | `record_sub_run`, `transition_state(..., Running)`, request allowlists, recursion depth |
| Child cannot message parent | mailbox router registration and `MessageTarget::Parent` resolution |
| Retry loop | `VerificationGate::verify`, `max_retries`, `DelegationRetry.reason` |
| Pause/resume stuck | `pause_flags`, child loop pause boundaries, `resume_children_of` |
| Cancellation leak | `cancel_tokens`, `cancel_children_of`, `cleanup_delegation` |
| Wrong aggregate | `AggregationStrategy` and `aggregate_results` in `coordination.rs` |
| Missing UI progress | `progress_broadcaster`, `DelegationProgress`, SSE event emission |

## Report Format

```
Delegation Trace: <delegation_id>

Pattern: <pattern>
Parent run: <parent_run_id>
Expected agents/tasks: <n>

Timeline:
- <ts> started: agents=<...>
- <ts> sub-run started: <run_id> agent=<agent_id> depth=<n> retry_of=<...>
- <ts> retry: <old> -> <new> attempt=<n> reason=<...>
- <ts> sub-run completed: <run_id> status=<status>
- <ts> delegation completed: succeeded=<n> failed=<n> status=<status>

Findings:
- <only concrete gaps or mismatches, with file/event evidence>

Limits:
- <state any data not present in the journal, such as per-sub-run token distribution>
```