Trace Delegation
Use this skill to reconstruct what the delegation engine actually did. Prefer durable evidence
from journal events and tracker-visible records over inferred architecture diagrams.
Current Implementation Map
| Concern |
Source |
| Engine, tracker, state transitions, verification/checkpoint gates |
crates/runtime/src/server/delegation/engine.rs |
| HTTP handlers for delegate/list/pause/resume |
crates/runtime/src/server/delegation/handlers.rs |
| Patterns, tiers, request/result types, aggregation |
crates/services/src/coordination.rs |
| Journal event builders and metadata fields |
crates/services/src/session_journal.rs |
| Parent/child messaging and broadcast groups |
crates/astra-messaging/src/ |
| Runtime lifecycle delegation hints |
crates/runtime/src/server/server_loop_host.rs |
Do not use the old path runtime/src/server/delegation_engine.rs; the engine lives under
runtime/src/server/delegation/.
Journal Events
These are the stable delegation events to group by metadata.delegation_id:
| Event |
Key Metadata |
DelegationStarted |
delegation_id, parent_run_id, pattern, agent_ids, agent_count |
DelegationSubRunStarted |
delegation_id, sub_run_id, parent_run_id, agent_id, status, depth, retry_of |
DelegationRetry |
delegation_id, original_run_id, retry_run_id, agent_id, attempt, reason |
DelegationSubRunCompleted |
delegation_id, sub_run_id, agent_id, status, error, output_preview |
DelegationCompleted |
delegation_id, pattern, total_sub_runs, succeeded, failed, aggregated_status, aggregated_output_preview |
Important limitation: journal sub-run completion events do not include per-sub-run token counts.
DelegationResult and HTTP DelegationResponse carry token totals, but a journal-only trace
cannot reliably reconstruct token distribution.
Locate Delegation Data
python3 - <<'PY'
import glob, json, os
events = []
names = {
"DelegationStarted",
"DelegationSubRunStarted",
"DelegationRetry",
"DelegationSubRunCompleted",
"DelegationCompleted",
}
for path in glob.glob(os.path.expanduser("~/.astra/sessions/*.jsonl")):
with open(path, "r", encoding="utf-8", errors="replace") as fh:
for line_no, line in enumerate(fh, 1):
try:
event = json.loads(line)
except Exception:
continue
if event.get("type") in names:
meta = event.get("metadata") or {}
events.append((event.get("ts", ""), path, line_no, event.get("type"), meta))
for ts, path, line_no, typ, meta in events[-80:]:
did = str(meta.get("delegation_id", ""))[:16]
rid = meta.get("sub_run_id") or meta.get("parent_run_id") or meta.get("retry_run_id") or ""
print(f"{ts[:19]} | {typ:27s} | {did:16s} | {rid} | {path}:{line_no}")
PY
Resolve the target:
| Target |
Action |
| Delegation ID |
Filter events where metadata.delegation_id matches |
| Parent run ID |
Find DelegationStarted.metadata.parent_run_id, then collect its delegation IDs |
| Session ID |
Read that session journal only |
last or omitted |
Use the most recent DelegationStarted |
Patterns And Aggregation
Pattern names emitted by the engine:
| Pattern |
Journal string |
Notes |
CoordinationPattern::FanOut |
fan_out |
Parallel agents, aggregated after all complete |
Pipeline |
pipeline |
Current engine executes through the sequential path using stage agent IDs |
Sequential |
sequential |
Ordered agents, optional stop-on-success |
AdversarialReview |
adversarial_review |
Producer/reviewer rounds with max rounds |
Fork |
fork |
N tasks sharing parent context, one user-tier agent, constrained recursion |
Aggregation strategies in current code are FirstSuccess, AllResults, Consensus, and
LlmGuided. Do not report Merge or VoteOnBest; those are stale names.
Trace Workflow
- Build a timeline from journal events sorted by timestamp and line number.
- For each
DelegationStarted, verify the expected pattern and agent set.
- For each expected child, confirm a
DelegationSubRunStarted event. Missing starts mean the
engine registered a delegation but the sub-run did not enter Running.
- For each started child, confirm terminal
DelegationSubRunCompleted. Missing completion means
pause, cancellation, crash, timeout, or unfinished state.
- Fold
DelegationRetry into the original child. A retry should mark the original as
verification_failed and create a retry_of relationship on the retry run.
- Confirm
DelegationCompleted totals match the completed child set. A mismatch is a journal or
aggregation bug.
- If pause/resume is involved, inspect
pause_delegations_handler, resume_delegations_handler,
pause_flags, cancel_tokens, and cleanup_delegation in engine.rs.
Failure Classification
| Symptom |
Check |
| Delegation never starts |
Engine configured in AppState, handler ownership check, request validation |
| No child runs |
record_sub_run, transition_state(..., Running), request allowlists, recursion depth |
| Child cannot message parent |
mailbox router registration and MessageTarget::Parent resolution |
| Retry loop |
VerificationGate::verify, max_retries, DelegationRetry.reason |
| Pause/resume stuck |
pause_flags, child loop pause boundaries, resume_children_of |
| Cancellation leak |
cancel_tokens, cancel_children_of, cleanup_delegation |
| Wrong aggregate |
AggregationStrategy and aggregate_results in coordination.rs |
| Missing UI progress |
progress_broadcaster, DelegationProgress, SSE event emission |
Report Format
Delegation Trace: <delegation_id>
Pattern: <pattern>
Parent run: <parent_run_id>
Expected agents/tasks: <n>
Timeline:
- <ts> started: agents=<...>
- <ts> sub-run started: <run_id> agent=<agent_id> depth=<n> retry_of=<...>
- <ts> retry: <old> -> <new> attempt=<n> reason=<...>
- <ts> sub-run completed: <run_id> status=<status>
- <ts> delegation completed: succeeded=<n> failed=<n> status=<status>
Findings:
- <only concrete gaps or mismatches, with file/event evidence>
Limits:
- <state any data not present in the journal, such as per-sub-run token distribution>
1---2name: trace-delegation3description: Trace Astra's current delegation engine: sub-run hierarchy, fan_out/pipeline/sequential/adversarial_review/fork patterns, journal events, verification retries, pause/resume, cancellation, and aggregation. Use when debugging delegated agents, child runs, team orchestration, or delegation progress.4---56# Trace Delegation78Use this skill to reconstruct what the delegation engine actually did. Prefer durable evidence9from journal events and tracker-visible records over inferred architecture diagrams.1011## Current Implementation Map1213| Concern | Source |14| --- | --- |15| Engine, tracker, state transitions, verification/checkpoint gates | `crates/runtime/src/server/delegation/engine.rs` |16| HTTP handlers for delegate/list/pause/resume | `crates/runtime/src/server/delegation/handlers.rs` |17| Patterns, tiers, request/result types, aggregation | `crates/services/src/coordination.rs` |18| Journal event builders and metadata fields | `crates/services/src/session_journal.rs` |19| Parent/child messaging and broadcast groups | `crates/astra-messaging/src/` |20| Runtime lifecycle delegation hints | `crates/runtime/src/server/server_loop_host.rs` |2122Do not use the old path `runtime/src/server/delegation_engine.rs`; the engine lives under23`runtime/src/server/delegation/`.2425## Journal Events2627These are the stable delegation events to group by `metadata.delegation_id`:2829| Event | Key Metadata |30| --- | --- |31| `DelegationStarted` | `delegation_id`, `parent_run_id`, `pattern`, `agent_ids`, `agent_count` |32| `DelegationSubRunStarted` | `delegation_id`, `sub_run_id`, `parent_run_id`, `agent_id`, `status`, `depth`, `retry_of` |33| `DelegationRetry` | `delegation_id`, `original_run_id`, `retry_run_id`, `agent_id`, `attempt`, `reason` |34| `DelegationSubRunCompleted` | `delegation_id`, `sub_run_id`, `agent_id`, `status`, `error`, `output_preview` |35| `DelegationCompleted` | `delegation_id`, `pattern`, `total_sub_runs`, `succeeded`, `failed`, `aggregated_status`, `aggregated_output_preview` |3637Important limitation: journal sub-run completion events do not include per-sub-run token counts.38`DelegationResult` and HTTP `DelegationResponse` carry token totals, but a journal-only trace39cannot reliably reconstruct token distribution.4041## Locate Delegation Data4243```bash44python3 - <<'PY'45import glob, json, os4647events = []48names = {49 "DelegationStarted",50 "DelegationSubRunStarted",51 "DelegationRetry",52 "DelegationSubRunCompleted",53 "DelegationCompleted",54}55for path in glob.glob(os.path.expanduser("~/.astra/sessions/*.jsonl")):56 with open(path, "r", encoding="utf-8", errors="replace") as fh:57 for line_no, line in enumerate(fh, 1):58 try:59 event = json.loads(line)60 except Exception:61 continue62 if event.get("type") in names:63 meta = event.get("metadata") or {}64 events.append((event.get("ts", ""), path, line_no, event.get("type"), meta))6566for ts, path, line_no, typ, meta in events[-80:]:67 did = str(meta.get("delegation_id", ""))[:16]68 rid = meta.get("sub_run_id") or meta.get("parent_run_id") or meta.get("retry_run_id") or ""69 print(f"{ts[:19]} | {typ:27s} | {did:16s} | {rid} | {path}:{line_no}")70PY71```7273Resolve the target:7475| Target | Action |76| --- | --- |77| Delegation ID | Filter events where `metadata.delegation_id` matches |78| Parent run ID | Find `DelegationStarted.metadata.parent_run_id`, then collect its delegation IDs |79| Session ID | Read that session journal only |80| `last` or omitted | Use the most recent `DelegationStarted` |8182## Patterns And Aggregation8384Pattern names emitted by the engine:8586| Pattern | Journal string | Notes |87| --- | --- | --- |88| `CoordinationPattern::FanOut` | `fan_out` | Parallel agents, aggregated after all complete |89| `Pipeline` | `pipeline` | Current engine executes through the sequential path using stage agent IDs |90| `Sequential` | `sequential` | Ordered agents, optional stop-on-success |91| `AdversarialReview` | `adversarial_review` | Producer/reviewer rounds with max rounds |92| `Fork` | `fork` | N tasks sharing parent context, one user-tier agent, constrained recursion |9394Aggregation strategies in current code are `FirstSuccess`, `AllResults`, `Consensus`, and95`LlmGuided`. Do not report `Merge` or `VoteOnBest`; those are stale names.9697## Trace Workflow98991. Build a timeline from journal events sorted by timestamp and line number.1002. For each `DelegationStarted`, verify the expected pattern and agent set.1013. For each expected child, confirm a `DelegationSubRunStarted` event. Missing starts mean the102 engine registered a delegation but the sub-run did not enter `Running`.1034. For each started child, confirm terminal `DelegationSubRunCompleted`. Missing completion means104 pause, cancellation, crash, timeout, or unfinished state.1055. Fold `DelegationRetry` into the original child. A retry should mark the original as106 `verification_failed` and create a `retry_of` relationship on the retry run.1076. Confirm `DelegationCompleted` totals match the completed child set. A mismatch is a journal or108 aggregation bug.1097. If pause/resume is involved, inspect `pause_delegations_handler`, `resume_delegations_handler`,110 `pause_flags`, `cancel_tokens`, and `cleanup_delegation` in `engine.rs`.111112## Failure Classification113114| Symptom | Check |115| --- | --- |116| Delegation never starts | Engine configured in `AppState`, handler ownership check, request validation |117| No child runs | `record_sub_run`, `transition_state(..., Running)`, request allowlists, recursion depth |118| Child cannot message parent | mailbox router registration and `MessageTarget::Parent` resolution |119| Retry loop | `VerificationGate::verify`, `max_retries`, `DelegationRetry.reason` |120| Pause/resume stuck | `pause_flags`, child loop pause boundaries, `resume_children_of` |121| Cancellation leak | `cancel_tokens`, `cancel_children_of`, `cleanup_delegation` |122| Wrong aggregate | `AggregationStrategy` and `aggregate_results` in `coordination.rs` |123| Missing UI progress | `progress_broadcaster`, `DelegationProgress`, SSE event emission |124125## Report Format126127```128Delegation Trace: <delegation_id>129130Pattern: <pattern>131Parent run: <parent_run_id>132Expected agents/tasks: <n>133134Timeline:135- <ts> started: agents=<...>136- <ts> sub-run started: <run_id> agent=<agent_id> depth=<n> retry_of=<...>137- <ts> retry: <old> -> <new> attempt=<n> reason=<...>138- <ts> sub-run completed: <run_id> status=<status>139- <ts> delegation completed: succeeded=<n> failed=<n> status=<status>140141Findings:142- <only concrete gaps or mismatches, with file/event evidence>143144Limits:145- <state any data not present in the journal, such as per-sub-run token distribution>146```
matrixorigin