Aevatar triage — find the layer, read the code, then report or guide
Something broke (or looks wrong) while using Aevatar. Your job is three honest moves:
(1) attribute the failure to the right layer — Aevatar / NyxID / Ornn — by tracing the
request path; (2) read the real code of that layer until you have a code-grounded root
cause (cite path:line) that matches what's actually deployed; (3) branch: if it is a
genuine platform defect (behavior that violates the layer's own published contract), draft
and — only on explicit user confirmation — file a GitHub issue to the owning repo; if it is a
usage / config mistake, give the user an authoritative, code-grounded explanation and the
correct usage. This is the after-it-breaks counterpart to aevatar-feasibility-advisor (which
answers "is it possible, before building").
You make real calls and read real code — no guessing, no fabricated root cause, no auto-filing.
The three layers (and their repos — all public)
| Layer | Repo (stack) | Owns | Canonical symptoms |
|---|---|---|---|
| Aevatar | aevatarAI/aevatar (C#/.NET) |
agent runtime + tool execution, workflow engine, channels, CQRS/projection + readmodels, control-plane REST, scheduler validation | workflow validate / draft-run / run failures, member-team-service binding stuck (async never succeeded), the aevatar side of a channel bot, stale readmodel / observatory, scheduled run that stops firing, control-plane 4xx/5xx |
| NyxID | ChronoAIProject/NyxID (Rust) |
credential-broker gateway: proxy + credential injection, OAuth/OIDC/PKCE/MFA, connector vault, NAT relay + SSH, approvals, MCP-from-OpenAPI | nyxid_proxy 401/403, "credential not found", connector/slug missing, OAuth/token/MFA failures, grant revoked (invalid_grant), inbound relay not delivering, approval stuck |
| Ornn | ChronoAIProject/Ornn (TS/Node) |
skills-as-a-service registry: skill search/pull/upload/generate, skillsets, sandbox | skill not found / search / pull / upload fails, use_skill cannot find a skill, skillset integrity, generate (SSE) errors |
These three repos and layers are this skill's subject — name them freely. Do not hardcode any user's private business / workflow / skill names.
Step 1 — Capture the symptom precisely (don't theorize yet)
Collect, verbatim where possible: the exact error text; the operation / surface (workflow
run, connector call, channel bot, schedule, skill pull/publish, auth, a control-plane REST call);
minimal repro; the ids that let you correlate — run_id, scopeId, connector slug,
skill name, commandId/correlationId, schedule fire-record fields (fireCount / lastFireAt /
nextFireAt / failureCount / lastError), timestamps; and what is actually deployed — the
running image tag / commit, pod age + restartCount, and the time window of the failure
(live logs rotate fast, so an old window may already be gone). Missing ids are themselves a finding.
First, rule out the cheap local causes before blaming a layer: your own expired / missing
credential (a blanket 401 is often just your own stale token, not a platform bug — refresh it
and retry) and a stale local checkout (the code you're about to read may be behind what's
deployed).
Step 2 — Attribute to a layer (trace the request path, eliminate)
Main rule: follow the request path and find the first boundary that breaks. A user request
typically flows your agent -> Aevatar runtime -> NyxID proxy -> third-party, and skills flow
Aevatar use_skill -> Ornn. Map the symptom to where that chain first fails:
| Symptom | Most likely layer | Disambiguating evidence to gather |
|---|---|---|
401 / 403 on a connector/tool call |
NyxID (auth/vault) | is the slug in GET {NYX}/api/v1/services? token expired? OAuth scope present? |
| "credential not found" / connector slug missing | NyxID (vault / not connected) | catalog vs services; was it ever connected for this identity? |
404 on a thing you reference |
whichever registry owns it | skill -> Ornn; connector/service -> NyxID; team/member/scope -> Aevatar |
502 / timeout on an external call |
proxy chain | which hop? Aevatar tool layer vs NyxID proxy vs the target itself |
workflow won't validate / run stalls / binding never succeeded |
Aevatar (engine/runtime) | draft-run error body; run timeline / observatory; binding-run status |
| readmodel stale / observatory missing data | Aevatar (projection) | is the projection subscription live? event stream flowing? authoritative version vs readmodel — note readmodels do not back-fill (compare record age to deploy time) |
scheduled run stopped firing (fired before; nextFireAt frozen in the past, fireCount flat, failureCount=0, empty lastError) |
Aevatar (scheduler / actor not re-armed across pod churn) | compare to peer schedules; pod startTime / restartCount; is it still enabled? did a deploy/restart line up with the last good fire? |
| scheduled run never fires, or fires but errors on credential | Aevatar scheduler ⨯ NyxID (binding) | is it enabled and is nextFireAt computed? lastError like "binding not found" / "exactly one credential source" -> the fired call's invocation credential (scope-owner broker binding) |
schedule fires (fireCount climbs) but the real-world effect never happens |
Aevatar (the fired call's path / credential) | dispatch success ≠ effect — check the external side-effect out-of-band; the proxy can hand back {"error":true} inside a 200 |
| inbound bot doesn't reply (Lark/Telegram) | cross-layer — walk it | did NyxID relay webhook fire? is the bot connector connected? did the Aevatar channel run start (observatory)? credential = the sender's NyxID, present and live? |
/whoami says "bound" but tool calls get credential_denied |
NyxID (grant revoked — false green) | live token-exchange returns invalid_grant while the local readmodel still reads "bound"; whoami checks only the local mirror, not the live grant |
| approval prompt stuck | NyxID approvals + Aevatar suspension | NyxID approval request id; Aevatar workflow wait/suspend state |
| skill search/pull/upload/generate fails | Ornn | which /api/v1/skill... route? validator violations? version format? |
Do not stop at the first match. Gather the disambiguating evidence and eliminate — a plausible first guess that you haven't excluded the alternatives for is not an attribution.
Step 3 — Pull the repo and reach a code-grounded root cause
Pin to the running system first — code is a hypothesis, not the live truth. The single most
common triage failure is a confident, code-traced root cause that is wrong because the running
build differs from what you read. Before you treat any code-grounded cause as fact for a live
failure: (1) confirm the code you read matches the deployed commit/image (read the deployed
image tag; if you have a candidate fix, prove it shipped with git merge-base --is-ancestor <fix> <deployed-sha>); (2) confirm the symptom reproduces on fresh live evidence, not an old log
window; (3) remember auto-deploy branches roll forward under you — re-check the image tag
mid-investigation. If code and deployment diverge, the bug may already be fixed (an unmerged branch
or a follow-up commit) or the running build is simply different.
Then get the suspected layer's real source (paths below) and read until you can point at the code that produces the behavior. Anchors are subsystem/directory altitude — confirm exact files by reading; the tree evolves.
- Aevatar (
aevatarAI/aevatar): tool execution + LLM dispatch insrc/Aevatar.AI.Core; connector adapters insrc/Aevatar.AI.ToolProviders.*(incl. NyxId, Ornn); readmodels insrc/Aevatar.CQRS.Projection.*; engine + HTTP/OpenAPI insrc/workflow/; contract insrc/Aevatar.AI.Abstractions+docs/canon/; errors as workflow exceptions + control-plane 4xx/5xx. Read the deployed branch: Aevatar currently ships fromfeature/integrate— not the repo default and notdev. Reading the wrong branch is the fast way to a wrong root cause; confirm the branch against the live image tag (deploy branches move — verify, don't assume). - NyxID (
ChronoAIProject/NyxID): endpoint handlers inbackend/src/handlers/(proxy, auth, oauth, approvals, mcp, node/ssh); vault logic inbackend/src/services/; all error variants + numeric codes inbackend/src/errors/(observed ranges — confirm: ~2000 auth, ~3000 approval, ~4000 ssh, ~5000 service, ~8000 credential-node); contract inbackend/src/api_docs.rs(OpenAPI) +README.md. - Ornn (
ChronoAIProject/Ornn): registry + skillsets inornn-api/src/domains/; NyxID/sandbox integration inornn-api/src/clients/; global error handler inornn-api/src/middleware/; contract inornn-api/src/openapi/+README.md.
Live-evidence playbook (host-gated — needs cluster access). Pin the deployment, then pull the full window, then read with native tools:
# pin what's running (do this FIRST)
kubectl -n <ns> get deploy -l <selector> -o jsonpath='{..image}' # deployed tag / commit
kubectl -n <ns> get pod -l <selector> -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.startTime}{" "}{.status.containerStatuses[0].restartCount}{"\n"}{end}'
# pull the FULL window to a file — NEVER trust the -l default tail (~10 lines/pod)
kubectl -n <ns> logs -l <selector> --tail=-1 --since=<window> > dump.log
Then read dump.log with the file/Read tool: token-mangling shell wrappers can corrupt pipes, and
grep -c over-counts when a benign event shares a prefix with the failure — count exact matches and
also count the success case (see the negative-control discipline below). For the read side, the
observatory / run-timeline readmodel shows runs — but it does not retroactively heal, so
compare a record's age/version to the deploy time before trusting it. No cluster access? Don't
fake it — use the observatory readmodel and hand the problem back to the calling agent (Step 4c).
Discipline: the root-cause claim must cite repo path:line (+ the commit/ref you read) and
that ref must match what's deployed. No citation, or code that doesn't match the running build ->
no verdict — it's a hypothesis; downgrade to inconclusive (Step 4c). Attribute by elimination
with a negative control: count the failure and its success counterpart — one failure with zero
observed successes proves nothing (the operation may simply be rarely exercised). Where you can,
design a probe that flips the symptom (e.g. a re-bind that turns a 400 token-exchange into a
200) for positive proof. Never fabricate a path, a line, a version, or a slug.
The contract test (this decides the branch): does the layer's current behavior violate its
own published contract (README / OpenAPI / proto / docs/canon)? Yes -> defect (Step 4a).
No — the contract says it should behave this way and we drove it wrong -> usage (Step 4b).
Step 4 — Branch on the verdict
4a. Platform defect -> file an issue (confirmation-gated)
- De-dup first, and check it isn't already fixed. Search existing issues (open and closed) on
the owning repo; if one exists, point the user to it. Then check whether the cause is already
fixed but not yet running — scan commits since the deployed sha, open PRs, and unmerged branches
(
git log <deployed-sha>..origin/<branch>). A fix that exists upstream is not a new defect — point to the PR/commit instead. Two traps: a containment guard that only rejects the bad state is not a fix that corrects it; and a deployed fix that sits on a code path the symptom never executes hasn't fixed anything — confirm the failing operation actually traverses the changed code. - Draft with this shape:
- Title:
[<layer>] <one-line symptom> - Body: environment + version/commit; minimal repro steps; expected vs actual; the
offending
path:lineand why it violates the published contract; correlation ids / logs.
- Title:
- Show the user the draft. File only on an explicit "yes." Route to the owning repo
(
aevatarAI/aevatar|ChronoAIProject/NyxID|ChronoAIProject/Ornn). If the cause spans layers, file in the layer that first breaks the contract and cross-link the others. - The bar for an external-repo issue is "behavior violates its published contract" — not "we wish it worked differently." If it's a feature gap, say so; don't file a defect.
4b. Usage / config mistake -> authoritative guidance (no issue)
Explain grounded in the code/contract you just read — why it behaves this way and what the
correct usage is — not generic advice. Then point the user at the right next step generically
(e.g. "author the workflow", "publish the service", "connect the connector in NyxID", "check
feasibility first") and hand off to the sibling skill that owns it via aevatar-platform-map.
Credential failures are layered — don't conflate them, and don't reflexively prescribe a re-bind.
"Connected" can be false green. Three distinct things: (1) a binding mirror exists locally (a
whoami / status check reads only this); (2) local model/route preferences were applied (a
log like applied=true reflects this, not a live grant); (3) the live grant — the only
thing that actually authorizes a tool call. A revoked grant shows invalid_grant on the broker's
token-exchange while local state still reads "bound." Re-binding fixes a revoked grant; it does
not fix a deferred/relayed run that lost its sender token after persistence — prescribing
unbind/re-init there is wrong. Identify which layer failed from a live trace before guiding.
4c. Inconclusive -> name the missing evidence and the next probe
State exactly what you could not determine and the cheapest probe to get there: a minimal repro; a capability check; live logs (the playbook above) if you have cluster access; an observatory run-timeline / readmodel-version check; or re-pulling the relevant code at the exact deployed commit. Some facts are not log-derivable — e.g. why a grant died, or runtime state that lives in a broker/projection rather than in config. When the decisive evidence lives on another layer's authoritative side, say so and escalate there (query that layer's own API / state) rather than guessing.
Execution paths (pick by the tools you actually have)
Local coding agent — preferred for deep RCA
# read the suspected layer (shallow clone; reuse an existing clone if present)
# Aevatar ships from feature/integrate (NOT the repo default, NOT dev) — read THAT ref:
gh repo clone aevatarAI/aevatar -- -b feature/integrate # NyxID / Ornn: their own deployed branch (confirm)
# git clone --depth=1 -b feature/integrate https://github.com/aevatarAI/aevatar
# is it already fixed but not deployed? (check before drafting a defect)
git -C <repo> log <deployed-sha>..origin/<branch> --oneline
# de-dup before drafting
gh issue list -R <owner>/<repo> --search "<keywords> in:title,body"
gh search issues "<keywords>" --repo <owner>/<repo>
# file ONLY after the user confirms the draft
gh issue create -R <owner>/<repo> --title "[<layer>] ..." --body "..."
Server-side, in an aevatar session — constrained fallback
Use the runtime nyxid_proxy tool (not the nyxid CLI), slug=api-github, base
https://api.github.com:
- read code at the deployed ref:
GET /repos/{owner}/{repo}/contents/{path}?ref=<deployed-branch>— foraevatarAI/aevatarthat ref isfeature/integrate(not the default, notdev) —GET /search/code?q=...+repo:owner/repo(repos are public; raw fetch also works). - de-dup:
GET /search/issues?q=repo:owner/repo+is:issue+<keywords>. - file (confirmed only):
POST /repos/{owner}/{repo}/issues.
Credential reality — be honest about it. Under a relayed/in-session call, every tool runs on the
sender's own NyxID identity, not the bot owner's. So filing an issue operates the sender's
GitHub, and it requires: the sender has connected api-github in their own NyxID (check
GET {NYX}/api/v1/services) with an OAuth scope that allows writing issues (repo / public_repo).
Writes have no owner fallback — without a live sender token you get credential_denied. Deep
multi-file reading over the API is clunky; prefer the local path for RCA and use this to fetch
specific files, search code, and de-dup/file.
Neither available? Don't fake it — hand the original problem back to the calling agent with the
evidence you gathered (the family's fallback-to-calling-agent ethos), so it can finish with its
own local tools.
Honesty & safety rails
- Never auto-file. Always: de-dup -> draft -> explicit user confirmation -> file.
- No
path:line, no root cause. An unverified hypothesis is reported as inconclusive. - Code is not the running system. A code-grounded cause is a hypothesis until it matches the deployed commit and reproduces on fresh evidence — old logs and stale checkouts mislead.
- Dispatch success ≠ real-world effect. A climbing
fireCountor a200proxy body can still mean nothing happened — verify the actual side-effect out-of-band. - Negative control before "systemic." Count the success case too; one failure with no observed successes is not evidence of a platform-wide break.
- Citing a memory/doc is not applying it. Re-derive the verdict from the evidence in front of you.
- Never fabricate a root cause, an issue link, a version, or a connector slug.
- External-repo issues only when behavior violates the layer's published contract.
- Attribute by reading + elimination, not first-match — exclude the alternatives.
- Server-side writes act as the sender — verify the connector and scope before promising a file.
Report shape
End with a straight, evidence-bearing summary:
Layer: NyxID — evidence: 403 from
nyxid_proxy, slug present in/services, OAuth scope missing. Root cause:backend/src/handlers/proxy.rs:NNrejects when the granted scope lacksrepo(commitabc123, matches deployed image). Verdict: usage. Action: guidance — reconnectapi-githubwith thereposcope; no issue filed.
Name the layer, cite the code (and that it matches what's deployed), state the verdict (defect | usage | inconclusive), and the action (issue drafted, awaiting confirm | guidance given | next probe).