Agent Observability
Rule
The observability system auto-instruments every agent run with zero configuration. Traces, automated evals, and feedback collection work out of the box. All data lives in the app's own SQL database — no external services required. Templates can optionally export to Langfuse, Datadog, or any OTel-compatible platform.
Five Pillars
1. Traces
Every runAgentLoop() call is automatically instrumented via instrumentAgentLoop() in packages/core/src/observability/traces.ts. It captures:
- agent_run span — top-level parent with total duration and cost
- llm_call span — model name, token counts (input, output, cache read/write), cost
- tool_call spans — one per action invocation, with duration and success/error
The run span is NAMED by what started it, because the trace list shows that name
and nothing else: a scheduled job is background_automation_run:<job name>, a
chat turn is agent_run, and a turn a feature sent on the user's behalf is
agent_run:<usageLabel> when the caller named it:
sendToAgentChat({
message: "Enrich this record from the web",
newTab: true,
usageLabel: "crm:enrich-record", // → usage row label + `agent_run:crm:enrich-record`
});
Content (prompts, tool args, tool results) is redacted by default. Opt in through the declared observability config domain:
// server/plugins/config.ts
import { defineAppConfig } from "@agent-native/core/server";
export default defineAppConfig({
observability: {
enabled: true,
capturePrompts: false,
captureToolArgs: true, // capture action input args
captureToolResults: false, // include results and full error text in local tool spans
evalSampleRate: 0.05, // 5% of runs get LLM-as-judge eval
inferredSentimentEnabled: false,
inferredSentimentSampleRate: 0,
inferredSentimentModel: "gpt-6-luna",
},
});
Two things are recorded whatever the flags say, because a failure nobody can classify is not observable:
- A failed tool span always keeps a signature: the first line of the error,
redacted (credentials, emails, long opaque ids) and capped at 500
characters.
captureToolResultsonly decides whether the rest is kept. The span's metadata says which (__tool_error_detail:full|signature), and the read path returns it aserrorDetail(full|signature|withheld|unrecorded) so "withheld on purpose" is never read as "nothing was recorded".$ai_*telemetry omits error text even when capture is enabled. Optional host-owned OTel run spans also use fixed code-derived messages and retainagent.error_code/agent.error_cause. - A stop that waits on the user is not an error. An
input_requiredoutcome (question, approval, connection) records theagent_runspan asstatus: "paused"with the reason interminal_code, noterror. Anything that counts failed runs must treatpausedas non-error; anerrorevent that follows the pause still makes the run an error.
From an error report to the failing run
A report is only useful if it names where it happened. Four pieces make that true, so nobody has to ask the reporter for an example:
- The failure packet. Every server
captureError()carriesextra.failureContext(app,route,actionName,automationName,threadId,runId,requestId,userScope,threadUrl, build, environment,errorCode,failureClass); see the tracking skill for how it is derived.threadUrlishttps://<app host>/?thread=<chat_threads.id>, and the Analytics issue page links it. The same fields ($ai_trace_id= run id,thread_id) are on the$ai_tracefor the run. - A copyable report on the client.
formatClientFailureReport()from@agent-native/core/client/failure-reportbuilds the plain-text packet (app, thread link, run, request, code, time, build, and the inspection call) for a "Copy details" button. The chat run-error card's existing "Copy debug info" button uses it; call it from any other error card. It needs only what the card already knows (message,errorCode,runId,requestId) and falls back to the open thread. The error-screen "Open GitHub issue" template adds the same thread link, build and time. A failed action carrieserror.requestId(the response'sx-agent-native-request-id, also on the server capture'sfailureContext.requestId). - Read by id. Dispatch's
get-agent-thread-debugtakes athreadIdor the copied run id (run-…) and returns the run'sterminalReasonand terminal event, its events, trace spans (failed tools carry a signature, see above), feedback and checkpoints;list-agent-run-failuresfinds candidates. Both are owner-scoped. Without Dispatch,GET /_agent-native/observability/traces/:runIdreturns the same spans for the caller's own runs. To debug another user's run from a run id, email, or object link, use thedebug-runsskill. - Alerts name examples. The chat-health Slack alert lists the latest failed
turns as
threadUrl (run id, code); if they cannot be read it says so.
Optional inferred sentiment
Self-hosted apps default to no inferred sentiment. First-party apps hosted on
agent-native.com automatically classify 100% of eligible user replies with
gpt-6-luna; an explicit stored inferredSentimentEnabled: false remains an
opt-out. Deployment overrides are AGENT_NATIVE_INFERRED_SENTIMENT=on|off,
AGENT_NATIVE_INFERRED_SENTIMENT_SAMPLE_RATE=0..1, and
AGENT_NATIVE_INFERRED_SENTIMENT_MODEL=<model>; off is always the emergency
kill switch.
Classification uses only the original visible user text, capped at 2,000 characters, with no tools, temperature 0, an eight-token output, and a five second timeout. It skips attachment-only turns, internal continuations, chained background chunks, and first turns that have no preceding response to attribute. The managed Builder engine runs the classifier after the main response has streamed, so it does not contend with the user's response.
Successful classifications emit a content-free $ai_sentiment tracking event:
sentiment:positive,negative, orneutralmethod:llmmodel/$ai_model: model that generated the preceding assistant responserun_id/$ai_trace_id: preceding response runthread_id/$ai_session_id: conversationclassification_trigger_run_id: run started by the classified user replyclassifier_modelandclassifier_engine: classifier attribution
No raw message, prompt, or response text is persisted or tracked.
A classification that cannot complete emits $ai_sentiment_failed with the same
identity properties and a coarse reason: engine_unavailable,
model_unsupported, timeout, parse_failed, or empty. It never carries
content, so a drop in $ai_sentiment is a count to read, not a silence to
guess at. The run's own engine classifies when it serves the classifier model;
otherwise the hosted Builder engine does. model_unsupported means the engine
is up but refuses the configured classifier model: its catalog does not list it
(the default is typed against the Builder catalog in model-config.ts, so a
retired id fails the build), or the call fails with a model-rejection
errorCode (builder_model_unauthorized, model_not_found,
not_found_error). It is a configuration error to fix, never an outage to
retry; any other failed call stays engine_unavailable.
2. Feedback
Explicit — AgentKit's assistant-message action bar renders inline thumbs up/down controls. A thumbs-down can collect a reason, and feedback includes the run and message sequence for trace linking. The shared AgentKitAssistantChat host submits it through the existing feedback action. The popover also offers one-click reason chips and a Copy details button (the failure-report packet with run, thread link and build, plus the note) so a "this chat was not good" report is one paste.
A vote for a run whose trace was never persisted is still saved, unlinked from the run (runId null), with traceMissing in the response and the stored value and trace_missing / unverified_run_id on $ai_feedback. A run that exists but belongs to someone else still answers 404.
Implicit — computeSatisfactionScore(threadId) computes a Frustration Index (0-100) from conversation signals:
- Rephrasing detection (weight 30): consecutive similar user messages
- Abandonment (weight 20): session ends shortly after agent response
- Sentiment (weight 15): negative language patterns
- Length trend (weight 15): declining message lengths
- Retry patterns (weight 20): "try again", "no that's wrong"
Score interpretation: 0-20 healthy, 20-40 friction, 40-60 dissatisfied, 60+ broken.
Satisfaction scoring fires automatically after each feedback POST with a threadId.
Human audit and refinement
For a human output audit, provide one table row per run with a distilled ask, result, inferred or explicit sentiment, audit state, and app-native preview. The same row contract works for standalone app and workspace roll-ups; keep single-app surfaces independent of workspace chrome.
Treat ordinary feedback separately from admin audit verdicts. Only admins can vote, mark audited, approve improvements, or apply them, enforced server-side with app-scoped roles. Let admins add reasons to downvotes across many rows, then run one synthesis that groups patterns, cites evidence, suggests update targets (skills, instructions, memories, data dictionaries, certified dashboards, or creative context), and shows before/after diffs. Allow feedback and regeneration before explicit approval; never auto-apply.
The shipped single-app human review surface is the Human review tab in the
shared observability dashboard. It lists the persisted ask and answer for each
run, records thumbs/text feedback through the existing feedback endpoint, and
saves explicit instruction changes as draft rows through the
save-observability-instruction-update action. Drafts are reviewable artifacts;
they never update agent behavior automatically. Agents can read the same table
through list-observability-reviews.
3. Evals
Three layers, configured via evalSampleRate in the observability config:
Automated (every run): Deterministic scorers that run after every traced run:
tool_success_rate— % of tool calls without errorsstep_efficiency— 1.0 for no-tool runs; penalizes excessive LLM iterations for tool-using runslatency_score— normalized against 10s/tool baselinecost_efficiency— normalized against 50 centicents/tool baselineerror_recovery— 1.0 if the run recovered from tool errors or had none
LLM-as-judge (sampled): Runs on evalSampleRate fraction of runs. Calls the configured engine with a judge prompt that scores against custom criteria.
Dataset evaluation: runDatasetEval(datasetId) runs a golden dataset through the agent and scores each case.
Promote a surprising production run into a CI eval case (promote-trace-eval / agent-native eval promote <runId> --write ...) rather than only inspecting it. Truncated runs fail closed.
Custom criteria use natural language rubrics:
const criteria: EvalCriteria = {
name: "helpfulness",
description: "Was the response helpful and complete?",
rubric: "0.0 = completely unhelpful, 0.5 = partially helpful, 1.0 = fully resolved the user's need",
};
Evals (CI gate)
The three layers above score real production runs after the fact. For an active, deterministic gate, use the first-class *.eval.ts primitive from @agent-native/core/eval (source: packages/core/src/eval/*). It runs the actual agent loop against fixed inputs and exits non-zero below threshold, so it gates CI/deploys.
// evals/faq.eval.ts
import { defineEval, contains, llmJudge } from "@agent-native/core/eval";
export default defineEval({
name: "answers the FAQ",
input: { prompt: "What is your return policy?" },
threshold: 0.7,
scorers: [contains("30 days"), llmJudge({ criteria: "accuracy" })],
});
- Built-in scorers:
exactMatch/contains/usesTool(pure JS) andllmJudge(provider-agnostic judge). - Custom scorers:
createScorerwith the 4-steppreprocess → analyze → generateScore → generateReasonpipeline (onlygenerateScoreis required). - Run as a gate:
agent-native eval [pattern] [--json] [--threshold N]— discovers**/*.eval.tsandevals/*.ts, runs the agent, and exits non-zero if any eval is below its threshold. An app with no eval files exits0. Complements (does not replace) the post-hoc scoring inevals.ts. Promote a surprising production run withagent-native eval promote <runId> --write evals/from-trace.eval.tsrather than only inspecting it. See the Evals doc.
4. Experiments
A/B testing with sticky user-level assignment:
import { insertExperiment, updateExperiment } from "@agent-native/core/observability";
const exp = {
id: crypto.randomUUID(),
name: "sonnet-vs-haiku",
status: "draft" as const,
variants: [
{ id: "control", weight: 50, config: { model: "claude-sonnet-4-6" } },
{ id: "treatment", weight: 50, config: { model: "claude-haiku-4-5-20251001" } },
],
metrics: ["cost", "latency", "satisfaction"],
assignmentLevel: "user" as const,
startedAt: null,
endedAt: null,
createdAt: Date.now(),
};
await insertExperiment(exp);
// Move it to "running" when ready to start collecting assignments.
await updateExperiment(exp.id, { status: "running" });
The agent loop reads active experiments via resolveActiveExperimentConfig() and applies the variant's model override automatically. Assignment uses consistent hashing — same user always gets the same variant.
Compute results with POST /_agent-native/observability/experiments/:id/results.
Hosted default-model experiment (code-level)
observability/hosted-model-experiment.ts runs one cross-app A/B on first-party
hosted domains with no DB row: users on the default Builder model at default
effort are split by a stable hash of their email. Each run carries the arm in
experiment_ids / experiment_variants, and in hosted_model_experiment_variant
regardless of other experiments on the turn. The singular experiment_id /
experiment_variant tags keep naming only the other experiment, as they did
before the hosted test. The DB
path above skips requests that pin a model; this one does not, because chat
clients always send the default model, so it keys off the resolved default.
Other model or effort choices are left alone. The kill switch is the declared
observability.hostedModelExperiment config (auto | on | off), env alias
AGENT_NATIVE_HOSTED_MODEL_EXPERIMENT.
In production, experiment management routes require the caller's email in the
comma-separated AGENT_NATIVE_EXPERIMENT_ADMIN_EMAILS allowlist. This gate is
separate from normal app/org admin roles because an experiment affects every
user in that deployment.
5. Dashboard
ObservabilityDashboard is a React component with 5 tabs:
- Overview — metric cards (runs, cost, latency, tool success, thumbs up rate, eval score)
- Conversations — trace list with drill-down to span detail
- Evals — eval stats and criteria breakdown bars
- Experiments — experiment list with status badges, drill-down to results
- Feedback — feedback stream, thumbs ratio, category badges
Add a dashboard route to any template:
// app/routes/observability.tsx
import { ObservabilityDashboard } from "@agent-native/core/client/observability";
export default function ObservabilityPage() {
return (
<div className="min-h-screen bg-background p-6">
<ObservabilityDashboard />
</div>
);
}
API Endpoints
All auto-mounted at /_agent-native/observability/*:
| Method | Path | Purpose |
|---|---|---|
| GET | / |
Overview stats |
| GET | /traces |
List trace summaries |
| GET | /traces/:runId |
Trace detail (summary + spans) |
| GET | /traces/:runId/evals |
Evals for a run |
| POST | /traces/:runId/promote |
Promote a completed run into a CI eval case |
| POST | /feedback |
Submit feedback |
| GET | /feedback |
List feedback entries |
| GET | /feedback/stats |
Feedback aggregation |
| GET | /satisfaction |
Satisfaction scores |
| GET | /evals/stats |
Eval statistics |
| POST | /experiments |
Create experiment |
| GET | /experiments |
List experiments |
| GET | /experiments/:id |
Experiment detail |
| PUT | /experiments/:id |
Update experiment status |
| POST | /experiments/:id/results |
Compute experiment results |
| GET | /experiments/:id/results |
Get experiment results |
All endpoints support ?since=N (ms timestamp) and ?limit=N query params.
SQL Tables
9 tables created automatically via ensureObservabilityTables():
agent_trace_spans— individual trace spansagent_trace_summaries— aggregated run summariesagent_feedback— explicit user feedbackagent_satisfaction_scores— computed frustration indexagent_evals— evaluation resultsagent_eval_datasets— golden test datasetsagent_experiments— experiment definitionsagent_experiment_assignments— user → variant assignmentsagent_experiment_results— computed metric results
All tables are PostgreSQL-compatible and strictly additive.
Key Files
| File | Purpose |
|---|---|
packages/core/src/observability/types.ts |
Shared type definitions |
packages/core/src/observability/store.ts |
SQL tables + CRUD |
packages/core/src/observability/traces.ts |
Auto-instrumentation |
packages/core/src/observability/posthog-ai.ts |
$ai_trace / $ai_span / survey sent emission, content bounding, $ai_error |
packages/core/src/observability/failure-context.ts |
The server failure packet (buildFailureContext, withFailureContext), applied by captureError() |
packages/core/src/shared/failure-report.ts |
The packet type, the ?thread= link, and the plain-text report format |
packages/core/src/client/failure-report.ts |
The browser's packet and formatClientFailureReport() for "Copy details" |
packages/core/src/tracking/failure-counters.ts |
Windowed failure counters (action_error_counts, credential_state_counts) |
packages/core/src/observability/feedback.ts |
Feedback + Frustration Index |
packages/core/src/observability/evals.ts |
Eval engine (3 layers) |
packages/core/src/observability/experiments.ts |
A/B testing system |
packages/core/src/observability/routes.ts |
HTTP API handlers |
packages/core/src/client/observability/ObservabilityDashboard.tsx |
Admin dashboard |
packages/core/src/client/observability/ThumbsFeedback.tsx |
Inline feedback buttons |
packages/core/src/client/observability/useObservability.ts |
React Query hooks |
Export to External Platforms
Core emits gen_ai.* semantic convention spans and deliberately registers no OpenTelemetry provider or exporter itself (observability/tracing.ts). To reach Langfuse, Datadog, Grafana, New Relic, or any OTel-compatible backend, the app registers its own TracerProvider.
There is no framework config field for an export endpoint or token, and there should not be: the backend credential belongs in the vault and in the app's own provider wiring, not in a config object or a settings row.
Live OpenTelemetry Spans (Optional)
The agent loop emits live OpenTelemetry spans for every run, model call, and tool call, so a host that already runs an OTel collector sees agent activity alongside its other distributed traces.
This layer is optional and no-op by default:
@opentelemetry/apiis an optional dependency. If it isn't installed, the span helpers degrade to silent no-ops — they never throw into the agent loop.- Even with the api package installed, it ships a default no-op tracer. Spans become real only once the host registers a
TracerProvider(via@opentelemetry/sdk-nodeor similar). The framework deliberately does not depend on the heavy SDK/exporter packages and never registers a provider itself — instrumentation is opt-in by the embedding app.
The loop emits invoke_agent (with gen_ai.provider.name (the engine), gen_ai.conversation.id, gen_ai.request.model, gen_ai.usage.*, agent.run_id, agent.user_id), execute_tool {tool} (gen_ai.tool.name, gen_ai.tool.call.id + status), and chat {model} (gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.response.finish_reasons, gen_ai.usage.*) spans, each finished with OK/ERROR status. This is purely additive to the in-house agent_trace_spans / agent_trace_summaries tables. Source: packages/core/src/observability/tracing.ts + traces.ts. See the Observability doc for the full table.
Tracking Bridge
Instrumented agent loops emit server-side tracking events for every run through
track() from @agent-native/core/tracking, so configured PostHog, Agent-Native
Analytics, Mixpanel, Amplitude, and webhook providers receive them through the
same best-effort fan-out as other tracking events.
- Events:
$ai_traceper run,$ai_spanper tool call, and$ai_generationper model call. Every node carries the run id as$ai_trace_idand links upward through$ai_parent_idso a backend can rebuild the tree.$ai_session_idis the thread; the browser session is separate and ships as$session_id, read fromX-Agent-Native-Session-IdviaRequestContext.browserSessionId. The agent chat and the action client both send that header, so a UI action call and the agent's own call during one visit share a session.setAnalyticsSessionId()from@agent-native/core/client/analyticspins a custom id and opts it out of the 30-minute idle rotation. Emission lives inposthog-ai.ts. - Each event is stamped with when it happened, not when the run flushed. The
whole tree is emitted in one burst at run end, so
track()takes anoccurredAtand the trace tree keeps a real timeline. - Model-call spans preserve
createdAtandendedAtas epoch milliseconds, alongsidedurationMs. Their$ai_generationevents carry the matchingcreated_at_ms,ended_at_ms, andduration_msproperties so latency can be compared over time without reconstructing request boundaries. - Agent-Native Analytics shape: the same event lands in
analytics_eventswith mirrored query-friendly properties such asrun_id,thread_id,cost_cents_x100,duration_ms,tool_calls,successful_tools,failed_tools, andstatus. A content-freetoolsarray includes at most 50 tool names, start offsets, durations, statuses, and coarse error classes; interrupted calls are finalized as errors, and failed runs still emit with zero or known usage.tools_truncatedmarks longer runs while the rollup counts remain complete. Delegated runs adddelegation_protocol,caller_app,a2a_task_id, andparent_run_idwhen available.parent_turn_idis separate because one logical turn may span multiple concrete runs.
Constraints that are not visible from the emit site:
Run failure telemetry carries identifiers, not error text.
error_detailanderror_messageare absent from tracking.$ai_error.messageis fixed text derived from itsterminal_code;causecomes from the failure taxonomy. Failed tool results in$ai_output_stateand generation input transcripts are replaced with an omission marker, and thetoolsarray retains only its error class. Localagent_runs.error_detailandagent_trace_spans.error_messageremain available to owner-scoped debugging and the Observability UI. Monitoring run/gateway captures use the omission policy documented in tracking.The trace event carries no latency, tokens, or cost under
$ai_*. PostHog DERIVES those from a trace's children: its trace query sums$ai_latencyover every event whose$ai_parent_idis the trace or is absent, and sums tokens/cost over$ai_generation/$ai_embeddingonly. An$ai_latencyon the$ai_traceevent is therefore added to its own children's and reports roughly twice the real duration. Run totals ride along asduration_ms,input_tokens,output_tokens, andcost_usdfor the backends that do no such aggregation.The generation's
$ai_latencyis model time, not run time. Tool calls are siblings under the same trace and PostHog adds their latency to the generation's, so tool duration is subtracted out.duration_mson the same event is still the full run — the two differ on purpose.$ai_http_statusis absent, not defaulted, when the status is unknown. A generation that streamed to completion reports 200, and the call the run died in reports whatever status the engine named (EngineError.statusCode, or a provider SDK error'sstatus). A failure that carried no status — a socket drop, an SDK throw — omits the field: a defaulted 200 would report the drop as a healthy call, and a defaulted 500 would invent a rejection the provider never made. Only the failing round-trip claims the error's status; earlier calls that completed keep their 200.PostHog's
$ai_*latency fields are seconds; ours are milliseconds.$ai_latencyand$ai_time_to_first_tokenare seconds;duration_msandtime_to_first_token_msare the millisecond siblings the first-party dashboards read. Feeding a millisecond value to a seconds field is invisible in the payload and inflates the metric 1000x.Custom properties never take an
$ai_prefix. That namespace belongs to PostHog's schema; a name it does not define today it may define tomorrow with a different meaning. Ours are plain (duration_ms,input_truncated,spans_dropped), which also keeps them out of PostHog's$ai_*aggregation.$ai_tracecarries no$ai_input_state/$ai_output_state. Content rides the generations; a trace-level copy repeated the run's prompt and answer on a second event. PostHog reads a trace's input and output from that event and from nowhere else, so the visible cost is that traces list with a null input and a conversation is titled from the first generation's$ai_inputinstead. Measured, not assumed — check whether that title is still wrong before trading the duplication back.One generation per model round-trip. Engines that bracket their calls with
model_streamget one$ai_generationeach, with that call's tools as its children. Only an engine that never brackets falls back to a single aggregate generation covering the whole run, and it is reported as one.Messages are rewritten into PostHog's shape before they ship.
toPostHogMessages()inposthog-ai.tsmaps engine parts onto the OpenAI/Anthropic conventions PostHog reads:tool-callbecomestool_calls, and atool-result— which the engine has to carry inside ausermessage, becauseEngineMessagehas notoolrole — becomes its ownrole: "tool"message. Skipping this is not cosmetic: PostHog dumps raw JSON for shapes it does not know, and the byte-ceiling rescue inboundAiContentkeeps "the last user message", which in engine shape is the last tool result rather than the question. Attachment bodies become a marker naming the media type and size — base64 renders as nothing in PostHog and spends the whole ceiling.A tool call and its result pair on the id the MODEL issued. The span id is a separate namespace that never appears in the transcript, so emitting it on
tool_calls[].idleaves PostHog with a call and a result that never match and every tool call renders with no output. Span id is the fallback only for emitters that report no call id.Disabled capture omits the field rather than sending an empty one. An empty array is indistinguishable from a run that genuinely had no messages. Truncated content is marked, and a run over the span cap stamps
spans_dropped— a truncated run must not read as a complete one. The one deliberate exception is a tool span's$ai_output_statewithcaptureToolResultsoff: it carries an explicit "withheld" marker, because an absent output state reads as a tool that returned nothing, and the tool did answer.A tool that RETURNS an error envelope is a success here.
$ai_is_errorandfailed_toolsfollow the tool event'sisError, which the agent sets when an action throws. An action that returns{ error: ... }instead is counted as a healthy call by every rollup on this page. Fix it in the action (fail()), not by teaching this layer to sniff payload shapes.The structural tool-call list ships even when content capture is off. Backends derive their tool tags from tool-call blocks inside the output choices and from nothing else, so tool names (without arguments) are always emitted. The parallel first-party
toolsarray stays because the dashboards read it; that duplication is deliberate, not cleanup.Only thumbs carry
sentiment. All four feedback types are reported, but a category follow-up to a thumbs-down is detail about the same vote — counting it again inflates the metric.Never invent an external id to make an integration light up. Survey-based feedback is emitted only when a real survey id is configured, and nothing is sent otherwise.
Do not build a separate LLM-observability ingestion API unless there is a clear reason the tracking provider registry cannot express the use case. Keep prompt, tool input, and model output content out of tracking by default; use the existing observability config flags for local trace content capture.