A365 Code Validator

A365 Code Validator

Trigger phrases — any of these will activate this skill automatically:

• "validate a365 code"
• "run a365 code validator"
• "check a365 observability code"
• "debug a365 activity missing"
• "debug MAC activity for this agent"
• "validate Agent 365 telemetry"
• "check why Agent Activity is empty"
• "check A365 exporter flags"
• "validate gen_ai agent id"
• "check A365 span types"

Overview

This skill performs a report-first validation of an existing agent codebase to answer:

"If this agent runs now, will its spans reach the A365 backend and become eligible for MAC Activity?"

It is designed for incident response. It checks the exact classes of bugs that cause silent or confusing A365 Activity failures:

1. Exporter configured but not actually enabled.
2. gen_ai.agent.id set to a Blueprint ID instead of the runtime Agent Identity / Source Agent ID.
3. Generic HTTP/OpenAI spans emitted without A365 semantic operations.
4. S2S/OBO endpoint mismatch.
5. Missing token resolver or wrong S2S token shape.
6. Stale local Blueprint ID or missing Blueprint permissions in the target tenant.
7. Inheritable permission policy configured without grants on the Blueprint service principal.
8. MAC reporting expectations confused with raw ingest success.

No source code is modified during validation. After the report, the skill asks whether to apply safe fixes, create a fix plan only, or stop.

Output is support-safe (applies to every phase)

This report may be pasted into a customer or partner thread, so it must be safe to share: never print internal cluster/database names, Kusto queries, private endpoint or route templates, correlation IDs, or real tenant/agent IDs. Report file/function names and public, supportable signals only. This is non-negotiable — it holds even when a user asks for "just the raw details."

Phase 0 — Create and Display Task List

Show the user this checklist BEFORE Phase 1. Exactly one task is in progress at a time. Claude Code uses TaskCreate / TaskUpdate; VS Code Copilot Chat / GitHub Copilot CLI should show and update a markdown checklist.

TaskCreate: "Detect stack and A365 artifacts"
TaskCreate: "Run static A365 code validator"
TaskCreate: "Inspect identity binding and semantic spans"
TaskCreate: "Check live blueprint permissions and runtime state"
TaskCreate: "Summarize blockers and next fixes"
TaskCreate: "Offer guided remediation"

Phase 1 — Detect Stack and A365 Artifacts

Mark task in progress: "Detect stack and A365 artifacts"

Read, in parallel when possible:

• .a365-workspace-detection.local.json if present
• a365.config.json
• a365.generated.config.json
• .env, .env.example, .env.production, .env.local
• appsettings.json, appsettings.Development.json
• package.json
• requirements.txt, pyproject.toml
• *.csproj

Detect:

Mark task complete.

Phase 2 — Run Static A365 Code Validator

Mark task in progress: "Run static A365 code validator"

Run the validator from the target project root.

When running from the plugin source (Claude Code / marketplace plugin), use:

node ${CLAUDE_PLUGIN_ROOT}/hooks/stop/validate-a365-code-validator.js

If the runtime cannot expand ${CLAUDE_PLUGIN_ROOT}, run with the absolute plugin path:

node /path/to/agent365-skills/plugins/agent365/hooks/stop/validate-a365-code-validator.js

When running from an open-standard .agents/skills/a365-code-validator/ install, use the standalone copy that ships with the skill:

node .agents/skills/a365-code-validator/references/a365-code-validator.js

Parse the JSON output. Treat findings by severity:

Important: Findings are not stop-hook failures. A validation run is successful when it accurately reports what is wrong.

Mark task complete.

Phase 3 — Inspect Identity Binding and Semantic Spans

Mark task in progress: "Inspect identity binding and semantic spans"

Read the reference checklist:

${CLAUDE_PLUGIN_ROOT}/skills/a365-code-validator/references/validation-checklist.md

Then inspect source for these required MAC Activity inputs:

3.1 Exporter activation

Python must have either explicit code:

use_microsoft_opentelemetry(
    enable_a365=True,
    a365_enable_observability_exporter=True,
)

or a runtime env guarantee:

ENABLE_A365_OBSERVABILITY_EXPORTER=true

Node.js must have:

useMicrosoftOpenTelemetry({
  a365: {
    enabled: true,
    enableObservabilityExporter: true,
  },
});

.NET must have:

"EnableAgent365Exporter": true

3.2 Identity binding

Verify the same runtime Agent Identity is used consistently:

token azp/appid/oid == route /agents/{agentId} == span gen_ai.agent.id

The following is wrong for S2S/MAC Activity:

gen_ai.agent.id = <blueprint id>

Blueprint belongs in:

microsoft.a365.agent.blueprint.id

3.3 Semantic span coverage

MAC Activity needs supported A365 operations:

invoke_agent
chat
execute_tool
output_messages

Generic HTTP spans alone are not enough. Flag code that only sets baggage but never creates or auto-generates supported semantic spans.

3.4 Activity/run context coverage

If the developer expects MAC Activity or Defender agent-activity views (not just exporter HTTP 200), inspect whether real runs carry the public Agent 365 activity attributes:

When reviewing a mixed app, separate the two objectives:

S2S/export success proves telemetry plumbing.
AI teammate/HumanToAgent or EventToAgent run context proves Activity/reporting eligibility.

Do not require Teams-specific values for autonomous agents; require equivalent logical run context instead.

Caller identity caveat: for HumanToAgent, user.id is the caller ID used for "who ran this agent" reporting. If it is missing or set to the agent identity / agent user instead of the human caller object ID, export can succeed while caller/user Activity remains blank or incomplete.

3.5 S2S vs OBO transport mode

Do not expose or require internal service URLs in the report. Validate only the structural intent:

Report whether the code appears to select the correct transport mode (useS2SEndpoint / a365_use_s2s_endpoint / UseS2SEndpoint) for S2S, without printing backend route templates. For Python S2S, prefer a365_use_s2s_endpoint=True in code rather than relying only on A365_USE_S2S_ENDPOINT=true in runtime environment. Also check the token shape: the S2S observability token must come from the 3-hop FMI exchange (principal == runtime Agent Identity), not a bare client-credentials call — see checklist §5 for the failure signatures.

3.6 Blueprint permission inheritance wording

Do not claim that application app roles can never inherit from blueprints. Current public Entra Agent ID docs describe both inheritableScopes and inheritableRoles. If a repo or tenant shows delegated scope inheritance working but S2S app roles requiring direct assignment, report it as a provisioning/configuration behavior to verify, not as the intended scalable model.

Mark task complete.

Phase 4 — Check Live Blueprint Permissions and Prepare Runtime Verification

Mark task in progress: "Check live blueprint permissions and runtime state"

Live Blueprint permission check (read-only Microsoft Graph)

The static validator can only inspect files. When the project has a non-empty agentBlueprintId and tenantId, complement it with the a365 CLI's read-only Microsoft Graph diagnostics. These commands verify the tenant state without changing permissions:

a365 query-entra blueprint-scopes
a365 query-entra inheritance

• blueprint-scopes lists the delegated scopes and application roles actually granted on the Blueprint service principal. This is the authoritative permissions view; do not infer grants from requiredResourceAccess.
• inheritance verifies that every configured resource uses kind=allAllowed for scopes and roles and that the Blueprint service principal has something to inherit. It exits non-zero when any resource has Effective inheritance: NONE or BROKEN.

Run these checks by default only when:

1. a365 query-entra inheritance --help confirms the installed CLI supports the command,
2. an existing Azure/a365 sign-in is available for the target tenant, and
3. the check will not trigger a new interactive login.

The validator must not install or update the CLI, start an interactive sign-in, run a365 setup permissions, or issue Graph POST/PATCH/DELETE requests. If a prerequisite is missing, report Blueprint permissions as not checked, explain why, and provide the two commands for the developer to run after a365-setup refreshes the CLI/login.

If the local config is missing, copied from another project, or suspected stale, use the read-only name-based resolver instead:

a365 query-entra blueprint-scopes --agent-name "<agent-name>" --tenant-id "<tenant-id>"
a365 query-entra inheritance --agent-name "<agent-name>" --tenant-id "<tenant-id>"

Compare the live Blueprint ID printed by the command with the local agentBlueprintId, but do not print either real ID in the support-safe report:

Interpret permissions in the context of authMode:

Treat a missing required OtelWrite grant as critical. Treat a non-zero inheritance result for another required resource as high. Permission names and resource display names are safe to summarize, but redact tenant, Blueprint, application, service-principal, and agent IDs.

For remediation, report the CLI guidance without applying it:

• Run a365 setup requirements when the output points to a missing wids claim or CLI consent.
• Run the relevant a365 setup permissions ... flow as a Global Administrator to reconcile grants and inheritance.
• For a stale Blueprint ID, hand off to a365-setup so the developer can explicitly choose the correct reuse/re-run/fresh path. Do not rewrite the ID automatically.

References:

• List inheritable permissions — Microsoft Graph
• Configure inheritable permissions for Blueprints
• Inheritable permissions and effective grants

Runtime verification commands

Generate commands appropriate for the project.

Local / App Service env check

For Azure App Service:

az webapp config appsettings list `
  -g <resource-group> `
  -n <app-name> `
  --query "[?name=='ENABLE_A365_OBSERVABILITY_EXPORTER' || name=='EnableAgent365Exporter' || name=='A365_USE_S2S_ENDPOINT']"

For local PowerShell:

$env:ENABLE_A365_OBSERVABILITY_EXPORTER="true"
$env:OTEL_LOG_LEVEL="INFO"
$env:A365_OBSERVABILITY_LOG_LEVEL="debug"

SDK log grep

Select-String .a365-observability.log -Pattern "Agent365Exporter|identity groups|Obtained token|export|No token|No eligible"

Backend verification handoff

If backend verification is required, keep customer-facing validation on public/supportable surfaces:

1. SDK/exporter logs show export success and no chunk failures,
2. direct OTel responses have partialSuccess empty/null where applicable,
3. Microsoft Defender Advanced Hunting CloudAppEvents shows the expected AgentId / TargetAgentId / AlternateId after indexing.

If the user is on an internal service team, they may also use their team's approved dashboard/playbook to confirm request acceptance and downstream delivery, but do not print those internal queries or endpoints in the validator report.

(Report only support-safe signals — see Output is support-safe above.)

If code checks pass but Activity is still empty — verify tenant-side

Most "exporter returns 200/sent but nothing shows in MAC" incidents are not code bugs. When the code checks above pass, have the user confirm, in the target tenant:

Provenance (verified 2026-07-06): the observability resource-SP GUID, the Agent365.Observability.OtelWrite scope, and the ingestion-lag figure below are preview-era Agent 365 facts and will change. Re-verify against current Agent 365 onboarding docs before quoting them to a customer.

• Licensing state — confirm the target tenant/user meets the current Agent 365 licensing prerequisites from public onboarding docs.
• Tenant enrollment state — confirm Agent 365 is enabled for the target tenant through the supported onboarding or admin process.
• Observability resource SP present — az ad sp show --id 9b975845-388f-4429-889e-eab1ef63949c returns a service principal in the tenant. A 404 / AADSTS500011 ("resource principal … not found") means the observability app isn't provisioned there — an onboarding step, not a code fix.
• Ingestion lag — Defender CloudAppEvents populates before the admin center; give it ~5 min.

Mark task complete: "Check live blueprint permissions and runtime state".

Phase 5 — Summarize Blockers and Next Fixes

Mark task in progress: "Summarize blockers and next fixes"

Return a concise report. Default to the short format below; include the optional details block only when the user asks for deeper debugging or when a blocker needs a code pointer to be actionable.

## A365 Code Validator Result

**Status:** PASS / PASS WITH RISKS / BLOCKED

**Blockers**
1. **<short title>** — <one-sentence impact>.  
   Fix: <one-sentence fix>.
2. **<short title>** — <one-sentence impact>.  
   Fix: <one-sentence fix>.

**High-risk checks**
- <short item> — <why to verify>.

**Identity binding**
- Runtime agent id: <source / missing / from config>
- Blueprint id: <source / missing / from config>
- Verdict: <correct | wrong | unclear>

**Blueprint permissions**
- Live Blueprint: <current | stale/missing | not checked>
- Grants: <verified summary | missing required permission | not checked>
- Effective inheritance: <OK | NONE | BROKEN | not checked>

**Semantic spans**
- Found: <invoke_agent/chat/execute_tool/output_messages or "none">
- Missing: <ops or "none">

**Recommended fix order**
1. <fix root cause first>
2. <then fix config/env>
3. <then verify spans/identity>

**Optional details**
- Files: `<file>:<line or function>`, ...
- Blueprint check: <read-only command result or reason it was skipped>
- Runtime check: <local/App Service env check or SDK log grep>
- Backend verification: use the team's approved telemetry playbook; do not paste internal
  cluster names, private endpoints, or correlation IDs into this report.

Style rules:

• Keep the default report to 8 bullets or fewer.
• Put details under Optional details, not in the main blocker list.
• Use exact file/function names for evidence, but do not paste long code blocks.
• Redact per Output is support-safe above (no internal infra, tenant/agent IDs, or correlation IDs).
• A skipped live Graph check is a verification gap, not automatically a code blocker.
• Do not suggest a365 publish unless the agent is an AI Teammate / M365 package flow; blueprint-based observability does not use a365 publish.

Mark task complete.

Phase 6 — Offer Guided Remediation

Mark task in progress: "Offer guided remediation"

If there are no critical or high findings, say:

No blocking fixes needed. Keep the runtime verification commands handy for live validation.

Then mark the task complete and stop.

If there are critical or high findings, ask the user with AskUserQuestion:

I found blockers. What would you like me to do next?

Options:

Default: make_fix_plan.

Safe fixes allowed after confirmation

Only apply these automatically when the user chooses apply_safe_fixes:

Blueprint permissions are never a safe code fix. Do not run a365 setup permissions, mutate Graph, grant admin consent, or rewrite a Blueprint ID from this skill. Report the mismatch and hand off to a365-setup or the appropriate administrator.

1. Python exporter kwarg

   • If code calls use_microsoft_opentelemetry(...) or builds kwargs for that call and already sets enable_a365=True, add the explicit exporter flag:

     a365_enable_observability_exporter=True
     

     or:

     kwargs["a365_enable_observability_exporter"] = True
     

   • Preserve existing code style.

2. Environment templates

   • Add missing, commented or empty template entries only:

     ENABLE_A365_OBSERVABILITY_EXPORTER=true
     

   • For S2S mode, add the version-appropriate S2S transport setting only if the code/reference for that project already names it. Do not invent a setting name.

3. Config field scaffolding for runtime agent identity

   • If the project already has a settings/config class, add a clearly named optional field such as agent_identity_app_id / AGENT_IDENTITY_APP_ID.
   • Use an empty default or placeholder; never hardcode real IDs.
   • Update .env.example if present.

Fixes that require a second explicit question

Ask a focused follow-up before editing:

1. Which runtime agent identity config name to use

   • Suggested default: AGENT_IDENTITY_APP_ID / agent_identity_app_id.
   • Ask if the repo already uses a different naming convention.

2. Token resolver wiring

   • If the project already has a token provider with the required exchange flow, ask whether to wire an observability token resolver to it.
   • If no such provider exists, do not generate a full token service silently. Provide a plan and ask the user to confirm a follow-up implementation.

3. Semantic spans

   • Ask whether to add only top-level invoke_agent / output_messages scopes first, or also wrap tools and LLM calls.
   • Prefer incremental fixes: identity/exporter first, semantic span enrichment second.

Fix report

After applying fixes, show:

**Applied fixes**
- ...

**Still needs user input**
- ...

**Validation to rerun**
- `validate a365 code`
- runtime exporter log check

Run the smallest available project tests if the edited repo has an obvious test command and the user did not decline validation. Otherwise, state that no obvious test command was run.

Mark task complete.