Agt E2e Encryption

AGT E2E Encrypted Inter-Agent Communication

Overview

Kars agents communicate via the official Microsoft Agent Governance Toolkit (AGT) — a decentralized, E2E encrypted messaging protocol using Signal Protocol (X3DH + Double Ratchet). Transport is provided through @microsoft/agent-governance-sdk (upstream) wrapped by the in-repo @kars/mesh provider, selected at runtime via createMeshTransport().

OpenClaw runtimes go through the TypeScript SDK; Hermes runtimes go through runtimes/agt-mesh-python/ (kars-agt-mesh), a Python AGT MeshClient at byte-for-byte wire-format parity with the TS SDK. The same relay/registry pair serves both. Cross-runtime bidi (OpenClaw ↔ Hermes) is proven on AKS and local kind — see tests/e2e/interop/.

History note: Earlier releases used a vendored fork of @agentmesh/sdk with 8+ local patches under vendor/. All upstream-relevant fixes have landed in AGT (PRs through the toolkit's 3.x line) and the vendored tree has been removed. The bug table below is preserved as a regression checklist — if any of these symptoms reappear, treat as a regression in the official SDK.

Protocol Flow

Parent Agent                    Sub-Agent
     │                              │
     │── Register + upload prekeys ──│── Register + upload prekeys
     │                              │
     │── Registry search by name ───│
     │── Fetch target's prekeys ────│
     │── X3DH key exchange ─────────│
     │── Send KNOCK via relay ──────│── Receive KNOCK, auto-accept
     │── Encrypt with Double Ratchet│
     │── Send encrypted msg ────────│── Decrypt with Double Ratchet
     │                              │── Call local LLM via router
     │── Receive encrypted reply ◄──│── Encrypt reply, send via relay

Infrastructure

• Relay: agentmesh-relay in agentmesh namespace (WebSocket, routes encrypted blobs)
• Registry: agentmesh-registry in agentmesh namespace (agent discovery, prekey storage)
• Router proxy: Inference router proxies /agt/relay (WS) and /agt/registry/* (HTTP)

Encryption Details

• Identity: Ed25519 signing + X25519 key exchange per agent
• Key Exchange: X3DH (Extended Triple Diffie-Hellman) with signed prekeys
• Message Encryption: Double Ratchet with XSalsa20-Poly1305 (libsodium)
• Session: KNOCK protocol for policy-gated session establishment
• Forward Secrecy: Per-message key rotation via Double Ratchet

Historical bug catalogue (now-upstream regression checklist)

All originally fixed via patches in vendor/ (removed in 2026-05). Listed here as a regression-detection cheatsheet against the current official @microsoft/agent-governance-sdk:

Debugging

• Check relay logs: kubectl logs -n agentmesh -l app=agentmesh-relay
• Check registry logs for prekey uploads: kubectl logs -n agentmesh -l app=agentmesh-registry
• Check sub-agent relay listener: kubectl exec <pod> -c openclaw -- cat /tmp/agt-relay-listener.log
• Parent plugin logs: look for [plugins] AGT relay: lines

Trust Scoring

• AGT_TRUST_THRESHOLD env var (default: 500 in sandbox, 0 = accept all)
• KNOCK handler evaluates peer trust score via registry lookup
• Anonymous agents (no OAuth) have score 0 — fail open when registry lookup fails
• Verified agents (Tier 1, GitHub OAuth) have score 600+

Mesh-message metrics

The inference router exports two Prometheus counters that the Headlamp Mesh Topology and operator-CLI topology use:

• kars_mesh_messages_sent_total
• kars_mesh_messages_received_total

Defined in inference-router/src/metrics.rs (AGT_MESH_MESSAGES_*) and incremented in inference-router/src/routes/mesh.rs next to the atomic MeshMetrics::messages_{sent,received} counters. The sandbox=<name> label is added at scrape time by deploy/monitoring/podmonitor-sandbox-router.yaml.

Counted: KNOCK frames, X3DH bundle exchange, each mesh_send call, explicit sendHeartbeat() ticks (every 30 s, scheduled from mesh-plugin/src/agt-transport.ts — vanilla AGT doesn't auto-heartbeat).

Not counted: WebSocket Ping/Pong keepalives (short-circuited with continue before the counter increments), and registry HTTP calls (/v1/agents/..., /v1/agents/{did}/heartbeat) which go over HTTP, not the WS relay.

Why sent ≫ received early on: a fresh sandbox emits ≥ 1 KNOCK per known peer plus a 30 s heartbeat tick, but only receives back the relay's KNOCK-ack until a real bidirectional conversation starts. Counters live in the router process and reset on pod restart.

Cross-runtime mesh (Hermes ↔ OpenClaw)

Wire format

Both sides are pinned to the same upstream commit via vendor/agt/pin.json::sha — a single source of truth across TS, Python, and Rust.

Prekey-clobber guard (DO NOT BYPASS)

MeshClient.connect() takes an exclusive fcntl.flock on <identity_dir>/.mesh-prekeys.lock (runtimes/agt-mesh-python/src/kars_agt_mesh/client.py::_acquire_prekey_writer_lock). A second Python process attempting to start a MeshClient for the same identity raises MeshTransportError naming the holder PID.

DO NOT debug a live Hermes pod by running kubectl exec ... python3 -c "from kars_runtime_hermes.plugin import mesh; mesh._get_or_init_client(); ...". The secondary process would (before the guard) generate fresh X3DH key material and PUT it to the registry, clobbering the daemon's bundle and causing silent Decrypt failed for did:mesh:... log entries with no traceback. Cost the team several hours of debugging in 2026-06; see docs/internal/security-audits/2026-06-06-cross-runtime-mesh-aks.md.

Correct ways to inspect a live daemon:

# Identity / DID
kubectl exec <pod> -c agent -- cat /sandbox/.hermes/.agt/identity.json | jq

# Operator trust store (peers seen by the daemon's mesh_worker)
ADMIN=$(kubectl get secret -n kars-<sb> admin-token -o jsonpath='{.data.token}' | base64 -d)
kubectl exec <pod> -c agent -- sh -c "curl -sS -H 'Authorization: Bearer $ADMIN' http://127.0.0.1:8443/agt/trust"

# Daemon log
kubectl logs <pod> -c agent

AKS-specific Entra requirements

Hermes pods on AKS require two RBAC arms for the mesh to bring up the verified tier:

1. The pod's workload-identity client (Entra Agent App auto-provisioned per sandbox) — must have Azure AI User on the resource group. Auto-granted by the controller when you set KarsAuthConfig.spec.foundryRbac in the sandbox namespace. The CRD field landed in commit 496cc92.
2. The Foundry project's MI — needs Azure AI User on the resource group for Memory Store operations. This is a one-time Portal step per project (see the Foundry Memory Store Auth section in the repo's CLAUDE.md).

The entrypoint exchanges the projected SA token for an Entra Agent App token (audience <app-id>/.default) and POSTs it to /agt/registry/v1/registry/verify. Success surfaces as tier=verified, verified_app_id=<guid> on the operator panel.

kars push --only runtime-hermes is hermetic

Out of the box (no manual git clone, no bash runtimes/build-agt-wheels.sh):

kars push --only runtime-hermes --apply

The CLI auto-clones microsoft/agent-governance-toolkit@kars-sdk-pop-signing@<pin sha> (via cli/src/lib/agt-bootstrap.ts::ensureAgtRepo) and builds runtimes/wheels/*.whl (via ensureAgtWheels) before docker build. Cached by runtimes/wheels/.agt-sha so re-runs are no-ops.

End-to-end harnesses

If either regresses, the first thing to suspect is the AGT pin — vendor/agt/pin.json::sha must match upstream microsoft/agent-governance-toolkit:kars-sdk-pop-signing. Pre-existing image layers may have a stale SDK; kars push --only runtime-hermes --apply (or runtime-* per affected runtime) refreshes them.