Add Supervisor

add-supervisor — Add a Supervisor to a Project

Adds a supervisor agent to an existing AgentOps Stacks project. A supervisor routes user queries across the project's agents (and other managed sub-agents like Genie spaces or Knowledge Assistants).

The supervisor is a custom LangGraph agent — a hand-written supervisor graph that is just another agent: served as a Databricks App via MLflow AgentServer, declared in databricks.yml, and gated by the same CI eval loop as every other agent. This is a post-scaffold pattern, applied as the project matures — like eval gates, governance, and monitoring. It is not part of bundle init.

When to use

• The project already has ≥1 agent (databricks.yml + .agentops-stacks/manifest.yml exist).
• The user wants a single entry point that routes to multiple specialists.
• Usually invoked after /add-agent has produced a second agent.

Step 0 — Is a supervisor even warranted? (run this first)

Multi-agent orchestration is powerful but easy to reach for too early. The Big Book of AgentOps names "overcomplex architecture — using supervisor agents and multi-agent systems when a simple sequential chain would suffice" as an explicit anti-pattern (orchestration overhead, harder debugging, possible loops between agents), and its first guiding principle is Start Narrow, Expand Deliberately (prefer deterministic over probabilistic). So before scaffolding, check:

• Only one real specialist? → don't add a supervisor; keep the single agent.
• Fixed, known order of steps? → a sequential chain / deterministic router in one agent is simpler, cheaper, and easier to evaluate. Don't add a supervisor.
• Routing that genuinely depends on the request (intent classification across ≥2 non-overlapping specialists, dynamic hand-off, chaining across domains)? → a supervisor is warranted. Proceed.

State your reasoning to the user. If a supervisor isn't warranted, say so and stop — recommend the simpler shape instead.

The pattern — custom LangGraph (GA)

A hand-written supervisor via langgraph-supervisor's create_supervisor (or a raw StateGraph router). You own the routing loop, so you control state, guardrails, retries, and human-in-the-loop. Sub-agents can be a databricks_langchain.GenieAgent, a remote serving endpoint (a deployed sibling agent), or an in-process ReAct agent. It deploys and is eval-gated exactly like any other agent — no extra workflow.

Why only custom? A Databricks-managed "Supervisor API" pattern was considered and removed: that API is deprecated and reaches end of life on 2026-09-30, and Databricks' own guidance is to build multi-agent systems as custom agents on Databricks Apps — which is exactly this pattern. (The manifest.yml still records type: so a future GA managed pattern can be added without a contract change.)

Workflow

1. Run Step 0 — decide whether a supervisor is warranted. If not, stop.
2. Locate the project — find databricks.yml in the current dir or a parent.
3. List existing agents — show what's under src/agents/ (candidate routes).
4. Gather inputs:
   • Supervisor name — must match ^[a-z][a-z0-9_]{2,}$ (and not collide with an existing agent).
   • Routes — comma-separated sub-agent names (≥1 required). Local agents are validated; names that aren't local agents are assumed managed sub-agents (Genie/KA/endpoint).
   • Source agent — which existing agent's App shape to base on (default: first found).
5. Run the script:

   python plugin/skills/agentops-stacks/scripts/add_supervisor.py \
     --name <supervisor_name> \
     --routes <agent_a,agent_b> \
     [--from <source_agent>] \
     --project-dir <project_root>
   

6. Guide customization (below) and relay the script's next-steps output unchanged.

What the script does

1. Copies a source agent as the App shape, renaming references (parity with add_agent.py).
2. Overwrites graph.py with the supervisor graph, tools.py with a supervisor stub, and agent.py with a stateless supervisor handler.
3. Adds langgraph-supervisor to pyproject.toml.
4. Appends an experiment + app resource to databricks.yml.
5. Records the supervisor in .agentops-stacks/manifest.yml. → CI's detect_patterns → eval_gate picks it up automatically (it has eval/gates.yml).

After adding

• Edit graph.py to point each route at its real backend (GenieAgent, remote endpoint, or a ReAct sub-agent).
• Add a routing-accuracy scorer to eval/gates.yml. Routing decisions are a structured output, so evaluate them programmatically (accuracy / F1 / confusion matrix over labeled expected-route examples) rather than with an LLM judge — per the Big Book's tiered-evaluation guidance. A supervisor whose gate doesn't test routing is rubber-stamping its core function.
• uv sync; databricks bundle validate -t dev && databricks bundle deploy -t dev.

Error handling

• Name collides with an existing agent → abort.
• Invalid name format → abort with the pattern hint.
• No --routes → abort (a supervisor with no routes can't start).
• No databricks.yml found → abort (not an AgentOps Stacks project).
• No existing agents to base on → abort (scaffold an agent first).
• Manifest already has a supervisor: block → leave it; tell the user to edit by hand.

Security posture (state this to the user)

• You own guardrails; scope each sub-agent's auth via MLflow resources=[...], least-privilege per endpoint.
• Flow the end user's identity through to sub-agents (on-behalf-of-user, or an explicit non-LLM-controlled ID filter — never take the scoping value from model output). This is the Big Book's two-level permission model: agent/tool least-privilege and end-user identity passthrough.
• Fully within the workspace boundary.

Reference

• Pattern deep-dive: docs/supervisor-patterns.md in the rendered project (scaffolded by the template).
• Custom multi-agent apps: https://docs.databricks.com/aws/en/agents/agent-framework/multi-agent-apps