Toolset Design

Toolset Design Guide

This skill codifies the methodology for designing MCP tools and toolsets in this repository. The core principle is eval-first, tools-second: prove the need before writing the code.

Phase 1: Validate the need with evals

Before writing any toolset code, write eval tasks that represent what a user would actually ask an LLM to do in the target domain. This serves two purposes:

1. Baseline: Run the evals with only the existing toolsets enabled (core, config, etc.). If the LLM can already accomplish the tasks using pods_exec, resources_list, or other generic tools, you may not need dedicated tools at all.

2. Gap identification: The tasks where the baseline fails (or produces poor results) reveal the actual gaps that new tools should fill.

How to write eval tasks

Create tasks under evals/tasks/<domain>/ using the mcpchecker/v1alpha2 format. See existing tasks in evals/tasks/ for examples at different difficulty levels.

Design tasks that represent real user workflows, not tool-shaped requests:

# BAD: This is testing a tool, not a workflow
prompt:
  inline: Run helm list --all-namespaces

# GOOD: This is testing what a user would actually ask
prompt:
  inline: What Helm releases are deployed across the cluster?

# BETTER: This tests whether the LLM can solve a real problem
prompt:
  inline: >
    The application in namespace "payments" was working yesterday but
    is failing after a recent Helm upgrade. Investigate what changed
    in the latest release and identify the issue.

Task difficulty should span the range:

• Easy: Simple queries (list resources, show config)
• Medium: Filtered/analyzed output (find specific entries matching a pattern, correlate data across sources)
• Hard: Multi-step diagnosis (investigate root cause, identify misconfiguration, suggest fix)

Running the baseline

# Run evals with only core+config toolsets to establish baseline
mcpchecker check evals/core-eval-testing/<agent>/eval-core.yaml --label-selector suite=<your-domain>

What to look for in baseline results:

• If the LLM uses pods_exec successfully -> dedicated tool adds only convenience
• If the LLM fails because it doesn't know the right commands -> tool descriptions add value
• If the LLM succeeds but output is noisy/unparsed -> filtering/pagination adds value
• If the LLM can't accomplish the task at all -> new capability is needed

Decision framework

Phase 2: Design the tools

The key question: tools vs. prompts

Before designing tools, ask whether a prompt would solve the problem instead.

• Tools execute operations and return results. They're for actions the LLM needs to perform.
• Prompts gather diagnostic data and frame it for LLM analysis. They're for workflows where the LLM needs context to reason about a problem.

See the kubevirt toolset for a good example: vm-troubleshoot is a prompt (not a tool) that gathers VM status, pod logs, and events, then presents them with analysis instructions. The core toolset's cluster-health-check prompt follows the same pattern.

Rule of thumb: If the value is in what data to gather and how to think about it, use a prompt. If the value is in executing a specific operation safely, use a tool.

Anti-pattern: instructions instead of data. A prompt or tool that returns a to-do list for the model ("call resources_get, then pods_list…") wastes tokens, adds round-trips, and hands a deterministic step to a stochastic actor. Instead, use dynamic context injection — the handler pre-fetches the data and injects it into the prompt text so the model receives facts to analyze, not steps to follow.

# BAD: output is a to-do list for the model
"To troubleshoot, call resources_get on the VM, then pods_list,
 then events_list, and look for errors."

# GOOD: the server gathers what it knows is needed
<VM + VMI + virt-launcher pod + logs + events, already fetched>
"Analyze the data above: check printableStatus, pod restarts, Warning events."

If the server can know in advance what to fetch, it should fetch it.

Tool consolidation: fewer tools is better

Every tool added to the MCP server increases cognitive load for the LLM. The LLM must read every tool's name and description to decide which one to use. Fewer, well-designed tools outperform many narrow ones.

The enum/subcommand pattern

When multiple operations share the same parameters and differ only in the action performed, consolidate them into a single tool with an enum parameter.

The kubevirt toolset demonstrates this well:

• vm_lifecycle with action: start | stop | restart (3 operations, 1 tool)
• vm_guest_info with info_type: all | os | filesystem | users | network (5 operations, 1 tool)

See pkg/toolsets/kubevirt/vm/lifecycle/tool.go and pkg/toolsets/kubevirt/vm/guestagent/tool.go for implementation.

When to consolidate:

• Operations share the same required parameters (namespace, name, etc.)
• Operations are in the same conceptual category (same CLI binary, same resource type)
• The LLM doesn't need to distinguish between them at the tool-selection level

When NOT to consolidate:

• Operations have fundamentally different parameter shapes
• You need different access control per operation (read-only vs destructive)
• The operations serve different user intents (diagnosis vs modification)

Grouping heuristic: Group by the CLI binary or API the tool wraps. If you're wrapping subcommands of the same CLI tool, that's one MCP tool with a subcommand enum. Don't create separate MCP tools for list-x and list-y when they're both subcommands of the same binary with identical base parameters.

Do core toolset tools already cover this?

Before designing any new tool, check whether the core toolset already handles the use case. The two most common sources of unnecessary tools:

1. CRUD on Kubernetes resources

The core toolset provides resources_list, resources_get, resources_create_or_update, resources_delete, and resources_scale. These work with any Kubernetes resource type via GVK (group/version/kind) parameters. If the proposed tool is essentially "list CRDs of type X" or "get a specific custom resource", it's already covered.

A dedicated tool for a specific resource type is only justified when:

• The tool does significant post-processing beyond what resources_list/resources_get return (e.g., correlating data across multiple resources, computing derived fields)
• The tool provides a domain-specific workflow that spans multiple API calls (e.g., kubevirt's vm-troubleshoot prompt gathers VM + VMI + pod + events in one shot)
• The resource requires specialized input that the generic tools can't express (e.g., Helm releases aren't plain Kubernetes resources)

If none of these apply, the tool is just a thinner wrapper around resources_get and should not be added.

2. Running commands in pods

The core toolset provides pods_exec which can run arbitrary commands in pods. A dedicated tool wrapping a pod exec is only justified if it provides value beyond what pods_exec offers:

Tool parameter design

Required vs optional parameters:

• Make parameters required only when the tool cannot function without them
• Use sensible defaults for optional parameters (e.g., head defaults to 100 lines)
• Use DependentRequired for conditional requirements (see pkg/toolsets/kiali/tools/list_or_get_resources.go)

Parameter naming conventions:

• Use snake_case for parameter names
• Use the same name as the upstream CLI flag where possible
• Common shared parameters: namespace, name, container, pattern, head, tail

Conditional parameters with enums: When a tool uses an enum subcommand and different subcommands need different parameters, document which parameters apply to which subcommand in the tool description. Make conditionally-required parameters technically optional and validate in the handler.

Tool annotations

Every tool must set appropriate annotations:

• ReadOnlyHint: true for diagnostic/query tools
• DestructiveHint: true for tools that delete or modify resources
• IdempotentHint: true if repeated calls with same args have no additional effect
• OpenWorldHint: true if the tool interacts with a live cluster (most tools)

See pkg/api/toolsets.go for annotation definitions.

Tool descriptions

Write descriptions for the LLM, not for humans reading docs:

• First sentence: What the tool does (verb phrase)
• Second sentence: When/why to use it (context)
• Keep it concise: The description consumes tokens on every LLM call

Returning results

Use api.NewToolCallResultStructured(result, err) for tools that return typed data. This populates both text content and structured content for MCP clients.

Avoid manual json.Marshal + api.NewToolCallResult -- that's the older pattern.

Phase 3: Validate the design with evals

After implementing the tools, update the eval tasks to assert the new tools are used:

assertions:
  toolsUsed:
    - server: kubernetes
      toolPattern: "my_domain_.*"  # Assert the new tools are called
  minToolCalls: 1
  maxToolCalls: 20

Then run the evals again. Compare against the baseline:

• Do the evals pass at a higher rate?
• Does the LLM use the new tools correctly, or does it still fall back to pods_exec?
• Are there tools you added that the LLM never uses? Those might not be needed.

Each toolset gets its own suite (e.g. suite: my-domain) and a corresponding eval-<suite>.yaml per agent. Create evals/core-eval-testing/<agent>/eval-my-domain.yaml for each agent that has eval configs. The eval file should include the core and config task sets alongside the new suite's tasks (see existing files for the pattern).

Phase 4: Implementation checklist

Once the design is validated, the implementation follows a mechanical checklist. See CLAUDE.md for full details on each step. The key files to touch:

1. Toolset package: pkg/toolsets/<name>/toolset.go -- implements api.Toolset interface
2. Tool definitions: pkg/toolsets/<name>/<group>/tools.go -- tool schemas and handlers
3. Module import: pkg/mcp/modules.go -- add blank import for auto-registration
4. Snapshot test: pkg/mcp/toolsets_test.go -- add to TestGranularToolsetsTools
5. Generate snapshots: UPDATE_TOOLSETS_JSON=1 go test -count=1 -v ./pkg/mcp
6. Update docs: make update-readme-tools
7. Eval configs: evals/core-eval-testing/<agent>/eval-<suite>.yaml (one per toolset per agent)
8. Eval tasks: evals/tasks/<domain>/*/task.yaml

Design review checklist

When reviewing a toolset PR, verify:

• Eval-first: Are there eval tasks? Do they test real user workflows, not tool-shaped requests?
• Baseline tested: Was the domain tested with existing tools first? Is there evidence the new tools improve outcomes?
• Tool count justified: Could any tools be consolidated using enum parameters? Does each tool provide value beyond pods_exec?
• Prompts considered: Would a diagnostic prompt serve better than (or alongside) some tools?
• Annotations correct: Are read-only tools marked ReadOnlyHint: true? Are destructive tools marked appropriately?
• Descriptions LLM-friendly: Do descriptions explain what the tool does and when to use it?
• Results use NewToolCallResultStructured: Not manual json.Marshal + NewToolCallResult
• Container targeting: If exec-ing into multi-container pods, is the correct container selected?
• Snapshot tests added: Is the toolset in TestGranularToolsetsTools with a snapshot file?
• All eval configs updated: evals/core-eval-testing/<agent>/eval-<suite>.yaml (one per toolset per agent)