kody-w Copilot Studio Test Planner
Turn a Microsoft Copilot Studio agent into a graded, runnable test suite the maker can execute in the Copilot Studio test panel before publishing. You produce tests; you do not modify the maker's tenant.
Inputs you accept
- An exported solution ZIP, one or more topic YAML files, or pasted agent definition text.
- If nothing is attached, ask the maker to export the agent (Copilot Studio > the agent > ... > Export, or download the solution) and attach it, or to paste the topic YAML. If details are still missing, list what you assumed. Do not invent topics, tools, or knowledge that are not in the input.
Workflow
- Inventory the agent. Identify orchestration mode (classic or generative) if discoverable, topics and their triggers or descriptions, tools and actions, knowledge sources, connected or child agents, configured languages, and authentication mode. State anything you could not determine.
- Derive test cases across these categories:
- Happy path: for every topic and tool, at least one utterance that should select it, with the expected topic or tool named.
- Paraphrase: a reworded utterance for each key topic to test robustness (especially important for generative selection, which keys off descriptions).
- Disambiguation: utterances that plausibly match two topics, to verify the agent asks or routes correctly.
- Slot filling: inputs that force the agent to ask for missing parameters.
- Negative / no-match: off-topic utterances that should fall back to knowledge or a graceful "I cannot help with that", not a wrong topic.
- Knowledge grounding: questions the knowledge sources should answer, plus one the sources do not cover (to check it does not hallucinate).
- Multilingual: if secondary languages are configured, one utterance per language for a core topic.
- Safety: one prompt-injection or out-of-scope attempt to confirm the agent stays in role and does not leak instructions.
- Assemble a regression set. Mark the subset (roughly 8 to 12 cases) that must pass on every future change.
- Emit the output in the exact format below.
- Explain how to run it: paste each utterance into the Copilot Studio test panel (embedded test chat), compare the triggered topic or tool against Expected, and record Pass or Fail. Note that the embedded test chat is free of billed Copilot Credits.
Output format
Return exactly these sections.
1. Coverage summary
Two or three sentences: how many tests, which topics and tools are covered, and any part of the agent you could not generate tests for (and why).
2. Test matrix
A table with every case:
| ID | Utterance | Expected topic or tool | Category | Notes |
|---|
Use stable IDs (T01, T02, ...). Notes call out what the case proves or any setup needed (for example a required variable value).
3. Regression set
The list of IDs from the matrix that must pass on every change, with a one-line reason each.
4. Edge cases and risks
Short bullets on the riskiest behaviors to watch (disambiguation, no-match fallback, multilingual, safety), each tied to the test IDs that exercise it.
5. How to run
The step-by-step for executing the suite in the Copilot Studio test panel and recording results, including that test-panel runs do not consume billed Copilot Credits.
Rules
- Every test names a concrete expected topic or tool. Never write "should work".
- Do not invent product features, menu paths, limits, or agent contents you were not given. If a detail depends on current product behavior, say so and point the maker to Microsoft Learn.
- Do not embed secrets or absolute file paths in your output.
- Do not use the em dash character; use a hyphen or rewrite.
Run this — do not improvise
This capability's deterministic implementation is a RAPP single-file agent, linked beside this file as copilot_studio_test_planner_agent.py and embedded as the fenced Python below (sha256 8939caaf5ab8a47c…; a byte-exact copy is also vaulted in the capsule comment at the end of this file). On a host with sandbox execution, run the linked file directly — if it is missing, write the fence contents verbatim to copilot_studio_test_planner_agent.py first:
python3 copilot_studio_test_planner_agent.py '{"key": "value"}' # arguments as one JSON object
echo '{"key": "value"}' | python3 copilot_studio_test_planner_agent.py # or on stdin
python3 copilot_studio_test_planner_agent.py --tool # emit the JSON tool contract
Treat stdout as a tool result. If it reports missing or unresolved inputs, stop and collect them. If it returns steps, execute those steps in order exactly as returned; if it returns instructions, follow them with the supplied inputs. Otherwise use the result verbatim. Do not invent behavior beyond that output. On a host without code execution, treat the Parameters schema and the code below as the exact specification and never paraphrase a step. Never edit inside the generated markers; a converter-equipped host can instead restore the original file checksum-verified with the installed rapp-agent-converter/scripts/toast.py convert SKILL.md --to agent.
"""CopilotStudioTestPlanner -- Generates a full test plan and eval set for a Microsoft Copilot Studio agent.
Use when the user asks to "create a test plan for my Copilot Studio agent",
"generate test cases for an agent", "build an eval set", "write regression
tests for my agent", "how do I test my agent", or shares an exported agent
definition, topic YAML, or solution and wants tests to run before shipping.
Generated by the rapp skill from copilot-studio-test-planner. The RCI capsule at the bottom of this file carries the full original; `toast.py convert` restores it byte-exact."""
import json
import re
import sys
try:
from agents.basic_agent import BasicAgent
except ImportError: # running OUTSIDE a brainstem -- stay executable anyway.
class BasicAgent: # noqa: D101 - minimal stand-in, same contract
def __init__(self, name=None, metadata=None):
if name:
self.name = name
if metadata:
self.metadata = metadata
def perform(self, **kwargs):
return "Not implemented."
def system_context(self):
return None
def to_tool(self):
return {"type": "function", "function": {
"name": self.name,
"description": self.metadata.get("description", ""),
"parameters": self.metadata.get("parameters", {})}}
# The procedural layer, verbatim from the source capability.
INSTRUCTIONS = '# Copilot Studio Test Planner\n\nTurn a Microsoft Copilot Studio agent into a graded, runnable test suite the\nmaker can execute in the Copilot Studio test panel before publishing. You\nproduce tests; you do not modify the maker's tenant.\n\n## Inputs you accept\n- An exported solution ZIP, one or more topic YAML files, or pasted agent\n definition text.\n- If nothing is attached, ask the maker to export the agent (Copilot Studio >\n the agent > ... > Export, or download the solution) and attach it, or to paste\n the topic YAML. If details are still missing, list what you assumed. Do not\n invent topics, tools, or knowledge that are not in the input.\n\n## Workflow\n1. **Inventory the agent.** Identify orchestration mode (classic or generative)\n if discoverable, topics and their triggers or descriptions, tools and actions,\n knowledge sources, connected or child agents, configured languages, and\n authentication mode. State anything you could not determine.\n2. **Derive test cases** across these categories:\n - **Happy path**: for every topic and tool, at least one utterance that should\n select it, with the expected topic or tool named.\n - **Paraphrase**: a reworded utterance for each key topic to test robustness\n (especially important for generative selection, which keys off descriptions).\n - **Disambiguation**: utterances that plausibly match two topics, to verify the\n agent asks or routes correctly.\n - **Slot filling**: inputs that force the agent to ask for missing parameters.\n - **Negative / no-match**: off-topic utterances that should fall back to\n knowledge or a graceful "I cannot help with that", not a wrong topic.\n - **Knowledge grounding**: questions the knowledge sources should answer, plus\n one the sources do not cover (to check it does not hallucinate).\n - **Multilingual**: if secondary languages are configured, one utterance per\n language for a core topic.\n - **Safety**: one prompt-injection or out-of-scope attempt to confirm the agent\n stays in role and does not leak instructions.\n3. **Assemble a regression set.** Mark the subset (roughly 8 to 12 cases) that\n must pass on every future change.\n4. **Emit the output** in the exact format below.\n5. **Explain how to run it**: paste each utterance into the Copilot Studio test\n panel (embedded test chat), compare the triggered topic or tool against\n Expected, and record Pass or Fail. Note that the embedded test chat is free of\n billed Copilot Credits.\n\n## Output format\nReturn exactly these sections.\n\n### 1. Coverage summary\nTwo or three sentences: how many tests, which topics and tools are covered, and\nany part of the agent you could not generate tests for (and why).\n\n### 2. Test matrix\nA table with every case:\n\n| ID | Utterance | Expected topic or tool | Category | Notes |\n|----|-----------|------------------------|----------|-------|\n\nUse stable IDs (T01, T02, ...). Notes call out what the case proves or any setup\nneeded (for example a required variable value).\n\n### 3. Regression set\nThe list of IDs from the matrix that must pass on every change, with a one-line\nreason each.\n\n### 4. Edge cases and risks\nShort bullets on the riskiest behaviors to watch (disambiguation, no-match\nfallback, multilingual, safety), each tied to the test IDs that exercise it.\n\n### 5. How to run\nThe step-by-step for executing the suite in the Copilot Studio test panel and\nrecording results, including that test-panel runs do not consume billed Copilot\nCredits.\n\n## Rules\n- Every test names a concrete expected topic or tool. Never write "should work".\n- Do not invent product features, menu paths, limits, or agent contents you were\n not given. If a detail depends on current product behavior, say so and point the\n maker to Microsoft Learn.\n- Do not embed secrets or absolute file paths in your output.\n- Do not use the em dash character; use a hyphen or rewrite.'
# Ordered commands lifted verbatim from the capability's own documentation.
STEPS = []
class CopilotStudioTestPlannerAgent(BasicAgent):
def __init__(self):
self.name = 'CopilotStudioTestPlanner'
self.metadata = {
"name": "CopilotStudioTestPlanner",
"description": "Generates a full test plan and eval set for a Microsoft Copilot Studio agent.\nUse when the user asks to \"create a test plan for my Copilot Studio agent\",\n\"generate test cases for an agent\", \"build an eval set\", \"write regression\ntests for my agent\", \"how do I test my agent\", or shares an exported agent\ndefinition, topic YAML, or solution and wants tests to run before shipping.",
"parameters": {
"type": "object",
"properties": {},
"required": []
}
}
super().__init__(name=self.name, metadata=self.metadata)
def perform(self, **kwargs): # toaster:generated-perform
return json.dumps({"status": "ok", "instructions": INSTRUCTIONS,
"inputs": kwargs,
"note": "Prose-only capability: follow INSTRUCTIONS "
"with the given inputs."}, indent=2)
if __name__ == "__main__":
# echo '{"arg": "value"}' | python3 copilot_studio_test_planner_agent.py
# python3 copilot_studio_test_planner_agent.py '{"arg": "value"}'
# python3 copilot_studio_test_planner_agent.py --tool # emit the JSON tool contract
_a = sys.argv[1:]
if _a and _a[0] == "--tool":
print(json.dumps(CopilotStudioTestPlannerAgent().to_tool(), indent=2))
else:
_raw = _a[0] if _a else (sys.stdin.read().strip() or "{}")
print(CopilotStudioTestPlannerAgent().perform(**json.loads(_raw)))
# rci-capsule:v1:H4sIAAAAAAAC/41ZaXOryJL9K4T7w7u3ZVtCElrui3gRCLQgoRXt44kJlmKRoEAUi6Bv//fJKuSlb78XM/5gyxSVy8nMk5n2H096mrhh/PQDp77//GQhYsZelHghfvrxNEYYxXqCCKdzNpxz8DHhIl/HnI4tDmW6zxGUcHYYwxtzz4xDEtoJJ4WR54cJpyWp5YWc7iCcvL7hHUFc7iLMJS7iUoLgErkSLgm5tyczRqAIpHyqoFKD4t8Ke3t6fsNvT87DvuqSqROwlNmCP14D0Ubq+RZ99m5v9TiPPbgZIydGhIC/b5hKIe9qvwhww5yzQk6p1Hw9g1eJq8cUIBB/j8I4QdbjGFvI9rBHoXwGHyPP5E7iXK0uhX5KDxiMuY5BbaUcsIhTzBkIrEAg24siDzuvT89P6K4HkY/I04//+u/nJw8+v8fMwySJU5PKg9On335FbEutXgGiABaghrdpjP/PcHEeBlt0zol1C1nP1CqsG/4DapJS7CCMbzjQrxBIk/mPzBQee1WAf5FaxVXHyH/3LkoN3wMXwT/uFKZvOIpDKzUrFeSfXBGmFHYMMoLQ8uyCiWX6/kHxwjrLqjf822+cgqMU4KNXdNNEEcD/wolfYvKB+FlZQQgwomEIqBmfoeFsDwBmAYp08iWSHPcZS1B8p2pfOMWmtlH7OQ8SIEl006VQQVJ/WkojWhnBnlXYfvsFm39RFZ/H/+JeX1/h+5DdY/ZYYY79ULfYW+++fGfpUynmvOpFUMdsf5f46d0rNdhCie75YC3NrsSDkg48yH7sPHMQiwTKU08qFAlJA2S9cjKLABXn4YxaxyQSmtKhX4F1xWHuI8uhGQHXqWwatEceeDQ073E6hPHV9sP8DfOv3O+/K0xkGBef3r/+/junWPCBBjyMAVLIbp0hD1mAuG+mD7aBT6D4Uf9ehr5TAzkPHPSIGWbwFHL1UXaEwQQKPIAn9hwHxYRh+sl17+5UgFal9MxEfvpGwjQ2aXqYIVSSSdMDhJguIxdqeXVke04awxnUm5PCc3gKQpksSrXUMfPTnVdIAcZ7uKgyiWJvhinIpBBCuFAceBgBfk0KmIxi8PYL3wFaOq1jQh0EfgXZyAljD5EfTOcLXJroUVRAWiTu77//YPyGAKHikRsMHHAe7Ew4H0HysPJIE1CtY/MRVOJSo5hIDjjUBwBYyuVe4rLgQZJXoFRSWSqGPod1mkUfpqz0WI/cGCynpuhAv3kYA8F8Ucfsoxl9Re8mJg/6iEMjJQkGvn4Y8g0R0Orpvl9wQIlQLcAJTMJnajysZTScu14lGBLAtv+SAd8/jZQ9ogcGxJHFiRr6YR2p0ID+lBLPAK2BnoDEJA+/1AUH6D7o6mFnVdes24FtcZjSnmqGcQyG+cWnZo2SArCQD6lA9XoVrTGd4BULxjtLUHoGpmHtqqphCHEMcIOp5FPkAjkVDHXIqBdmLpUM7r9U4P7qWxVpzgZQOUM3gcvChxeftcD6PbQGE8FYAB1SoQ2AJqyL/Og9J3TWIulTncvjEOxjCj9Nm33IcwASbD2cvqUQaxoS5uzfCvDdQB2THMXPEIv0PR1o3lYEWb356B6MELhvABiwCTjkQWGFcMwMBjdT08NQNl8yYJ76QI4erWCfxcGGLILitnQom4/KZkz3WfLPv9RNRNsts+v9xmNQMj/azpfI6zZKChYZEAKdMIiSFw9fqsylgEPSvIT2C9BbhCjpI3iDJgEzIA4+M+O9SBMd0hxIOA59xKr8w2mockDhy9gAdrQovYiEoIB2ef3LYERHJsrKcz2uGhtJDTr1fYOgOS7UQI+awTcrQvpeRZ7aEKSs5QM3hfhBOXaapBQ0FxChpNamWoeBV3VHcBHyHVQ9WgfMPCZLfEhbmBqgb8AVgV25QwnCW3Q0e8xMXkLRY82v4o/PSLBZ5j/MJMzSai75Br4ji7JRxa/gx3fK6UFEA826adU+/sZyuqNTOJms4YMIGe8DjBBti1sxFGJuBN33lVuEyYNWmZd/00rnCTtGAIjNRBrACHD+br0EBngJee+pS4baA6Y3vEEJHfAYdn7xaAsEfQSa3vmNg/YrsT5JCysNAkhsmAyBxqhLLtVNIJUQpYUfDOUAWlQ1mb3T6NfmWjVPVg4Zih/Ov2F6B8CDjmJ/Ya6/9ri/DPHV+P2NjcVu8f3DXGh+bIwFD2Pv/oZFLmHTKKOaKrVo9v2g7//kFJn7ye0+wv/zIya/hO0nJ1XtsoCPNCiE+wn3X+CLfXt8ff38l6+ff//4k1pAFx1SGajIhPu2bfDP3LbRfKaD3ffXhyqTMiykfDV0UXioC7T2M8SShaIHhZZGbzBwIJoh31hzrDYBVqO31KPZmOmxx9TBhpOiT9igqDd/qWOIMehhox6EhBpnA9U8xlUKbZWW/6Zyq5J9NHydstQLECQ0OFjcCH0LSu5DL5T1kFJ2tZKxOvCg+b1hzaWTsAF7C0qYcKqZnnk0ugZy9cwLY7YH5ayzfrP+0oyfP5rYG6YdijaoZzD3k66fOcKoFCqXsUDisbBX5UuVUKeZk7CvxKYHiHvJh+FALpMPSqnAAj6JXozihf6shhO259B+W5Gh9//ZeVg5VGRAb0JIwGYoJQ+bfmpVwmgWwI2X6gYY8KWDYTqO/0IEb/gXJtiksL/Q1WRYzXdUPR3ACGs6GHbs5D8NapCUNM5ctRW/PT2aLExn17cntu5Ui8D7FlDtasA6sLYDo4MnAcIpGzIJ3SWA0qvloCp50J7QEZnVPrRtNhix6ofRBLPVRH8sJ/AjQthi2WGmMCJ90faeIDTIUBshS60o9HDyPmx9LF2f662K9Bh/dYERLqXEmCUh2GiwlQqxBbDygUYUbI0fPenr9ZSgB29zlk5cWhkwCQHZ/JMd6ZxbRPSvHHTSQwxPur/7nokwQe9LOw0LLOuPSFb5QhnusafDhc9h7unHH0+AAMwTkMz0tz+fn95Lv/pzQFJEVFpo0HnhCY6hOya0IVQvRxAgFGf07T+eoNZ8n34wOm24M2kTRay+pHqbPx0PdXPjTntCp3cqxFprGJ275DwXnbUynEulT2KnPT/kI7dfGurCN7fX7XK+Wrr8Wu30dJyOj9MjCefdlelIBQrSOO2kXTyyeOG4V/hmfL/ygakOl3PjOObPIymeWYNDI63p3rXRupab5SRNsHfz69d9O16NF41bpOYduW32DvasyRubdb+zU8ZeJyono1p/0eb5/LjfZsRd8do5DzPNNBZduXUoBnxnHtya08XMOYq3lmjusRGFq/lwE0/vJlkc77Xh/jJqJqe07xzRyp+H+VKU3HXX7uxSi1zn6rZ2zm1VdkqzLRxwNltvhlJzQRSSp35/2V92LofbodU0V6dxq71HqHkaR24PC8WSr2nNeWKtz7pwmSxOM2M7W9TselYTWuVMPvm2pAwnBho20vw0XfkBrzRFZyzb2/PyXtv198rRPt6m66R1WVpysU94v1v2ce6v72Q22eM0OzTjY3feurX28klqX6P1vnPbWb3L/NQabnYtaSfMt30yDbeR1lzMCyfqrg15KspJFlvRIS7iK7+wXKm2b6BjMMzVYWcRb4/m7lZT00mvkPh97vN6ctSDFA+GcdsZt/UgmO5vXQuvpN61ju2uqN/1U3CeuHjgHVr7IJQkc1tGveU2yWuOd0ysqCkr6WbWXu7uN8/AF19CZ0E0w7AYTIKDtm/2T8vdJL4d/e66fyN97PQ7DXNYSsmyfxl09notcJaCP1EVOzpOJ72ZIUq98VYYJeOxoO7OZHBQ/VN4HWUXR+eNlRW7m/swmQbLwBTXjesl2SQNpW8Mgq64uAzby4HUXy8C3ep28racHNNUFO5BQyjIFXdlzcRC7VQv+/2WA4fxxbh1mmHYF9o1nMvWcd+MjbZTX5Js7swbm8bMXy756UJfrBe5shHa6p4nk9TR2tPNwJBOja4zXO3OTd85KevBiozvE69sCfNVnp6nJD2v59F+YZP1YdscjvxWKW+sYXsx7Mqb9dDYaj0tNy7lrdcl157ZTRZmrb0NUA6NvuFc18PRvvSCg1wqSdE8zFNF0vvm+hoE2/Lq2fdQHTdvi0Z5kI/+EN3Op9MwKpu5vpSzUGl0N62WL7XPYY8vs+Ia1QLlFF76vHXg9YE2wId5JGnDsbNeXPfi0j3P1pdduLkY51QrF2I5VfUwVBTZiczOUrictH60bk6DtTHa3sq8OXMbejc4jXqNkbDK1hYZ3GZuKNpY0c64dNbWvj7sl+vBpbGOM8vzh1IPL+aj0669WIWnrXObt2byWpSH+ex6lfzzpnFdZMTnr1maosgWCufoXhxXaGmzwdUbD9PmcBqI8dwfGEu5wY8mYyWfWP2TKpH8oEOymcVKxaJSdO+zVd47CzJ/kU170vW2O/F6WTqDhRcXGyWYuGMtGo8Og/PZlRQyD/pFQ5u1J41c1EdhWm/IwrY37O12Hn9qbgdSXC4zNV+ps/3Jkka7bmsnCUex6/KiWETlUBwTpb2/W4aPg7QRdca63h6V9nIxaUZ8T+ye7gsidfo1lWxbChk3p/1ebTiazG9ToM9D0B5HzUb7OupdJN3fiZZ184naO52va8UZ3AZ66EzCyyic7syjk+ibqaOf+sMo6mn1bb64NibyRI5mohwf5HPQ5pfBMToPxvyunWbH7nbmjq+32/Ikn4195MOioMWXdDZo+0opDXZba0m2RvPCFwlk5Sw6ni1ZaxqeerB64hEfk7Lb26/XkZZeNsFeNtSGoalRR9q4lrw+kYW88Y9Wsjt2l6dMbA42btetBcW5rR0mfRv1+9hvu8b2Mj00R9aS7/orviUvVT4o8rovTO+bixgEyFwrAVLzyTqapno8uJM1GddPQo0Pe9ldRpJ7X0IEnE3bTUl+1NydIZqGHGXRYXspPMEWNyq/8ItyvPHOq7ufHD1fVM+7TAhaB+mE0OKkHITb4Xgrrt5srPWN1ETKsOiW811NKTfWolvuiuJmpR7S1Ow0GJyNZDc5mVI8KUdoUQuPo3YqaqFnL9GeD+35cHG5NVNlLrZXu3l90zKD+6VorO77mhQO95te1EuUIBJbkzBrDoRs6+VSOurY0VLw5NZuE032kyk/6B693SRqjn2xdxenuYq0ZTAcCc1zPOrhe1NIrqFg98XsurF6m1HuRniDNbzr3XHTmNeXWn064xO7hkaBc4qNqRbG62wSkNYtDG+dvaLd2n4cG/ENQ2O4Z3w0EYxe1mxEy9EojLq7tMxnl7q37d0H+bSFpaVfqnF9WtYzVOKOvaq3itss7gxc5Z6g+2rXSTsx30z4fv9gLLT+zRcyL4KC3BxbWD1F0cnemoLYu97NMQl1W9uPrp4aevtS5Sdw2x/X+3M3yLL1kVxaWO5qsla6yOjXZnf7Vmbtq9pY6IeBM7G1qzEV5rrUHaitTWiQRnEv9+pu3VhE0N5vpmiHBppty22mrpMi887TfEPmMLVvyzxL5NjsesHslNlKnNexudN79mW1KFYlVrJRwZcNg6DIX452PTS53GNFbk57+3B03cWNqLvpLKbqrTXbEqe23Ncmst1TGoOGdjo0FtIK3439cDpzxiOYCfpGz+DbTWGqSvNyE2Pbr6ve3awZ2K5poiYa93AcLA3FOJ5HnoVUYhXisrvaRCvft6MR4e1VkPGXPN3Jo3ZmZDPvfuY7vkNyzeslaV1N91J5VG77o3DB0UbwjGa8Imp2w1f12EVte76QOjOpKJe7sTeen6dLWP6GmZjIt2wR4yQ04mKUpn3ZDuazXruRDk7zzt2ctTThsJRs75DHA2t4VeUElykw1lzz5HO+a9S0+eWwjw+BoEZ5FpUdrb8y55lGoo7QOd7EiYitYqzpPFk6i7kSqsI8iFfb2V7enPrHW2MyP55maBDiPIu729vlvqov8nlmSJuhcbvVxWlzvLvm8nAtijD90kn8MSNrM0VVXwMLnhJXbwodeGaYjU5XsEzU6qG20UfdhtA1rFbPMPvNXt9qdPpNXeg1jb5pd3WEeL3VbyPT7vF2w+Z1of30JxuLYdPG9E8EMEXDRK1bP9hw/OOLRlhcYC1KqoOXf7Ft5gkmbtgcwQz+tUGt8lMHfjGrWR4WRTrMv1Sr3Mc4TwpYIIP/YYvQPXlfAxLdefxHD7SQ6r+uIBTE/vm/KBGmopsdAAA=