microsoft-foundry Testing CLI Compatibility
Validates that the standalone Copilot CLI works correctly with this repo's skills and MCP servers.
Quick Start
# Run all tests (including MCP)
./deployment/scripts/test-cli-compatibility.ps1
# Skip MCP server tests (faster, no tool approval needed)
./deployment/scripts/test-cli-compatibility.ps1 -SkipMcp
What It Tests
| # | Test | What It Checks |
|---|---|---|
| 1 | CLI Version | copilot command is installed and responds |
| 2 | Custom Instructions | .github/copilot-instructions.md is loaded (identifies project name) |
| 3 | Skills | All .github/skills/*/SKILL.md files are visible (count auto-detected) |
| 4 | MCP Servers | playwright and microsoftdocs servers are connected |
Prerequisites
- Copilot CLI installed:
npm install -g @anthropic-ai/copilot-clior via GitHub - Authenticated: Run
copilot loginif not already logged in - MCP servers synced: Run the
syncing-mcp-serversskill first if MCP tests fail
How It Works
The script uses copilot -p "<prompt>" -s --no-auto-update to invoke the CLI in non-interactive mode:
-p— Single prompt, exits after response-s— Silent mode, output only the agent response (no stats)--no-auto-update— Skip update check for faster execution--allow-all-tools— Used only for MCP test (needs tool access)
Each test sends a prompt asking the CLI to list what it can see, then validates the response against expected values.
When to Run
- After adding or modifying skills in
.github/skills/ - After syncing MCP servers with the
syncing-mcp-serversskill - When a user reports CLI issues (missing agents, skills, or tools)
- As a smoke test after CLI updates (
copilot update)
Manual Testing
If the script isn't available, you can test manually:
# Check version
copilot --version
# Test custom instructions
copilot -p "What project is this repo for?" -s
# Test skills
copilot -p "List all custom skills" -s
# Test MCP (needs --allow-all-tools)
copilot -p "List all MCP servers" -s --allow-all-tools
Troubleshooting
| Issue | Fix |
|---|---|
copilot not found |
Install CLI or add to PATH |
| Custom instructions not loaded | Ensure you're in the repo directory |
| Agents missing | No custom agents expected — this repo uses skills + copilot-instructions only |
| Skills missing | Check .github/skills/*/SKILL.md files exist with valid frontmatter |
| MCP servers not connected | Run syncing-mcp-servers skill to sync .vscode/mcp.json → ~/.copilot/mcp-config.json |