Test Artifact

Test Artifact in Sandbox

Use this workflow to test artifacts from this repository against testing_sandbox/.

Prerequisites

• GitHub Copilot CLI: copilot must be available in your PATH
• Authenticated with GitHub Copilot
• Shell note: these examples are written for macOS/Linux shells. If rg is unavailable, use grep with separate -e patterns instead of regex alternation that begins with -.
• Session note: avoid set -u for Copilot test commands in VS Code terminals; some prompt hooks reference unset variables and can interrupt scripted runs.

Default Model

Use gpt-5-mini for all test executions unless the user specifies a different model. After launching copilot, set the model by running /model gpt-5-mini as the first command in the session. Do not use /model gpt-5 mini (with a space).

Scope

• Artifacts under test may come from agents/, prompts/, skills/, or instructions/.
• Sandbox target is testing_sandbox/ (Next.js app with TypeScript, Tailwind, and docs structure).
• Output reports are written to docs/testingResults/ at the repository root.

Mutation Policy

• You may modify sandbox agent-configuration files:
  • testing_sandbox/AGENTS.md
  • testing_sandbox/CLAUDE.md
  • testing_sandbox/.github/**
  • testing_sandbox/.claude/**
• Treat application/runtime files as read-only by default:
  • testing_sandbox/app/**
  • testing_sandbox/public/**
  • testing_sandbox/package.json
  • testing_sandbox/tsconfig.json
• Only modify application/runtime files when the artifact being tested explicitly requires it.

Workflow

1) Select Artifact and Success Criteria

• Identify the artifact and its intended behavior.
• Define concrete pass/fail checks before running the test.

2) Validate Artifact Format

Before installing, verify the artifact's frontmatter meets the format rules for its type:

• .instructions.md files — must have both description and applyTo fields. If either is missing, stop and report the missing field(s) before proceeding.
• .agent.md files — must have name, description, and model fields.
• SKILL.md files — must have name and description fields; name must exactly match the parent directory name.
• .prompt.md files — must have a description field; model is strongly recommended.

If any required field is missing, fail the test at this step and include the validation error in the feedback report.

3) Prepare Sandbox Install Target

• Install/copy the artifact into sandbox-compatible paths as needed.
• For .instructions.md testing, install into testing_sandbox/.github/instructions/.
• For agent testing, install into testing_sandbox/.github/agents/.
• For skill testing, install into testing_sandbox/.github/skills/<name>/SKILL.md.

4) Execute Artifact Intent

Use the copilot CLI to exercise the artifact. Run from the testing_sandbox/ directory so the sandbox codebase is available as context.

Resolve the repository root first so the workflow is stable even if your current shell directory drifts:

REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT/testing_sandbox"

Before each run, ensure no previous Copilot CLI process is still active:

pkill -f '/opt/homebrew/Caskroom/copilot-cli/.*/copilot' || true
pkill -x copilot || true
ps aux | grep -E '[ /]copilot($| )' | grep -v grep || echo 'no copilot CLI processes'

General invocation pattern:

REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT/testing_sandbox"
copilot

Once inside the Copilot session, set the model first:

/model gpt-5-mini

Then submit the task description:

<describe the task the artifact is meant to handle>

By artifact type:

• Prompts — Paste the prompt content as the input. Compare output against intended behavior.
• Agents — Describe the agent's role and a representative task; pass as the input.
• Instructions / Skills — First install the artifact into the sandbox (Step 3), then invoke a task that the instruction is meant to shape. Check that output reflects the instruction constraints.

Overriding the model:

Use /model <model-name> within the Copilot session before submitting the task.

Execution modes

Use one of the two modes below.

Mode A: Manual interactive (human-operated)

copilot

Inside the Copilot session, run these in order:

/model gpt-5-mini
<artifact test input>
/exit

Mode B: Scripted interactive (automation-friendly, preferred for reproducible reports)

Use this mode when you need a deterministic transcript written to disk while still setting the model as the first in-session command:

REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT/testing_sandbox"

OUT="$REPO_ROOT/docs/testingResults/YYYY-MM-DD-<artifact-name>-cli-output.txt"
INPUT_FILE="$(mktemp)"
TIMEOUT_BIN="$(command -v gtimeout || command -v timeout || true)"

{ printf '/model gpt-5-mini\n'; cat "$REPO_ROOT/prompts/<artifact>.prompt.md"; printf '\n/exit\n'; } > "$INPUT_FILE"

if [ -n "$TIMEOUT_BIN" ]; then
  "$TIMEOUT_BIN" 300 copilot --allow-all-tools --output-format text < "$INPUT_FILE" > "$OUT" 2>&1 || true
else
  copilot --allow-all-tools --output-format text < "$INPUT_FILE" > "$OUT" 2>&1
fi

rm -f "$INPUT_FILE"

After the run, verify transcript completeness before evaluating behavior:

OUT="$REPO_ROOT/docs/testingResults/YYYY-MM-DD-<artifact-name>-cli-output.txt"
wc -c "$OUT"
wc -l "$OUT"
tail -n 40 "$OUT"

ps aux | grep -E '[ /]copilot($| )' | grep -v grep || echo 'no copilot CLI processes'

Use these minimum infrastructure gates:

• transcript has at least 100 bytes
• transcript has at least 5 lines
• no active Copilot CLI process remains after the run

If any gate fails, classify the attempt as infrastructure failure (not artifact behavior), clean up active Copilot CLI processes, and retry up to 3 attempts with backoff (10s, 20s, 30s).

If all 3 Mode B attempts fail infrastructure gates, switch to Mode A (manual interactive), document the fallback in the report, and keep the verdict scoped to infrastructure reliability unless artifact behavior was fully observed.

Capture the session output verbatim for inclusion in the feedback report (Step 6). Include both:

• The transcript file path

• The exact command used to execute the test

• Verify the captured behavior against the success criteria from Step 1.

Stuck session handling

If output appears stalled:

1. Verify whether a copilot process is still active.
2. Check whether the transcript file is still growing.
3. If truly stuck, terminate only Copilot CLI processes (do not kill unrelated processes) and rerun with Mode B.

Example checks:

ps aux | grep -i '[c]opilot'
ls -l "$REPO_ROOT/docs/testingResults/YYYY-MM-DD-<artifact-name>-cli-output.txt"

Example targeted stop:

pkill -f '^/opt/homebrew/Caskroom/copilot-cli/.*/copilot$' || true
pkill -x copilot || true

5) Evaluate Quality

Assess:

• Correctness against intended purpose
• Instruction fidelity (did the agent follow constraints?)
• Practicality and maintainability of resulting changes
• Safety and policy alignment

6) Publish Feedback Report

Create docs/testingResults/YYYY-MM-DD-<artifact-name>.md with:

• Artifact under test
• Date
• Sandbox setup performed
• Test actions executed
• Observed behavior
• Pass/fail verdict
• Recommended improvements

Report Template

# <artifact-name> test results

- Date: YYYY-MM-DD
- Artifact: <path>
- Sandbox target: testing_sandbox/

## Setup
<what was installed or configured>

## Test Actions
<what was executed>

```bash
<exact copilot command used>

Observed Behavior

Verdict

• Pass/Fail: <pass|fail|partial>
• Rationale:

Recommended Improvements

To score this run with quantitative section-by-section feedback, invoke `eval-artifacts` with the transcript path after the report is written:

eval-artifacts: transcript: docs/testingResults/YYYY-MM-DD--cli-output.txt

## Optional Cleanup

- If you modified sandbox files during testing, you may reset sandbox changes with:

```bash
git checkout testing_sandbox/

• Only clean up files changed by the current test run.