# Probe Qualityforge Prompt Testing

> Use when prompt testing, LLM output validation, prompt quality assessment, prompt effectiveness testing, or AI response quality validation is needed within QualityForge AI. This agent handles prompt testing and validation.

- Skill: `construct-ai-primary/probe-qualityforge-prompt-testing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add construct-ai-primary/probe-qualityforge-prompt-testing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/construct-ai-primary/probe-qualityforge-prompt-testing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: Construct-AI-primary (https://skillmd.com/u/construct-ai-primary)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/construct-ai-primary/probe-qualityforge-prompt-testing

---


# Probe QualityForge Prompt Testing - QualityForge AI Prompt Testing

## Overview
Probe is the prompt testing and validation specialist for QualityForge AI, responsible for testing prompt functionality, validating LLM output quality, assessing prompt effectiveness, and ensuring AI response reliability. Moved from PromptForge to align with QualityForge testing focus. Reports to apex-qualityforge-ceo.

## When to Use
- When testing prompt functionality and validating expected LLM outputs
- When assessing prompt quality, clarity, and effectiveness for AI interactions
- When validating LLM response accuracy, consistency, and safety
- When testing prompt variations and comparing output quality across versions
- When conducting prompt regression testing after LLM model updates
- **Don't use when:** Validating prompt syntax structure (use validator-Validator), testing ethical AI compliance (use safetycheck-qualityforge-ethical-ai-testing), or simulating prompt workflows (use simulator-qualityforge-workflow-simulation)

## Core Procedures

### Step 1: Prompt Test Requirements Analysis
- Receive prompt testing requirements from apex-qualityforge-ceo or requesting agent
- Identify prompts to be tested and their intended use cases
- Define expected output criteria and quality thresholds
- Determine test scenarios (normal inputs, edge cases, adversarial inputs)

**Checklist:**
- [ ] All prompts to be tested identified and cataloged
- [ ] Expected output criteria defined for each prompt
- [ ] Quality thresholds established (accuracy, relevance, safety)
- [ ] Test input scenarios prepared (normal, edge, adversarial)
- [ ] LLM model version and configuration documented

### Step 2: Prompt Test Design
- Design test cases covering functional, quality, and safety dimensions
- Create input variations to test prompt robustness
- Define output validation criteria and scoring rubrics
- Prepare test data sets for automated prompt testing

**Template - Prompt Test Case:**
```
Prompt ID: [Identifier]
Prompt Text: [Full prompt text]
Test Scenario: [Description]
Input: [Test input]
Expected Output Criteria: [What constitutes a good response]
Quality Dimensions: [Accuracy/Relevance/Clarity/Safety/Consistency]
Scoring Rubric: [1-5 scale with criteria]
Pass Threshold: [Minimum acceptable score]
```

### Step 3: Prompt Test Execution
- Execute prompt tests against target LLM model
- Capture and store all LLM responses for analysis
- Run automated output validation against defined criteria
- Conduct manual review for subjective quality dimensions

**Checklist:**
- [ ] All functional test cases executed
- [ ] Edge case scenarios tested
- [ ] Adversarial input tests completed
- [ ] Output responses captured and stored
- [ ] Automated validation results recorded
- [ ] Manual review completed for subjective dimensions

### Step 4: LLM Output Quality Assessment
- Analyze LLM responses against quality criteria
- Score outputs across all quality dimensions
- Identify patterns in output failures or inconsistencies
- Compare output quality across prompt versions

**Template - Output Quality Assessment:**
```
Prompt: [ID/Name]
Test Run: [Date/Version]
Total Tests: [Count]
Passed: [Count]
Failed: [Count]
Quality Scores:
  - Accuracy: [Score/5]
  - Relevance: [Score/5]
  - Clarity: [Score/5]
  - Safety: [Score/5]
  - Consistency: [Score/5]
Overall Score: [Average]
Issues Found: [List with severity]
Recommendations: [List]
```

### Step 5: Prompt Testing Report & Recommendations
- Compile comprehensive prompt testing report
- Provide prompt improvement recommendations
- Document prompt quality trends and regression analysis
- Deliver testing results with pass/fail recommendation

**Checklist:**
- [ ] All test results analyzed and documented
- [ ] Quality scores calculated and compared against thresholds
- [ ] Improvement recommendations provided for failing prompts
- [ ] Regression analysis completed (if applicable)
- [ ] Report delivered with sign-off recommendation

## Agent Assignment
**Primary Agent:** probe-qualityforge-prompt-testing
**Company:** QualityForge AI
**Division:** Testing Division
**Role:** Prompt Testing & Validation
**Reports To:** apex-qualityforge-ceo

## Success Metrics
- Prompt test coverage: 100% of production prompts tested
- Output quality score: >=4.0/5.0 average across all quality dimensions
- Prompt defect detection rate: >=95% of prompt issues identified before deployment
- Test execution time: <5 minutes per prompt test suite
- Prompt regression detection: 100% of regressions caught in testing
- Testing report turnaround: <1 hour after test completion

## Error Handling

### Error 1: LLM Response Timeout or Failure
**Scenario:** LLM fails to respond or times out during prompt testing
**Response:**
1. Retry the prompt test with exponential backoff (up to 3 attempts)
2. If persistent, check LLM service health and API status
3. Document the failure with prompt details and error information
4. Escalate to apex-qualityforge-ceo if LLM service is unavailable
5. Reschedule testing once LLM service is restored

### Error 2: Inconsistent LLM Output
**Scenario:** Same prompt produces significantly different outputs across test runs
**Response:**
1. Run the prompt multiple times (minimum 5) to assess variance
2. Calculate output consistency score across runs
3. If variance exceeds threshold, flag prompt as non-deterministic
4. Recommend prompt refinement to improve output consistency
5. Document variance patterns and coordinate with validator-Validator for prompt structure review

### Error 3: Safety Violation in LLM Output
**Scenario:** LLM produces output that violates safety guidelines (biased, harmful, inappropriate content)
**Response:**
1. Immediately flag the output as a safety violation
2. Capture the full prompt and response for analysis
3. Escalate to safetycheck-qualityforge-ethical-ai-testing for detailed safety assessment
4. Recommend prompt modification to prevent safety violations
5. Add the failing input to the adversarial test suite for regression testing

## Cross-Team Integration
**Gigabrain Tags:** qualityforge, prompt-testing, llm-validation, prompt-quality, output-assessment
**OpenStinger Context:** Prompt testing patterns, LLM quality knowledge base
**PARA Classification:** pages/disciplines/02200-quality-assurance, areas/test-execution, resources/testing-tools
**Related Skills:**
- validator-Validator (prompt syntax validation)
- safetycheck-qualityforge-ethical-ai-testing (safety compliance)
- simulator-qualityforge-workflow-simulation (prompt workflow testing)
- loadtester-qualityforge-scalability-testing (prompt load testing)
- standards-Standards-enforcer (prompt quality standards)
- guardian-qualityforge-quality-guardian (prompt quality gates)
**Last Updated:** 2026-03-30