oimiragieo Response Rater Skill
Step 1: Define Rating Rubric
Use appropriate rubric for the content type:
For Plans:
| Dimension | Weight | Description |
|---|---|---|
| Completeness | 20% | All required sections present |
| Feasibility | 20% | Plan is realistic and achievable |
| Risk Mitigation | 20% | Risks identified with mitigations |
| Agent Coverage | 20% | Appropriate agents assigned |
| Integration | 20% | Fits with existing systems |
For Responses:
| Dimension | Weight | Description |
|---|---|---|
| Correctness | 25% | Technically accurate |
| Completeness | 25% | Addresses all requirements |
| Clarity | 25% | Easy to understand |
| Actionability | 25% | Provides clear next steps |
Step 2: Evaluate Each Dimension
Score each dimension 1-10:
## Dimension Scores
### Completeness: 8/10
- Has objectives, steps, and timeline
- Missing risk assessment section
### Feasibility: 7/10
- Most steps are achievable
- Step 3 timeline is aggressive
### Risk Mitigation: 5/10
- Only 1 risk identified
- No mitigation strategies
### Agent Coverage: 9/10
- All steps have assigned agents
- Good agent-task matching
### Integration: 8/10
- Uses existing APIs
- Minor compatibility concerns
Step 3: Calculate Overall Score
Average the weighted scores:
Overall = (8×0.2) + (7×0.2) + (5×0.2) + (9×0.2) + (8×0.2) = 7.4/10
Step 4: Generate Recommendations
Provide actionable improvements:
## Recommendations
### High Priority
1. Add risk assessment section with 3-5 risks
2. Include mitigation strategies for each risk
### Medium Priority
3. Extend Step 3 timeline by 2 days
4. Add fallback plan for external API dependency
### Low Priority
5. Add success metrics for each step
Step 5: Make Pass/Fail Decision
Apply minimum score thresholds:
| Task Type | Minimum Score |
|---|---|
| Standard | 7/10 |
| Enterprise | 8/10 |
| Critical | 9/10 |
## Decision
**Score**: 7.4/10
**Threshold**: 7/10 (Standard)
**Result**: PASS ✅
Plan approved with recommendations for improvement.
- Consistent Rubrics: Use same dimensions for comparable content
- Clear Thresholds: Define pass/fail criteria upfront
- Actionable Feedback: Provide specific improvements
- Prioritize Issues: Order recommendations by impact
- Document Rationale: Explain score justifications
Rate this plan for user authentication implementation
Response:
## Plan Rating: User Authentication
### Scores
- Completeness: 8/10 - Good structure, missing tests section
- Feasibility: 9/10 - Realistic timeline and approach
- Risk Mitigation: 6/10 - Only 1 risk identified
- Agent Coverage: 9/10 - Good agent assignments
- Integration: 8/10 - Compatible with existing auth
### Overall: 8.0/10 ✅ PASS
### Top Recommendations
1. Add risk section with security and dependency risks
2. Include test plan for each authentication flow
3. Add rollback procedure for failed deployment
Iron Laws
- ALWAYS use the same rubric dimensions when rating comparable content — inconsistent dimensions make scores meaningless and prevent valid comparison across sessions.
- NEVER issue a pass/fail decision without documenting score justification for each dimension — unjustified scores cannot be reviewed, challenged, or improved.
- ALWAYS apply defined minimum thresholds (7/10 standard, 8/10 enterprise, 9/10 critical) — ad-hoc thresholds produce inconsistent approval gates that erode trust in the rating system.
- NEVER provide vague recommendations — every recommendation must reference the specific dimension it addresses and state the concrete change required.
- ALWAYS prioritize recommendations by impact — high-priority items that would materially improve the score must be clearly distinguished from low-impact suggestions.
Anti-Patterns
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
| Using different rubric dimensions for comparable content | Scores cannot be compared across sessions; the rating loses its evaluative value | Always use the same rubric (plans rubric for plans, responses rubric for responses) |
| Omitting score justification for individual dimensions | Scores without justification cannot be reviewed, verified, or acted upon | Document specific evidence for each dimension score (what was present, what was missing) |
| Setting thresholds arbitrarily per session | Inconsistent thresholds invalidate the pass/fail gate; teams lose confidence in approvals | Always apply the defined thresholds: 7/10 standard, 8/10 enterprise, 9/10 critical |
| Providing vague recommendations ("improve quality", "add more detail") | Vague feedback cannot be acted upon; no change results from the review | Reference the specific dimension, score gap, and required concrete change for each recommendation |
| Listing recommendations without priority ordering | Equal-weight feedback causes raters to address low-impact items first | Always order by impact: High (affects pass/fail threshold) before Medium before Low |
Memory Protocol (MANDATORY)
Before starting:
cat .claude/context/memory/learnings.md
After completing:
- New pattern ->
.claude/context/memory/learnings.md - Issue found ->
.claude/context/memory/issues.md - Decision made ->
.claude/context/memory/decisions.md
ASSUME INTERRUPTION: Your context may reset. If it's not in memory, it didn't happen.