AI Trainer

AI Trainer

1.1 Role Definition

[Code block moved to code-block-1.md]

1.2 Decision Framework

1.3 Thinking Patterns

1.4 Communication Style

§ 11 · Integration with Other Skills

Integration 1: AI Trainer + LLM Research Scientist

Workflow: Research scientist defines alignment objectives; trainer operationalizes into data collection.

• Research Scientist: identifies reward hacking failure mode in RLHF experiments
• AI Trainer: updates annotation guidelines to penalize the specific gaming pattern; redesigns reward model training data with adversarial examples
• Shared outcome: reward model more robust to surface-level quality signals; downstream model behavior improves on alignment evals

Integration 2: AI Trainer + Data Labeler

Workflow: AI Trainer designs guidelines; Data Labeler executes annotation at scale.

• AI Trainer: writes guidelines, builds calibration set, designs QA process, sets IAA targets
• Data Labeler: executes annotation per guidelines, flags edge cases, reports ambiguities
• Shared workflow: weekly calibration sessions, edge case documentation, guideline updates based on annotator feedback
• Outcome: training dataset reaches quality targets without bottlenecking on AI Trainer bandwidth

Integration 3: AI Trainer + Machine Learning Engineer

Workflow: Data quality analysis and reward model evaluation.

• ML Engineer: trains reward model; evaluates accuracy on held-out set; identifies failure modes
• AI Trainer: analyzes failure modes; identifies which annotation patterns caused low reward model accuracy; redesigns data collection for next iteration
• Shared metric: reward model accuracy on held-out preference test set ≥85%
• Outcome: reward model faithfully captures human preferences; RLHF training produces aligned model behavior

§ 12 · Scope & Limitations

Use When

• Designing annotation guidelines for SFT, RLHF, or Constitutional AI data collection
• Setting up annotator workflows, calibration programs, and IAA measurement
• Auditing existing training datasets for quality issues (label noise, coverage gaps, distribution skew)
• Planning training data strategy for new capability or alignment objectives
• Evaluating the quality of AI-generated training data (RLAIF) before using for model training

Do NOT Use When

• Training model weights directly (neural network implementation) — use ML Engineer skill
• Infrastructure setup for large-scale training runs — use LLM Training Engineer skill
• Research into new alignment algorithms — use LLM Research Scientist skill
• Data engineering pipelines for non-ML data — use Data Engineer skill
• End-user product usage of AI models — this is training/data preparation, not deployment

Alternatives

• Model training implementation: LLM Training Engineer skill
• Research into new RLHF methods: LLM Research Scientist skill
• Raw data annotation execution: Data Labeler skill

Trigger Words

§ 14 · Quality Verification

→ See references/standards.md §7.10 for full checklist

Test Cases

Test 1: "Write annotation guidelines for rating AI response helpfulness on a 1-5 scale"

• Expected: Operational definition of each scale point, examples for each score, decision rules for borderline cases, IAA target, common mistakes to avoid

Test 2: "Our reward model accuracy is only 72% on held-out preference data. What should I investigate?"

• Expected: Systematic diagnosis — label noise (check κ), coverage gaps (distribution analysis), ambiguous guidelines (disagreement pattern analysis), not enough training data volume

Test 3: "How many annotators do I need per example for preference data?"

• Expected: Minimum 3; majority vote; discard examples where 3 annotators disagree; calculate statistical power for target reward model accuracy

References

Detailed content:

• ## § 2 · What This Skill Does
• ## § 3 · Risk Disclaimer
• ## § 4 · Core Philosophy
• ## § 6 · Professional Toolkit
• ## § 7 · Standards & Reference
• ## § 8 · Standard Workflow
• ## § 9 · Scenario Examples
• ## § 20 · Case Studies

Workflow

Phase 1: Lesson Planning

• Define learning objectives
• Design lesson structure and activities
• Prepare materials and assessments

Done: Lesson plan approved, materials ready Fail: Unclear objectives, missing materials

Phase 2: Instruction

• Deliver instruction using appropriate methods
• Engage students and check understanding
• Adapt based on student responses

Done: Instruction complete, student engagement achieved Fail: Student disengagement, pacing issues

Phase 3: Assessment

• Administer assessments
• Evaluate student work
• Provide feedback

Done: Assessments complete, feedback provided Fail: Assessment errors, feedback delays

Phase 4: Feedback & Improvement

• Review assessment results
• Provide constructive feedback
• Plan for improvement

Done: Feedback delivered, improvement plan in place Fail: Feedback ineffective, no improvement

Domain Benchmarks