Chapter Evaluator

Chapter Evaluator Skill

Evaluate educational chapters through dual lenses: the Student Experience (engagement, clarity, confidence) and the Teacher Perspective (pedagogy, objectives, assessment). Output structured analysis with ratings, gaps, and actionable improvements.

When to Use

• Analyzing a chapter's overall quality before publication
• Identifying why content "feels off" (too short, boring, disconnected)
• Planning improvements to existing chapters
• Comparing chapters against quality standards
• User asks to "evaluate", "review", "analyze", or "assess" a chapter

Evaluation Process

Step 1: Gather Chapter Content

Read all lesson files in the chapter directory:

ls -la <chapter-path>/*.md | grep -v summary | grep -v README | grep -v quiz

For each lesson file, extract:

• YAML frontmatter (learning objectives, cognitive load, skills, layer)
• Word count
• Section structure (headings)
• Try With AI prompts
• Hands-on exercises
• Code examples

Step 2: Student Perspective Analysis

Evaluate as a beginner encountering this content for the first time.

2.1 Engagement Score (1-10)

Check for:

• Opening hook (does first paragraph grab attention?)
• Real-world scenarios (why does this matter to ME?)
• Story/narrative flow vs disconnected facts
• Visual breaks (diagrams, tables, code blocks)
• Pacing variety (concept → hands-on → concept)
• Comparative Value (vs alternatives like VS Code/Copilot)

2.2 Length Assessment

Word count benchmarks:

• Conceptual lesson: 1,000-1,400 words
• Hands-on lesson: 1,200-1,600 words
• Installation/setup: 800-1,200 words (focused)
• Capstone: 1,400-1,800 words

2.3 Clarity Score (1-10)

Check for:

• Jargon introduced before defined
• Logical flow between paragraphs
• Transitions between sections
• Prerequisites assumed vs stated
• Safety Checks: No concatenated commands or risky copy-pastes

2.4 Hands-On Effectiveness (1-10)

Check for:

• Step-by-step instructions (numbered, clear)
• Expected output/results shown
• Troubleshooting guidance
• Connection to concepts just learned

2.5 Progression Clarity (1-10)

Check for:

• Opening connections ("In Lesson N-1, you learned X. Now...")
• Running example threaded through chapter
• Skills building on each other
• Clear "what's next" at lesson end

2.6 Confidence Score (1-10)

Check for:

• Practice opportunities before moving on
• Verification steps ("you should see X")
• Real-world application examples
• "Try it yourself" prompts

Step 3: Teacher Perspective Analysis

Evaluate as an instructional designer assessing pedagogical soundness.

3.1 Learning Objectives Quality (1-10)

Check for:

• Bloom's taxonomy verb alignment (Remember → Create)
• Measurable criteria ("can explain", "can create", "can distinguish")
• Assessment method specified
• Objectives actually taught in lesson content

3.2 Cognitive Load Management (1-10)

Benchmarks by proficiency:

• A1-A2: 3-5 new concepts per lesson
• B1-B2: 5-7 new concepts per lesson
• C1-C2: 7-10 new concepts per lesson

Check for:

• New concepts counted in frontmatter
• Concepts introduced one at a time
• Practice before new concept introduced
• Chunking of complex procedures

3.3 Scaffolding Quality (1-10)

Check for:

• Prerequisites listed and actually prerequisite
• Concepts introduced before used
• Increasing complexity curve
• Prior knowledge activated before new content

3.4 Pedagogical Layer Appropriateness (1-10)

Check for:

• Layer declared in frontmatter
• Content matches layer expectations
• Layer progression through chapter (L1 → L2 → L3 → L4)
• No premature automation (L3 content in early lessons)

3.5 Try With AI Effectiveness (1-10)

Check for:

• 2-3 prompts per lesson (not 1, not 5+)
• Prompts reference lesson content specifically
• Progressive difficulty across prompts
• "What's you're learning" explanations present

3.6 Assessment/Verification Quality (1-10)

Check for:

• "Expected output" shown for commands
• "You should see X" confirmations
• Error states explained
• End-of-lesson checkpoint

Dimension Criticality & Publication Gate

CRITICAL: Not all dimensions are equally important for publication. Use this gate to determine if content is ready.

Gate Dimensions (MUST BE 7+)

These dimensions BLOCK publication if below 7/10. Fix these first.

Important Dimensions (6+)

These should be strong but minor issues are fixable.

Enhancement Dimensions (5+)

These are nice-to-have; publication doesn't require perfection here.

• Progression Clarity (5+)
• Hands-On Effectiveness (5+)
• Confidence (5+)
• Try With AI Effectiveness (5+)

Publication Decision Logic

Use this decision tree AFTER scoring all dimensions:

IF any gate dimension (Clarity, Scaffolding, Layer) < 7:
  → REVISE: Content not ready
  → Fix the failing dimension(s)
  → Re-evaluate

ELSE IF (Engagement < 6) AND (Hands-On < 6):
  → CONDITIONAL PASS: Functional but needs improvement
  → Content is usable; improvements recommended
  → Can publish with revision plan

ELSE IF any important dimension (Objectives, Assessment, Load) < 5:
  → CONDITIONAL PASS: Missing elements but learnable
  → Flag for revision; can publish

ELSE:
  → PASS ✅: Ready for publication
  → All gate dimensions 7+
  → Most important dimensions 6+

Example Decision

Chapter Evaluation Results:

• Clarity: 8 ✅
• Scaffolding: 7 ✅
• Layer Appropriateness: 8 ✅
• Engagement: 5 (below ideal)
• Cognitive Load: 7 ✅
• Learning Objectives: 6 ✅
• Assessment: 7 ✅

Decision: PASS ✅ — All gate dimensions 7+. Engagement is low, but structure is solid. Recommend: Add more compelling examples in next revision.

Step 4: Gap Analysis

After scoring, identify specific missing elements:

Content Gaps

• Missing examples (concept taught but not demonstrated)
• Missing hands-on (theory without practice)
• Missing "why" (what but not why it matters)
• Missing troubleshooting (happy path only)
• Missing transitions (lessons don't connect)

Structural Gaps

• Missing opening hook
• Missing running example continuity
• Missing "What's Next" closure
• Missing visual elements (all text, no diagrams/tables)
• Missing code examples for technical content

Pedagogical Gaps

• Objectives not assessed
• Cognitive overload unaddressed
• Layer mismatch (content doesn't match declared layer)
• Prerequisites not actually prerequisite
• Try With AI prompts disconnected from content

Step 5: Generate Improvement Recommendations

For each gap, provide:

1. Problem: What's missing or wrong
2. Impact: How it affects learning (high/medium/low)
3. Fix: Specific action to address
4. Effort: Estimated work (low: <30min, medium: 30-90min, high: >90min)
5. Priority: 1 (critical), 2 (important), 3 (nice-to-have)

Output Format

Generate analysis in this structure:

# Chapter Evaluation: [Chapter Name]

## Executive Summary

[1 paragraph: Overall quality assessment, key strengths, critical issues, recommendation]

## Student Analysis

### Scores

| Dimension | Score | Verdict |
|-----------|-------|---------|
| Engagement | X/10 | [One-line summary] |
| Length | [Short/Right/Long] | [One-line summary] |
| Clarity | X/10 | [One-line summary] |
| Hands-On | X/10 | [One-line summary] |
| Progression | X/10 | [One-line summary] |
| Confidence | X/10 | [One-line summary] |

**Overall Student Experience**: X/10

### Detailed Findings

[Specific observations per dimension with examples from content]

### Student Pain Points

1. [Specific issue from student perspective]
2. [Specific issue from student perspective]
...

## Teacher Analysis

### Scores

| Dimension | Score | Verdict |
|-----------|-------|---------|
| Learning Objectives | X/10 | [One-line summary] |
| Cognitive Load | X/10 | [One-line summary] |
| Scaffolding | X/10 | [One-line summary] |
| Layer Appropriateness | X/10 | [One-line summary] |
| Try With AI | X/10 | [One-line summary] |
| Assessment | X/10 | [One-line summary] |

**Overall Pedagogical Quality**: X/10

### Detailed Findings

[Specific observations per dimension with examples from content]

### Pedagogical Concerns

1. [Specific issue from teacher perspective]
2. [Specific issue from teacher perspective]
...

## Gap Analysis

### Content Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

### Structural Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

### Pedagogical Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

## Improvement Recommendations

### Priority 1 (Critical)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...

### Priority 2 (Important)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...

### Priority 3 (Nice-to-Have)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...

## Publication Decision

### Gate Status
| Gate | Dimension | Score | Status |
|------|-----------|-------|--------|
| 🚧 BLOCK if <7 | Clarity | X/10 | ✅/❌ |
| 🚧 BLOCK if <7 | Scaffolding | X/10 | ✅/❌ |
| 🚧 BLOCK if <7 | Layer Appropriateness | X/10 | ✅/❌ |

### Publication Verdict
**Status**: [PASS ✅ | CONDITIONAL | REVISE]
**Recommendation**: [Ready for publication | Fix gates first | Needs revision plan]

## Next Steps

If PASS:
- [ ] Ready for publication
- [ ] Note: Optional improvements in Priority 3 section above

If CONDITIONAL:
- [ ] Content is functional
- [ ] Recommended: Address Priority 1 issues in next iteration
- [ ] Can publish now; plan revision cycle

If REVISE:
- [ ] STOP: Fix gate dimensions first
- [ ] [Gate dimension 1]: [Specific action]
- [ ] [Gate dimension 2]: [Specific action]
- [ ] Use remediation skills: [skill-1, skill-2]
- [ ] Re-evaluate after fixes

## Summary Metrics

| Metric | Value |
|--------|-------|
| Total Lessons | X |
| Average Word Count | X |
| Student Score | X/10 |
| Teacher Score | X/10 |
| Overall Score | X/10 |
| Gate Pass? | Yes/No |
| Critical Issues | X |
| Estimated Fix Time | X hours |

Quality Reference

Compare evaluated chapters against high-quality reference lessons. The skill should automatically identify and read a reference lesson from Part 1 or Part 6 for comparison when available.

Reference lesson patterns to look for:

• 01-agent-factory-paradigm/01-digital-fte-revolution.md
• 33-introduction-to-ai-agents/01-what-is-an-ai-agent.md

Resources

references/

See references/ for detailed rubrics:

• student-rubric.md - Detailed student perspective evaluation criteria
• teacher-rubric.md - Detailed teacher perspective evaluation criteria
• word-count-benchmarks.md - Word count guidelines by lesson type