chapter-evaluator

name	chapter-evaluator
description	Evaluate educational chapters from dual student and teacher perspectives. This skill should be used when analyzing chapter quality, identifying content gaps, or planning chapter improvements. Reads all lessons in a chapter directory and provides structured analysis with ratings, gap identification, and prioritized recommendations.

Chapter Evaluator Skill

Evaluate educational chapters through dual lenses: the Student Experience (engagement, clarity, confidence) and the Teacher Perspective (pedagogy, objectives, assessment). Output structured analysis with ratings, gaps, and actionable improvements.

When to Use

• Analyzing a chapter's overall quality before publication
• Identifying why content "feels off" (too short, boring, disconnected)
• Planning improvements to existing chapters
• Comparing chapters against quality standards
• User asks to "evaluate", "review", "analyze", or "assess" a chapter

Evaluation Process

Step 1: Gather Chapter Content

Read all lesson files in the chapter directory:

ls -la <chapter-path>/*.md | grep -v summary | grep -v README | grep -v quiz

For each lesson file, extract:

• YAML frontmatter (learning objectives, cognitive load, skills, layer)
• Word count
• Section structure (headings)
• Try With AI prompts
• Hands-on exercises
• Code examples

Step 2: Student Perspective Analysis

Evaluate as a beginner encountering this content for the first time.

2.1 Engagement Score (1-10)

Score	Criteria
9-10	Compelling hook, real-world relevance clear, I want to keep reading
7-8	Interesting enough, some engaging moments, minor dry spots
5-6	Functional but forgettable, reads like documentation
3-4	Boring, walls of text, no compelling reason to continue
1-2	Would abandon after first section

Check for:

• Opening hook (does first paragraph grab attention?)
• Real-world scenarios (why does this matter to ME?)
• Story/narrative flow vs disconnected facts
• Visual breaks (diagrams, tables, code blocks)
• Pacing variety (concept → hands-on → concept)

2.2 Length Assessment

Verdict	Criteria
Too Short	Missing examples, concepts unexplained, abrupt endings, "I don't understand"
Just Right	Each concept has sufficient depth, examples clarify, natural flow
Too Long	Repetitive explanations, over-elaborated points, could cut 30%+

Word count benchmarks:

• Conceptual lesson: 1,000-1,400 words
• Hands-on lesson: 1,200-1,600 words
• Installation/setup: 800-1,200 words (focused)
• Capstone: 1,400-1,800 words

2.3 Clarity Score (1-10)

Score	Criteria
9-10	Crystal clear, no re-reading needed, "aha" moments
7-8	Mostly clear, occasional re-read for complex parts
5-6	Understandable with effort, some confusing sections
3-4	Frequently confused, missing context, jargon unexplained
1-2	Cannot follow, assumes knowledge I don't have

Check for:

• Jargon introduced before defined
• Logical flow between paragraphs
• Transitions between sections
• Prerequisites assumed vs stated

2.4 Hands-On Effectiveness (1-10)

Score	Criteria
9-10	Clear steps, achievable, builds confidence, "I did it!"
7-8	Mostly clear, minor ambiguity, successful completion likely
5-6	Workable but confusing steps, may need to troubleshoot
3-4	Missing steps, unclear what to do, likely to get stuck
1-2	Cannot complete without external help

Check for:

• Step-by-step instructions (numbered, clear)
• Expected output/results shown
• Troubleshooting guidance
• Connection to concepts just learned

2.5 Progression Clarity (1-10)

Score	Criteria
9-10	Clear path from start to mastery, each lesson builds on previous
7-8	Generally progressive, minor jumps between lessons
5-6	Some logical progression, noticeable gaps
3-4	Disconnected lessons, unclear how they relate
1-2	Random ordering, no clear learning path

Check for:

• Opening connections ("In Lesson N-1, you learned X. Now...")
• Running example threaded through chapter
• Skills building on each other
• Clear "what's next" at lesson end

2.6 Confidence Score (1-10)

Score	Criteria
9-10	"I can definitely do this now" - ready to apply independently
7-8	"I mostly understand and could figure out the rest"
5-6	"I kind of get it but would need help applying it"
3-4	"I'm confused about when/how to use this"
1-2	"I have no idea what I just read"

Check for:

• Practice opportunities before moving on
• Verification steps ("you should see X")
• Real-world application examples
• "Try it yourself" prompts

Step 3: Teacher Perspective Analysis

Evaluate as an instructional designer assessing pedagogical soundness.

3.1 Learning Objectives Quality (1-10)

Score	Criteria
9-10	SMART objectives, measurable, aligned to content and assessment
7-8	Clear objectives, mostly measurable, good alignment
5-6	Objectives present but vague or partially aligned
3-4	Weak objectives, not measurable, poor alignment
1-2	Missing or meaningless objectives

Check for:

• Bloom's taxonomy verb alignment (Remember → Create)
• Measurable criteria ("can explain", "can create", "can distinguish")
• Assessment method specified
• Objectives actually taught in lesson content

3.2 Cognitive Load Management (1-10)

Score	Criteria
9-10	Appropriate concepts for level, well-scaffolded, no overload
7-8	Generally appropriate, minor overload moments
5-6	Some cognitive overload, too many concepts at once
3-4	Significant overload, concepts piled without consolidation
1-2	Overwhelming, no chance of retention

Benchmarks by proficiency:

• A1-A2: 3-5 new concepts per lesson
• B1-B2: 5-7 new concepts per lesson
• C1-C2: 7-10 new concepts per lesson

Check for:

• New concepts counted in frontmatter
• Concepts introduced one at a time
• Practice before new concept introduced
• Chunking of complex procedures

3.3 Scaffolding Quality (1-10)

Score	Criteria
9-10	Perfect progression, each concept builds on previous, no gaps
7-8	Good scaffolding, minor jumps that students can bridge
5-6	Some scaffolding gaps, requires prior knowledge not taught
3-4	Significant gaps, assumes knowledge not in prerequisites
1-2	No scaffolding, concepts appear randomly

Check for:

• Prerequisites listed and actually prerequisite
• Concepts introduced before used
• Increasing complexity curve
• Prior knowledge activated before new content

3.4 Pedagogical Layer Appropriateness (1-10)

Layer	Expected Characteristics
L1 (Foundation)	Manual-first, understand before automate, no AI shortcuts
L2 (Collaboration)	AI as Teacher/Student/Co-Worker, learning through interaction
L3 (Intelligence)	Pattern recognition, creating reusable intelligence (skills/subagents)
L4 (Orchestration)	Capstone, combining components, spec-driven development

Check for:

• Layer declared in frontmatter
• Content matches layer expectations
• Layer progression through chapter (L1 → L2 → L3 → L4)
• No premature automation (L3 content in early lessons)

3.5 Try With AI Effectiveness (1-10)

Score	Criteria
9-10	Prompts directly extend lesson, specific, build skills
7-8	Good prompts, mostly connected to content
5-6	Generic prompts, loosely connected
3-4	Copy-paste prompts, don't match lesson
1-2	Missing or irrelevant prompts

Check for:

• 2-3 prompts per lesson (not 1, not 5+)
• Prompts reference lesson content specifically
• Progressive difficulty across prompts
• "What you're learning" explanations present

3.6 Assessment/Verification Quality (1-10)

Score	Criteria
9-10	Clear verification at each step, students know if they succeeded
7-8	Good verification for most exercises
5-6	Some verification, students may be unsure of success
3-4	Weak verification, students can't tell if they're on track
1-2	No verification, students have no idea if they succeeded

Check for:

• "Expected output" shown for commands
• "You should see X" confirmations
• Error states explained
• End-of-lesson checkpoint

Step 4: Gap Analysis

After scoring, identify specific missing elements:

Content Gaps

• Missing examples (concept taught but not demonstrated)
• Missing hands-on (theory without practice)
• Missing "why" (what but not why it matters)
• Missing troubleshooting (happy path only)
• Missing transitions (lessons don't connect)

Structural Gaps

• Missing opening hook
• Missing running example continuity
• Missing "What's Next" closure
• Missing visual elements (all text, no diagrams/tables)
• Missing code examples for technical content

Pedagogical Gaps

• Objectives not assessed
• Cognitive overload unaddressed
• Layer mismatch (content doesn't match declared layer)
• Prerequisites not actually prerequisite
• Try With AI prompts disconnected from content

Step 5: Generate Improvement Recommendations

For each gap, provide:

1. Problem: What's missing or wrong
2. Impact: How it affects learning (high/medium/low)
3. Fix: Specific action to address
4. Effort: Estimated work (low: <30min, medium: 30-90min, high: >90min)
5. Priority: 1 (critical), 2 (important), 3 (nice-to-have)

Output Format

Generate analysis in this structure:

# Chapter Evaluation: [Chapter Name]

## Executive Summary

[1 paragraph: Overall quality assessment, key strengths, critical issues, recommendation]

## Student Analysis

### Scores

| Dimension | Score | Verdict |
|-----------|-------|---------|
| Engagement | X/10 | [One-line summary] |
| Length | [Short/Right/Long] | [One-line summary] |
| Clarity | X/10 | [One-line summary] |
| Hands-On | X/10 | [One-line summary] |
| Progression | X/10 | [One-line summary] |
| Confidence | X/10 | [One-line summary] |

**Overall Student Experience**: X/10

### Detailed Findings

[Specific observations per dimension with examples from content]

### Student Pain Points

1. [Specific issue from student perspective]
2. [Specific issue from student perspective]
...

## Teacher Analysis

### Scores

| Dimension | Score | Verdict |
|-----------|-------|---------|
| Learning Objectives | X/10 | [One-line summary] |
| Cognitive Load | X/10 | [One-line summary] |
| Scaffolding | X/10 | [One-line summary] |
| Layer Appropriateness | X/10 | [One-line summary] |
| Try With AI | X/10 | [One-line summary] |
| Assessment | X/10 | [One-line summary] |

**Overall Pedagogical Quality**: X/10

### Detailed Findings

[Specific observations per dimension with examples from content]

### Pedagogical Concerns

1. [Specific issue from teacher perspective]
2. [Specific issue from teacher perspective]
...

## Gap Analysis

### Content Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

### Structural Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

### Pedagogical Gaps

| Gap | Lesson(s) | Impact |
|-----|-----------|--------|
| [Missing element] | L0X | High/Med/Low |
...

## Improvement Recommendations

### Priority 1 (Critical)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...

### Priority 2 (Important)

| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...



| # | Problem | Fix | Effort | Lesson(s) |
|---|---------|-----|--------|-----------|
| 1 | [Issue] | [Action] | Low/Med/High | L0X |
...



| Metric | Value |
|--------|-------|
| Total Lessons | X |
| Average Word Count | X |
| Student Score | X/10 |
| Teacher Score | X/10 |
| Overall Score | X/10 |
| Critical Issues | X |
| Estimated Fix Time | X hours |

Quality Reference

Compare evaluated chapters against high-quality reference lessons. The skill should automatically identify and read a reference lesson from Part 1 or Part 6 for comparison when available.

Reference lesson patterns to look for:

• 01-agent-factory-paradigm/01-digital-fte-revolution.md
• 33-introduction-to-ai-agents/01-what-is-an-ai-agent.md

Resources

references/

See references/ for detailed rubrics:

• student-rubric.md - Detailed student perspective evaluation criteria
• teacher-rubric.md - Detailed teacher perspective evaluation criteria
• word-count-benchmarks.md - Word count guidelines by lesson type