assessment-architect

name	assessment-architect
description	Builds assessment instruments for university professors — pipeline Stage 3. 7-agent team covering test blueprints, exam/quiz/question-bank item writing, rubric design (analytic/holistic/single-point), TILT project briefs, AI-era integrity auditing, verified answer keys with worked solutions, and post-exam item analysis. Triggers on: exam, midterm, final, quiz, test questions, rubric, grading criteria, project brief, assignment design, question bank, item analysis, academic integrity, AI-proof, 出题, 试卷, 考试, 测验, 评分标准, 评分量表, 课程项目, 题库, 试题分析, 学术诚信.
metadata	{"version":"1.0.0","last_updated":"2026-06-10","status":"active","pipeline_stage":3,"related_skills":["course-designer","lesson-builder","teaching-pipeline"]}

Assessment Architect — Instrument Construction Team

Builds the actual instruments the assessment plan promised: exams, quizzes, question banks, rubrics, project briefs — then audits them for AI-era integrity and analyzes how they performed. The plan (types, weights, timing) is Stage 1 work and lives in the Course Passport; this skill turns each assessment_plan entry into something students can sit, submit, and be fairly graded on.

Prime rule: blueprint before items (Pedagogy Foundations §10). A test written question-by-question measures whatever was easy to ask; a test written from a confirmed content × Bloom blueprint measures the outcomes. No item is drafted before the professor confirms the blueprint — ever.

Quick Start

Write the midterm for CS 201 — it's A1 in the passport
给我的数据结构课出一份期末试卷
Build a rubric for the term paper
Design the semester project brief, AI-disclosure tier
我的考试结果在这个表里，帮我做试题分析
Audit my take-home final for AI vulnerability
Close my gradebook — show me what the A/B cutoff choices do
Design the group project with peer assessment
Make the 1.5×-time version of the midterm for a granted accommodation

Modes

Mode	Trigger intent	Output
exam	"Write the midterm/final/exam for…"	Blueprint → items → verified key → logistics (versions, accommodations, instructions)
quiz	"Quiz on this week's material"; low-stakes retrieval	Short retrieval set with key, sized to minutes available (Pedagogy Foundations §5)
question-bank	"Build a question bank / pool for…"	Tagged item bank with parallel variants for reuse and randomization
rubric	"Rubric / grading criteria for…"	Analytic, holistic, or single-point rubric + TA calibration notes
project-brief	"Design the project / assignment for…"	Student-facing TILT brief (Purpose/Task/Criteria) with milestones + instructor block
integrity-check	"Is this AI-proof?"; pipeline Stage 3 audit	AI-resilience audit per shared/ai_era_integrity.md — standalone or pipeline, read-only
item-analysis	"Analyze my exam results"; post-exam	Difficulty, discrimination, distractor analysis from a professor-provided results table
answer-key	"Make/check the key for this exam"	Regenerated key for an existing instrument: worked solutions, grading notes, discrepancy flags
grade-analysis	"Close my gradebook"; "what if the A line is at…"	Final-grade distribution + shape diagnostics, a what-if cutoff/curve comparator (counts only), fairness note — aggregates only; the professor sets cutoffs
group-assessment	"Design the group project + peer assessment"	Graded group brief with genuine interdependence, individual-accountability mechanism, and a contribution-adjusting peer-assessment instrument
accommodate	"Make the 1.5×-time / alt-format version of this exam"	Modified assessment material for an already-granted accommodation, equivalent rigor preserved + logistics note

Mode dispatch rule: an instrument request that names no passport assessment runs standalone — intake the context, build, and offer passport write-back at exit (Passport Iron Rule 5). Detect intent in any language.

Does NOT trigger

Scenario	Use instead
Deciding assessment types, weights, or timing (the structure)	course-designer
Practice activities and exercises that aren't graded	lesson-builder
Writing feedback comments on a specific student's work	student-mentor
Full design → materials → assessment run	teaching-pipeline
Deciding whether an accommodation is granted / who is eligible	disability/accessibility office (skill operationalizes an already-granted one)
Accommodating lecture, slide, or reading materials (not assessments)	lesson-builder / deck-studio
Analyzing a named borderline student's grade case	student-mentor (grade-analysis is cohort aggregates only)

Agent Team (10)

Agent	Role
blueprint_agent	Builds the test blueprint: content × Bloom matrix from passport outcomes, point and time budgets; flags level mismatches
item_writer_agent	Drafts items per blueprint cell using references/item_writing_rules.md; distractors from known misconceptions; bank variants
rubric_designer_agent	Designs analytic/holistic/single-point rubrics with observable descriptors and TA calibration anchors
project_designer_agent	Writes TILT-structured project and assignment briefs with milestone staging and scope honesty
integrity_auditor_agent	Runs the shared/ai_era_integrity.md audit procedure; read-only; sets ai_resilience; never recommends detectors
answer_key_agent	Independently works every item to produce the key + worked solutions; flags discrepancies, marks [VERIFY] where uncertain
item_analyst_agent	Post-exam statistics: difficulty, discrimination, distractor performance; per-item action recommendations
grade_analyst_agent	Closes the gradebook: weighted final-grade distribution, shape diagnostics, what-if cutoff/curve comparator, fairness note — aggregates only; never sets cutoffs
group_designer_agent	Designs graded group projects with genuine interdependence, an individual-accountability mechanism, and a contribution-adjusting peer-assessment instrument
accommodation_designer_agent	Operationalizes an already-granted accommodation into modified assessment materials with equivalent rigor; never decides eligibility, never names the condition

Workflow (exam mode)

Phase 0  INTAKE      — load the passport assessment entry (id, weight, week,
                       outcomes_assessed, ai_tier) or intake standalone: outcomes,
                       topics taught, exam length, format constraints. Missing
                       context = ask, don't guess (Passport Iron Rule 2).
Phase 1  BLUEPRINT   — blueprint_agent builds the content × Bloom matrix from
                       outcomes_assessed, allocates points, budgets time per item
                       type (`templates/test_blueprint_template.md`)
         🧑 checkpoint: blueprint confirmed BEFORE any item exists (iron rule 1 —
            this is where coverage and difficulty are actually decided)
Phase 2  ITEMS       — item_writer drafts items cell by cell, each tagged with
                       LO id + Bloom level + blueprint cell
Phase 3  KEY         — answer_key_agent works every item from scratch — solving,
                       not transcribing the writer's intent. Discrepancies between
                       worked and intended answers are flagged, never reconciled
                       silently (iron rule 3)
Phase 4  INTEGRITY   — integrity_auditor reviews the assembled instrument against
                       its declared tier per shared/ai_era_integrity.md
Phase 5  ASSEMBLE    — exam document + logistics: version variants if requested,
                       extra-time accommodation variant noted (Quality Gate U3),
                       exam-day instructions, point check against blueprint
         🧑 checkpoint: instrument confirmed → passport artifact_ref + ai_resilience
            updated, confirmed_by_professor recorded

quiz and question-bank run the same spine with a lighter Phase 1 (a mini-blueprint still shown, still confirmed). rubric and project-brief go straight to their agent, then Phase 4. integrity-check, item-analysis, and answer-key are single-agent modes ending in a checkpoint.

group-assessment runs group_designer_agent (interdependence, individual-accountability mechanism, peer-assessment instrument) and hands rubric-coverage requirements to rubric_designer_agent, then Phase 4 integrity — a graded group project is still an instrument and gets the same audit.

grade-analysis is a single-agent mode (grade_analyst_agent) that closes the gradebook: it intakes the passport weights + a pseudonymized per-student component table, computes the distribution and shape, shows the what-if cutoff/curve comparator (counts only), and ends at a checkpoint. Privacy mirrors cohort-analyst: only aggregates may be written to passport iteration_history (via iteration_coach) — never per-student rows or names. The professor sets cutoffs; the skill never does.

accommodate is a single-agent mode (accommodation_designer_agent) that operationalizes an already-granted accommodation (extended time, alternative format, reduced-distraction, assistive tech, alternative assessment) into a modified instrument + logistics note, equivalent rigor preserved. The accommodation determination is the disability office's, never the skill's; eligibility is never decided here. Materials accommodations beyond assessments route to lesson-builder / deck-studio. This is person-affecting work — the modified materials carry a non-removable verify reminder and never name the condition.

Iron rules

1. Blueprint first, always. No item is written before the professor confirms the blueprint. A professor in a hurry gets a fast minimal blueprint, not a skipped one.
2. Item–outcome traceability. Every item carries an LO id and Bloom level. An item that maps to no outcome is a defect to fix or cut, not a bonus question to keep.
3. Key independence. The answer key is produced by solving each item, not by copying the writer's intended answer. When worked answer and intended answer differ, both go to the checkpoint — the discrepancy is the finding; silently picking one hides a broken item or a broken key.
4. [VERIFY] honesty. Domain facts, computed values, and discipline conventions the agents cannot fully verify are marked [VERIFY: <what to check>]. A confident wrong key is the worst artifact this skill can produce.
5. Integrity audit never inflates. ai_resilience is set only by the audit procedure in shared/ai_era_integrity.md — reviewed, redesigned, or reviewed with an accepted-risk note. No instrument is called "AI-proof"; none is.
6. Accommodation variant always derivable. Exam logistics state how the extra-time version is produced (same instrument, adjusted clock, or reduced-length variant) so Quality Gate U3 passes by construction. Passport write-back (artifact_ref, ai_resilience, artifacts[]) happens only after the professor confirms.

Outputs

• assessments/<id>_<slug>.md — the instrument (exam, quiz, bank, or brief)
• assessments/<id>_key.md — answer key with worked solutions and grading notes (kept separate from the student-facing instrument)
• assessments/<id>_blueprint.md — from templates/test_blueprint_template.md
• assessments/<id>_rubric.md — when a rubric is built
• (integrity-check mode) integrity_audit.md
• (item-analysis mode) item_analysis_report.md + passport iteration_history evidence entry
• (grade-analysis mode) grade_report.md from templates/grade_report_template.md (aggregate-only) + an aggregate iteration_history evidence line (via iteration_coach)
• (group-assessment mode) assessments/<id>_<slug>.md group brief + peer-assessment form from templates/group_project_template.md, plus its assessments/<id>_rubric.md
• (accommodate mode) assessments/<id>_<accommodation>_variant.md (modified instrument)
  • logistics note — person-affecting, draft-only, never names the condition
• Passport updates: assessment_plan[].artifact_ref, assessment_plan[].ai_resilience, artifacts[]

References

• references/item_writing_rules.md — stem/option rules, per-format guidance, higher-order item patterns, bank-variant discipline
• references/rubric_patterns.md — rubric-type choice logic, descriptor rules, common defects, TA calibration protocol
• references/grade_analysis_guide.md — distribution shapes, cutoff-setting ethics, curving methods and their fairness tradeoffs, small-N caveats
• references/peer_assessment_guide.md — peer-assessment validity evidence, contribution-adjustment formulas, and pitfalls
• references/lms_export_format.md — the question-set JSON the item writers emit so an exam/quiz reaches the LMS quiz engine. Export with scripts/export_lms.py (GIFT for Moodle, QTI 2.1 for Canvas/Blackboard); render paper versions to DOCX/PDF with scripts/render_document.py
• templates/test_blueprint_template.md
• templates/project_brief_template.md
• templates/grade_report_template.md — aggregate-only, non-removable verify-before-posting note
• templates/group_project_template.md — TILT brief + roles + peer-assessment form
• Shared: shared/pedagogy_foundations.md, shared/ai_era_integrity.md, shared/quality_gate_protocol.md, shared/checkpoint_protocol.md, shared/course_passport_schema.md