القائمة
Build a governed, representative AI evaluation set with provenance, expected behavior, edge cases, affected groups, adjudication, versioning, and limits.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
| name | aipom-golden-dataset-builder |
| description | Build a governed, representative AI evaluation set with provenance, expected behavior, edge cases, affected groups, adjudication, versioning, and limits. |
| type | component |
| category | evaluation-and-evidence |
| phase | 3 |
| status | active |
| operating_level | ["product-team","initiative"] |
| audience | ["Product Manager","Team Lead","Research","Data","Engineering","Domain Experts","Privacy","AI Governance"] |
| best_for | ["Creating repeatable pre-release evaluation","Improving unrepresentative test cases","Governing expected answers and expert disagreement"] |
| evidence_required | ["Behavior contract and real-use evidence","Data readiness and provenance","Failure incidents and affected perspectives","Expert adjudication capacity"] |
| produces | ["Versioned evaluation set","Coverage and provenance record","Adjudication and maintenance rules"] |
| assessment_questions | ["EVAL-02","EVAL-03","EVAL-04","CTX-03"] |
| maturity_move | {"from":"emerging","to":"repeatable"} |
| estimated_time | 90-180 min plus maintenance |
| group_size | 4-10 |
| depends_on | ["aipom-behavior-contract-builder","aipom-data-readiness-audit"] |
| combine_with | ["aipom-eval-scorecard-builder","aipom-evaluation-strategy-advisor","aipom-production-evidence-review"] |
| sources | ["https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence"] |
Build a governed set of representative evaluation cases with inputs, context, expected behavior, acceptable variation, prohibited outcomes, provenance, coverage, adjudication, and lifecycle rules. “Golden” means reviewed reference evidence, not infallible truth.
Convenient happy-path examples create stable scores and false confidence. A useful set represents real use, edge cases, affected groups, adversarial conditions, uncertainty, and consequential failure modes.
Use after defining behavior and data readiness, before repeatable evaluation or launch decisions. Use production evidence to revise the set without contaminating protected holdouts.
Include product, evaluation and data specialists, domain experts, affected-user representation, engineering, and privacy or governance partners proportionate to the data and consequences.
Bring behavior contracts, real-use samples, incidents, edge and adversarial cases, affected-group evidence, source lineage, permissions, evaluation goals, expert guidance, and known disagreements.
Use for retrieval, classification, summarization, recommendation, decision support, agents, safety testing, and workflow evaluation.
Use aipom-eval-scorecard-builder for scoring and decision rules, aipom-evaluation-strategy-advisor for coverage, and aipom-production-evidence-review to add emerging cases responsibly.
Diagnose missing, stale, conflicting, inaccessible, excessive, sensitive, or untrusted context and recommend the next source, package, data, or lifecycle intervention.
Assess whether data is fit for a specific AI product decision across provenance, quality, access, representativeness, consent, privacy, freshness, and operations.
Recommend the product, model, workflow, human, and production evaluations needed for an AI decision, based on behavior, consequences, evidence gaps, and lifecycle stage.
Diagnose AI portfolio imbalance, premature scaling, weak evidence, vendor exposure, and zombie pilots; recommend where to explore, validate, scale, pause, or stop.
Design preventive controls, detection, triage, containment, rollback, communication, investigation, remediation, learning, and reporting for AI incidents.
Turn scattered AI ambition into an evidence-aware product strategy thesis with choices, boundaries, outcomes, assumptions, and next bets. Use when direction or non-goals are unclear.
