aiwg-training

Generate Datasheet, Model Card, and Data Statement from a dataset manifest

Deterministically rebuild a dataset from its manifest and verify fixity equivalence

Create a versioned training dataset with manifest, fixity, provenance, and archive snapshot

End-to-end training dataset pipeline — acquire sources through publication

Detect training-eval overlap against benchmark sets before dataset publication

Generate SFT training examples from raw sources using Self-Instruct / Evol-Instruct / SQuAD / STaR patterns

Convert canonical training examples to Alpaca format for training frameworks

Convert canonical training examples to ChatML format for training frameworks

Convert canonical training examples to JSONL format for training frameworks

Convert canonical training examples to Parquet format for training frameworks

Convert canonical training examples to ShareGPT format for training frameworks

Generate preference pairs (chosen/rejected) for DPO/KTO/ORPO/SimPO training

Large-scale synthetic training data generation with Model Collapse recursion guard

Acquire a training data source with license validation and delegate ingest to the semantic memory kernel

GRADE quality assessment adapted for individual training examples