LLM-Training-Data-Eval

Generate data_types, loader, and executor code for a new dataset based on Round 1 inspection results. Use after the user confirms the modality match and field mapping from data-inspect.

Run the full evaluation pipeline on the dataset using the code from Rounds 2-3. Collect results, analyze findings, and present the audit report. Use after metric-design is confirmed.

Inspect a user-provided dataset, detect its modality and downstream task, find similar open-source datasets, and propose field mappings to the evaluation framework. Use when the user gives a dataset path and wants to start the agentic data audit workflow.

Design and implement evaluation metrics for a downstream task, based on the data_types and executor from Round 2. May add or modify executors as needed. Use after pipeline-generate is confirmed.