Menü
Run the full evaluation pipeline on the dataset using the code from Rounds 2-3. Collect results, analyze findings, and present the audit report. Use after metric-design is confirmed.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Basierend auf der SOC-Berufsklassifikation
Generate data_types, loader, and executor code for a new dataset based on Round 1 inspection results. Use after the user confirms the modality match and field mapping from data-inspect.
Inspect a user-provided dataset, detect its modality and downstream task, find similar open-source datasets, and propose field mappings to the evaluation framework. Use when the user gives a dataset path and wants to start the agentic data audit workflow.
Design and implement evaluation metrics for a downstream task, based on the data_types and executor from Round 2. May add or modify executors as needed. Use after pipeline-generate is confirmed.
| name | audit-run |
| description | Run the full evaluation pipeline on the dataset using the code from Rounds 2-3. Collect results, analyze findings, and present the audit report. Use after metric-design is confirmed. |
Critical rules for this round:
Rounds 2-3 gave you: data_types, loader, executor(s), and metrics. Now run everything and produce the audit report.
Before running, confirm:
scripts/run_full_test.py DATASETS config with correct:
executor/__init__.pyrun_full_test.py (including in the --metric all path)Run the modality-level metrics that apply to this dataset. These are in modalities/<modality>/metrics/.
Not all Layer 1 metrics apply to every dataset -- use the analysis from Round 1 Step 3c to decide which ones to run.
Run all task-level metrics using the unified entry point:
# Run all metrics at once
python evaluate.py <modality> <dataset> all
# Or submit as a job
PARTITION=t-cpu-new NUM_GPUS=0 CPUS=16 ./submit_eval.sh <modality> <dataset> all
If some metrics need GPU (e.g., diversity with large embedding models):
PARTITION=TDS NUM_GPUS=1 ./submit_eval.sh <modality> <dataset> diversity
Monitor the job and check for errors in the log files at results/<dataset>/.
Read all result files from results/<dataset>/:
format_check_results.jsonexecutability_results.jsondiversity_results.jsonFor each metric, note:
Summarize findings for the user:
Do NOT just trust the numbers. Re-read the dataset from scratch with the metric results in hand, and cross-validate:
Re-read the dataset: go back and read the actual data entries again -- not from memory of Round 1, but fresh. This time you have the metric results, so read with specific questions: "the format_check says 15% error rate -- do I see format issues when I read random entries?" Read entries from beginning, middle, and end of the file.
Read ALL error samples: for each metric that reports errors, read every error sample in the result JSON. For each one, go back to the raw data entry and verify: is this a real problem, or is the metric/executor wrong?
Read passing samples with suspicion: read at least 20-30 samples that passed all checks. Look specifically for problems that SHOULD have been caught but weren't. Are there obvious quality issues the metrics missed?
Cross-check distributions against data reality: if a metric reports an unusual distribution (e.g., 99% pass rate on format but 40% fail on executability), does that match what you see when browsing the actual data? If not, something is off.
Check for blind spots: compare what you observe in the data now with what the metrics report. Are there quality problems visible in the data that do NOT show up in any metric result? If so, the metrics have a gap.
If the results do not match reality:
Only present the report after you are confident the results reflect the actual state of the data.