dataset-prep

name	dataset-prep
description	Utilities for converting raw interaction data to pyKT format and validating dataset integrity. Use when preparing new datasets for Knowledge Tracing experiments.

pyKT Dataset Preparation

Prepare and validate datasets for Knowledge Tracing experiments with pyKT framework.

Quick Start

Check Preprocessing Status

python scripts/preprocess.py --status --data-dir ../data

Preprocess a Dataset

# Single dataset
python scripts/preprocess.py --dataset assist2009 --pykt-path ../pykt-toolkit

# Multiple datasets
python scripts/preprocess.py --dataset assist2009 assist2015 ednet

# All available datasets
python scripts/preprocess.py --all

Validate Dataset

# Validate entire dataset directory
python scripts/validate_dataset.py --dir ../data/assist2009

# Validate specific file
python scripts/validate_dataset.py --file ../data/assist2009/data.txt --type raw
python scripts/validate_dataset.py --file ../data/assist2009/train_valid_sequences.csv --type csv

Scripts

preprocess.py

Wrapper for pyKT's preprocessing pipeline with batch support.

Option	Description
--dataset NAME	Dataset(s) to preprocess
--all	Process all available datasets
--status	Show preprocessing status only
--data-dir PATH	Data directory location
--pykt-path PATH	pykt-toolkit installation path
--min-seq-len N	Minimum sequence length (default: 3)
--maxlen N	Maximum sequence length (default: 200)
--kfold N	Number of CV folds (default: 5)
--list	List supported datasets with download URLs

validate_dataset.py

Validate data format and report statistics.

Option	Description
--file PATH	Single file to validate
--dir PATH	Dataset directory to validate
--type raw/csv	File type (auto-detected)
--json	Output results as JSON

Supported Datasets

Dataset	Type	Description
assist2009	Q+C	ASSISTments 2009-2010 Math
assist2012	Q+C	ASSISTments 2012-2013
assist2015	C	ASSISTments Skill Builder
assist2017	Q+C	ASSISTments Competition
algebra2005	Q+C	KDD Cup Algebra
bridge2algebra2006	Q+C	KDD Cup Bridge to Algebra
statics2011	C	Andes Physics
nips_task34	Q+C	Eedi Education Challenge
ednet	Q+C	TOEIC English (Riiid)
junyi2015	Q+C	Junyi Academy K-12 Math
slepemapy	Q+C	Geography
poj	C	Programming Judge

Type: Q+C = Questions + Concepts, C = Concepts only

References

• Data format details: See references/data_format.md
• Dataset configurations: See references/datasets_config.md

Workflow

1. Download raw data from source (see --list for URLs)
2. Place in pykt-toolkit/data/{dataset_name}/
3. Preprocess with preprocess.py
4. Validate with validate_dataset.py
5. Train models using pyKT