test-data-generation-validation

name	Test Data Generation & Validation
description	Generate real Cassandra 5.0 test data using Docker containers, export SSTables with proper directory structure, validate parsing against sstabledump, and manage test datasets. Use when working with test data generation, dataset creation, SSTable export, validation, fixture management, or sstabledump comparison.

Test Data Generation & Validation

This skill provides guidance on generating real Cassandra 5.0 test data and validating parsing correctness.

When to Use This Skill

• Generating test data with specific schemas
• Creating test fixtures for property tests
• Exporting SSTables from Cassandra
• Validating parsed data against sstabledump
• Managing test datasets
• Creating reproducible test scenarios

Overview

CQLite uses real Cassandra 5.0 instances to generate test data, ensuring:

• Format correctness (real Cassandra writes)
• Edge case coverage (nulls, empty values, large values)
• Compression validation (actual compressed SSTables)
• Schema variety (all CQL types)

Test Data Workflow

See dataset-generation.md for complete workflow details.

Quick Start

cd test-data

# 1. Start clean Cassandra 5 with schemas
./scripts/start-clean.sh

# 2. Generate data (N rows per table)
ROWS=1000 ./scripts/generate.sh

# 3. Export SSTables
./scripts/export.sh

# 4. Shutdown and clean volumes
./scripts/shutdown-clean.sh

Generation Scripts

start-clean.sh

Starts Cassandra 5.0 container and applies schemas.

What it does:

1. Starts cassandra-5-0 container via docker-compose
2. Waits for Cassandra to be healthy
3. Applies schemas from schemas/core.list
4. Verifies keyspaces and tables created

Environment variables:

• SCHEMA_SET=core - Use curated schema list (default)
• SCHEMA_SET=all - Use all *.cql files

Example:

# Use default core schemas
./scripts/start-clean.sh

# Use all schemas
SCHEMA_SET=all ./scripts/start-clean.sh

generate.sh

Generates test data using Python data generator.

What it does:

1. Connects to running Cassandra container
2. Generates type-correct data for each table
3. Inserts rows using prepared statements
4. Flushes memtables to SSTables
5. Produces metadata.yml with row counts

Environment variables:

• ROWS=N - Rows per table (default: varies by SCALE)
• TABLES=table1,table2 - Generate for specific tables only
• SCALE=SMALL|MEDIUM|LARGE - Preset sizes

Example:

# Generate 1000 rows per table
ROWS=1000 ./scripts/generate.sh

# Generate only for specific tables
TABLES=simple_table,collection_table ROWS=500 ./scripts/generate.sh

# Use LARGE scale preset
SCALE=LARGE ./scripts/generate.sh

export.sh

Exports SSTables from Cassandra data directory.

What it does:

1. Stops Cassandra to ensure consistent snapshot
2. Copies SSTables from container to datasets/sstables/
3. Preserves directory structure (keyspace/table/files)
4. Copies metadata.yml
5. Creates metadata about dataset

Output structure:

test-data/datasets/
├── metadata.yml          # Generated by generate.sh
├── sstables/
│   ├── test_basic/
│   │   └── simple_table/
│   │       ├── *-Data.db
│   │       ├── *-Index.db
│   │       ├── *-Statistics.db
│   │       ├── *-Summary.db
│   │       └── *-TOC.txt
│   ├── test_collections/
│   └── test_timeseries/

shutdown-clean.sh

Stops Cassandra and removes Docker volumes.

What it does:

1. Stops all containers
2. Removes Docker volumes (clean slate)
3. Prepares for next generation cycle

Use when:

• Done with current dataset
• Want to regenerate from scratch
• Cleaning up after tests

Test Schemas

Schemas in test-data/schemas/:

basic-types.cql

Simple table with all primitive types:

• Partition key: uuid
• No clustering
• Columns: int, text, timestamp, boolean, etc.

collections.cql

Collection types:

• list
• set
• map<text, int>
• Nested frozen collections

time-series.cql

Time-series pattern:

• Partition key: sensor_id
• Clustering: timestamp (DESC)
• Columns: temperature, humidity, pressure

wide-rows.cql

Wide partition testing:

• Single partition key
• Many clustering rows (1000+)
• Tests pagination and offset handling

Custom Schemas

Add your own:

# Create schema
echo "CREATE TABLE test_keyspace.my_table (...);" > schemas/my-schema.cql

# Add to core.list
echo "my-schema.cql" >> schemas/core.list

# Generate
./scripts/start-clean.sh
./scripts/generate.sh

Validation Workflow

See validation-workflow.md for complete validation process.

Validate Against sstabledump

# 1. Generate sstabledump reference
sstabledump test-data/datasets/sstables/keyspace/table/*-Data.db \
    > reference.json

# 2. Parse with cqlite
cargo run --bin cqlite -- \
    --data-dir test-data/datasets/sstables/keyspace/table \
    --schema test-data/schemas/schema.cql \
    --out json > cqlite.json

# 3. Compare (ignoring formatting)
jq -S '.' reference.json > ref-sorted.json
jq -S '.' cqlite.json > cql-sorted.json
diff ref-sorted.json cql-sorted.json

Automated Validation

Run validation script:

# Validate all test tables
cargo test --test sstable_validation

# Validate specific table
cargo test --test sstable_validation -- simple_table

Property Testing

Generate random data for property tests:

use proptest::prelude::*;

proptest! {
    #[test]
    fn test_row_parsing_roundtrip(
        partition_key in any::<i32>(),
        text_value in "\\PC*",  // Any valid unicode
        int_value in any::<i32>(),
    ) {
        // Generate test data in Cassandra
        insert_test_row(partition_key, &text_value, int_value)?;
        flush_memtable()?;
        
        // Parse with cqlite
        let parsed = parse_sstable()?;
        
        // Validate roundtrip
        assert_eq!(parsed.get_int("partition_key"), partition_key);
        assert_eq!(parsed.get_text("text_col"), text_value);
        assert_eq!(parsed.get_int("int_col"), int_value);
    }
}

Dataset Packaging

Package datasets for CI or distribution:

# Package current dataset
./scripts/package_datasets.sh

# Output: test-data/cqlite-test-data-v5.0-<date>.tar.gz

Contents:

• All SSTables
• metadata.yml
• Schema files
• README with generation parameters

CI Integration

Smoke Test

Quick validation in CI:

# Use packaged dataset
tar xzf cqlite-test-data-v5.0.tar.gz

# Run core tests
./scripts/ci-one-shot-smoke.sh

# Validates:
# - Basic parsing
# - All CQL types
# - Compression
# - Collections

See test-data/scripts/CI_SMOKE_TEST_USAGE.md for details.

Common Scenarios

Scenario 1: Test New CQL Type

# 1. Add column to schema
echo "ALTER TABLE test_basic.simple_table ADD duration_col duration;" \
    >> schemas/basic-types.cql

# 2. Regenerate data
./scripts/start-clean.sh
./scripts/generate.sh
./scripts/export.sh

# 3. Validate parsing
cargo test --test sstable_validation

Scenario 2: Test Large Values

# Generate with specific row size
ROWS=100 SCALE=LARGE ./scripts/generate.sh

# Validates:
# - Large text values (1MB+)
# - Large blob values
# - Large collections (1000+ elements)

Scenario 3: Test Edge Cases

# Modify generate_comprehensive_test_data.py
def generate_edge_cases(session):
    # Null values
    session.execute("INSERT INTO table (pk) VALUES (?)", [uuid.uuid4()])
    
    # Empty collections
    session.execute("INSERT INTO table (pk, tags) VALUES (?, [])", 
                   [uuid.uuid4()])
    
    # Empty strings
    session.execute("INSERT INTO table (pk, name) VALUES (?, '')", 
                   [uuid.uuid4()])

PRD Alignment

Supports Milestone M1 (Core Reading Library):

• 95% test coverage goal
• All CQL types validated
• Real Cassandra data ensures format correctness

Supports All Milestones:

• Regression testing with frozen datasets
• Property-based testing for edge cases
• CI integration for PR validation

Troubleshooting

Cassandra Won't Start

# Check logs
docker logs cassandra-5-0

# Common issue: Port 9042 in use
lsof -i :9042
# Kill process or change port in docker-compose-cassandra5.yml

Generation Fails

# Check generator logs
cat test-data/logs/data_generation.log

# Verify schema applied
docker exec cassandra-5-0 cqlsh -e "DESCRIBE KEYSPACES;"

Export Produces No Files

# Verify data exists in container
docker exec cassandra-5-0 ls -la /var/lib/cassandra/data/

# Check if flush happened
docker logs cassandra-5-0 | grep flush

Dataset Repository

Packaged datasets available at:

https://github.com/pmcfadin/cqlite/releases/tag/test-data-v5.0

Download for:

• CI without Docker
• Reproducible benchmarks
• Offline development

Next Steps

When creating new tests:

1. Design schema in schemas/
2. Generate data with generate.sh
3. Export SSTables with export.sh
4. Write parser test
5. Validate with sstabledump
6. Add to CI smoke test suite

See documentation:

• dataset-generation.md - Full workflow
• validation-workflow.md - Validation process