nlp-dataset-analyzer

name	nlp-dataset-analyzer
description	Analyzes NLP datasets before model training or annotation pipeline iteration. Provides insights on data volume, text content distribution, label distribution, and data quality issues. Use when: (1) Starting a new NLP project, (2) Before training a model, (3) After data collection/annotation, (4) Evaluating dataset quality for text classification, NER, QA, or other NLP tasks.
license	Apache 2.0
allowed-tools	["Read","Bash","Grep","Glob"]

NLP Dataset Analyzer

Overview

A specialized skill for analyzing NLP datasets to help you quickly understand data characteristics and identify potential issues before model training or data annotation.

Usage Instructions

When the user provides an NLP dataset path, follow the workflow below to perform comprehensive analysis and generate a standardized Markdown report.

Analysis Workflow (Execute in Order)

Step 1: Data Loading and Initial Inspection

Objective: Understand the basic dataset information

Actions:

1. Use the Read tool to load the data file (supports JSON/JSONL/CSV)
2. If the file is large (>1MB), read only the first 100-1000 rows as a sample
3. Check data format and field structure

Analysis Content:

• File format (JSON/JSONL/CSV/other)
• Total number of samples (if quickly obtainable)
• Field list and preliminary type inference
• Identify possible text fields (common names: text, content, sentence, question, answer, input, etc.)
• Identify possible label fields (common names: label, labels, category, tags, entities, output, etc.)

Example Commands:

# JSONL format quick view
head -n 5 dataset.jsonl

# View JSON structure
head -n 1 dataset.jsonl | python3 -m json.tool

Step 2: Dataset Overview Statistics

Objective: Obtain basic statistical information

Actions: Use Bash tool to run simple commands

JSONL Format:

# Line count
wc -l dataset.jsonl

# File size
ls -lh dataset.jsonl

# JSON structure sampling
head -n 3 dataset.jsonl | jq .

CSV Format:

# Line count (including header)
wc -l dataset.csv

# Column count
head -1 dataset.csv | awk -F',' '{print NF}'

# View first few rows
head -n 5 dataset.csv

Record Content:

• Total sample count
• File size (MB/GB)
• Data format confirmation
• Field confirmation

Step 3: Text Content Deep Analysis

Objective: Understand text features and quality

3.1 Text Length Analysis

Use temporary Python script to analyze text length distribution:

JSONL Format Example:

import json
import statistics

lengths_char = []
lengths_word = []

with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        data = json.loads(line)
        text = data.get('text', '')  # Adjust field name as needed
        lengths_char.append(len(text))
        lengths_word.append(len(text.split()))

print(f"Characters - Min: {min(lengths_char)}, Max: {max(lengths_char)}, Mean: {statistics.mean(lengths_char):.1f}, Median: {statistics.median(lengths_char):.1f}")
print(f"Words - Min: {min(lengths_word)}, Max: {max(lengths_word)}, Mean: {statistics.mean(lengths_word):.1f}, Median: {statistics.median(lengths_word):.1f}")

One-liner Execution:

python3 -c 'import json
import statistics
lengths_char = []
lengths_word = []
with open("dataset.jsonl", "r", encoding="utf-8") as f:
    for line in f:
        data = json.loads(line)
        text = data.get("text", "")
        lengths_char.append(len(text))
        lengths_word.append(len(text.split()))
print(f"Characters - Min: {min(lengths_char)}, Max: {max(lengths_char)}, Mean: {statistics.mean(lengths_char):.1f}, Median: {statistics.median(lengths_char):.1f}")
print(f"Words - Min: {min(lengths_word)}, Max: {max(lengths_word)}, Mean: {statistics.mean(lengths_word):.1f}, Median: {statistics.median(lengths_word):.1f}")'

Report Content:

• Character count statistics: min, max, mean, median
• Word count statistics (based on space tokenization)
• Length distribution description (e.g., "Most samples are between 50-200 words")

3.2 Vocabulary Statistics

Sampling analysis (recommended: first 1000 samples to avoid long processing time on large datasets):

from collections import Counter
import json

words = []
sample_count = 0
max_samples = 1000

with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        if sample_count >= max_samples:
            break
        data = json.loads(line)
        text = data.get('text', '')
        words.extend(text.lower().split())
        sample_count += 1

vocab_size = len(set(words))
word_freq = Counter(words)
top_words = word_freq.most_common(20)
low_freq_words = sum(1 for w, c in word_freq.items() if c <= 2)
low_freq_ratio = low_freq_words / len(word_freq) * 100 if word_freq else 0

print(f"Total vocabulary: {vocab_size}")
print(f"Top 20 frequent words: {[w for w, c in top_words]}")
print(f"Low-frequency word ratio: {low_freq_ratio:.1f}%")

Report Content:

• Total vocabulary size (unique tokens)
• Top 10-20 frequent words
• Low-frequency word ratio (words appearing ≤2 times)

3.3 Text Quality Check

Check common quality issues:

import json

total = 0
empty_count = 0
texts = []

with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        data = json.loads(line)
        text = data.get('text', '')
        texts.append(text)
        total += 1
        if not text.strip():
            empty_count += 1

duplicates = total - len(set(texts))
very_short = sum(1 for t in texts if len(t) < 5)
very_long = sum(1 for t in texts if len(t) > 10000)

print(f"Empty texts: {empty_count} ({empty_count/total*100:.1f}%)")
print(f"Exact duplicates (100%): {duplicates} ({duplicates/total*100:.1f}%)")
print(f"Very short texts (<5 chars): {very_short} ({very_short/total*100:.1f}%)")
print(f"Very long texts (>10000 chars): {very_long} ({very_long/total*100:.1f}%)")

Check Items:

• Number and percentage of empty or whitespace-only texts
• Number and percentage of exact duplicates (100% match)
• Abnormally short samples (<5 characters)
• Abnormally long samples (>10000 characters)

3.4 Advanced Duplication Analysis

Objective: Detect near-duplicates and provide deduplication recommendations

Near-Duplicate Detection (>90% similarity):

For datasets with <10k samples, check for near-duplicates:

import json
from difflib import SequenceMatcher

def text_similarity(text1, text2):
    """Calculate similarity ratio between two texts"""
    return SequenceMatcher(None, text1, text2).ratio()

# Load all samples
samples = []
with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for idx, line in enumerate(f):
        data = json.loads(line)
        samples.append({
            'idx': idx,
            'text': data.get('text', ''),
            'label': data.get('label', ''),
            'full_data': data
        })

# Find exact and near duplicates
exact_dups = {}  # 100% match
near_dups = []   # >90% but <100% match

for i in range(len(samples)):
    for j in range(i+1, len(samples)):
        text_i = samples[i]['text']
        text_j = samples[j]['text']

        if text_i == text_j:
            # Exact duplicate
            key = text_i
            if key not in exact_dups:
                exact_dups[key] = []
            exact_dups[key].append((i, j, samples[i]['label'], samples[j]['label']))
        else:
            # Check similarity
            similarity = text_similarity(text_i, text_j)
            if similarity > 0.9:
                near_dups.append({
                    'idx1': i,
                    'idx2': j,
                    'similarity': similarity,
                    'text1_preview': text_i[:50] + '...' if len(text_i) > 50 else text_i,
                    'text2_preview': text_j[:50] + '...' if len(text_j) > 50 else text_j,
                    'label1': samples[i]['label'],
                    'label2': samples[j]['label']
                })

print(f"\n=== Duplication Analysis ===")
print(f"Exact duplicates (100%): {len(exact_dups)} unique texts with duplicates")
print(f"Near duplicates (>90%): {len(near_dups)} pairs")

# Show examples
if exact_dups:
    print("\nExact Duplicate Examples:")
    for idx, (text, instances) in enumerate(list(exact_dups.items())[:3]):
        print(f"\n  Text: {text[:60]}...")
        print(f"  Appears {len(instances)+1} times at indices:", end=" ")
        indices = set([inst[0] for inst in instances] + [inst[1] for inst in instances])
        print(list(indices)[:5])
        # Check label consistency
        labels = set([inst[2] for inst in instances] + [inst[3] for inst in instances])
        if len(labels) > 1:
            print(f"  ⚠️  WARNING: Same text has different labels: {labels}")

if near_dups:
    print("\nNear Duplicate Examples (>90% similar):")
    for dup in near_dups[:3]:
        print(f"\n  Similarity: {dup['similarity']:.2%}")
        print(f"  Text 1 (idx {dup['idx1']}): {dup['text1_preview']}")
        print(f"  Text 2 (idx {dup['idx2']}): {dup['text2_preview']}")
        if dup['label1'] != dup['label2']:
            print(f"  ⚠️  Different labels: {dup['label1']} vs {dup['label2']}")

For Large Datasets (>10k samples):

Use sampling or hashing for efficiency:

import hashlib
from collections import defaultdict

# Use text hash for exact duplicates (faster)
hash_groups = defaultdict(list)

with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for idx, line in enumerate(f):
        data = json.loads(line)
        text = data.get('text', '')
        text_hash = hashlib.md5(text.encode()).hexdigest()
        hash_groups[text_hash].append(idx)

# Count exact duplicates
exact_dup_groups = {k: v for k, v in hash_groups.items() if len(v) > 1}
total_duplicates = sum(len(v) - 1 for v in exact_dup_groups.values())

print(f"Exact duplicates: {total_duplicates} duplicate samples")
print(f"Duplicate groups: {len(exact_dup_groups)}")

Deduplication Recommendations:

Based on analysis results, provide recommendations:

1. For Exact Duplicates (100%):

   • If labels are consistent: Safe to remove duplicates, keep one copy
   • If labels are inconsistent: Data quality issue - investigate and fix labels
   • Recommendation: Keep the first occurrence or most recent (if timestamp available)

2. For Near Duplicates (>90% similarity):

   • Review manually if count is small (<100 pairs)
   • If labels differ: Likely annotation inconsistency - review and correct
   • If labels same: Consider whether minor differences are meaningful
   • Options:
     • Keep both if differences are semantically important
     • Keep longer version if one is truncated
     • Merge if differences are typos/formatting

3. Deduplication Priority:

   • High priority: Exact duplicates with same labels (safe to remove)
   • Medium priority: Exact duplicates with different labels (fix first, then dedupe)
   • Low priority: Near duplicates (requires manual review)

Report Content:

• Exact duplicate count and percentage
• Near duplicate pairs (>90% similarity)
• Label consistency analysis for duplicates
• Specific deduplication recommendations
• Indices/IDs of duplicate samples for removal

3.5 Special Fields Analysis

Objective: Analyze distribution and data types of non-standard fields

Auto-detect All Fields:

import json
from collections import Counter, defaultdict

# Collect all field names and their value samples
all_fields = defaultdict(list)
field_types = defaultdict(Counter)
field_missing = Counter()

total_samples = 0
with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        data = json.loads(line)
        total_samples += 1

        # Track all fields
        for key, value in data.items():
            all_fields[key].append(value)

            # Detect field type
            if value is None:
                field_types[key]['null'] += 1
            elif isinstance(value, bool):
                field_types[key]['boolean'] += 1
            elif isinstance(value, int):
                field_types[key]['integer'] += 1
            elif isinstance(value, float):
                field_types[key]['float'] += 1
            elif isinstance(value, list):
                field_types[key]['list'] += 1
            elif isinstance(value, dict):
                field_types[key]['dict/json'] += 1
            elif isinstance(value, str):
                # Check if string contains JSON
                if value.strip().startswith(('{', '[')):
                    try:
                        json.loads(value)
                        field_types[key]['json_string'] += 1
                    except:
                        # Check if it's unescaped JSON
                        if '{"' in value or '["' in value:
                            field_types[key]['unescaped_json'] += 1
                        else:
                            field_types[key]['string'] += 1
                else:
                    field_types[key]['string'] += 1

print("=== Field Analysis ===\n")
print(f"Total samples: {total_samples}")
print(f"Total fields detected: {len(all_fields)}\n")

# Identify core fields (text, label) and special fields
core_fields = ['text', 'label', 'labels', 'input', 'output', 'content', 'sentence']
special_fields = [f for f in all_fields.keys() if f not in core_fields]

if special_fields:
    print(f"Special fields found: {len(special_fields)}")
    print(f"Fields: {special_fields}\n")

    for field in special_fields:
        print(f"\n--- Field: '{field}' ---")

        # Type distribution
        types = field_types[field]
        print(f"Type distribution:")
        for type_name, count in types.most_common():
            percentage = count / total_samples * 100
            print(f"  {type_name}: {count} ({percentage:.1f}%)")

        # Missing rate
        present_count = len(all_fields[field])
        missing_count = total_samples - present_count
        if missing_count > 0:
            print(f"⚠️  Missing in {missing_count} samples ({missing_count/total_samples*100:.1f}%)")

        # Value distribution (for categorical fields)
        values = all_fields[field]
        if types.get('string', 0) > 0 or types.get('integer', 0) > 0:
            unique_values = set(str(v) for v in values if v is not None)
            print(f"Unique values: {len(unique_values)}")

            if len(unique_values) <= 20:  # Categorical field
                value_dist = Counter(str(v) for v in values if v is not None)
                print(f"Value distribution:")
                for val, count in value_dist.most_common(10):
                    print(f"  {val}: {count} ({count/len(values)*100:.1f}%)")

        # JSON field validation
        if types.get('json_string', 0) > 0 or types.get('unescaped_json', 0) > 0:
            print("🔍 JSON Field Details:")
            if types.get('json_string', 0) > 0:
                print(f"  ✅ Valid JSON strings: {types['json_string']}")
            if types.get('unescaped_json', 0) > 0:
                print(f"  ⚠️  Unescaped JSON detected: {types['unescaped_json']} samples")
                print(f"     Recommendation: Need to unescape before parsing")

            # Sample a valid JSON to show structure
            for val in values:
                if isinstance(val, str) and val.strip().startswith(('{', '[')):
                    try:
                        parsed = json.loads(val)
                        print(f"  Sample JSON structure: {type(parsed).__name__}")
                        if isinstance(parsed, dict):
                            print(f"  Keys: {list(parsed.keys())[:5]}")
                        break
                    except:
                        pass
else:
    print("No special fields detected (only standard text/label fields)")

Report Content:

• List of all detected fields (core + special)
• For each special field:
  • Data type distribution (string/int/float/list/dict/json_string/unescaped_json/mixed)
  • Missing rate
  • Value distribution (for categorical fields with <20 unique values)
  • For JSON fields:
    • Valid JSON count
    • Unescaped JSON count (needs fixing)
    • Mixed type count (inconsistent format)
    • Sample JSON structure
• Data quality warnings:
  • Fields with high missing rate (>10%)
  • Fields with mixed types (inconsistent format)
  • Fields with unescaped JSON (parsing issues)

Step 4: Special Fields and Advanced Analysis

See sections 3.4 and 3.5 above for:

• Advanced duplication detection (exact and near-duplicates)
• Special fields analysis and type validation

Step 5: Label Distribution Analysis

Objective: Understand label distribution and task type

4.1 Identify Task Type

Automatically determine based on label field format:

• Single string/number → Single-label classification
• List of strings → Multi-label classification
• Nested structure (with start/end/entity_type, etc.) → Sequence labeling (NER)
• question-answer field pairs → QA task

4.2 Category Distribution Statistics (Classification Tasks)

from collections import Counter
import json

labels = []

with open('dataset.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        data = json.loads(line)
        label = data.get('label')  # Adjust field name as needed
        labels.append(label)

label_dist = Counter(labels)
total = sum(label_dist.values())

print(f"Number of classes: {len(label_dist)}")
print("\nClass distribution:")
for label, count in label_dist.most_common():
    percentage = count / total * 100
    print(f"  {label}: {count} ({percentage:.2f}%)")

# Imbalance detection
if label_dist:
    max_count = max(label_dist.values())
    min_count = min(label_dist.values())
    imbalance_ratio = max_count / min_count if min_count > 0 else float('inf')
    print(f"\nImbalance ratio: {imbalance_ratio:.1f}:1 (largest class / smallest class)")

    # Rare class warning
    rare_classes = [label for label, count in label_dist.items() if count < 10]
    if rare_classes:
        print(f"⚠️  Rare classes (sample count <10): {rare_classes}")

Analysis Content:

• Number of classes
• Sample count and percentage for each class
• Class imbalance degree (largest class / smallest class ratio)
• Rare class warning (classes with <10 samples)

4.3 Sequence Labeling Analysis (If Applicable)

For NER and similar tasks, analyze:

• Entity type distribution
• Average number of entities per sample
• Entity length distribution

4.4 Multi-label Analysis (If Applicable)

For multi-label classification tasks, analyze:

• Distribution of number of labels per sample
• Label co-occurrence patterns
• Frequency of individual labels

Step 6: Data Quality Assessment

Objective: Systematically evaluate data quality and identify potential issues

Completeness Check

• All samples contain required text fields
• All samples contain label fields (if supervised learning)
• Missing value ratio is acceptable (<5%)

Consistency Check

• All samples have consistent data format
• Label format is consistent (e.g., all strings or all numbers)
• Encoding is consistent (no garbled text)

Anomaly Detection

• Check if all labels are in expected category list
• Check for obvious annotation errors
• Identify statistical outliers (extreme lengths, abnormal patterns)

Quality Scoring Reference:

• Excellent (90-100): Sufficient samples, balanced classes, no obvious issues
• Good (70-89): Reasonably sufficient samples, slightly imbalanced, minor issues
• Needs Improvement (50-69): Insufficient samples, moderately imbalanced, several issues
• Poor (<50): Severely insufficient, severely imbalanced, serious issues

Step 7: Generate Analysis Report

Strictly follow this Markdown template for output:

# NLP Dataset Analysis Report

## 📊 Dataset Overview

- **File Path**: [path]
- **Data Format**: [JSON/JSONL/CSV]
- **Total Samples**: [count]
- **Dataset Size**: [file size, e.g., 1.2 MB]
- **Analysis Time**: [timestamp]

## 📝 Text Content Analysis

### Text Length Statistics
- **Characters**: min=[X], max=[X], mean=[X], median=[X]
- **Words**: min=[X], max=[X], mean=[X], median=[X]
- **Length Distribution**: [description, e.g., "80% of samples are between 50-200 words"]

### Vocabulary Statistics
- **Total Vocabulary**: [X] unique tokens (based on [X] sampled samples)
- **Top 10 Frequent Words**: [word1, word2, word3, ...]
- **Low-Frequency Word Ratio**: [X]% (words appearing ≤2 times)

### Text Quality

[If no issues]
✅ **No major quality issues**

[If issues found]
⚠️ **Issues Found**:
- Empty texts: [X] samples ([X]%)
- Very short texts (<5 chars): [X] samples ([X]%)
- Very long texts (>10000 chars): [X] samples ([X]%)

### Duplication Analysis

#### Exact Duplicates (100% Match)
- **Total Exact Duplicates**: [X] samples ([X]%)
- **Unique Duplicate Groups**: [X] groups

[If exact duplicates exist]
**Top Duplicate Groups**:
| Text Preview | Count | Labels | Indices |
|--------------|-------|--------|---------|
| [First 50 chars...] | [X] | [label1, label2, ...] | [idx1, idx2, ...] |
| ... | ... | ... | ... |

[If none]
✅ No exact duplicates found

#### Near-Duplicates (>90% Similarity)
- **Total Near-Duplicate Pairs**: [X] pairs
- **Cross-Label Near-Duplicates**: [X] pairs (potential annotation inconsistency)

[If near-duplicates exist]
**Top Similar Pairs** (sorted by similarity):
| Similarity | Text 1 (Label) | Text 2 (Label) | Indices |
|------------|----------------|----------------|---------|
| [95.3%] | [Text 1 preview...] ([label1]) | [Text 2 preview...] ([label2]) | [idx1, idx2] |
| ... | ... | ... | ... |

[If cross-label near-duplicates exist]
⚠️ **Annotation Inconsistency Warning**: [X] near-duplicate pairs have different labels, which may indicate inconsistent annotation

[If none]
✅ No significant near-duplicates found (>90% similarity)

#### Deduplication Recommendations

[Based on findings, provide specific recommendations]

**Recommendation**: [Keep/Remove] duplicates based on the following strategy:

- **For Exact Duplicates**:
  - [If same labels] Consider removing duplicates to avoid data leakage and reduce redundancy
  - [If different labels] Keep all instances but flag for manual review - indicates annotation inconsistency

- **For Near-Duplicates**:
  - [If same labels and >95% similar] Consider removing to reduce redundancy
  - [If different labels] Keep but review for annotation consistency
  - [If 90-95% similar] Generally keep unless they are trivially similar (e.g., only differ in punctuation)

**Expected Impact After Deduplication**:
- Estimated dataset size after removing exact duplicates: [X] samples (reduction: [X]%)
- Estimated dataset size after removing high-similarity near-duplicates (>95%): [X] samples (reduction: [X]%)
- Improved dataset quality by reducing redundancy
- Reduced risk of data leakage between train/test splits

### Special Fields Analysis

[If special fields detected beyond standard text/label fields]

#### Field Overview
| Field Name | Data Type Distribution | Missing Rate | Notes |
|------------|------------------------|--------------|-------|
| [field1] | string: [X]%, int: [X]%, ... | [X]% | [e.g., "Contains JSON strings"] |
| [field2] | ... | [X]% | ... |
| ... | ... | ... | ... |

#### JSON Field Validation

[If JSON fields detected]

**Field: [field_name]**
- **Valid JSON Strings**: [X] samples ([X]%)
- **Unescaped/Malformed JSON**: [X] samples ([X]%)
- **Plain Text (Non-JSON)**: [X] samples ([X]%)

[If issues exist]
⚠️ **JSON Validation Issues**:
- [X] samples contain unescaped JSON that may cause parsing errors
- [X] samples have mixed content (partial JSON + text)

**Example of Unescaped JSON**:

Sample [idx]: [example text showing the issue]

**Recommendation**:
- [If >10% unescaped] Consider preprocessing to properly escape JSON strings before use
- [If mixed types] Standardize field format to either pure JSON or pure text

[If none]
✅ All JSON fields are properly formatted

#### Value Distribution for Key Fields

[For categorical or important fields beyond labels]

**Field: [field_name]**
- **Unique Values**: [X]
- **Top 10 Values**:
  | Value | Count | Percentage |
  |-------|-------|------------|
  | [value1] | [X] | [X]% |
  | ... | ... | ... |

[If abnormal distribution detected]
⚠️ **Distribution Note**: [description of any skewness or unusual patterns]

## 🏷️ Label Distribution Analysis

### Task Type
[Text Classification/Sequence Labeling/Question Answering/Text Generation/Other]

### Class Statistics

| Class | Sample Count | Percentage |
|-------|--------------|------------|
| [label1] | [count] | [percentage]% |
| [label2] | [count] | [percentage]% |
| ... | ... | ... |

### Imbalance Analysis

- **Number of Classes**: [X] classes
- **Imbalance Ratio**: [X]:1 (largest class / smallest class)
- **Imbalance Level**: [Low (<5:1) / Medium (5-20:1) / High (>20:1)]

[If rare classes exist]
⚠️ **Rare Class Warning**:
- [label1]: Only [X] samples
- [label2]: Only [X] samples

## 🔍 Data Quality Assessment

### ✅ Passed Checks

- [Check item 1, e.g., "All samples contain text field"]
- [Check item 2, e.g., "Data format is consistent"]
- [Check item 3]

### ⚠️ Issues Requiring Attention

[If issues exist]
- **[Issue 1]**: [description] - Impact: [impact description]
- **[Issue 2]**: [description] - Impact: [impact description]

[If none]
No issues requiring special attention

### ❌ Critical Issues

[If critical issues exist]
- **[Critical issue]**: [description] - Recommendation: [fix suggestion]

[If none]
No critical issues

### Data Quality Score

**Overall Score**: [score]/100

**Rating**: [Excellent/Good/Needs Improvement/Poor]

## 💡 Recommendations and Next Steps

### Key Findings

1. [Finding 1, e.g., "Dataset has moderate size with 1000 samples"]
2. [Finding 2, e.g., "Slight class imbalance exists (5:1 ratio)"]
3. [Finding 3, e.g., "Overall text quality is good with only 2% empty texts"]

### Improvement Recommendations

- [ ] **[Recommendation 1]**: [specific action, e.g., "Clean empty text samples"] - Expected Effect: [e.g., "Improve data quality, avoid training interference"]
- [ ] **[Recommendation 2]**: [specific action, e.g., "Apply oversampling for minority classes or use class weighting"] - Expected Effect: [e.g., "Improve model performance on minority classes"]
- [ ] **[Recommendation 3]**: [specific action, e.g., "Remove exact duplicates with same labels"] - Expected Effect: [e.g., "Reduce data redundancy by [X]%, prevent data leakage"]
- [ ] **[Recommendation 4]**: [if applicable, e.g., "Fix unescaped JSON in [field_name] field"] - Expected Effect: [e.g., "Ensure proper JSON parsing, avoid preprocessing errors"]
- [ ] **[Recommendation 5]**: [specific action] - Expected Effect: [effect description]

### Pre-Training Considerations

- [Consideration 1, e.g., "Recommend using class-weighted loss function"]
- [Consideration 2, e.g., "Recommend maintaining class proportions when splitting validation set"]
- [Consideration 3, e.g., "Ensure train/validation/test splits do not contain duplicate or near-duplicate samples across splits"]
- [Consideration 4, if applicable, e.g., "Preprocess JSON fields before training to ensure consistent formatting"]
- [Consideration 5]

### Related Resources

- 📋 [Data Quality Checklist](references/data_quality_checklist.md) - Complete quality checklist and common issue solutions

---

*Analysis completed at [current timestamp]*

Supported Data Formats

• JSONL: One JSON object per line (recommended, most common)
• JSON: Single JSON array or object
• CSV/TSV: Tabular data with headers

Supported Task Types

Automatically identifies and adapts based on data structure:

• Text Classification: Single-label or multi-label classification
• Sequence Labeling: NER, POS tagging, word segmentation, etc.
• Question Answering: QA pairs
• Text Generation: Translation, summarization, etc. (input-output pairs)
• Dialogue Systems: Multi-turn conversation data
• Other: Any NLP dataset containing text fields

Large Dataset Handling

When the dataset is very large (>100k samples or >100MB):

1. Sampling Analysis: Randomly sample 1000-5000 samples for detailed analysis (vocabulary stats, quality checks, etc.)
2. Streaming Statistics: For basic statistics (sample count, length distribution, label distribution), can process full dataset
3. Stratified Sampling: For classification tasks, sample proportionally by class to maintain representativeness

JSONL Format Sampling Commands:

# Random sample 1000 lines
shuf -n 1000 large_dataset.jsonl > sample.jsonl

# Get total line count without loading entire file
wc -l large_dataset.jsonl

CSV Format Sampling Commands:

# Keep header and sample 1000 rows
head -n 1 data.csv > sample.csv
tail -n +2 data.csv | shuf -n 1000 >> sample.csv

Common Issues Handling

Q: How to handle Chinese or other non-English data?

A: Basic statistics (character count, sample count, label distribution) are language-agnostic. Vocabulary analysis uses simple space tokenization. For languages like Chinese that don't use spaces for word boundaries, consider:

• Use character-level statistics instead of word-level
• Or integrate tokenization tools (e.g., jieba for Chinese)
• High-frequency word analysis has limited value for Chinese; can skip

Q: How to handle nested JSON structures?

A: Adjust field access paths based on actual structure. For example:

# Simple structure
text = data.get('text')

# Nested structure
text = data.get('conversation', [])[0].get('text', '')

# Multi-level nesting
text = data.get('data', {}).get('content', {}).get('text', '')

Q: What if data format is non-standard?

A:

1. First use Read tool to view a few samples and understand the actual format
2. Adjust field access paths in code
3. If format varies greatly, may need preprocessing to unify format

Q: What to do after discovering data quality issues?

A: Refer to the "Improvement Recommendations" section in the report and Data Quality Checklist for specific solutions

Q: How to handle commas in text fields for CSV files?

A: Use Python's csv module or pandas to correctly parse quoted fields:

import csv
with open('data.csv', 'r', encoding='utf-8') as f:
    reader = csv.DictReader(f)
    for row in reader:
        text = row['text']

Important Notes

• File Path: Confirm correct file path, use absolute or relative path
• Large File Handling: For files >100MB, prioritize sampling analysis to avoid memory issues
• Privacy Protection: For sensitive data, don't display raw text content in reports
• Encoding Issues: If encountering parsing errors, check file encoding (should be UTF-8)
• Field Identification: Flexibly adjust field names based on actual data structure (text/content/sentence, etc.)
• Time Estimate: Small datasets (<10k samples) typically analyze in <1 minute; large datasets may take longer

Advanced Tips

Quick Data Format Validation

# Check if JSONL format is correct (each line is valid JSON)
cat dataset.jsonl | python3 -c "import sys, json; [json.loads(line) for line in sys.stdin]" && echo "Format correct" || echo "Format error"

# View JSON fields
head -n 1 dataset.jsonl | jq 'keys'

Quick Statistics (Without Python Scripts)

# Statistics on label distribution (assuming label field is 'label')
cat dataset.jsonl | jq -r '.label' | sort | uniq -c | sort -rn

# Statistics on text length distribution
cat dataset.jsonl | jq -r '.text | length' | sort -n | head -20

Detect Data Leakage

# Check if two files have duplicate lines
comm -12 <(sort train.jsonl) <(sort test.jsonl) | wc -l

Advanced Reference

For systematic quality assessment and problem solving, refer to:

• Data Quality Checklist - Complete quality checklist, common issues and solutions, quality scoring standards

Usage Example

User: Analyze this NLP dataset: sentiment_data.jsonl

Claude:

1. Read data file, check format and fields
2. Run statistical commands to get basic information
3. Execute Python scripts for deep analysis
4. Generate standardized Markdown report
5. Provide targeted improvement recommendations

The report will cover data volume, text features, label distribution, quality issues, and improvement suggestions.