parallel-file-processor

name	parallel-file-processor
description	Process multiple files in parallel with aggregation and progress tracking. Use for batch file operations, directory scanning, ZIP handling, and parallel data processing with 2-3x performance improvement.
type	reference
version	1.1.0
category	development
related_skills	["data-pipeline-processor","yaml-workflow-executor","engineering-report-generator"]
capabilities	[]
requires	[]
tags	[]

Parallel File Processor

Quick Start

from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd

def process_csv(file_path: Path) -> dict:
    """Process a single CSV file."""
    df = pd.read_csv(file_path)
    return {'file': file_path.name, 'rows': len(df), 'columns': len(df.columns)}

# Get all CSV files
files = list(Path('data/raw/').glob('*.csv'))

# Process in parallel
results = []
with ThreadPoolExecutor(max_workers=8) as executor:
    futures = {executor.submit(process_csv, f): f for f in files}
    for future in as_completed(futures):
        results.append(future.result())

print(f"Processed {len(results)} files")

When to Use

• Processing large numbers of files (100+ files)
• Batch operations on directory contents
• Extracting data from multiple ZIP archives
• Aggregating results from parallel operations
• CPU-bound file transformations
• IO-bound file operations with proper concurrency

Related Skills

• data-pipeline-processor - Data transformation
• yaml-workflow-executor - Workflow automation
• engineering-report-generator - Report generation