| name | parallel-file-processor |
| description | Process multiple files in parallel with aggregation and progress tracking. Use for batch file operations, directory scanning, ZIP handling, and parallel data processing with 2-3x performance improvement. |
| type | reference |
| version | 1.1.0 |
| category | development |
| related_skills | ["data-pipeline-processor","yaml-workflow-executor","engineering-report-generator"] |
| capabilities | [] |
| requires | [] |
| tags | [] |
Parallel File Processor
Quick Start
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd
def process_csv(file_path: Path) -> dict:
"""Process a single CSV file."""
df = pd.read_csv(file_path)
return {'file': file_path.name, 'rows': len(df), 'columns': len(df.columns)}
files = list(Path('data/raw/').glob('*.csv'))
results = []
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {executor.submit(process_csv, f): f for f in files}
for future in as_completed(futures):
results.append(future.result())
print(f"Processed {len(results)} files")
When to Use
- Processing large numbers of files (100+ files)
- Batch operations on directory contents
- Extracting data from multiple ZIP archives
- Aggregating results from parallel operations
- CPU-bound file transformations
- IO-bound file operations with proper concurrency
Related Skills
