parallel-file-processor-mode-selection

name	parallel-file-processor-mode-selection
description	Sub-skill of parallel-file-processor: Mode Selection (+2).
version	1.1.0
category	development
type	reference
scripts_exempt	true

Mode Selection (+2)

Mode Selection

Workload Type	Recommended Mode	Reason
File I/O	THREAD_POOL	IO-bound, threads avoid GIL issues
Data parsing	THREAD_POOL	Pandas releases GIL during IO
CPU computation	PROCESS_POOL	Bypasses GIL for true parallelism
Network requests	ASYNC	Best for many concurrent connections
Simple operations	SEQUENTIAL	Overhead may exceed benefit

Worker Count

import os

# IO-bound (reading files, network)
io_workers = os.cpu_count() * 2

# CPU-bound (heavy computation)
cpu_workers = os.cpu_count()

# Memory-constrained (large files)
memory_workers = max(2, os.cpu_count() // 2)

Batch Size

• Small files (<1MB): Large batches (500-1000)
• Medium files (1-100MB): Medium batches (50-100)
• Large files (>100MB): Small batches (10-20) or one at a time