parallel

name	parallel
description	Run multiple commands in parallel and aggregate outputs. Load when tasks are independent and can run concurrently.

Parallel Execution

Run independent commands concurrently for efficiency.

Working Pattern (ddgs)

mkdir -p /tmp/parallel-work

(ddgs text -q "query1" -m 15 > /tmp/parallel-work/01.txt) &
PID1=$!
(ddgs text -q "query2" -m 15 > /tmp/parallel-work/02.txt) &
PID2=$!
(ddgs text -q "query3" -m 15 > /tmp/parallel-work/03.txt) &
PID3=$!
(ddgs text -q "query4" -m 15 > /tmp/parallel-work/04.txt) &
PID4=$!

wait $PID1 $PID2 $PID3 $PID4
cat /tmp/parallel-work/0*.txt

Generic Pattern

mkdir -p /tmp/my-temp

(cmd1 > /tmp/my-temp/r1.txt) &
PID1=$!
(cmd2 > /tmp/my-temp/r2.txt) &
PID2=$!
(cmd3 > /tmp/my-temp/r3.txt) &
PID3=$!

wait $PID1 $PID2 $PID3
cat /tmp/my-temp/r*.txt

Batch Pattern (Loop)

mkdir -p /tmp/parallel-work
PIDS=()

for item in "${ITEMS[@]}"; do
    (process "$item" > "/tmp/parallel-work/r_$item.txt") &
    PIDS+=($!)
done

wait "${PIDS[@]}"
cat /tmp/parallel-work/r_*.txt

With Timeout

(timeout 60s cmd > /tmp/my-temp/result.txt) &

Error Handling

(cmd > /tmp/my-temp/r.txt) &
PID=$!

if wait $PID; then
    echo "Success"
    cat /tmp/my-temp/r.txt
else
    echo "Failed with exit code: $?"
fi

Guidelines

Guideline	Reason
Use 4-6 concurrent	Sweet spot for most systems
Unique temp dir per run	Avoids file collisions
Use absolute paths	Avoids cwd confusion
Always wait	Ensures all processes finish
Parenthesize each bg job	(cmd &) not { cmd & }

⚠️ Critical: Use subshells for background jobs
Pattern (command > output.txt) & works reliably.
Pattern { command > output.txt & } does NOT work with ddgs.

⚠️ Danger: Temp file race conditions
Never use $$ or variable temp paths in filenames across parallel invocations.