Use when user needs to process data at scale. Triggers on: batch processing, data ingestion, pipeline, parallel processing, GPU acceleration, video processing, PDF processing, large-scale.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/zilliztech/milvus-marketplace --skill ray
The command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Use when user needs to process data at scale. Triggers on: batch processing, data ingestion, pipeline, parallel processing, GPU acceleration, video processing, PDF processing, large-scale.
Ray - Distributed Data Processing Framework
All data processing tasks (batch import, video processing, PDF parsing, etc.) should use Ray for orchestration.
When to Use Ray
Scenario
Use Ray?
Processing < 100 items
Optional, simple loops work
Processing 100 - 10,000 items
Recommended, single-machine multi-core parallel
Processing > 10,000 items
Required, can scale to cluster
Need GPU acceleration
Required, Ray manages GPU resources
Multi-step pipeline
Recommended, clear DAG orchestration
Installation
pip install "ray[data]"
Core Concepts
1. Ray Task - Parallel Functions
Turn regular functions into parallelizable tasks:
import ray
ray.init()
@ray.remotedefprocess_file(file_path):
# Process single filereturn result
# Process 1000 files in parallel
files = ["file1.pdf", "file2.pdf", ...]
futures = [process_file.remote(f) for f files]
results = ray.get(futures)
in
# Wait for all to complete
2. Ray Actor - Stateful Services
Suitable for scenarios that need model loading (model loads only once):
@ray.remote(num_cpus=2) # Each task uses 2 CPUsdefcpu_task():
...
GPU Tasks
@ray.remote(num_gpus=1) # Each task uses 1 GPUdefgpu_task():
...
@ray.remote(num_gpus=0.5) # Two tasks share 1 GPUdefsmall_gpu_task():
...
Actor Pool
# Create pool of 4 Actors with automatic load balancing
ds.map_batches(
MyActor,
compute=ray.data.ActorPoolStrategy(size=4, num_gpus=1)
)
Error Handling
@ray.remote(max_retries=3) # Auto-retry 3 times on failuredefunreliable_task():
...
# Error handling for batch tasks
futures = [process.remote(item) for item in items]
results = []
for future in futures:
try:
result = ray.get(future)
results.append(result)
except Exception as e:
print(f"Task failed: {e}")
results.append(None)
Monitoring
# Start Ray Dashboard
ray.init(dashboard_host="0.0.0.0", dashboard_port=8265)
# Visit http://localhost:8265 to view task status
