Piper framework for user-controllable distributed training that decouples parallelism strategy from runtime implementation using unified global training DAG intermediate representation. Use for distributed ML training, parallelism strategy design, and flexible training system architecture.
Piper framework for user-controllable distributed training that decouples parallelism strategy from runtime implementation using unified global training DAG intermediate representation. Use for distributed ML training, parallelism strategy design, and flexible training system architecture.
Fundamental Problem: Existing systems require manual strategy design + implementation, making adaptation difficult. General-purpose frameworks are tied to fixed parallelism strategies, hindering state-of-the-art integration.
Solution: Decouple strategy from runtime implementation using:
User declarations: Model annotations + scheduling directives
Unified IR: Global training DAG representing all computation/communication
Key Innovation: Joint scheduling of compute + communication in composed strategies.
Implementation Steps
Step 1: Model Annotation
import piper
@piper.annotate(
data_parallel=4, # Number of data parallel replicas
pipeline_parallel=8, # Number of pipeline stages
expert_parallel=2, # Number of expert parallel groups
zero_stage=3# ZeRO optimization stage)classTransformerModel:
def__init__(self, config):
self.layers = [TransformerLayer(config) for _ inrange(config.num_layers)]
defforward(self, x):
for layer inself.layers:
x = layer(x)
return x
Step 2: Directive Application
# Directive: transforms IR based on strategy@piper.directive("DualPipe",
schedule="interleaved_forward_backward",
communication_overlap=True,
memory_optimization="activation_checkpointing")
Deployment Speed: Strategy changes via annotations, not code modifications
Advantages vs Traditional Systems
Aspect
Traditional
Piper
Strategy implementation
Hand-coded runtime
declarative directives
Strategy adaptation
Code refactoring
Annotation modification
Novel strategies
Custom runtime needed
Directive + IR transform
Compute-communication scheduling
Sequential
Joint optimization
Memory optimization
Manual tuning
IR-level scheduling
Pitfalls
IR Complexity: Large models → complex DAG → compilation overhead
Directive Availability: New strategies require new directive implementations
Device Constraints: Compilation must respect physical device topology
Communication Scheduling: Overlapping compute/communication requires careful timing
Debugging: IR-level errors harder to trace than runtime-level errors
Use Cases
1. Foundation Model Pretraining
@piper.annotate(data_parallel=128, pipeline_parallel=8, zero_stage=3)@piper.directive("ZeRO-3", communication_overlap=True)# Large-scale GPT-style training
2. MoE Model Training
@piper.annotate(expert_parallel=16, data_parallel=4)@piper.directive("ExpertParallel", load_balancing="dynamic")# Mixture-of-experts training
3. Hybrid Strategy (DeepSeek-V3)
@piper.annotate(data_parallel=4, pipeline_parallel=8, expert_parallel=2)@piper.directive("DualPipe", interleaved_schedule=True)# Composed parallelism with communication overlap
