Menu
Bilingual guide for understanding LengthPoolSortDataset cross-rank length synchronization mechanism in multi-GPU training
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Bilingual guide for running and interpreting LLaVA-OneVision2 HF vs Megatron consistency checks across TP and PP settings
Bilingual guide for merging ViT + LLM into LlavaOnevision2 HF checkpoint and validating weight/inference consistency
Guide for writing clear, consistent git commit messages following this repository's conventions
Bilingual guide for the OFFLINE_PACKING_BMR and OFFLINE_PACKED_DATA environment variables that control LLaVA-OneVision2 training-side packing — what each gate does, why both must be enabled together, MBS=1 requirement, and the dead OFFLINE_PACKING_VQA branch
Bilingual guide for running offline_packing/auto_pipe.sh across multiple nodes to produce padding-free packed WebDataset shards for SFT, with Energon Metadataset assembly
Bilingual guide for understanding how cu_lengths controls attention behavior across ViT and LLM stages, and how patch_positions scope differs between the two
| name | length-pool-sort-dataset |
| description | Bilingual guide for understanding LengthPoolSortDataset cross-rank length synchronization mechanism in multi-GPU training |
| compatibility | opencode |
| metadata | {"domain":"distributed-training","framework":"megatron-energon","repo":"llava-onevision2"} |
Use this skill when analyzing, debugging, or tuning LengthPoolSortDataset — the cross-rank length synchronization mechanism used in this repository's training pipeline.
在分析、调试或调优本仓库训练 pipeline 中的跨 rank 长度同步机制 LengthPoolSortDataset 时，使用这个 skill。
This skill is specifically for:
aiak_training_llm/data/multimodal/length_sort_dataset.pylength_sort_pool_size > 0pool_size for optimal multi-GPU efficiency这个 skill 专门用于：
aiak_training_llm/data/multimodal/length_sort_dataset.pylength_sort_pool_size > 0 时训练速度提升pool_size 以获得最佳多卡效率上游 dataset → 累积 pool_size 个 sample → 按序列长度排序 → 用确定性 seed shuffle → 逐个 yield
for batch_idx, sample in enumerate(self.dataset):
pool.append(sample)
if len(pool) >= self.pool_size:
pool.sort(key=self.key_fn) # 1. 按长度排序
shuffle_seed = 42 + batch_idx # 2. 确定性 seed
random.Random(shuffle_seed).shuffle(pool) # 3. 同 seed shuffle
for s in pool:
yield s
pool.clear()
CrudeWebdataset → ShuffleBuffer → cook_crude_sample → encode_sample
→ LengthPoolSortDataset → BatchDataset → EpochizeDataset → LogSampleDataset
Inserted after encode_sample (where total_len / tokens are available) and before BatchDataset.
插在 encode_sample 之后（此时已有 total_len / tokens）、BatchDataset 之前。
Activated by: --length-sort-pool-size N (where N > 0).
通过 --length-sort-pool-size N（N > 0）激活。
In multi-GPU data-parallel training, all ranks must synchronize at each step (gradient all-reduce). If different ranks process samples of very different lengths, fast ranks idle waiting for slow ranks.
多卡数据并行训练中，所有 rank 每步都要同步（梯度 all-reduce）。如果不同 rank 处理的 sample 长度差异很大，快的 rank 空等慢的 rank。
无 pool sort:
Rank 0 step 100: 长度 200 → 0.5s ┐
Rank 1 step 100: 长度 5000 → 3.0s ├→ 所有 rank 等 3.0s
Rank 2 step 100: 长度 300 → 0.6s ┘
Sort + same-seed shuffle ensures all ranks yield samples of approximately the same length at the same time.
排序 + 同 seed shuffle 保证所有 rank 在同一时刻输出近似相同长度的 sample。
有 pool sort:
Rank 0 step 100: 长度 ~200 → 0.5s ┐
Rank 1 step 100: 长度 ~200 → 0.5s ├→ 所有 rank 等 0.5s
Rank 2 step 100: 长度 ~200 → 0.5s ┘
Each rank's data comes from different shards of the same dataset, so length distributions are approximately identical. After sorting, the i-th position in each rank's pool corresponds to the i-th quantile of its length distribution. Similar distributions → similar quantiles → similar lengths.
各 rank 的数据来自同一数据集的不同 shard，长度分布近似相同。排序后，各 rank pool 中第 i 个位置对应各自长度分布的第 i 个分位数。分布相似 → 分位数相似 → 长度相似。
Rank 0 排序后: [100, 102, 105, 108, ..., 4998, 5000]
Rank 1 排序后: [101, 103, 106, 109, ..., 4999, 5001]
Rank 2 排序后: [ 99, 104, 107, 110, ..., 4997, 5002]
shuffle_seed = 42 + batch_idx is identical across all ranks → same permutation applied to all pools. Since the i-th position had similar lengths, the shuffled output at the same position still has similar lengths.
shuffle_seed = 42 + batch_idx 对所有 rank 相同 → 相同排列应用于所有 pool。由于第 i 个位置的长度相似，shuffle 后同一位置的输出长度仍然相似。
permutation = [3, 0, 4, 1, 2]:
Rank 0 输出: [108, 100, 5000, 102, 105]
Rank 1 输出: [109, 101, 5001, 103, 106]
Rank 2 输出: [110, 99, 5002, 104, 107]
→ 同一位置长度近似一致
| 方案 | 效果 |
|---|---|
| 只 sort 不 shuffle | 所有 rank 先跑短 sample 后跑长 sample。前期 step 极快，后期 step 极慢。训练动态不稳定 |
| sort + shuffle | 各 step 长度随机但 rank 间一致。step 耗时平稳，rank 间同步开销小 |
| 不 sort 不 shuffle | 各 rank 同一 step 长度随机且不一致，快的等慢的 |
Sort solves cross-rank synchronization. Shuffle solves temporal uniformity. Both are required.
排序解决跨 rank 同步问题，shuffle 解决时序均匀性问题。二者缺一不可。
| pool_size | 跨 rank 长度同步精度 | 内存开销 | 首个 pool 输出延迟 |
|---|---|---|---|
| 小（~100） | 较差，各 rank 分位数估计不准 | 低 | 低 |
| 中（~1000-10000） | 好，推荐范围 | 中 | 中 |
| 大（~全量） | 完美同步 | 高 | 高 |
| 极限 = dataset 大小 | 等价全局排序 | 不实际 | 不实际 |
Larger pool_size → more accurate quantile estimation across ranks → better synchronization → less idle time.
pool_size 越大 → 各 rank 分位数估计越准 → 同步越好 → 空等越少。
Rule of thumb: pool_size should be significantly larger than batch_size, ideally 10x-100x.
经验法则：pool_size 应远大于 batch_size，理想情况下 10x-100x。
When num_workers > 1, each worker runs an independent LengthPoolSortDataset instance with its own pool.
当 num_workers > 1 时，每个 worker 运行独立的 LengthPoolSortDataset 实例，各自维护独立 pool。
Correctness: No issue. Each worker sorts its own shard subset independently.
Synchronization effect: Diluted. DataLoader interleaves outputs from multiple workers, partially disrupting the within-pool length ordering.
Recommendation: num_workers=1 gives the strongest synchronization effect. If num_workers > 1 is needed for I/O throughput, increase pool_size proportionally.
正确性：无问题。每个 worker 独立排序自己的 shard 子集。
同步效果：被稀释。DataLoader 交替取多个 worker 的输出，部分打乱 pool 内的长度顺序。
建议：num_workers=1 同步效果最强。如果需要 num_workers > 1 提升 I/O 吞吐，可按比例增大 pool_size。
save_state / restore_state delegate directly to the upstream dataset. The pool's internal state (accumulated but not yet yielded samples) is NOT saved.
save_state / restore_state 直接委托给上游 dataset。pool 内部状态（已累积但未 yield 的 sample）不会被保存。
On resume, up to pool_size - 1 samples may be re-ordered differently or skipped. This is generally negligible for large datasets.
恢复时，最多 pool_size - 1 个 sample 可能会被不同地排序或跳过。对于大数据集来说通常可以忽略。
key_fn=lambda s: getattr(s, "total_len", len(getattr(s, "tokens")))
Prefers total_len attribute (set by encode_sample), falls back to len(tokens).
优先使用 total_len 属性（由 encode_sample 设置），回退到 len(tokens)。
Is --length-sort-pool-size actually set and > 0?
What is the actual length distribution of the dataset? (High variance → more benefit from pool sort)
Are all ranks receiving data from the same underlying dataset with similar length distributions?
Is num_workers > 1 diluting the synchronization effect?
After enabling, did step time variance across steps decrease?
Is memory usage acceptable for the chosen pool_size?
--length-sort-pool-size 是否真正设置且 > 0？
数据集的实际长度分布如何？（方差越大 → pool sort 收益越大）
所有 rank 是否从同一底层数据集接收长度分布相似的数据？
num_workers > 1 是否稀释了同步效果？
启用后，各 step 的 step time 方差是否降低了？
所选 pool_size 的内存占用是否可接受？
When asked to analyze a training speed issue related to length sorting, return:
当被要求分析与长度排序相关的训练速度问题时，应返回：
Whether LengthPoolSortDataset is active and with what pool_size
The dataset's length distribution characteristics (high/low variance)
The relationship between pool_size, batch_size, and num_workers
Whether cross-rank synchronization is the bottleneck
Recommended pool_size adjustment if needed
LengthPoolSortDataset 是否激活，pool_size 是多少
数据集的长度分布特征（高/低方差）
pool_size、batch_size、num_workers 之间的关系
跨 rank 同步是否是瓶颈
如需要，推荐的 pool_size 调整