data-loading
Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.
来源信息
- 仓库
- aiming-lab/AutoResearchClaw
- 最近来源活动
- 2026年3月23日 01:46
- 检测到的 SKILL.md 语言
- 英语
- 星标
- 14,579
- 分支
- 1,698
安装方式
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
检查来源文件
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
正在显示 SKILL.md
SKILL.md
来源说明 · 只读预览- name
- data-loading
- description
- Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.
- metadata
- {"category":"tooling","trigger-keywords":"data,loading,dataloader,dataset,preprocessing,augmentation","applicable-stages":"10","priority":"6","version":"1.0","author":"researchclaw","references":"PyTorch Data Loading Tutorial, pytorch.org"}
## Efficient Data Loading Best Practice
1. Use num_workers = min(8, os.cpu_count()) for DataLoader
2. Enable pin_memory=True when using GPU
3. Use persistent_workers=True to avoid re-spawning
4. Pre-compute and cache transformations when possible
5. For image data: use torchvision.transforms.v2 (faster)
6. For large datasets: consider memory-mapped files or WebDataset
7. Profile with torch.utils.bottleneck to find I/O bottlenecks
在 GitHub 查看