| name | vlm-packing |
| description | Add, review, update, and validate VLM/MLLM packing behavior around the current MLLMDataKit design, including MLLMPackingSpec, MLLMPackingAssembler, MLLMPack metadata, block-causal masks, model-specific packed input preparation, token accounting, and step estimation. |
VLM Packing
Goal
Use MLLMPackingSpec through MLLMDataKit.build_dataset(data_spec) for standard
MLLM packing. This skill covers packing knobs and model-specific packed
semantics around the generic DataKit packer.
Required Inputs
- target recipe path;
- existing
MLLMDataSpec and MLLMPackingSpec wiring;
max_seq_len, strategy, open-pack limit, and buffer size;- packed batch fields consumed by model preparation;
- attention backend and position-id requirements;
- token accounting and total-step estimation path.
Ask only if packed attention semantics or accounting boundaries are unclear.
Workflow
1. Confirm DataKit Packing
Search:
rg -n \
"MLLMPackingSpec|MLLMPackingAssembler|MLLMPack|pack_segment_ids|source_sample_num|prepare_packed" \
recipes/<recipe> mvp_engine/kit
For standard MLLM recipes, packing is part of the DataKit pipeline. Active
configuration lives in MLLMPackingSpec.
2. Leave Generic Packing In DataKit
DataKit owns:
- grouping tokenized
MLLMSample objects;
- producing
MLLMPack;
- concatenating token fields in
MLLMPack.to_model_inputs();
- creating
pack_segment_ids and source_sample_num;
- merging media fields through
MLLMMediaHandler.merge_pack.
Recipe code consumes these packed fields and adds model-specific preparation.
3. Implement Model-Specific Packed Preparation
Recipe/model code may transform packed metadata into:
- block-causal masks;
- packed position ids;
- FlashAttention/cu-seqlens metadata;
- multimodal position rules;
- backend-specific mask patches;
- FLOPs, token, and loss accounting metadata.
Keep this near the model or engine path that prepares model inputs.
Validation
Soft checks:
- packing knobs map to
MLLMPackingSpec;
- custom algorithms use
MLLMPackingSpec(assembler_cls=...);
- packed samples include
pack_segment_ids and source_sample_num;
- collator pads packed metadata with inactive values;
- model preparation prevents cross-sample attention and preserves multimodal
position behavior;
- total/effective token counts and step estimation count packed outputs
consistently.
Hard checks when requested:
.venv/bin/python -m compileall -q mvp_engine/kit/mllm/data recipes/<recipe>
.venv/bin/python -m pytest recipes/<recipe>/tests/test_structure.py -q
.venv/bin/python -m pytest recipes/<recipe>/tests/test_smoke.py -q
Output
- State DataKit packing knobs and any custom assembler.
- State attention, position, and token accounting behavior.
- Report validation and untested packed-boundary risks.
Read On Demand
skills/kit/mllm-data-kit/references/packing.md: standard DataKit packing contract.references/packing_rules.md: detailed checks for custom packed model behavior.
