bones-seed-dataset

name

bones-seed-dataset

description

BONES-SEED dataset workflow for this IsaacLab-Imitation repo, including Hugging Face access, Skynet data paths, metadata counts, G1 archive layout, trajectory shortlists, CSV extraction, G1 joint-order validation, root/unit conversion, NPZ export, manifest/language sidecar generation, and MiniLM language embeddings. Use when the user mentions BONES-SEED, bones seed, bones-studio/seed, G1 CSV/NPZ conversion, language-conditioned planner dataset prep, or dataset metadata/joint-order issues.

BONES-SEED Dataset

Use this skill when preparing or inspecting BONES-SEED G1 trajectories for the language-conditioned planner.

Ground Truth

• Hugging Face dataset repo: bones-studio/seed
• Full metadata inventory: 142220 trajectories.
• seed_metadata_v004.csv has 142221 lines including header.
• seed_metadata_v002_temporal_labels.jsonl has 142220 lines.
• move_g1_path is nonempty for all 142220 metadata rows.
• BONES-SEED source FPS is treated as 120.0; exported NPZ references use 50.0.
• The curated paper subset currently used by this repo is 24 trajectories.

On Skynet, the working data root is:

/coc/flash12/fwu91/Research/IsaacLab/data/bones_seed_paper24

Known downloaded raw assets:

raw/g1.tar.gz                                      # about 22G
raw/metadata/seed_metadata_v004.csv                # 146411608 bytes
raw/metadata/seed_metadata_v002_temporal_labels.jsonl

Important Files

• Pipeline orchestrator: scripts/rlopt/run_bones_seed_language_pipeline.py
• Subset prep and metadata annotation: scripts/prepare_bones_seed_subset.py
• Generic CSV to NPZ wrapper: scripts/prepare_lafan1_from_csv.py
• Batched Isaac Sim conversion: scripts/batch_csv_to_npz.py
• Single-motion converter: scripts/csv_to_npz.py
• G1 joint-order source: source/isaaclab_imitation/isaaclab_imitation/assets/robots/unitree_joint_order.py
• Manifest loader utilities: source/isaaclab_imitation/isaaclab_imitation/tasks/manager_based/imitation/lafan1_manifest.py

Dataset Layout

For a prepared subset, expect:

<data_root>/
  raw/
    g1.tar.gz
    metadata/seed_metadata_v004.csv
    metadata/seed_metadata_v002_temporal_labels.jsonl
    g1/*.csv
  curated/*.timeline.json
  npz/g1/*.npz
  manifests/*.json
  language/*_language.json
  language/*_minilm_goal_embeddings.pt
  g1_hl_diffsr/                 # zarr/cache dataset produced by env use

Use quick counts:

ssh skynet 'root=/coc/flash12/fwu91/Research/IsaacLab/data/bones_seed_paper24; printf "csv="; find "$root/raw/g1" -type f -name "*.csv" 2>/dev/null | wc -l; printf "npz="; find "$root/npz/g1" -type f -name "*.npz" 2>/dev/null | wc -l; ls -lh "$root/manifests" "$root/language" 2>/dev/null || true'

Count full metadata:

ssh skynet 'python3 - <<'"'"'PY'"'"'
import csv
from pathlib import Path
p = Path("/coc/flash12/fwu91/Research/IsaacLab/data/bones_seed_paper24/raw/metadata/seed_metadata_v004.csv")
with p.open(newline="", encoding="utf-8") as f:
    rows = list(csv.DictReader(f))
print(len(rows))
print(sum(1 for r in rows if r.get("move_g1_path", "").strip()))
PY'

Downloading

Prefer the existing pipeline downloader or direct authenticated HTTP when hf is broken. Do not print token values.

The pipeline downloads these files if missing:

metadata/seed_metadata_v004.csv
metadata/seed_metadata_v002_temporal_labels.jsonl
g1.tar.gz

Skynet token file:

/nethome/fwu91/.hf_token

If checking access manually, use HEAD/range requests or a tiny metadata read first. Avoid re-downloading g1.tar.gz when it already exists and has nonzero size.

Shortlist Format

scripts/prepare_bones_seed_subset.py expects a JSON list. Each item must contain at least filename; language fields and events are preserved when present:

[
  {
    "filename": "Neutral_stoop_down_001__A057",
    "overview_description": "stoop down from neutral",
    "events": [],
    "num_events": 0
  }
]

The paper24 shortlist is generated by scripts/rlopt/run_bones_seed_language_pipeline.py from PAPER24_FILENAMES.

Conversion Workflow

Run conversion through the IsaacLab environment or the cluster wrapper. The direct script form is:

pixi run -e isaaclab python scripts/prepare_bones_seed_subset.py \
  --shortlist /path/to/bones_seed_paper24.timeline.json \
  --archive /path/to/g1.tar.gz \
  --csv_dir /path/to/raw/g1 \
  --npz_dir /path/to/npz/g1 \
  --manifest_path /path/to/manifests/g1_bones_seed_paper24_manifest.json \
  --language_path /path/to/language/g1_bones_seed_paper24_language.json \
  --metadata_csv /path/to/raw/metadata/seed_metadata_v004.csv \
  --headless \
  --device cuda:0 \
  --skip_existing

Use --no_convert only to stage CSVs and write the language sidecar without launching Isaac Sim. Use --overwrite only when intentionally refreshing CSVs or NPZs.

After manifest and language sidecar exist, build language embeddings:

pixi run -e isaaclab python scripts/rlopt/build_language_goal_embeddings.py \
  --manifest /path/to/manifests/g1_bones_seed_paper24_manifest.json \
  --language_sidecar /path/to/language/g1_bones_seed_paper24_language.json \
  --require_language_sidecar_matches \
  --backend sentence-transformer \
  --output /path/to/language/g1_bones_seed_paper24_minilm_goal_embeddings.pt

Joint Order and Units

The BONES-SEED G1 CSV header must match this repo's canonical 29-DOF Unitree G1 SDK joint order from unitree_joint_order.py.

prepare_bones_seed_subset.py validates the header against:

Frame,
root_translateX, root_translateY, root_translateZ,
root_rotateX, root_rotateY, root_rotateZ,
<UNITREE_G1_29DOF_SDK_JOINT_NAMES with "_dof" suffix>

The script converts:

• root translations: centimeters to meters
• root rotations: degrees to quaternions using BONES seed-viewer convention
• joint positions: degrees to radians
• output CSV shape: 36 columns, then NPZ export through prepare_lafan1_from_csv.py

The manifest is annotated with:

"source_dataset": "bones-studio/seed",
"joint_order": {
  "source": "BONES-SEED G1 CSV header columns after root fields",
  "matches": "...",
  "joint_names": [...]
},
"root_rotation_convention": "...",
"loader_kwargs": {
  "chunk_size": 1,
  "shard_size": 512
}

Do not bypass joint-order validation. If the expected/actual header differs, inspect source/isaaclab_imitation/.../unitree_joint_order.py and the BONES CSV header before editing conversion code.

Pipeline Modes

The BONES pipeline can be limited by skip flags:

# data prep + embeddings + skill pretrain + commander pretrain only
EXTRA_PIPELINE_ARGS='--skip-low-level --skip-rollout-ft'

# data prep only, if the manifest does not already exist
EXTRA_PIPELINE_ARGS='--skip-skill --skip-commander --skip-low-level --skip-rollout-ft'

# reuse prepared data
EXTRA_PIPELINE_ARGS='--skip-data-prep'

When the full pipeline runs, stages are:

1. download/reuse metadata and g1.tar.gz
2. write shortlist
3. prepare selected CSVs, NPZs, manifest, and language sidecar
4. build MiniLM embeddings
5. train skill encoder with train_hl_skill_diffsr.py
6. train skill commander with train_skill_commander.py
7. train low-level IPMD bilinear policy
8. run rollout finetune/eval

Validation Checks

Use these checks before training:

python3 - <<'PY'
import json
from pathlib import Path
root = Path("/coc/flash12/fwu91/Research/IsaacLab/data/bones_seed_paper24")
manifest = json.loads((root / "manifests/g1_bones_seed_paper24_manifest.json").read_text())
language = json.loads((root / "language/g1_bones_seed_paper24_language.json").read_text())
print("manifest_entries", len(manifest["dataset"]["trajectories"]["lafan1_csv"]))
print("language_motions", len(language["motions"]))
print("metadata", manifest.get("metadata", {}))
PY

Expected for paper24: manifest_entries=24, language_motions=24, and npz/g1 should contain 24 .npz files.

Common Failure Modes

• EULA prompt/EOF during Isaac Sim conversion: ensure ACCEPT_EULA=Y, PRIVACY_CONSENT=Y, and OMNI_KIT_ACCEPT_EULA=YES are set in the job environment.
• hf CLI broken on Skynet: use direct HTTP or Python urllib with HF_TOKEN.
• g1.tar.gz scan is slow: avoid full tar enumeration when metadata is enough; extract only selected CSVs.
• Existing manifest causes data prep skip in run_bones_seed_language_pipeline.py: delete/rename the manifest or pass a new data root if intentional regeneration is needed.
• Short trajectories and zarr chunking: rely on manifest loader_kwargs.chunk_size=1; do not pad NPZs unless explicitly needed.