lance-format/lance-huggingface

ملف lance-format

آخر نشاط مصدر مسجل: ١٨ مايو ٢٠٢٦ في ١٩:٥٥

آخر تحديث لفهرس SkillsMP: ٢٦ أكتوبر ٢٠٢٦ في ١١:٣٧

skills مجمعة: ١

نجوم GitHub: ٢
تفرعات GitHub: ٣

ملف README للمستودع

مصدر ملف README للمستودع: README.md

قراءة الملف الأصلي

عرض ملف README كاملًا

# lance-huggingface LanceDB conversions of standard ML datasets, published under the [`lance-format`](https://huggingface.co/lance-format) Hugging Face organization. ## Data organization You can either include a `README.md` with [dataset cards](https://huggingface.co/docs/hub/en/datasets-cards) directly in a raw Lance dataset, or adopt a Hugging Face–style directory structure, placing Lance datasets for different splits in the `/data` directory: ```sh my_dataset/ ├── README.md └── data/ ├── train.lance ├── test.lance └── validation.lance ``` ## Datasets in this repo Every folder is a self-contained converter — `dataprep.py` pulls the source from the Hub, writes a Lance dataset with inline multimodal data + indices, and uploads the bundle to `lance-format/<name>`. Cards in `HF_DATASET_CARD.md` follow the pattern of the existing converters (laion-1m, openvid). | Folder | HF repo | Modality | Embeddings & indices | |---|---|---|---| | `mnist/` | [`lance-format/mnist-lance`](https://huggingface.co/datasets/lance-format/mnist-lance) | image classification | CLIP ViT-B/32 image emb · IVF_PQ · BTREE/BITMAP on label | | `cifar10/` | [`lance-format/cifar10-lance`](https://huggingface.co/datasets/lance-format/cifar10-lance) | image classification | CLIP ViT-B/32 image emb · IVF_PQ · BTREE/BITMAP on label | | `fashion_mnist/` | [`lance-format/fashion-mnist-lance`](https://huggingface.co/datasets/lance-format/fashion-mnist-lance) | image classification | CLIP ViT-B/32 image emb · IVF_PQ · BTREE/BITMAP on label | | `imagenet1k_val/` | [`lance-format/imagenet-1k-val-lance`](https://huggingface.co/datasets/lance-format/imagenet-1k-val-lance) | image classification (50k val) | CLIP ViT-B/32 image emb · IVF_PQ | | `pascal_voc_2012/` | [`lance-format/pascal-voc-2012-segmentation-lance`](https://huggingface.co/datasets/lance-format/pascal-voc-2012-segmentation-lance) | semantic segmentation | CLIP ViT-B/32 image emb · IVF_PQ | | `flickr30k/` | [`lance-format/flickr30k-lance`](https://huggingface.co/datasets/lance-format/flickr30k-lance) | image-caption | CLIP image+text · IVF_PQ on both · FTS on caption | | `coco_captions_2017/` | [`lance-format/coco-captions-2017-lance`](https://huggingface.co/datasets/lance-format/coco-captions-2017-lance) | image-caption | CLIP image+text · IVF_PQ on both · FTS on caption | | `squad_v2/` | [`lance-format/squad-v2-lance`](https://huggingface.co/datasets/lance-format/squad-v2-lance) | question-answering | MiniLM question emb · IVF_PQ · FTS on question/context | | `triviaqa/` | [`lance-format/trivia-qa-lance`](https://huggingface.co/datasets/lance-format/trivia-qa-lance) | question-answering | MiniLM question emb · IVF_PQ · FTS on question | | `ms_marco/` | [`lance-format/ms-marco-v2.1-lance`](https://huggingface.co/datasets/lance-format/ms-marco-v2.1-lance) | passage retrieval | MiniLM query emb · IVF_PQ · FTS on query/selected_passage | | `laion-1M/` | [`lance-format/laion-1m`](https://huggingface.co/datasets/lance-format/laion-1m) | image-caption (1M) | CLIP image emb · IVF_PQ · FTS on caption | | `openvid_hf/` | [`lance-format/openvid-lance`](https://huggingface.co/datasets/lance-format/openvid-lance) | text-to-video | video blobs · 1024-d emb · IVF_PQ · FTS on caption | | `lerobot/xvla-soft-fold/` | [`lance-format/lerobot-xvla-soft-fold`](https://huggingface.co/datasets/lance-format/lerobot-xvla-soft-fold) | robotics episodes | episode video blobs | | `fineweb_edu/` | [`lance-format/fineweb-edu`](https://huggingface.co/datasets/lance-format/fineweb-edu) | text corpus | Cohere embeddings · IVF_PQ · FTS | ## Shared helpers — `_common/` - `embeddings.py` — GPU-batched OpenCLIP and sentence-transformer encoders (single-model load + L2-normalized output). - `indexing.py` — `IVF_PQ` / FTS / BTREE / BITMAP builders with sensible defaults derived from row count and embedding dim, following [docs.lancedb.com/performance](https://docs.lancedb.com/performance). - `schemas.py` — `pa.field` helpers for fixed-size embedding columns and blob-tagged fields. - `upload.py` + `upload_and_cleanup.sh` — wrap `hf upload-large-folder`, then remove the local copy on success so disk frees up between datasets. - `image_classification.py` and `image_caption.py` — generic write paths for the two most common image-dataset shapes. - `run_all.sh` — orchestrator: runs conversions sequentially (single GPU) and overlaps uploads via a `flock`-serialized queue (HF documentation discourages parallel large-folder uploads). ## Performance / best practices The shared helpers pin a few values from the [LanceDB performance guide](https://docs.lancedb.com/performance): - Iterator ingest with several-thousand-row batches (`pa.RecordBatch.from_pydict`) — one commit per dataset, not per row. - `max_bytes_per_file = 8 GiB` so individual Lance fragments stay manageable. - `IVF_PQ` partitions sized by `sqrt(num_rows)` clamped to 16/64/256/512/1024 buckets; sub-vectors at `dim/8` (8-dim PQ chunks) by default. - Embeddings are L2-normalized and indexed with `metric=cosine`. - FTS index built with `with_position=False`, `remove_stop_words=False` to keep the index small. - BITMAP for low-cardinality categorical columns (≤ ~1000 distinct values), BTREE everywhere else. - All multimodal data — image bytes, masks, embeddings — stays inline in the Lance dataset, **not** as sidecar files. ## Running a conversion ```bash # Convert a single dataset with embeddings and indices, then push to the Hub. python mnist/dataprep.py --overwrite --push # Or run several in sequence, with parallel uploads: _common/run_all.sh mnist cifar10 fashion_mnist ``` Set `HF_TOKEN` (or run `hf auth login`) before pushing to the Hub.

جارٍ تحميل ملف README للمستودع…

Skills في هذا المستودع

مطوّرو البرمجيات

١ فئات مهنية · 100% مصنفة

skill

المهنة

الوصف

آخر تحديث

hf-dataset-card-writer

مطوّرو البرمجيات

Use when creating or revising an HF_DATASET_CARD.md file in this repo. Defines the six-section card structure (Search, Curate, Evolve, Train, Versioning, Materialize a subset), the writing voice, and the example style rules for Lance-format datasets published…

لغة النص الأصلي: الإنجليزية

2026-05-18

عرض ١ من أصل ١ skills مجمعة.

التثبيت بمساعدة مساعد ذكي

انسخ هذا الطلب إلى المساعد الذكي الذي تستخدمه.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/lance-format/lance-huggingface.