القائمة
Partition a large dataset into N balanced shards using integer key modulo arithmetic for reproducible, class-interleaved splits across CSV files
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
Memory-efficient Keras generator that streams sharded CSV files in chunks, renders strokes to images on-the-fly, and yields batches for training on datasets too large for memory
Stack 1D convolutions for local feature extraction before bidirectional LSTMs to classify variable-length stroke sequences into hundreds of doodle categories
Normalize raw stroke coordinates to 0-255 range, resample at uniform arc-length spacing, then apply Ramer-Douglas-Peucker simplification
Render stroke sequences to grayscale images with temporal intensity encoding where earlier strokes are brighter and later strokes fade to encode drawing order
Score each face in a video as anomalous by computing L2 distance from the embedding centroid of all faces, then convert to probability via logistic function
Extract per-frame CNN features then classify the temporal sequence with stacked GRU layers and a boolean mask for variable-length video inputs
استنادا إلى تصنيف SOC المهني
| name | cv-deterministic-hash-partitioning |
| description | Partition a large dataset into N balanced shards using integer key modulo arithmetic for reproducible, class-interleaved splits across CSV files |
When a dataset is too large to fit in memory (e.g., 50M doodles across 340 classes), deterministic hash partitioning splits it into N shards using key_id % N. Each shard gets a balanced mix of all classes, is reproducible without storing the split, and can be processed independently. Combined with streaming append, this builds sharded files without loading the full dataset.
import pandas as pd
import numpy as np
from tqdm import tqdm
N_SHARDS = 100
categories = [...] # list of 340 class names
for class_idx, category in enumerate(tqdm(categories)):
df = pd.read_csv(f"train_{category}.csv", nrows=30000)
df["label"] = class_idx
df["shard"] = (df["key_id"] // 10**7) % N_SHARDS
for k in range(N_SHARDS):
chunk = df[df["shard"] == k].drop(["key_id", "shard"], axis=1)
mode = "w" if class_idx == 0 else "a"
header = class_idx == 0
chunk.to_csv(f"train_shard_{k}.csv.gz",
mode=mode, header=header, index=False,
compression="gzip")
key_id // 10^7 % N_SHARDS// 10^7 to avoid sequential correlation