메뉴
Partition a large dataset into N balanced shards using integer key modulo arithmetic for reproducible, class-interleaved splits across CSV files
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Memory-efficient Keras generator that streams sharded CSV files in chunks, renders strokes to images on-the-fly, and yields batches for training on datasets too large for memory
Stack 1D convolutions for local feature extraction before bidirectional LSTMs to classify variable-length stroke sequences into hundreds of doodle categories
Normalize raw stroke coordinates to 0-255 range, resample at uniform arc-length spacing, then apply Ramer-Douglas-Peucker simplification
Render stroke sequences to grayscale images with temporal intensity encoding where earlier strokes are brighter and later strokes fade to encode drawing order
Score each face in a video as anomalous by computing L2 distance from the embedding centroid of all faces, then convert to probability via logistic function
Extract per-frame CNN features then classify the temporal sequence with stacked GRU layers and a boolean mask for variable-length video inputs
| name | cv-deterministic-hash-partitioning |
| description | Partition a large dataset into N balanced shards using integer key modulo arithmetic for reproducible, class-interleaved splits across CSV files |
When a dataset is too large to fit in memory (e.g., 50M doodles across 340 classes), deterministic hash partitioning splits it into N shards using key_id % N. Each shard gets a balanced mix of all classes, is reproducible without storing the split, and can be processed independently. Combined with streaming append, this builds sharded files without loading the full dataset.
import pandas as pd
import numpy as np
from tqdm import tqdm
N_SHARDS = 100
categories = [...] # list of 340 class names
for class_idx, category in enumerate(tqdm(categories)):
df = pd.read_csv(f"train_{category}.csv", nrows=30000)
df["label"] = class_idx
df["shard"] = (df["key_id"] // 10**7) % N_SHARDS
for k in range(N_SHARDS):
chunk = df[df["shard"] == k].drop(["key_id", "shard"], axis=1)
mode = "w" if class_idx == 0 else "a"
header = class_idx == 0
chunk.to_csv(f"train_shard_{k}.csv.gz",
mode=mode, header=header, index=False,
compression="gzip")
key_id // 10^7 % N_SHARDS// 10^7 to avoid sequential correlation