菜单
Memory-efficient Keras generator that streams sharded CSV files in chunks, renders strokes to images on-the-fly, and yields batches for training on datasets too large for memory
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | cv-chunked-csv-image-generator |
| description | Memory-efficient Keras generator that streams sharded CSV files in chunks, renders strokes to images on-the-fly, and yields batches for training on datasets too large for memory |
When training on millions of images stored as raw data in CSV files (stroke coordinates, pixel arrays), loading everything into memory is infeasible. A streaming generator reads sharded CSVs in chunks, converts each chunk to images on-the-fly, and yields (X, y) batches. Random shard permutation ensures epoch-level shuffling without random access to the full dataset.
import pandas as pd
import numpy as np
import json
def image_generator(shard_ids, batch_size, size=64, n_classes=340):
while True:
for k in np.random.permutation(shard_ids):
filename = f"train_shard_{k}.csv.gz"
for chunk in pd.read_csv(filename, chunksize=batch_size):
chunk["drawing"] = chunk["drawing"].apply(json.loads)
X = np.zeros((len(chunk), size, size, 1), dtype=np.float32)
for i, strokes in enumerate(chunk["drawing"]):
X[i, :, :, 0] = render_strokes(strokes, size)
X = X / 255.0
y = np.eye(n_classes)[chunk["label"].values]
yield X, y
train_gen = image_generator(range(100), batch_size=256)
model.fit(train_gen, steps_per_epoch=800, epochs=50)
batch_size rowssteps_per_epochtf.data.Dataset.from_generator().prefetch(2) for GPU pipeline overlapworkers=4, use_multiprocessing=True if rendering is CPU-boundStack 1D convolutions for local feature extraction before bidirectional LSTMs to classify variable-length stroke sequences into hundreds of doodle categories
Partition a large dataset into N balanced shards using integer key modulo arithmetic for reproducible, class-interleaved splits across CSV files
Normalize raw stroke coordinates to 0-255 range, resample at uniform arc-length spacing, then apply Ramer-Douglas-Peucker simplification
Render stroke sequences to grayscale images with temporal intensity encoding where earlier strokes are brighter and later strokes fade to encode drawing order
Score each face in a video as anomalous by computing L2 distance from the embedding centroid of all faces, then convert to probability via logistic function
Extract per-frame CNN features then classify the temporal sequence with stacked GRU layers and a boolean mask for variable-length video inputs