Exécutez n'importe quel Skill dans Manus
en un clic

Exécutez n'importe quel Skill dans Manus en un clic

Commencer

data-pipelines

Étoiles5

Forks0

Mis à jour1 juin 2026 à 21:46

Build reliable, resumable data connectors and transforms (ETL)

Installation

Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.

Exécuter dans Manus

Source

New1Direction

New1Direction/korgex

Ouvrir le dépôt GitHub Voir les dépôts du créateur

Téléchargement

Exécuter dans Manus

Métiers associésSOC

Basé sur la classification professionnelle SOC

Développeurs de logicielsProfessions informatiques et mathématiques·SOC 15-1252

SKILL.md

readonly

name	data-pipelines
description	Build reliable, resumable data connectors and transforms (ETL)
version	1
trust	built-in

A data pipeline moves/transforms data between systems. The failure modes are about correctness and recovery, not just throughput.

Idempotent + resumable. Pipelines fail partway. Design so re-running produces the same result (upsert by key, not blind insert) and can resume from a checkpoint rather than restarting from zero.
Validate at the edges. Check schema/types/ranges on ingest; decide up front what to do with bad records — reject, quarantine, or repair — never silently drop.
Idempotency keys + watermarks. Track what's been processed (a high-water mark or per-record key) so incremental runs don't double-count or miss late data.
Batch sensibly. Process in chunks with bounded memory; don't load an entire dataset into RAM. Stream where you can.
Make it observable. Record counts in/out/rejected per stage; a pipeline that silently processes 0 rows is a common, costly bug. Alert on anomalies.
Schema evolution. Upstream schemas change — handle added/removed fields gracefully (defaults, versioning) rather than crashing the whole run.
Separate extract / transform / load so each stage is testable in isolation, and keep raw inputs so you can reprocess after a transform bug.

Red flag: a non-resumable pipeline that must restart from scratch on any failure, or one with no record counts so you can't tell correct from silently-empty.

Plus depuis ce dépôt

même dépôt

react-doctor

New1Direction/korgex

Catch bad React in your changes — run react-doctor's deterministic scan and fix what it flags

2026-06-055

security-scan

New1Direction/korgex

Run a verifiable security scan (vulns, secrets, misconfig) over code you wrote or deps you added, before calling the work done

2026-06-035

browser-automation

New1Direction/korgex

Drive a real browser verifiably — navigate, snapshot, act BY INDEX, extract; every step provable

2026-06-025

verifiable-orchestration

New1Direction/korgex

Run multi-agent work as ONE provable causal DAG — parallel subagents, an Orchestrate graph, an immutable spec-seed

2026-06-025

building-an-mcp-server

New1Direction/korgex

Author or connect a Model Context Protocol server so an agent gains new tools

2026-06-015

ci-cd-pipelines

New1Direction/korgex

Build automated pipelines that gate merges and ship reliably

2026-06-015

name	data-pipelines
description	Build reliable, resumable data connectors and transforms (ETL)
version	1
trust	built-in

A data pipeline moves/transforms data between systems. The failure modes are about correctness and recovery, not just throughput.

Idempotent + resumable. Pipelines fail partway. Design so re-running produces the same result (upsert by key, not blind insert) and can resume from a checkpoint rather than restarting from zero.
Validate at the edges. Check schema/types/ranges on ingest; decide up front what to do with bad records — reject, quarantine, or repair — never silently drop.
Idempotency keys + watermarks. Track what's been processed (a high-water mark or per-record key) so incremental runs don't double-count or miss late data.
Batch sensibly. Process in chunks with bounded memory; don't load an entire dataset into RAM. Stream where you can.
Make it observable. Record counts in/out/rejected per stage; a pipeline that silently processes 0 rows is a common, costly bug. Alert on anomalies.
Schema evolution. Upstream schemas change — handle added/removed fields gracefully (defaults, versioning) rather than crashing the whole run.
Separate extract / transform / load so each stage is testable in isolation, and keep raw inputs so you can reprocess after a transform bug.

Red flag: a non-resumable pipeline that must restart from scratch on any failure, or one with no record counts so you can't tell correct from silently-empty.