// ALWAYS read and follow this skill before acting. SQL database pipeline
| name | sql-database-pipeline-workflow |
| description | ALWAYS read and follow this skill before acting. SQL database pipeline |
ALWAYS start with Find source (find-source) SKILL — identify the database, explore available tables, and pick what to load
find-source) — classify the database type, explore schemas and tables, gather connection details, pick first table and destinationcreate-sql-database-pipeline) — scaffold with dlt init sql_database, write code, set up credentials, test load, choose backenddebug-pipeline) — run it, inspect traces and load packages, fix connection or driver errorsvalidate-data) — inspect schema and data, fix types and column mappings, iterate until correctexplore_db.py, inspect_schema.py, any throwaway .py files created during exploration). Keep only the pipeline script and config files.adjust-table) — remove dev limits, add incremental loading with a cursor column, configure merge keys, fix column types and schemaadd-table) — add more tables or views from the same database into the pipelinequery_adapter_callback to filter rows at SQL level, table_adapter_callback to modify schema, or add_map to transform rows after extraction; see create-sql-database-pipeline — "Add transformation callbacks" sectionview-data) — query and explore loaded data using dlt dataset API, ibis, or raw SQLdeploy-workspace when the pipeline needs modification before deploying) — pipeline name and destination are already known; skip find-source discovery and go straight to the relevant fix skill (debug-pipeline, adjust-table, or add-table).When the user's needs go beyond this toolkit, hand over to:
validate-data or view-data, when the user wants interactive notebooks, charts, dashboards, or deeper analysis with marimovalidate-data or view-data, when the user wants to model the ingested data into a CDM or run cross-source transformationsvalidate-data, when the user wants ongoing validation, check contracts, or quality guarantees on every pipeline loadcreate-sql-database-pipeline or debug-pipeline): when the user wants to run the pipeline on dltHub right away — a working pipeline is enough to deployadjust-table, incremental loading, add-table, or a subsequent debug-pipeline run): when the pipeline is refined and the user wants to deploy or schedule it on dltHubALWAYS read and follow this skill before acting. Data quality conventions
ALWAYS read and follow this skill before acting. Data quality workflow
Deploy dlt pipelines to dltHub Platform. Use when the user says "deploy to dltHub", "launch on dltHub", "run on dltHub", "schedule pipeline", or wants to deploy a pipeline or notebook to dltHub.
ALWAYS read and follow this skill before acting. Profiles
ALWAYS read and follow this skill before acting. Deploy to dltHub Platform
ALWAYS read and follow this skill before acting. Filesystem pipeline workflow