Debug and inspect a dlt SQL database pipeline after running it. Use after a pipeline run (success or failure) to inspect traces, load packages, schema, and diagnose errors like connection failures, missing credentials, driver issues, or failed jobs.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Debug and inspect a dlt SQL database pipeline after running it. Use after a pipeline run (success or failure) to inspect traces, load packages, schema, and diagnose errors like connection failures, missing credentials, driver issues, or failed jobs.
Check that table_names is set on the source, not filtered via .with_resources().
sql_database() without table_names reflects the entire schema before filtering — slow on large databases. Always pass the tables you need upfront:
# good — reflects only these two tables
source = sql_database(table_names=["family", "clan"])
# slow — reflects all tables in the schema first, then filters
source = sql_database().with_resources("family", "clan")
Check the backend — this is the most common cause of slow pipelines.
The default sqlalchemy backend fetches rows one at a time through Python objects. Switch to a faster backend:
Backend
Speed
Notes
sqlalchemy
baseline
Default; safe but slow
pyarrow
~20–30x faster
Best general upgrade; also needs numpy (uv add numpy)
connectorx
~2x faster than pyarrow
Rust-based; great for large MySQL/PostgreSQL tables; uses its own connection string format
Apply by passing backend= to sql_table or sql_database:
source = sql_database(table_names=["orders"], backend="pyarrow")
# or
source = sql_database(table_names=["orders"], backend="connectorx")
uv run dlthub local pipeline trace <pipeline_name> -vv
Shows credentials resolution, step timing, and failures.
Load packages
Each pipeline run generated one or more load packages. Use trace tool to find their ids.
uv run dlthub local pipeline load-package <pipeline_name> -v # most recent package
uv run dlthub local pipeline load-package <pipeline_name> <load_id> -v # specific package
Shows package state, per-job details (table, file type, size, timing), and error messages for failed jobs. With -v also shows schema updates applied.
uv run dlthub local pipeline failed-jobs <pipeline_name>
Scans all packages for failed jobs and displays error messages from the destination.
Inspecting raw load files
Load packages are stored at ~/.dlt/pipelines/<pipeline_name>/load/loaded/<load_id>/. Job files live in completed_jobs/ and failed_jobs/ subdirectories.
File format depends on the destination:
Format
Default for
File extension
INSERT VALUES
duckdb, postgres, redshift, mssql, motherduck
.insert_values.gz
JSONL
bigquery, snowflake, filesystem
.jsonl.gz
Parquet
athena, databricks (also supported by duckdb, bigquery, snowflake)
Useful for verifying data transformations and debugging destination errors.
Clean up after debugging
Before moving on, revert all debugging settings YOU introduced:
.dlt/config.toml — restore log_level to its previous value (or remove if you added it)
Pipeline script — remove progress="log" from dlt.pipeline() if you added it. Remove .add_limit(N) if you added it for debugging.
Do NOT remove settings the user had before you started.
Next steps
Load successful → use validate-data to inspect schema and data, or hand over to explore-data (data-exploration toolkit) to jump straight into charts and analysis
Config/secrets missing → revisit create-sql-database-pipeline — "Set up config and secrets" section
No pipeline exists yet → use create-sql-database-pipeline to scaffold one first
