| name | Incremental Model Strategy Selector |
| description | Selects and configures optimal incremental model strategies |
| version | 1.0.0 |
| category | Transformation |
| skillId | SK-DEA-019 |
| allowed-tools | ["Read","Write","Edit","Glob","Grep","Bash"] |
| graph | {"domains":["domain:data-engineering"],"specializations":["specialization:data-engineering-analytics"],"skillAreas":["skill-area:dbt-modeling","skill-area:etl-pipelines"],"roles":["role:analytics-engineer","role:data-engineer"],"workflows":["workflow:data-pipeline-deployment"]} |
Incremental Model Strategy Selector
Overview
Selects and configures optimal incremental model strategies. This skill optimizes data transformation efficiency through proper incremental processing patterns.
Capabilities
- Incremental strategy selection (append, merge, delete+insert)
- Partition pruning optimization
- Unique key configuration
- On_schema_change handling
- Full refresh scheduling
- Lookback window optimization
- Late-arriving data handling
Input Schema
{
"modelCharacteristics": {
"sourceType": "string",
"updatePattern": "append|update|delete",
"volumeGB": "number",
"updateFrequency": "string"
},
"platform": "snowflake|bigquery|redshift",
"existingModel": "object"
}
Output Schema
{
"strategy": "append|merge|delete+insert",
"config": "object",
"partitionStrategy": "object",
"refreshSchedule": "object",
"dbtConfig": "object"
}
Target Processes
- Incremental Model Setup
- dbt Model Development
- Pipeline Migration
Usage Guidelines
- Analyze source data update patterns
- Measure data volume and update frequency
- Select strategy based on characteristics
- Configure appropriate lookback windows
Best Practices
- Use append for insert-only sources
- Use merge for sources with updates
- Configure partition pruning for large tables
- Schedule periodic full refreshes for data correction
- Handle late-arriving data with appropriate lookback