data-engineering

name	data-engineering
description	Data engineering, machine learning, AI, and MLOps. From data pipelines to production ML systems and LLM applications.
triggers	["data engineering","machine learning","ml","ai","mlops","spark","airflow","llm","rag","langchain"]
parameters	{"role":{"type":"string","enum":["data-engineer","ml-engineer","ai-engineer"],"required":true},"experience":{"type":"string","enum":["beginner","intermediate","advanced"],"required":false,"default":"beginner"}}
outputs	{"learning_path":{"type":"array"},"tech_stack":{"type":"object"},"projects":{"type":"array"}}
retry	{"max_attempts":3,"backoff":"exponential"}
observability	{"log_level":"info","metrics":["path_completion_rate"]}
level	advanced
prerequisites	["programming-basics","python-advanced"]
sasmp_version	1.3.0
bonded_agent	01-core-paths
bond_type	PRIMARY_BOND

Data Engineering Skill

Quick Reference

Role	Focus	Timeline	Entry From
Data Engineer	Pipelines, Infra	12-24 mo	Backend Dev
ML Engineer	Models, Features	12-24 mo	Data Scientist
AI Engineer	LLMs, Agents	6-12 mo	Any Developer

---

Learning Paths

Data Engineer

[1] SQL Mastery (4-6 wk)
 │  └─ Window functions, CTEs, optimization
 │
 ▼
[2] Python for Data (4-6 wk)
 │  └─ Pandas, file formats, scripting
 │
 ▼
[3] ETL/ELT Pipelines (6-8 wk)
 │  └─ Extract, transform, load patterns
 │
 ▼
[4] Big Data: Spark (8-12 wk)
 │  └─ PySpark, DataFrames, partitioning
 │
 ▼
[5] Data Warehouse (4-6 wk)
 │  └─ Star schema, dbt, Snowflake/BQ
 │
 ▼
[6] Orchestration (4-6 wk)
    └─ Airflow/Prefect, scheduling, monitoring

2025 Stack: Python + Spark + Airflow + dbt + Snowflake/BigQuery

---

ML Engineer

[1] Python + NumPy (4-6 wk)
 │
 ▼
[2] Math Foundations (6-8 wk)
 │  └─ Linear algebra, calculus, statistics
 │
 ▼
[3] Classical ML (8-12 wk)
 │  └─ scikit-learn, XGBoost, evaluation
 │
 ▼
[4] Deep Learning (8-12 wk)
 │  └─ PyTorch, CNNs, Transformers
 │
 ▼
[5] MLOps (6-8 wk)
    └─ MLflow, model serving, monitoring

2025 Stack: Python + PyTorch + scikit-learn + MLflow + W&B

---

AI Engineer (2025 Hot Path)

[1] LLM Fundamentals (2-3 wk)
 │  └─ Tokens, embeddings, context windows
 │
 ▼
[2] Prompt Engineering (2-3 wk)
 │  └─ Few-shot, CoT, structured output
 │
 ▼
[3] RAG Systems (3-4 wk)
 │  └─ Embeddings, vector DBs, retrieval
 │
 ▼
[4] AI Agents (4-6 wk)
 │  └─ Tool calling, agent loops, memory
 │
 ▼
[5] Production Deploy (ongoing)
    └─ Evaluation, guardrails, monitoring

2025 Stack: Python + LangChain/LlamaIndex + OpenAI/Anthropic + ChromaDB

---

2025 Tool Matrix

Data Processing

Tool	Scale	Use Case
Pandas	<10GB	Prototyping, small data
Polars	<100GB	Fast local processing