AI pipeline architecture design — development pipelines, production pipelines, data stores, model registry, CI/CD for AI, and non-functional requirements. This skill should be used when the user asks to "design AI pipelines", "architect ML pipelines", "select data stores for AI", "design model registry", "implement CI/CD for ML", "define AI pipeline requirements", or mentions MLOps, training pipeline, inference pipeline, feature pipeline, Blue and Gold deployment, or pipeline patterns.
AI pipeline architecture design — development pipelines, production pipelines, data stores, model registry, CI/CD for AI, and non-functional requirements. This skill should be used when the user asks to "design AI pipelines", "architect ML pipelines", "select data stores for AI", "design model registry", "implement CI/CD for ML", "define AI pipeline requirements", or mentions MLOps, training pipeline, inference pipeline, feature pipeline, Blue and Gold deployment, or pipeline patterns.
model
opus
context
fork
allowed-tools
["Read","Write","Edit","Glob","Grep","Bash"]
AI Pipeline Architecture: Development & Production Pipelines for AI Systems
AI pipeline architecture defines how data flows through AI systems — from raw ingestion through model training and serving to production monitoring. This skill produces comprehensive pipeline architecture documentation covering development pipelines (experimentation to model artifact), production pipelines (data ingestion to prediction delivery), data store selection, model registry design, CI/CD strategy, and measurable requirements.
Principio Rector
El pipeline es la arquitectura. El modelo es solo un componente dentro de el. La mayoria del esfuerzo en sistemas de IA de produccion esta en la infraestructura de datos, no en el algoritmo. Un pipeline mal disenado convierte un buen modelo en un sistema fragil. Un pipeline bien disenado permite que modelos mediocres evolucionen.
Filosofia de Pipeline Architecture
Dos pipelines, un registro. Development pipeline y production pipeline son sistemas distintos con requisitos distintos (experimentacion vs. confiabilidad). El model registry es el puente que los conecta. Sin registro, no hay reproducibilidad.
Data quality es el primer gate, no el ultimo test. En pipelines de IA, basura entra, basura sale — con la agravante de que el modelo amplifica los sesgos de datos malos. Quality gates al inicio del pipeline, no al final.
Blue y Gold, no YOLO deploy. Desplegar un modelo nuevo directamente a produccion es irresponsable. La estrategia Blue (produccion) y Gold (staging con validacion) garantiza que ningun modelo llega a usuarios sin pasar gates automatizados.
Inputs
The user provides a system or project name as $ARGUMENTS. Parse $1 as the system/project name used throughout all output artifacts.
Designs the deployment strategy connecting development artifacts to production serving.
Blue Pipeline (Production): Currently serving, fully validated, rollback target.
Gold Pipeline (Staging): New version under validation, receives shadow/canary traffic.
Validation gates:
Model validation: accuracy, AUC, fairness, robustness meet thresholds
Feature validation: distributions match expected profiles
Data quality: input data passes schema and quality checks
Performance: latency and throughput within SLA
Security: no new vulnerabilities, access controls intact
Regression: no degradation vs. current Blue performance
Promotion flow: Gold passes all gates → canary traffic → gradual shift → full promotion → Gold becomes Blue → previous Blue archived.
Key decisions:
Canary percentage and duration
Automated vs. manual gate approval
Rollback trigger criteria
Pipeline-level vs. model-level deployment
S6: Requirements Framework (AP/NF/SEC/CP)
Defines measurable requirements across four categories with thresholds and objectives.
Performance (AP-1 to AP-13): Data processing speed, model accuracy, fairness, explainability, robustness.
Non-Functional (NF-1 to NF-9): Availability (>99.9%), recovery time (<1 min), fault detection (<0.5 secs), drift detection (<1 hour), pipeline isolation.
Security (SEC-1 to SEC-6): PKI for external interfaces, audit logging, adversarial protection, data access controls, model extraction monitoring.
Compliance (CP-1 to CP-7): Authorized data access, transaction archival, encryption at rest/in use, audit trails, model governance workflows.
Trade-off Matrix
Decision
Enables
Constrains
When to Use
Batch Pipeline
Simple, cost-effective, easy debugging
High latency, not real-time
Offline analytics, nightly retraining
Streaming Pipeline
Real-time predictions, low latency
Complex, exactly-once semantics hard
Real-time fraud, recommendations
Hybrid Pipeline
Best of both, flexible
Two systems to maintain, consistency
Most production AI systems
Feature Store
Consistency, reuse, drift monitoring
Infra overhead, governance cost
Multiple models sharing features
Blue & Gold CI/CD
Safe deployments, instant rollback
Doubled infrastructure during validation
All production AI systems
Single Model Registry
Central governance, clear lineage
Single point of failure, access bottleneck
Standard team size
Distributed Registry
Team autonomy, reduced bottleneck
Consistency challenges, governance complexity
Large multi-team orgs
Assumptions
Team has or will build experience with ML pipeline orchestration
Infrastructure supports the compute requirements for training and inference
Data sources are identified and access is available or negotiable
Model registry and CI/CD are organizational priorities (not afterthoughts)
Monitoring and observability budget is allocated
Limits
Focuses on pipeline architecture, not infrastructure provisioning (see metodologia-infrastructure-architecture)
Does not design internal module structure (see metodologia-ai-software-architecture)
Does not select design patterns or tactics (see metodologia-ai-design-patterns)
Does not define testing strategy beyond pipeline gates (see metodologia-ai-testing-strategy)
Notebook-to-Production Migration:
Data scientists work in Jupyter notebooks; production requires orchestrated pipelines. Bridge with notebook-aware orchestrators (Papermill, Ploomber). Prioritize extracting feature engineering and model training into reusable pipeline stages.
Multi-Team Pipeline Ownership:
Different teams own different pipeline stages (data eng owns ingestion, ML eng owns training, platform owns serving). Clear data contracts between stages are essential. Feature store becomes the coordination point.
Real-Time + Batch Hybrid:
System needs both real-time predictions (online serving) and batch analytics (offline scoring). Lambda or Kappa architecture patterns. Feature store must support both online (low-latency) and offline (batch) serving.
Regulated Pipeline (Finance, Healthcare):
Every pipeline stage must produce audit-worthy artifacts. Data lineage tracking from source to prediction. Model governance gates require human approval before production promotion.
Validation Gate
Before finalizing delivery, verify:
Development and production pipelines are designed as separate systems connected by model registry
Each pipeline stage has defined inputs, outputs, and quality gates
Data store selection justified by workload characteristics (not technology preference)
Model registry design includes versioning, lineage, staging, and rollback
CI/CD strategy uses Blue & Gold (or equivalent staged deployment)
Validation gates cover model quality, data quality, performance, security, and regression
Requirements framework includes all four categories (AP, NF, SEC, CP) with thresholds
Training-serving skew risk explicitly addressed in pipeline design
Monitoring strategy covers all pipeline stages (not just model serving)
Pipeline architecture is deployable (not just theoretical)
Cross-References
metodologia-ai-software-architecture: Provides module structure context; receives pipeline as component
metodologia-ai-conops: Provides operational requirements and success metrics
metodologia-ai-design-patterns: Patterns applied within pipeline stages (Feature Store, Champion-Challenger)
metodologia-ai-testing-strategy: Testing strategy for pipeline validation and integration
Primary:A-02_AI_Pipeline_Architecture_Deep.html — Development pipeline diagram, production pipeline diagram, data store selection matrix, model registry design, Blue & Gold CI/CD flow, requirements framework tables.
Secondary: Pipeline stage contracts (.md), data store comparison matrix, model registry workflow diagram (Mermaid/PNG/SVG), requirements checklist.
Fuente: Avila, R.D. & Ahmad, I. (2025). Architecting AI Software Systems. Packt.
