using-neural-architectures

name	using-neural-architectures
description	The architecture selection router for CNNs, Transformers, RNNs, GANs, GNNs by data modality and constraints

Using Neural Architectures: Architecture Selection Router

<CRITICAL_CONTEXT> Architecture selection comes BEFORE training optimization. Wrong architecture = no amount of training will fix it.

This meta-skill routes you to the right architecture guidance based on:

• Data modality (images, sequences, graphs, etc.)
• Problem type (classification, generation, regression)
• Constraints (data size, compute, latency, interpretability)

Load this skill when architecture decisions are needed. </CRITICAL_CONTEXT>

When to Use This Skill

Use this skill when:

• ✅ Selecting an architecture for a new problem
• ✅ Comparing architecture families (CNN vs Transformer, RNN vs Transformer, etc.)
• ✅ Designing custom network topology
• ✅ Troubleshooting architectural instability (deep networks, gradient issues)
• ✅ Understanding when to use specialized architectures (GNNs, generative models)

DO NOT use for:

• ❌ Training/optimization issues (use training-optimization pack)
• ❌ PyTorch implementation details (use pytorch-engineering pack)
• ❌ Production deployment (use ml-production pack)

When in doubt: If choosing WHAT architecture → this skill. If training/deploying architecture → different pack.

---

How to Access Reference Sheets

IMPORTANT: All reference sheets are located in the SAME DIRECTORY as this SKILL.md file.

When this skill is loaded from: skills/using-neural-architectures/SKILL.md

Reference sheets like cnn-families-and-selection.md are at: skills/using-neural-architectures/cnn-families-and-selection.md

NOT at: skills/cnn-families-and-selection.md ← WRONG PATH

When you see a link like [cnn-families-and-selection.md](cnn-families-and-selection.md), read the file from the same directory as this SKILL.md.

---

Core Routing Logic

Step 1: Identify Data Modality

Question to ask: "What type of data are you working with?"

Data Type	Route To	Why
Images (photos, medical scans, etc.)	cnn-families-and-selection.md	CNNs excel at spatial hierarchies
Sequences (time series, text, audio)	sequence-models-comparison.md	Temporal dependencies need sequential models
Graphs (social networks, molecules)	graph-neural-networks-basics.md	Graph structure requires GNNs
Generation task (create images, text)	generative-model-families.md	Generative models are specialized
Multiple modalities (text + images)	architecture-design-principles.md	Need custom design
Unclear / Generic	architecture-design-principles.md	Start with fundamentals

Step 2: Check for Special Requirements

If any of these apply, address FIRST:

Requirement	Route To	Priority
Deep network (> 20 layers) unstable	normalization-techniques.md	CRITICAL - fix before continuing
Need attention mechanisms	attention-mechanisms-catalog.md	Specialized component
Custom architecture design	architecture-design-principles.md	Foundation before specifics
Transformer-specific question	transformer-architecture-deepdive.md	Specialized architecture

Step 3: Consider Problem Characteristics

Clarify BEFORE routing:

Ask:

• "How large is your dataset?" (Small < 10k, Medium 10k-1M, Large > 1M)
• "What are your computational constraints?" (Edge device, cloud, GPU availability)
• "What are your latency requirements?" (Real-time, batch, offline)
• "Do you need interpretability?" (Clinical, research, production)

These answers determine architecture appropriateness.

---

Routing by Data Modality

Images → CNN Families

Symptoms triggering this route:

• "classify images"
• "object detection"
• "semantic segmentation"
• "medical imaging"
• "computer vision"

Route to: See cnn-families-and-selection.md for CNN architecture selection and comparison.

When to route here:

• ANY vision task (CNNs are default for spatial data)
• Even if considering Transformers, check CNN families first (often better with less data)

Clarifying questions:

• "Dataset size?" (< 10k → Start with proven CNNs, > 100k → Consider ViT)
• "Deployment target?" (Edge → EfficientNet, Cloud → Anything)
• "Task type?" (Classification → ResNet/EfficientNet, Detection → YOLO/Faster-RCNN)

---

Sequences → Sequence Models Comparison

Symptoms triggering this route:

• "time series"
• "forecasting"
• "natural language" (NLP)
• "sequential data"
• "temporal patterns"
• "RNN vs LSTM vs Transformer"

Route to: See sequence-models-comparison.md for sequential model selection (RNN, LSTM, Transformer, TCN).

When to route here:

• ANY sequential data
• When user asks "RNN vs LSTM" (skill will present modern alternatives)
• Time-dependent patterns

Clarifying questions:

• "Sequence length?" (< 100 → RNN/LSTM/TCN, 100-1000 → Transformer, > 1000 → Sparse Transformers)
• "Latency requirements?" (Real-time → TCN/LSTM, Offline → Transformer)
• "Data volume?" (Small → Simpler models, Large → Transformers)

CRITICAL: Challenge "RNN vs LSTM" premise if they ask. Modern alternatives (Transformers, TCN) often better.

---

Graphs → Graph Neural Networks

Symptoms triggering this route:

• "social network"
• "molecular structure"
• "knowledge graph"
• "graph data"
• "node classification"
• "link prediction"
• "graph embeddings"

Route to: See graph-neural-networks-basics.md for GNN architectures and graph learning.

When to route here:

• Data has explicit graph structure (nodes + edges)
• Relational information is important
• Network topology matters

Red flag: If treating graph as tabular data (extracting features and ignoring edges) → WRONG. Route to GNN skill.

---

Generation → Generative Model Families

Symptoms triggering this route:

• "generate images"
• "synthesize data"
• "GAN vs VAE vs Diffusion"
• "image-to-image translation"
• "style transfer"
• "generative modeling"

Route to: See generative-model-families.md for GANs, VAEs, and Diffusion models.

When to route here:

• Goal is to CREATE data, not classify/predict
• Need to sample from distribution
• Data augmentation through generation

Clarifying questions:

• "Use case?" (Real-time game → GAN, Art/research → Diffusion, Fast training → VAE)
• "Quality vs speed?" (Quality → Diffusion, Speed → GAN)
• "Controllability?" (Fine control → StyleGAN/Conditional models)

CRITICAL: Different generative models have VERY different trade-offs. Must clarify requirements.

---

Routing by Architecture Component

Attention Mechanisms

Symptoms triggering this route:

• "when to use attention"
• "self-attention vs cross-attention"
• "attention in CNNs"
• "attention bottleneck"
• "multi-head attention"

Route to: See attention-mechanisms-catalog.md for attention mechanism selection and design.

When to route here:

• Designing custom architecture that might benefit from attention
• Understanding where attention helps vs hinders
• Comparing attention variants

NOT for: General Transformer questions → transformer-architecture-deepdive.md instead

---

Transformer Deep Dive

Symptoms triggering this route:

• "how do transformers work"
• "Vision Transformer (ViT)"
• "BERT architecture"
• "positional encoding"
• "transformer blocks"
• "scaling transformers"

Route to: See transformer-architecture-deepdive.md for Transformer internals and implementation.

When to route here:

• Implementing/customizing transformers
• Understanding transformer internals
• Debugging transformer-specific issues

Cross-reference:

• For sequence models generally → sequence-models-comparison.md (includes transformers in context)
• For LLMs specifically → yzmir/llm-specialist/transformer-for-llms (LLM-specific transformers)

---

Normalization Techniques

Symptoms triggering this route:

• "gradient explosion"
• "training instability in deep network"
• "BatchNorm vs LayerNorm"
• "normalization layers"
• "50+ layer network won't train"

Route to: See normalization-techniques.md for deep network stability and normalization methods.

When to route here:

• Deep networks (> 20 layers) with training instability
• Choosing between normalization methods
• Architectural stability issues

CRITICAL: This is often the ROOT CAUSE of "training won't work" - fix architecture before blaming hyperparameters.

---

Architecture Design Principles

Symptoms triggering this route:

• "how to design architecture"
• "architecture best practices"
• "when to use skip connections"
• "how deep should network be"
• "custom architecture for [novel task]"
• Unclear problem modality

Route to: See architecture-design-principles.md for custom architecture design fundamentals.

When to route here:

• Designing custom architectures
• Novel problems without established architecture
• Understanding WHY architectures work
• User is unsure what modality/problem type they have

This is the foundational skill - route here if other specific skills don't match.

---

Multi-Modal / Cross-Pack Routing

When Problem Spans Multiple Modalities

Example: "Text + image classification" (multimodal)

Route to BOTH:

1. sequence-models-comparison.md (for text)
2. cnn-families-and-selection.md (for images)
3. architecture-design-principles.md (for fusion strategy)

Order matters: Understand individual modalities BEFORE fusion.

When Architecture + Other Concerns

Example: "Select architecture AND optimize training"

Route order:

1. Architecture skill FIRST (this pack)
2. Training-optimization SECOND (after architecture chosen)

Why: Wrong architecture can't be fixed by better training.

Example: "Select architecture AND deploy efficiently"

Route order:

1. Architecture skill FIRST
2. ML-production SECOND (quantization, serving)

Deployment constraints might influence architecture choice - if so, note constraints during architecture selection.

---

Common Routing Mistakes (DON'T DO THESE)

Symptom	Wrong Route	Correct Route	Why
"My transformer won't train"	transformer-architecture-deepdive.md	training-optimization	Training issue, not architecture understanding
"Deploy image classifier"	cnn-families-and-selection.md	ml-production	Deployment, not selection
"ViT vs ResNet for medical imaging"	transformer-architecture-deepdive.md	cnn-families-and-selection.md	Comparative selection, not single architecture detail
"Implement BatchNorm in PyTorch"	normalization-techniques.md	pytorch-engineering	Implementation, not architecture concept
"GAN won't converge"	generative-model-families.md	training-optimization	Training stability, not architecture selection
"Which optimizer for CNN"	cnn-families-and-selection.md	training-optimization	Optimization, not architecture

Rule: Architecture pack is for CHOOSING and DESIGNING architectures. Training/deployment/implementation are other packs.

---

Red Flags: Stop and Clarify

If query contains these patterns, ASK clarifying questions before routing:

Pattern	Why Clarify	What to Ask
"Best architecture for X"	"Best" depends on constraints	"What are your data size, compute, and latency constraints?"
Generic problem description	Can't route without modality	"What type of data? (images, sequences, graphs, etc.)"
Latest trend mentioned (ViT, Diffusion)	Recency bias risk	"Have you considered alternatives? What are your specific requirements?"
"Should I use X or Y"	May be wrong question	"What's the underlying problem? There might be option Z."
Very deep network (> 50 layers)	Likely needs normalization first	"Are you using normalization layers? Skip connections?"

Never guess modality or constraints. Always clarify.

---

Recency Bias: Resistance Table

Trendy Architecture	When NOT to Use	Better Alternative
Vision Transformers (ViT)	Small datasets (< 10k images)	CNNs (ResNet, EfficientNet)
Vision Transformers (ViT)	Edge deployment (latency/power)	EfficientNets, MobileNets
Transformers (general)	Very small datasets	RNNs, CNNs (less capacity, less overfit)
Diffusion Models	Real-time generation needed	GANs (1 forward pass vs 50-1000 steps)
Diffusion Models	Limited compute for training	VAEs (faster training)
Graph Transformers	Small graphs (< 100 nodes)	Standard GNNs (GCN, GAT) simpler and effective
LLMs (GPT-style)	< 1M tokens of training data	Simpler language models or fine-tuning

Counter-narrative: "New architecture ≠ better for your use case. Match architecture to constraints."

---

Decision Tree

Start here: What's your primary goal?

┌─ SELECT architecture for task
│  ├─ Data modality?
│  │  ├─ Images → [cnn-families-and-selection.md](cnn-families-and-selection.md)
│  │  ├─ Sequences → [sequence-models-comparison.md](sequence-models-comparison.md)
│  │  ├─ Graphs → [graph-neural-networks-basics.md](graph-neural-networks-basics.md)
│  │  ├─ Generation → [generative-model-families.md](generative-model-families.md)
│  │  └─ Unknown/Multiple → [architecture-design-principles.md](architecture-design-principles.md)
│  └─ Special requirements?
│     ├─ Deep network (>20 layers) unstable → [normalization-techniques.md](normalization-techniques.md) (CRITICAL)
│     ├─ Need attention mechanism → [attention-mechanisms-catalog.md](attention-mechanisms-catalog.md)
│     └─ None → Proceed with modality-based route
│
├─ UNDERSTAND specific architecture
│  ├─ Transformers → [transformer-architecture-deepdive.md](transformer-architecture-deepdive.md)
│  ├─ Attention → [attention-mechanisms-catalog.md](attention-mechanisms-catalog.md)
│  ├─ Normalization → [normalization-techniques.md](normalization-techniques.md)
│  └─ General principles → [architecture-design-principles.md](architecture-design-principles.md)
│
├─ DESIGN custom architecture
│  └─ [architecture-design-principles.md](architecture-design-principles.md) (start here always)
│
└─ COMPARE architectures
   ├─ CNNs (ResNet vs EfficientNet) → [cnn-families-and-selection.md](cnn-families-and-selection.md)
   ├─ Sequence models (RNN vs Transformer) → [sequence-models-comparison.md](sequence-models-comparison.md)
   ├─ Generative (GAN vs Diffusion) → [generative-model-families.md](generative-model-families.md)
   └─ General comparison → [architecture-design-principles.md](architecture-design-principles.md)

---

Workflow

1. Clarify → Data modality, task type, dataset size, compute/latency constraints
2. Route by modality → Images/Sequences/Graphs/Generation → appropriate skill
3. Check critical issues → Deep network unstable? Fix normalization FIRST
4. Apply architecture skill → Follow routed skill, consider trade-offs
5. Cross-pack → training-optimization (training) or ml-production (deployment)

---

Rationalization Table

Rationalization	Reality	Counter
"Transformers are SOTA, recommend them"	SOTA on benchmark ≠ best for user's constraints	"Ask about dataset size and compute first"
"User said RNN vs LSTM, answer that"	Question premise might be outdated	"Challenge: Have you considered Transformers or TCN?"
"Just recommend latest architecture"	Latest ≠ appropriate	"Match architecture to requirements, not trends"
"Architecture doesn't matter, training matters"	Wrong architecture can't be fixed by training	"Architecture is foundation - get it right first"
"They seem rushed, skip clarification"	Wrong route wastes more time than clarification	"30 seconds to clarify saves hours of wasted effort"
"Generic architecture advice is safe"	Generic = useless for specific domains	"Route to domain-specific skill for actionable guidance"

---

Integration with Other Packs

After Architecture Selection

Once architecture is chosen, route to:

Training the architecture: → yzmir/training-optimization/using-training-optimization

• Optimizer selection
• Learning rate schedules
• Debugging training issues

Implementing in PyTorch: → yzmir/pytorch-engineering/using-pytorch-engineering

• Module design patterns
• Performance optimization
• Custom components

Deploying to production: → yzmir/ml-production/using-ml-production

• Model serving
• Quantization
• Inference optimization

Dynamic/growing architectures: → yzmir/dynamic-architectures/using-dynamic-architectures

• Networks that grow, prune, or adapt during training
• Continual learning and catastrophic forgetting prevention
• Module lifecycle management and progressive training

Before Architecture Selection

If problem involves:

Reinforcement learning: → yzmir/deep-rl/using-deep-rl FIRST

• RL algorithms dictate architecture requirements
• Value networks vs policy networks have different needs

Large language models: → yzmir/llm-specialist/using-llm-specialist FIRST

• LLM architectures are specialized transformers
• Different considerations than general sequence models

Architecture is downstream of algorithm choice in RL and LLMs.

---

Summary

Use this meta-skill to:

• ✅ Route architecture queries to appropriate specialized skill
• ✅ Identify data modality and problem type
• ✅ Clarify constraints before recommending
• ✅ Resist recency bias (latest ≠ best)
• ✅ Recognize when architecture is the problem (vs training/implementation)

Neural Architecture Specialist Skills

After routing, load the appropriate specialist skill for detailed guidance:

1. architecture-design-principles.md - Custom design, architectural best practices, skip connections, network depth fundamentals
2. attention-mechanisms-catalog.md - Self-attention, cross-attention, multi-head attention, attention in CNNs, attention variants comparison
3. cnn-families-and-selection.md - ResNet, EfficientNet, MobileNet, YOLO, computer vision architecture selection
4. generative-model-families.md - GANs, VAEs, Diffusion models, image generation, style transfer, generative modeling trade-offs
5. graph-neural-networks-basics.md - GCN, GAT, node classification, link prediction, graph embeddings, molecular structures
6. normalization-techniques.md - BatchNorm, LayerNorm, GroupNorm, training stability for deep networks (>20 layers)
7. sequence-models-comparison.md - RNN, LSTM, Transformer, TCN comparison, time series, NLP, sequential data
8. transformer-architecture-deepdive.md - Transformer internals, ViT, BERT, positional encoding, scaling transformers

Critical principle: Architecture comes BEFORE training. Get this right first.