| name | architecture-search |
| description | This skill should be used when the user asks about "neural architecture search", "NAS", "architecture design", "model selection", "Pareto analysis", "proxy metrics", "zero-cost NAS", "search space design", "DARTS", "ENAS", "EfficientNet", "architecture efficiency", "FLOP count", "parameter count", "model capacity", or when deciding which model family to use for a given task and constraint set. |
Architecture Search — NAS and Pareto-Optimal Design
Provides systematic methodology for neural architecture search, candidate evaluation using proxy metrics, and Pareto-front analysis to balance accuracy against deployment constraints.
Architecture Search Paradigms
Manual Design (Expert-Driven)
Use domain expertise to select from known architecture families:
- Tabular: gradient boosting → MLP → attention-based (TabNet)
- Vision: ResNet → EfficientNet → ViT (scale-dependent)
- Text: LSTM → Transformer → sparse attention (long context)
When to use: Dataset is small, domain is well-understood, latency constraint is tight.
Zero-Cost NAS (Proxy-Based)
Evaluate architectures without training using proxy signals computed in seconds:
- SNIP: gradient-based pruning score at initialization — correlates with trained accuracy
- GradNorm: magnitude of gradients at init — high GradNorm → good information flow
- SynFlow: path norm proxy — avoids "layer-collapse" in pruning-based search
When to use: Tight time budget (< 1 hour), many candidates to screen.
Differentiable NAS (DARTS-style)
Relax the discrete architecture search to a continuous optimization:
- Mixed operations at each edge with learnable architecture weights α
- Jointly optimize model weights W and architecture weights α
- After training, discretize by selecting argmax over α at each edge
When to use: Custom search space, sufficient compute (DARTS requires ~4 GPU days).
Evolutionary / RL-Based NAS
- Encode architectures as genomes; evolve a population using mutation and crossover
- Or train a controller RNN to generate architecture strings (Zoph et al.)
When to use: Largest compute budget, maximum potential for novel architectures.
Pareto-Front Analysis
For each candidate architecture, evaluate on two axes:
- Accuracy proxy (e.g., zero-cost score or validation accuracy on small proxy dataset)
- Efficiency proxy (e.g., latency on target hardware, FLOP count, parameter count)
A candidate is Pareto-optimal if no other candidate is strictly better on both axes simultaneously.
Selection rules:
- If latency constraint is hard: filter to candidates meeting constraint, then pick highest accuracy
- If no hard constraints: pick the "knee" of the Pareto curve (max marginal accuracy per FLOP)
See references/nas-algorithms.md for DARTS, ENAS, and evolutionary search implementation details.
See references/search-spaces.md for domain-specific search space definitions and operator libraries.
FLOP and Parameter Estimation
Quick estimation formulas:
- Linear/FC layer: FLOPs = 2 · in_features · out_features (multiply-add)
- Conv2d: FLOPs = 2 · C_in · C_out · K² · H_out · W_out
- Multi-Head Attention: FLOPs ≈ 4 · seq_len · d_model² + 2 · seq_len² · d_model
Parameters:
- Linear(in, out): in × out + out (weights + bias)
- Conv2d(C_in, C_out, K): C_in × C_out × K² + C_out
- MHA(d_model, n_heads): 4 × d_model²
Use these to quickly screen candidate architectures before running any training.
