Using Neural Architectures

Using Neural Architectures: Architecture Selection Router

This meta-skill routes you to the right architecture guidance based on:

• Data modality (images, sequences, graphs, etc.)
• Problem type (classification, generation, regression)
• Constraints (data size, compute, latency, interpretability)

Load this skill when architecture decisions are needed.

When to Use This Skill

Use this skill when:

• ✅ Selecting an architecture for a new problem
• ✅ Comparing architecture families (CNN vs Transformer, RNN vs Transformer, etc.)
• ✅ Designing custom network topology
• ✅ Troubleshooting architectural instability (deep networks, gradient issues)
• ✅ Understanding when to use specialized architectures (GNNs, generative models)

DO NOT use for:

• ❌ Training/optimization issues (use training-optimization pack)
• ❌ PyTorch implementation details (use pytorch-engineering pack)
• ❌ Production deployment (use ml-production pack)

When in doubt: If choosing WHAT architecture → this skill. If training/deploying architecture → different pack.

How to Access Reference Sheets

IMPORTANT: All reference sheets are located in the SAME DIRECTORY as this SKILL.md file.

When this skill is loaded from: skills/using-neural-architectures/SKILL.md

Reference sheets like cnn-families-and-selection.md are at: skills/using-neural-architectures/cnn-families-and-selection.md

NOT at: skills/cnn-families-and-selection.md ← WRONG PATH

When you see a link like [cnn-families-and-selection.md](cnn-families-and-selection.md), read the file from the same directory as this SKILL.md.

Core Routing Logic

Step 1: Identify Data Modality

Question to ask: "What type of data are you working with?"

Step 2: Check for Special Requirements

If any of these apply, address FIRST:

Step 3: Consider Problem Characteristics

Clarify BEFORE routing:

Ask:

• "How large is your dataset?" (Small < 10k, Medium 10k-1M, Large > 1M)
• "What are your computational constraints?" (Edge device, cloud, GPU availability)
• "What are your latency requirements?" (Real-time, batch, offline)
• "Do you need interpretability?" (Clinical, research, production)

These answers determine architecture appropriateness.

Routing by Data Modality

Images → CNN Families

Symptoms triggering this route:

• "classify images"
• "object detection"
• "semantic segmentation"
• "medical imaging"
• "computer vision"

Route to: See cnn-families-and-selection.md for CNN architecture selection and comparison.

When to route here:

• ANY vision task (CNNs are default for spatial data)
• Even if considering Transformers, check CNN families first (often better with less data)

Clarifying questions:

• "Dataset size?" (< 10k → Start with proven CNNs, > 100k → Consider ViT)
• "Deployment target?" (Edge → EfficientNet, Cloud → Anything)
• "Task type?" (Classification → ResNet/EfficientNet, Detection → YOLO/Faster-RCNN)

Sequences → Sequence Models Comparison

Symptoms triggering this route:

• "time series"
• "forecasting"
• "natural language" (NLP)
• "sequential data"
• "temporal patterns"
• "RNN vs LSTM vs Transformer"

Route to: See sequence-models-comparison.md for sequential model selection (RNN, LSTM, Transformer, TCN).

When to route here:

• ANY sequential data
• When user asks "RNN vs LSTM" (skill will present modern alternatives)
• Time-dependent patterns

Clarifying questions:

• "Sequence length?" (< 100 → RNN/LSTM/TCN, 100+ → Transformer with exact FlashAttention; long context → + RoPE scaling, not sparse/linear attention)
• "Latency requirements?" (Real-time → TCN/LSTM, Offline → Transformer)
• "Data volume?" (Small → Simpler models, Large → Transformers)

CRITICAL: Challenge "RNN vs LSTM" premise if they ask. Modern alternatives (Transformers, TCN) often better.

Graphs → Graph Neural Networks

Symptoms triggering this route:

• "social network"
• "molecular structure"
• "knowledge graph"
• "graph data"
• "node classification"
• "link prediction"
• "graph embeddings"

Route to: See graph-neural-networks-basics.md for GNN architectures and graph learning.

When to route here:

• Data has explicit graph structure (nodes + edges)
• Relational information is important
• Network topology matters

Red flag: If treating graph as tabular data (extracting features and ignoring edges) → WRONG. Route to GNN skill.

Generation → Generative Model Families

Symptoms triggering this route:

• "generate images"
• "synthesize data"
• "GAN vs VAE vs Diffusion"
• "image-to-image translation"
• "style transfer"
• "generative modeling"

Route to: See generative-model-families.md for GANs, VAEs, and Diffusion models.

When to route here:

• Goal is to CREATE data, not classify/predict
• Need to sample from distribution
• Data augmentation through generation

Clarifying questions:

• "Use case?" (Production text-to-image → SDXL/SD3/FLUX + LoRA; real-time/edge → distilled diffusion (LCM/Turbo) or a pretrained GAN niche; cross-modal generation → native multimodal (Chameleon/Gemini-class))
• "Quality vs speed?" (Quality → DiT-class diffusion; Speed → distilled diffusion before reaching for GAN — distillation closed most of the speed gap)
• "Controllability?" (Structural control → ControlNet on top of diffusion; identity / style → IP-Adapter or LoRA fine-tune)

CRITICAL: Different generative models have VERY different trade-offs. Don't reflexively recommend GAN for "fast" — distilled diffusion is the modern fast option. Don't recommend training Stable Diffusion from scratch.

Routing by Architecture Component

Attention Mechanisms

Symptoms triggering this route:

• "when to use attention"
• "self-attention vs cross-attention"
• "attention in CNNs"
• "attention bottleneck"
• "multi-head attention"

Route to: See attention-mechanisms-catalog.md for attention mechanism selection and design.

When to route here:

• Designing custom architecture that might benefit from attention
• Understanding where attention helps vs hinders
• Comparing attention variants

NOT for: General Transformer questions → transformer-architecture-deepdive.md instead

Transformer Deep Dive

Symptoms triggering this route:

• "how do transformers work"
• "Vision Transformer (ViT)"
• "BERT architecture"
• "positional encoding"
• "transformer blocks"
• "scaling transformers"

Route to: See transformer-architecture-deepdive.md for Transformer internals and implementation.

When to route here:

• Implementing/customizing transformers
• Understanding transformer internals
• Debugging transformer-specific issues

Cross-reference:

• For sequence models generally → sequence-models-comparison.md (includes transformers in context)
• For LLMs specifically → yzmir-llm-specialist (/llm-specialist router: LLM-specific architecture, fine-tuning, inference optimization, context management)

Normalization Techniques

Symptoms triggering this route:

• "gradient explosion"
• "training instability in deep network"
• "BatchNorm vs LayerNorm"
• "normalization layers"
• "50+ layer network won't train"

Route to: See normalization-techniques.md for deep network stability and normalization methods.

When to route here:

• Deep networks (> 20 layers) with training instability
• Choosing between normalization methods
• Architectural stability issues

CRITICAL: This is often the ROOT CAUSE of "training won't work" - fix architecture before blaming hyperparameters.

Architecture Design Principles

Symptoms triggering this route:

• "how to design architecture"
• "architecture best practices"
• "when to use skip connections"
• "how deep should network be"
• "custom architecture for [novel task]"
• Unclear problem modality

Route to: See architecture-design-principles.md for custom architecture design fundamentals.

When to route here:

• Designing custom architectures
• Novel problems without established architecture
• Understanding WHY architectures work
• User is unsure what modality/problem type they have

This is the foundational skill - route here if other specific skills don't match.

Multi-Modal / Cross-Pack Routing

When Problem Spans Multiple Modalities

Default route: multimodal-architectures.md — covers contrastive (CLIP / SigLIP), generative bolt-on (LLaVA / BLIP-2 / Flamingo), and native multimodal (Chameleon / Gemini-style) recipes, plus audio and video extensions.

Only fall back to component sheets if the user is building from scratch without using foundation models:

1. sequence-models-comparison.md (for text)
2. cnn-families-and-selection.md (for images)
3. architecture-design-principles.md (for hand-rolled fusion strategy)

Order matters: In 2026 the default for "vision + language" is SigLIP + MLP projector + open LLM, not designing fusion from first principles. Read the multimodal sheet before reaching for design-principles.

When Architecture + Other Concerns

Example: "Select architecture AND optimize training"

Route order:

1. Architecture skill FIRST (this pack)
2. Training-optimization SECOND (after architecture chosen)

Why: Wrong architecture can't be fixed by better training.

Example: "Select architecture AND deploy efficiently"

Route order:

1. Architecture skill FIRST
2. ML-production SECOND (quantization, serving)

Deployment constraints might influence architecture choice - if so, note constraints during architecture selection.

Common Routing Mistakes (DON'T DO THESE)

Rule: Architecture pack is for CHOOSING and DESIGNING architectures. Training/deployment/implementation are other packs.

Red Flags: Stop and Clarify

If query contains these patterns, ASK clarifying questions before routing:

Never guess modality or constraints. Always clarify.

Recency Bias: Resistance Table

Counter-narrative: "New architecture ≠ better for your use case. Match architecture to constraints."

Decision Tree

Start here: What's your primary goal?

┌─ SELECT architecture for task
│  ├─ Data modality?
│  │  ├─ Images → [cnn-families-and-selection.md](cnn-families-and-selection.md)
│  │  ├─ Sequences → [sequence-models-comparison.md](sequence-models-comparison.md)
│  │  ├─ Graphs → [graph-neural-networks-basics.md](graph-neural-networks-basics.md)
│  │  ├─ Generation → [generative-model-families.md](generative-model-families.md)
│  │  ├─ Multimodal (text + image / audio / video) → [multimodal-architectures.md](multimodal-architectures.md)
│  │  └─ Unknown → [architecture-design-principles.md](architecture-design-principles.md)
│  └─ Special requirements?
│     ├─ Deep network (>20 layers) unstable → [normalization-techniques.md](normalization-techniques.md) (CRITICAL)
│     ├─ Need attention mechanism → [attention-mechanisms-catalog.md](attention-mechanisms-catalog.md)
│     └─ None → Proceed with modality-based route
│
├─ UNDERSTAND specific architecture
│  ├─ Transformers → [transformer-architecture-deepdive.md](transformer-architecture-deepdive.md)
│  ├─ Attention → [attention-mechanisms-catalog.md](attention-mechanisms-catalog.md)
│  ├─ Normalization → [normalization-techniques.md](normalization-techniques.md)
│  └─ General principles → [architecture-design-principles.md](architecture-design-principles.md)
│
├─ DESIGN custom architecture
│  └─ [architecture-design-principles.md](architecture-design-principles.md) (start here always)
│
└─ COMPARE architectures
   ├─ CNNs (ResNet vs EfficientNet) → [cnn-families-and-selection.md](cnn-families-and-selection.md)
   ├─ Sequence models (RNN vs Transformer) → [sequence-models-comparison.md](sequence-models-comparison.md)
   ├─ Generative (GAN vs Diffusion) → [generative-model-families.md](generative-model-families.md)
   └─ General comparison → [architecture-design-principles.md](architecture-design-principles.md)

Workflow

1. Clarify → Data modality, task type, dataset size, compute/latency constraints
2. Route by modality → Images/Sequences/Graphs/Generation → appropriate skill
3. Check critical issues → Deep network unstable? Fix normalization FIRST
4. Apply architecture skill → Follow routed skill, consider trade-offs
5. Cross-pack → training-optimization (training) or ml-production (deployment)

Rationalization Table

Integration with Other Packs

After Architecture Selection

Once architecture is chosen, route to:

Training the architecture: → yzmir/training-optimization/using-training-optimization

• Optimizer selection
• Learning rate schedules
• Debugging training issues

Implementing in PyTorch: → yzmir/pytorch-engineering/using-pytorch-engineering

• Module design patterns
• Performance optimization
• Custom components

Deploying to production: → yzmir/ml-production/using-ml-production

• Model serving
• Quantization
• Inference optimization

Dynamic/growing architectures: → yzmir/dynamic-architectures/using-dynamic-architectures

• Networks that grow, prune, or adapt during training
• Continual learning and catastrophic forgetting prevention
• Module lifecycle management and progressive training

Before Architecture Selection

If problem involves:

Reinforcement learning: → yzmir/deep-rl/using-deep-rl FIRST

• RL algorithms dictate architecture requirements
• Value networks vs policy networks have different needs

Large language models: → yzmir/llm-specialist/using-llm-specialist FIRST

• LLM architectures are specialized transformers
• Different considerations than general sequence models

Architecture is downstream of algorithm choice in RL and LLMs.

Summary

Use this meta-skill to:

• ✅ Route architecture queries to appropriate specialized skill
• ✅ Identify data modality and problem type
• ✅ Clarify constraints before recommending
• ✅ Resist recency bias (latest ≠ best)
• ✅ Recognize when architecture is the problem (vs training/implementation)

Neural Architecture Specialist Skills

After routing, load the appropriate specialist skill for detailed guidance:

1. architecture-design-principles.md - Custom design, architectural best practices, skip connections, network depth fundamentals
2. attention-mechanisms-catalog.md - Self-attention, cross-attention, multi-head attention, attention in CNNs, attention variants comparison
3. cnn-families-and-selection.md - ResNet, EfficientNet, MobileNet, YOLO, computer vision architecture selection
4. generative-model-families.md - GANs, VAEs, Diffusion models, image generation, style transfer, generative modeling trade-offs
5. graph-neural-networks-basics.md - GCN, GAT, node classification, link prediction, graph embeddings, molecular structures
6. normalization-techniques.md - BatchNorm, LayerNorm, GroupNorm, training stability for deep networks (>20 layers)
7. multimodal-architectures.md - CLIP/SigLIP, LLaVA, BLIP-2 / Q-Former, Flamingo / Idefics, native multimodal (Chameleon / Gemini-style), audio + video extensions
8. sequence-models-comparison.md - RNN, LSTM, Transformer, TCN, Mamba/SSM, RWKV, Jamba; time series, NLP, sequential data
9. transformer-architecture-deepdive.md - Transformer internals, ViT (DINOv2/MAE/SigLIP-era), MoE (Mixtral/DeepSeek-MoE/OLMoE), positional encoding (RoPE/YaRN/NoPE)

Critical principle: Architecture comes BEFORE training. Get this right first.