# Attention Mechanisms

> Attention mechanisms in neural networks

- Skill: `ffsshhttiikk/attention-mechanisms` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ffsshhttiikk/attention-mechanisms`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ffsshhttiikk/attention-mechanisms/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: ffsshhttiikk (https://skillmd.com/u/ffsshhttiikk)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ffsshhttiikk/attention-mechanisms

---


## What I do

- Implement attention mechanisms
- Design transformer architectures
- Build sequence-to-sequence models
- Optimize attention computations
- Apply self-attention and cross-attention
- Create efficient attention patterns

## When to use me

Use me when:
- Building transformer models
- Working with sequence data
- Implementing NLP models
- Creating vision transformers
- Multi-modal learning

## Key Concepts

### Attention Formula
```
Attention(Q, K, V) = softmax(QK^T / √d_k)V

Q = Query (what we're looking for)
K = Key (what we're searching in)
V = Value (content to retrieve)
d_k = dimension of keys
```

### Multi-Head Attention
```python
import torch
import torch.nn as nn
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)
    
    def split_heads(self, x):
        batch_size = x.size(0)
        x = x.view(batch_size, -1, self.num_heads, self.d_k)
        return x.permute(0, 2, 1, 3)
    
    def forward(self, q, k, v, mask=None):
        Q = self.split_heads(self.W_q(q))
        K = self.split_heads(self.W_k(k))
        V = self.split_heads(self.W_v(v))
        
        # Attention scores
        scores = torch.matmul(Q, K.permute(0, 1, 3, 2))
        scores = scores / math.sqrt(self.d_k)
        
        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)
        
        attn_weights = torch.softmax(scores, dim=-1)
        attn_output = torch.matmul(attn_weights, V)
        
        # Merge heads
        attn_output = attn_output.permute(0, 2, 1, 3).contiguous()
        attn_output = attn_output.view(attn_output.size(0), -1, self.d_model)
        
        return self.W_o(attn_output)
```

### Attention Types
- **Self-Attention**: Q, K, V from same sequence
- **Cross-Attention**: Q from one, K, V from another
- **Causal Attention**: Masked for language modeling
- **Sparse Attention**: Computational efficiency
- **Linear Attention**: Kernel-based approximation

### Transformer Applications
- **BERT**: Bidirectional encoders
- **GPT**: Autoregressive decoders
- **T5**: Encoder-decoder
- **ViT**: Vision transformers
- **Stable Diffusion**: Cross-attention in diffusion