🔮The Codex
Attention Mechanism
The core technique that allows AI to focus on relevant parts of input text.
📖 Apprentice Explanation
Attention is how AI decides which words in a sentence are most important for understanding meaning. It's like how you focus on key words when reading quickly.
🧙 Archmage Notes
Self-attention computes query-key-value matrices to determine token relationships. Multi-head attention allows parallel attention patterns. Variants include flash attention, grouped-query attention, and multi-query attention for efficiency.