Transformer Attention: Why Self-Attention Works

Self-attention lets each token in a sequence look at every other token and decide how much to weigh them when computing its own output. For each token, you learn three linear projections of its embedding: Q (query, "what am I looking for?"), K (key, "what do I offer?"), and V (value, "my content to be mixed in"). Attention weights are computed as softmax of Q · K^T divided by sqrt(d_k), applied to V. Intuitively, token i's query vector is dot-producted against every other token's key vector to produce a similarity score; softmax turns these into weights summing to 1; then the output for token i is the weighted sum of all tokens' value vectors. The sqrt(d_k) scaling is easy to miss — without it, as dimension grows, the magnitude of dot products grows too, pushing softmax into saturation where gradients vanish and training stalls. Scaling keeps the distribution well-behaved. Multi-head attention runs h independent attentions in parallel, each with d/h-dimensional Q/K/V, then concatenates and projects. Different heads empirically learn different patterns — some heads attend to syntactic dependents, some to previous mention for coreference, some to positional neighbors. This gives the model richer representational capacity than one big attention. The crucial comparison to RNN: RNN processes tokens sequentially, so the path from token 1 to token 1000 is 999 recurrent steps — gradients vanish, long-range dependencies are hard, and training is inherently sequential. Self-attention makes every token see every other token in one layer, so the effective path length is 1, and all token computations can be parallelized on GPU. This is the reason Transformers train faster and learn longer dependencies better. The cost is O(n²) in sequence length for compute and memory, which is why long-context models use tricks like sparse attention, sliding windows, or linear approximations.