You
Why do we scale by 1/√dk in attention?
Codex · grounded in Week 7 sources
For large dk, dot products grow in magnitude and push softmax into regions with vanishing gradients. Dividing by √dk keeps the scores in a range where softmax stays sensitive12.
1Lecture 07 · Self-attention0:34:102Goodfellow · Reading 9.3pp. 339