# LLMs: Vocabulary Hub: [[Topics/LLMs/LLMs|LLMs]] - **Attention:** the transformer operation in which each token computes weighted mixes of other tokens' vectors, with weights from query–key dot products ([Vaswani et al.](https://arxiv.org/abs/1706.03762)). - **Backpropagation:** using the chain rule to compute the gradient of the loss with respect to every weight, layer by layer from the output back (Rumelhart, Hinton and Williams, 1986). - **Batch normalization:** normalizing each layer's activations over a mini-batch. The ResNet paper used it, which is why its authors said vanishing gradients were unlikely to be the cause of degradation. - **Bottleneck block:** a 3-layer residual block (1×1, 3×3, 1×1 convolutions) used in ResNet-50/101/152 ([arXiv](https://arxiv.org/abs/1512.03385)). - **Degradation problem:** the 2015 observation that deeper *plain* networks get higher **training** error than shallower ones. It isn't overfitting ([arXiv](https://arxiv.org/abs/1512.03385)). - **Embedding:** the learned vector for each token. GPT-3's embedding matrix is 50,257 tokens × 12,288 dimensions (≈ 617M weights). - **High-norm artifact tokens:** patch tokens in large ViTs that the model takes over to store global information, visible as spikes in attention maps ([arXiv](https://arxiv.org/abs/2309.16588)). - **Highway network:** a deep network with *gated* skip connections (Srivastava, Greff and Schmidhuber, 2015) ([arXiv](https://arxiv.org/abs/1505.00387)). - **Identity mapping / shortcut:** the skip path that passes x through unchanged so that the block outputs F(x) + x. A projection shortcut (1×1 conv) is used when shapes change. - **Loss landscape:** the loss plotted over directions in weight space. Li et al. 2018 showed skip connections make it smoother ([arXiv](https://arxiv.org/abs/1712.09913)). - **Pre-activation ResNet:** He et al.'s 2016 variant with BN and ReLU before each weight layer, which keeps the identity path clean ([arXiv](https://arxiv.org/abs/1603.05027)). - **Register tokens:** extra learnable tokens added to a ViT's input to give it scratch space, which removes artifact tokens ([arXiv](https://arxiv.org/abs/2309.16588)). - **Residual connection / skip connection:** adding a block's input to its output. It is in every modern transformer and diffusion U-Net. - **Residual stream:** Anthropic's 2021 term for the running sum of x and every layer's output in a transformer, which each layer reads from and writes to ([Anthropic](https://transformer-circuits.pub/2021/framework/index.html)). - **Shattered gradients:** Balduzzi et al.'s 2017 finding that gradients in deep plain nets become nearly uncorrelated (white-noise-like) with depth ([arXiv](https://arxiv.org/abs/1702.08591)). - **Vanishing / exploding gradients:** gradients shrinking or growing exponentially through many layers. Largely tamed by careful initialization and batch normalization before ResNet.