The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
arXiv:2502.19002v2 Announce Type: cross Abstract: Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding the differences and interactions among these blocks is important. In this paper, we uncover a clear Sharpness Disparity across these blocks, which emerges early in training and intriguingly persists throughout the training process. Motivated by this finding, we propose
Read original sourceSource record
Project analysis
Project analysis is not yet available. See the source record above.