The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
arXiv:2502.19002v2 Announce Type: cross Abstract: Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding the differences and interactions among these blocks is important. In this paper, we uncover a clear Sharpness Disparity across these blocks, which emerges early in training and intriguingly persists throughout the training process. Motivated by this finding, we propose
阅读原始来源来源记录
项目分析
暂无五维分析,可先查看上方来源资料。