WeWorkBuddy

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training

arXiv:2502.19002v2 Announce Type: cross Abstract: Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding the differences and interactions among these blocks is important. In this paper, we uncover a clear Sharpness Disparity across these blocks, which emerges early in training and intriguingly persists throughout the training process. Motivated by this finding, we propose

阅读原始来源
来源参考:The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training。暂未取得截图。
来源记录 · 预览待采集 · arxiv.org打开来源查看完整网页。

来源记录

项目分析

暂无五维分析,可先查看上方来源资料。