WeWorkBuddy

The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training

arXiv:2502.19002v2 Announce Type: cross Abstract: Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding the differences and interactions among these blocks is important. In this paper, we uncover a clear Sharpness Disparity across these blocks, which emerges early in training and intriguingly persists throughout the training process. Motivated by this finding, we propose

Read original source
Source reference for The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training. Screenshot not yet available.
Source record · preview pending · arxiv.orgOpen the source for the full page.

Source record

Project analysis

Project analysis is not yet available. See the source record above.