10 April 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function ( DeepNorm ) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.
1 November 2026 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Efficient deployment of 1-bit Large Language Models (LLMs) is hindered by activation outliers, which pose significant challenges to low-bit quantization. In this work, we investigate sparse and low-bit activation strategies tailored for 1-bit LLMs. We first introduce BitNet v2 a4.8, a hybrid quantization-sparsification architecture designed to mitigate quantization errors arising from outlier activation channels. BitNet v2 a4.8 utilizes 4-bit quantization for inputs to attention and feed-forward layers, while sparsifying intermediate activations and subsequently quantizing them to 8 bits. To further leverage the 4-bit computation capabilities of next-generation GPUs, we present BitNet v2, a novel framework that enables fully 4-bit activations for 1-bit LLMs. To address the outlier activation channels, we propose $/mathcal{H}$, a lightweight module that applies an online Hadamard transformation prior to quantization, effectively regularizing activation distributions into more Gaussian-like forms conducive to low-bit representation. Extensive experiments demonstrate that BitNet v2 a4.8 achieves performance comparable to BitNet b1.58 with similar training costs. Furthermore, BitNet v2, when trained from scratch with 8-bit activations, matches the performance of BitNet b1.58. Remarkably, training BitNet v2 directly with native 4-bit activations leads to only marginal performance degradation, while significantly reducing computational overhead during batched inference.