Li Dong

Li Dong

Auto-generated profile, based on publicly available information

Li Dong

Microsoft Research, Beijing, China

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
10 April 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Open access
HWHongyu Wang
Hongyu Wang
SMShuming Ma
Shuming Ma
LDLi Dong
Li Dong
SHShaohan Huang
Shaohan Huang
DZDongdong Zhang
Dongdong Zhang
FWFuru Wei
Furu Wei
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function ( DeepNorm ) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.

15 May 2018 in IEEE Transactions on Wireless Communications
Verified
Journal Article
Closed access
UCUrsula Challita
Ursula Challita
LDLi Dong
Li Dong
Walid SaadWalid Saad
Walid Saad
LTE in unlicensed spectrum using licensed assisted access LTE (LTE-LAA) is a promising approach to overcome the wireless spectrum scarcity. However, to reap the benefits of LTE-LAA, a fair coexistence mechanism with other incumbent WiFi deployments is required. In this paper, a novel deep learning approach is proposed for modeling the resource allocation problem of LTE-LAA small base stations (SBSs). The proposed approach enables multiple SBSs to proactively perform dynamic channel selection, carrier aggregation, and fractional spectrum access while guaranteeing fairness with existing WiFi networks and other LTE-LAA operators. Adopting a proactive coexistence mechanism enables future delay-tolerant LTE-LAA data demands to be served within a given prediction window ahead of their actual arrival time thus avoiding the underutilization of the unlicensed spectrum during off-peak hours while maximizing the total served LTE-LAA traffic load. To this end, a noncooperative game model is formulated in which SBSs are modeled as Homo Egualis agents that aim at predicting a sequence of future actions and thus achieving long-term equal weighted fairness with WLAN and other LTE-LAA operators over a given time horizon. The proposed deep learning algorithm is then shown to reach a mixed-strategy Nash equilibrium (NE), when it converges. Simulation results using real data traces show that the proposed scheme can yield up to 28% and 11% gains over a conventional reactive approach and a proportional fair coexistence mechanism, respectively. The results also show that the proposed framework prevents WiFi performance degradation for a densely deployed LTE-LAA network.

1 January 2023
Verified
Conference Paper
Open access
PDF file
YGYuxian Gu
Yuxian Gu
LDLi Dong
Li Dong
FWFuru Wei
Furu Wei
MHMinlie Huang
Minlie Huang
Yuxian Gu, Li Dong, Furu Wei, Minlie Huang. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

1 January 2022
Verified
Conference Paper
Open access
PDF file
JYJian Yang
Jian Yang
SHShaohan Huang
Shaohan Huang
SMShuming Ma
Shuming Ma
YYYuwei Yin
Yuwei Yin
LDLi Dong
Li Dong
FWFuru Wei
Furu Wei
Jian Yang, Shaohan Huang, Shuming Ma, Yuwei Yin, Li Dong, Dongdong Zhang, Hongcheng Guo, Zhoujun Li, Furu Wei. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.

10 November 2023 in International Journal of Machine Learning and Cybernetics
Verified
Journal Article
Closed access
PDF file
HBHangbo Bao
Hangbo Bao
LDLi Dong
Li Dong
WWWenhui Wang
Wenhui Wang
NYNan Yang
Nan Yang
SPSonghao Piao
Songhao Piao
FWFuru Wei
Furu Wei

2 September 2024 in Nonlinear Dynamics
Verified
Journal Article
Closed access
PDF file
JZJingjing Zeng
Jingjing Zeng
XCXingzhi Chen
Xingzhi Chen
LWLixiang Wei
Lixiang Wei
DLDong Li
Dong Li