Weiwei Deng

Weiwei Deng

Auto-generated profile, based on publicly available information

Weiwei Deng

Microsoft, Beijing, China

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
4 August 2023
Verified
Conference Paper
Open access
DWDong Wang
Dong Wang
KSKavé Salamatian
Kavé Salamatian
YXYunqing Xia
Yunqing Xia
WDWeiwei Deng
Weiwei Deng
QZQi Zhang
Qi Zhang
Although deep pre-trained language models have shown promising benefit in a large set of industrial scenarios, including Click-Through-Rate (CTR) prediction, how to integrate pre-trained language models that handle only textual signals into a prediction pipeline with non-textual features is challenging. Up to now, two directions have been explored to integrate multi-modal inputs in fine-tuning of pre-trained language models. One consists of fusing the outcome of language models and non-textual features through an aggregation layer, resulting into ensemble framework, where the cross-information between textual and non-textual inputs are learned only in the aggregation layer. The second one consists of splitting and transforming non-textual features into fine-grained tokens that are fed, along with textual tokens, directly into the transformer layers of language models. However, by adding additional tokens, this approach increases the complexity of the learning and inference. We propose in this paper, a novel framework, BERT4CTR, that addresses these limitations. The new framework leverages Uni-Attention mechanism to benefit from the interactions between non-textual and textual features, while maintaining low training and inference time-costs, through a dimensionality reduction. We demonstrate through comprehensive experiments on both public and commercial data that BERT4CTR outperforms significantly the state-of-the-art approaches to handle multi-modal inputs and is applicable to CTR prediction. In comparison with ensemble framework, BERT4CTR brings more than 0.4% AUC gain on both tested data sets with only 7% increase on latency.

4 August 2023
Verified
Conference Paper
Open access
JLJunyan Li
Junyan Li
LZLi Lyna Zhang
Li Lyna Zhang
JXJiahang Xu
Jiahang Xu
YWYujing Wang
Yujing Wang
SYShaoguang Yan
Shaoguang Yan
MYMao Yang
Mao Yang
Deploying pre-trained transformer models like BERT on downstream tasks in resource-constrained scenarios is challenging due to their high inference cost, which grows rapidly with input sequence length. In this work, we propose a constraint-aware and ranking-distilled token pruning method ToP, which selectively removes unnecessary tokens as input sequence passes through layers, allowing the model to improve online inference speed while preserving accuracy. ToP overcomes the limitation of inaccurate token importance ranking in the conventional self-attention mechanism through a ranking-distilled token distillation technique, which distills effective token rankings from the final layer of unpruned models to early layers of pruned models. Then, ToP introduces a coarse-to-fine pruning approach that automatically selects the optimal subset of transformer layers and optimizes token pruning decisions within these layers through improved L0 regularization. Extensive experiments on GLUE benchmark and SQuAD tasks demonstrate that ToP outperforms state-of-the-art token pruning and model compression methods with improved accuracy and speedups. ToP reduces the average FLOPs of BERT by 8.1X while achieving competitive accuracy on GLUE, and provides a real latency speedup of up to 7.4X on an Intel CPU. Code is available at https://github.com/microsoft/Moonlit/tree/main/ToP

22 April 2025
Verified
Conference Paper
Open access
JYJun Yin
Jun Yin
ZZZhengxin Zeng
Zhengxin Zeng
MLMingzheng Li
Mingzheng Li
HYHao Yan
Hao Yan
CLChaozhuo Li
Chaozhuo Li
WSWang Senzhang
Wang Senzhang

3 August 2025
Verified
Conference Paper
Open access
LWLu Wang
Lu Wang
DZDi Zhang
Di Zhang
FYFangkai Yang
Fangkai Yang
PZPu Zhao
Pu Zhao
JLJianfeng Liu
Jianfeng Liu
QZQi Zhang
Qi Zhang

13 May 2024
Verified
Conference Paper
Open access
YCYaoqi Chen
Yaoqi Chen
RZRuicheng Zheng
Ruicheng Zheng
QCQi Chen
Qi Chen
SXShuotao Xu
Shuotao Xu
QZQianxi Zhang
Qianxi Zhang
MYMao Yang
Mao Yang
Multi-index vector search has become the cornerstone for many applications, such as recommendation systems. Efficient search in such a multi-modal hybrid vector space is challenging since no single index design performs well for all kinds of vector data. Existing approaches to processing multi-index hybrid queries either suffer from algorithmic limitations or processing inefficiency. In this paper, we propose OneSparse, a unified multi-vector index query system that incorporates multiple posting-based vector indices, which enables highly efficient retrieval of multi-modal data-sets. OneSparse introduces a novel multi-index query engine design of inter-index intersection push-down. It also optimizes the vector posting format to expedite multi-index queries. Our experiments show OneSparse achieves more than 6x search performance improvement while maintaining comparable accuracy. OneSparse has already been integrated into Microsoft online web search and advertising systems with 5x+ latency gain for Bing web search and 2.0% Revenue Per Mille (RPM) gain for Bing sponsored search.

8 May 2025
Verified
Conference Paper
Open access
YLYuxuan Liu
Yuxuan Liu
TYTianchi Yang
Tianchi Yang
ZZZihan Zhang
Zihan Zhang
MSMinghui Song
Minghui Song
HHHaizhen Huang
Haizhen Huang
QZQi Zhang
Qi Zhang