Junzhuo Li

Junzhuo Li

Auto-generated profile, based on publicly available information

Junzhuo Li

School of New Media and Communication Tianjin University Tianjin China

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2023
Verified
Conference Paper
Open access
PDF file
XWXinwei Wu
Xinwei Wu
JLJunzhuo Li
Junzhuo Li
MXMinghui Xu
Minghui Xu
WDWeilong Dong
Weilong Dong
SWShuangzhi Wu
Shuangzhi Wu
DXDeyi Xiong
Deyi Xiong
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, Deyi Xiong. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

1 January 2023
Verified
Conference Paper
Open access
PDF file
SXShaoyang Xu
Shaoyang Xu
JLJunzhuo Li
Junzhuo Li
DXDeyi Xiong
Deyi Xiong
Shaoyang Xu, Junzhuo Li, Deyi Xiong. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

5 December 2023 in arXiv
Verified
Preprint
Open access
PDF file
XWXinwei Wu
Xinwei Wu
JLJunzhuo Li
Junzhuo Li
MXMinghui Xu
Minghui Xu
WDWeilong Dong
Weilong Dong
SWShuangzhi Wu
Shuangzhi Wu
DXDeyi Xiong
Deyi Xiong
Large language models pretrained on a huge amount of data capture rich knowledge and information in the training data. The ability of data memorization and regurgitation in pretrained language models, revealed in previous studies, brings the risk of data leakage. In order to effectively reduce these risks, we propose a framework DEPN to Detect and Edit Privacy Neurons in pretrained language models, partially inspired by knowledge neurons and model editing. In DEPN, we introduce a novel method, termed as privacy neuron detector, to locate neurons associated with private information, and then edit these detected privacy neurons by setting their activations to zero. Furthermore, we propose a privacy neuron aggregator dememorize private information in a batch processing manner. Experimental results show that our method can significantly and efficiently reduce the exposure of private data leakage without deteriorating the performance of the model. Additionally, we empirically demonstrate the relationship between model memorization and privacy neurons, from multiple perspectives, including model size, training time, prompts, privacy neuron distribution, illustrating the robustness of our approach.

7 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
SXShaoyang Xu
Shaoyang Xu
JLJunzhuo Li
Junzhuo Li
DXDeyi Xiong
Deyi Xiong
Multilingual pretrained language models serve as repositories of multilingual factual knowledge. Nevertheless, a substantial performance gap of factual knowledge probing exists between high-resource languages and low-resource languages, suggesting limited implicit factual knowledge transfer across languages in multilingual pretrained language models. This paper investigates the feasibility of explicitly transferring relatively rich factual knowledge from English to non-English languages. To accomplish this, we propose two parameter-free $\textbf{L}$anguage $\textbf{R}$epresentation $\textbf{P}$rojection modules (LRP2). The first module converts non-English representations into English-like equivalents, while the second module reverts English-like representations back into representations of the corresponding non-English language. Experimental results on the mLAMA dataset demonstrate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transferability across diverse non-English languages. We further investigate the working mechanism of LRP2 from the perspectives of representation space and cross-lingual knowledge neuron.

16 December 2022 in arXiv
Verified
Preprint
Open access
PDF file
WDWeilong Dong
Weilong Dong
XWXinwei Wu
Xinwei Wu
JLJunzhuo Li
Junzhuo Li
SWShuangzhi Wu
Shuangzhi Wu
CBChao Bian
Chao Bian
DXDeyi Xiong
Deyi Xiong
Massively multi-task learning with large language models has recently made substantial progress on few-shot generalization. However, this is usually performed in a centralized learning fashion, ignoring the privacy sensitivity issue of (annotated) data used in multiple tasks. To mitigate this issue, we propose FewFedWeight, a few-shot federated learning framework across multiple tasks, to achieve the best of both worlds: privacy preservation and cross-task generalization. FewFedWeight trains client models in isolated devices without sharing data. It broadcasts the global model in the server to each client and produces pseudo data for clients so that knowledge from the global model can be explored to enhance few-shot learning of each client model. An energy-based algorithm is further proposed to weight pseudo samples in order to reduce the negative impact of noise from the generated pseudo data. Adaptive model weights of client models are also tuned according to their performance. We use these model weights to dynamically aggregate client models to update the global model. Experiments on 118 NLP tasks show that FewFedWeight can significantly improve the performance of client models on 61% tasks with an average performance improvement rate of 30.5% over the baseline and substantially outperform FedAvg and other decentralized learning methods.