Ning Jiang

Ning Jiang


Research Focus & Metrics

Academic Career Length

11Years (2016-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 August 2024 in ACM Transactions on Knowledge Discovery From Data
Verified
Journal Article
Open access
ZZZiyu Zhao
Ziyu Zhao
YBYuqi Bai
Yuqi Bai
RXRuoxuan Xiong
Ruoxuan Xiong
QCQingyu Cao
Qingyu Cao
CMChao Ma
Chao Ma
KKKun Kuang
Kun Kuang
Estimating individual treatment effects in networked observational data is a crucial and increasingly recognized problem. One major challenge of this problem is violating the stable unit treatment value assumption (SUTVA), which posits that a unit’s outcome is independent of others’ treatment assignments. However, in network data, a unit’s outcome is influenced not only by its treatment (i.e., direct effect) but also by the treatments of others (i.e., spillover effect) since the presence of interference. Moreover, the interference from other units is always heterogeneous (e.g., friends with similar interests have a different influence than those with different interests). In this article, we focus on the problem of estimating individual treatment effects (including direct effect and spillover effect) under heterogeneous interference in networks. To address this problem, we propose a novel dual weighting regression (DWR) algorithm by simultaneously learning attention weights to capture the heterogeneous interference from neighbors and sample weights to eliminate the complex confounding bias in networks. We formulate the learning process as a bi-level optimization problem. Theoretically, we give a generalization error bound for the expected estimation error of the individual treatment effects. Extensive experiments on four benchmark datasets demonstrate that the proposed DWR algorithm outperforms the state-of-the-art methods in estimating individual treatment effects under heterogeneous network interference.

1 January 2022
Verified
Conference Paper
Open access
PDF file
YZYing Zhang
Ying Zhang
BZBaohang Zhou
Baohang Zhou
KSKehui Song
Kehui Song
XSXuhui Sui
Xuhui Sui
GZGuoqing Zhao
Guoqing Zhao
XYXiaojie Yuan
Xiaojie Yuan
Ying Zhang, Baohang Zhou, Kehui Song, Xuhui Sui, Guoqing Zhao, Ning Jiang, Xiaojie Yuan. Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.

25 December 2024 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
THTianshun Han
Tianshun Han
SGShengnan Gui
Shengnan Gui
YHYiqing Huang
Yiqing Huang
BLBaihui Li
Baihui Li
LLLijian Liu
Lijian Liu
JWJun Wan
Jun Wan
Speech-driven 3D facial animation has improved a lot recently while most related works only utilize acoustic modality and neglect the influence of visual and textual cues, leading to unsatisfactory results in terms of precision and coherence. We argue that visual and textual cues are not trivial information. Therefore, we present a novel framework, namely PMMTalk, using complementary P seudo M ulti- M odal features for improving the accuracy of facial animation. The framework entails three modules: PMMTalk encoder, cross-modal alignment module, and PMMTalk decoder. Specifically, the PMMTalk encoder employs the off-the-shelf talking head generation architecture and speech recognition technology to extract visual and textual information from speech, respectively. Following this, the cross-modal alignment module aligns the audio-image-text features at temporal and semantic levels. Subsequently, the PMMTalk decoder is employed to predict lip-syncing facial blendshape coefficients. Contrary to prior methods, PMMTalk only requires an additional random reference face image but yields more accurate results. Additionally, it is artist-friendly as it seamlessly integrates into standard animation production workflows by introducing facial blendshape coefficients. Finally, given the scarcity of 3D talking face datasets, we introduce a large-scale 3D C hinese A udio- V isual F acial A nimation (3D-CAVFA) dataset. Extensive experiments and user studies show that our approach outperforms the state of the art. Codes and datasets are available at PMMTalk.

16 April 2024 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
KWKun Wei
Kun Wei
BLBei Li
Bei Li
HLHang Lv
Hang Lv
QLQuan Lu
Quan Lu
NJNing Jiang
Ning Jiang
LXLei Xie
Lei Xie
Automatic Speech Recognition (ASR) in conversational settings presents unique challenges, including extracting relevant contextual information from previous conversational turns. Due to irrelevant content, error propagation, and redundancy, existing methods struggle to extract longer and more effective contexts. To address this issue, we introduce a novel conversational ASR system, extending the Conformer encoder-decoder model with cross-modal conversational representation. Our approach leverages a cross-modal extractor that combines pre-trained speech and text models through a specialized encoder and a modal-level mask input. This enables the extraction of richer historical speech context without explicit error propagation. We also incorporate conditional latent variational modules to learn conversational-level attributes such as role preference and topic coherence. By introducing both cross-modal and conversational representations into the decoder, our model retains longer context without information loss, achieving relative accuracy improvements of 8.8% and 23% on Mandarin conversation datasets HKUST and MagicData-RAMC, respectively, compared to the standard Conformer model.

21 August 2023 in IEEE Access
Verified
Journal Article
Open access
JPJiatian Pi
Jiatian Pi
JZJunyi Zeng
Junyi Zeng
QLQuan Lu
Quan Lu
NJNing Jiang
Ning Jiang
HWHaiying Wu
Haiying Wu
ZWZhiyou Wu
Zhiyou Wu
Deep face recognition models are vulnerable to adversarial samples generated by adversarial attack methods. However, current attack methods do not adequately represent the security problems of the deep FR models, because they either produce adversarial samples which are unnatural and easily perceived by human or have poor attack capabilities with low attack success rates on the black-box victim FR model. To achieve a good trade-off between the imperceptibility and attack capability, we propose Adv-Eye, a novel method for constructing adversarial facial images by adding natural eyeshadow to the orbital region. Adv-Eye consists of Makeup Generation Module, Makeup Blending Module, and Attack Module. Makeup Generation Module develops pre-makeup strategy to help adversarial generative networks (GANs) to accurately generate eyeshadow on the orbital image. Makeup Blending Module develops a multi-view image visual similarity evaluation method to improve the imperceptibility of the generated eyeshadow. In Attack Module, an ensemble attack strategy based on fine-grained meta-learning and input decay, is applied to improve attack capability under query-free black-box setting. From the experimental results under LADN dataset and MT dataset, compared with existing techniques, the adversarial samples generated by Adv-Eye not only significantly improve the visual quality, but also get 1.63% and 1.05% average success rates improving on the local black-box FR model, 5.33 and 5.22 average confidence point improving on the online commercial FR plateform respectively. The above results demonstrate that pre-makeup strategy and multi-view image visual similarity evaluation method effectively improve the imperceptibility of generated adversarial perturbations, and Attack Module effectively improves attack success rate while maintaining high image quality.

1 February 2023 in Frontiers in Neuroscience
Verified
Journal Article
Open access
JPJiatian Pi
Jiatian Pi
CLChaoyang Luo
Chaoyang Luo
FXFen Xia
Fen Xia
NJNing Jiang
Ning Jiang
HWHaiying Wu
Haiying Wu
ZWZhiyou Wu
Zhiyou Wu
In transferable black-box attacks, adversarial samples remain adversarial across multiple models and are more likely to attack unknown models. From this view, acquiring and exploiting multiple models is the key to improving transferability. For exploiting multiple models, existing approaches concentrate on differences among models but ignore the underlying complex dependencies. This exacerbates the issue of unbalanced and inadequate attacks on multiple models. To this problem, this paper proposes a novel approach, called Relational Graph Ensemble Attack (RGEA), to exploit the dependencies among multiple models. Specifically, we redefine the multi-model ensemble attack as a multi-objective optimization and create a sub-optimization problem to compute the optimal attack direction, but there are serious time-consuming problems. For this time-consuming problem, we define the vector representation of the model, extract the dependency matrix, and then equivalently simplify the sub-optimization problem by utilizing the dependency matrix. Finaly, we theoretically extend to investigate the connection between RGEA and the traditional multiple gradient descent algorithm (MGDA). Notably, combining RGEA further enhances the transferability of existing gradient-based attacks. The experiments using ten normal training models and ten defensive models on the labeled face in the wild (LFW) dataset demonstrate that RGEA improves the success rate of white-box attacks and further boosts the transferability of black-box attacks.