Yan Yan

Yan Yan

Auto-generated profile, based on publicly available information

Yan Yan

Illinois Institute of Technology, Chicago, IL, USA

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
28 June 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
YZYe Zhu
Ye Zhu
YWYu Wu
Yu Wu
NSNicu Sebe
Nicu Sebe
YYYan Yan
Yan Yan
We are perceiving and communicating with the world in a multisensory manner, where different information sources are sophisticatedly processed and interpreted by separate parts of the human brain to constitute a complex, yet harmonious and unified sensing system. To endow the machines with true intelligence, multimodal machine learning that incorporates data from various sources has become an increasingly popular research area with emerging technical advances in recent years. In this paper, we present a survey on multimodal machine learning from a novel perspective considering not only the purely technical aspects but also the intrinsic nature of different data modalities. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and then present the methodological advancements categorized by the combination of data modalities, such as Vision+Text , with slightly inclined emphasis on the visual data. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels, and provide an additional comparison in the light of their technical connections with the data nature, e.g., the semantic consistency between image objects and textual descriptions, and the rhythm correspondence between video dance moves and musical beats. We hope that the exploitation of the alignment as well as the existing gap between the intrinsic nature of data modality and the technical designs, will benefit future research studies to better address a specific challenge related to the concrete multimodal task, prompting a unified multimodal machine learning framework closer to a real human intelligence system.

6 February 2023 in ACM Transactions on Multimedia Computing, Communications, and Applications
Verified
Journal Article
Open access
NZNa Zheng
Na Zheng
XSXuemeng Song
Xuemeng Song
TSTianyu Su
Tianyu Su
WLWeifeng Liu
Weifeng Liu
YYYan Yan
Yan Yan
LNLiqiang Nie
Liqiang Nie
Egocentric early action prediction aims to recognize actions from the first-person view by only observing a partial video segment, which is challenging due to the limited context information of the partial video. In this article, to tackle the egocentric early action prediction problem, we propose a novel multi-modal adversarial knowledge distillation framework. In particular, our approach involves a teacher network to learn the enhanced representation of the partial video by considering the future unobserved video segment, and a student network to mimic the teacher network to produce the powerful representation of the partial video and based on that predicting the action label. To promote the knowledge distillation between the teacher and the student network, we seamlessly integrate adversarial learning with latent and discriminative knowledge regularizations encouraging the learned representations of the partial video to be more informative and discriminative toward the action prediction. Finally, we devise a multi-modal fusion module toward comprehensively predicting the action label. Extensive experiments on two public egocentric datasets validate the superiority of our method over the state-of-the-art methods. We have released the codes and involved parameters to benefit other researchers. 1

3 November 2022
Verified
Book Chapter
Closed access
PDF file
YSYuzhang Shang
Yuzhang Shang
DXDan Xu
Dan Xu
ZZZiliang Zong
Ziliang Zong
LNLiqiang Nie
Liqiang Nie
YYYan Yan
Yan Yan

3 November 2022
Verified
Book Chapter
Closed access
PDF file
YSYuzhang Shang
Yuzhang Shang
DXDan Xu
Dan Xu
Bin DuanBin Duan
Bin Duan
ZZZiliang Zong
Ziliang Zong
LNLiqiang Nie
Liqiang Nie
YYYan Yan
Yan Yan

7 February 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
HXHanyu Xuan
Hanyu Xuan
ZWZhiliang Wu
Zhiliang Wu
JYJian Yang
Jian Yang
BJBo Jiang
Bo Jiang
LLLei Luo
Lei Luo
YYYan Yan
Yan Yan
By observing a scene and listening to corresponding audio cues, humans can easily recognize where the sound is. To achieve such cross-modal perception on machines, existing methods take advantage of the maps obtained by interpolation operations to localize the sound source. As semantic object-level localization is more attractive for prospective practical applications, we argue that these map-based methods only offer a coarse-grained and indirect description of the sound source. Additionally, these methods utilize a single audio-visual tuple at a time during self-supervised learning, causing the model to lose the crucial chance to reason about the data distribution of large-scale audio-visual samples. Although the introduction of Audio-Visual Contrastive Learning ( AVCL ) can effectively alleviate this issue, the contrastive set constructed by randomly sampling is based on the assumption that the audio and visual segments from all other videos are not semantically related. Since the resulting contrastive set contains a large number of faulty negatives, we believe that this assumption is rough. In this paper, we advocate a novel proposal-based solution that directly localizes the semantic object-level sound source, without any manual annotations. The Global Response Map ( GRM ) is incorporated as an unsupervised spatial constraint to filter those instances corresponding to a large number of sound-unrelated regions. As a result, our proposal-based Sound Source Localization ( SSL ) can be cast into a simpler Multiple Instance Learning ( MIL ) problem. To overcome the limitation of random sampling in AVCL , we propose a novel Active Contrastive Set Mining ( ACSM ) to mine the contrastive sets with informative and diverse negatives for robust AVCL . Our approaches achieve state-of-the-art ( SOTA ) performance when compared to several baselines on multiple SSL datasets with diverse scenarios.

12 December 2023 in International Journal of Biological Macromolecules
Verified
Journal Article
Closed access
RRRathnayake A.C. Rathnayake
Rathnayake A.C. Rathnayake
ZZZhenghao Zhao
Zhenghao Zhao
NMNathan McLaughlin
Nathan McLaughlin
WLWei Li
Wei Li
YYYan Yan
Yan Yan
Rong WangRong Wang
Rong Wang