Bo He

Bo He

Auto-generated profile, based on publicly available information

Bo He

University of Maryland

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
23 October 2022
Verified
Book Chapter
Closed access
PDF file
SHShuaiyi Huang
Shuaiyi Huang
LYLuyu Yang
Luyu Yang
BHBo He
Bo He
SZSongyang Zhang
Songyang Zhang
XHXuming He
Xuming He
ASAbhinav Shrivastava
Abhinav Shrivastava

3 November 2021 in arXiv
Verified
Preprint
Open access
PDF file
BHBo He
Bo He
XYXitong Yang
Xitong Yang
ZWZuxuan Wu
Zuxuan Wu
HCHao Chen
Hao Chen
SLSer-Nam Lim
Ser-Nam Lim
ASAbhinav Shrivastava
Abhinav Shrivastava
Self-attention learns pairwise interactions to model long-range dependencies, yielding great improvements for video action recognition. In this paper, we seek a deeper understanding of self-attention for temporal modeling in videos. We first demonstrate that the entangled modeling of spatio-temporal information by flattening all pixels is sub-optimal, failing to capture temporal relationships among frames explicitly. To this end, we introduce Global Temporal Attention (GTA), which performs global temporal attention on top of spatial attention in a decoupled manner. We apply GTA on both pixels and semantically similar regions to capture temporal relationships at different levels of spatial granularity. Unlike conventional self-attention that computes an instance-specific attention matrix, GTA directly learns a global attention matrix that is intended to encode temporal structures that generalize across different samples. We further augment GTA with a cross-channel multi-head fashion to exploit channel interactions for better temporal modeling. Extensive experiments on 2D and 3D networks demonstrate that our approach consistently enhances temporal modeling and provides state-of-the-art performance on three video action recognition datasets.

18 November 2022 in arXiv
Verified
Preprint
Open access
PDF file
HCHao Chen
Hao Chen
MGMatt Gwilliam
Matt Gwilliam
BHBo He
Bo He
SLSer-Nam Lim
Ser-Nam Lim
ASAbhinav Shrivastava
Abhinav Shrivastava
Compression and reconstruction of visual data have been widely studied in the computer vision community, even before the popularization of deep learning. More recently, some have used deep learning to improve or refine existing pipelines, while others have proposed end-to-end approaches, including autoencoders and implicit neural representations, such as SIREN and NeRV. In this work, we propose Neural Visual Representation with Content-adaptive Embedding (CNeRV), which combines the generalizability of autoencoders with the simplicity and compactness of implicit representation. We introduce a novel content-adaptive embedding that is unified, concise, and internally (within-video) generalizable, that compliments a powerful decoder with a single-layer encoder. We match the performance of NeRV, a state-of-the-art implicit neural representation, on the reconstruction task for frames seen during training while far surpassing for frames that are skipped during training (unseen images). To achieve similar reconstruction quality on unseen images, NeRV needs 120x more time to overfit per-frame due to its lack of internal generalization. With the same latent code length and similar model size, CNeRV outperforms autoencoders on reconstruction of both seen and unseen images. We also show promising results for visual data compression. More details can be found in the project pagehttps://haochen-rye.github.io/CNeRV/