Sijie Zhu

Sijie Zhu

Auto-generated profile, based on publicly available information

Sijie Zhu

Center for Research in Computer Vision University of Central Florida USA

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
26 August 2023 in ACM Computing Surveys
Verified
Journal Article
Open access
CZCe Zheng
Ce Zheng
WWWenhan Wu
Wenhan Wu
CCChen Chen
Chen Chen
TYTaojiannan Yang
Taojiannan Yang
SZSijie Zhu
Sijie Zhu
MSMubarak Shah
Mubarak Shah
Human pose estimation aims to locate the human body parts and build human body representation (e.g., body skeleton) from input data such as images and videos. It has drawn increasing attention during the past decade and has been utilized in a wide range of applications including human-computer interaction, motion analysis, augmented reality, and virtual reality. Although the recently developed deep learning-based solutions have achieved high performance in human pose estimation, there still remain challenges due to insufficient training data, depth ambiguities, and occlusion. The goal of this survey article is to provide a comprehensive review of recent deep learning-based solutions for both 2D and 3D pose estimation via a systematic analysis and comparison of these solutions based on their input data and inference procedures. More than 260 research papers since 2014 are covered in this survey. Furthermore, 2D and 3D human pose estimation datasets and evaluation metrics are included. Quantitative performance comparisons of the reviewed methods on popular datasets are summarized and discussed. Finally, the challenges involved, applications, and future research directions are concluded. A regularly updated project page is provided: https://github.com/zczcwh/DL-HPE .

27 May 2021 in IEEE Transactions on Geoscience and Remote Sensing
Verified
Journal Article
Closed access
YSYu Shen
Yu Shen
SZSijie Zhu
Sijie Zhu
TYTaojiannan Yang
Taojiannan Yang
CCChen Chen
Chen Chen
DPDelu Pan
Delu Pan
Qian DuQian Du
Qian Du
Fast and effective responses are required when a natural disaster (e.g., earthquake and hurricane) strikes. Building damage assessment from satellite imagery is critical before relief effort is deployed. With a pair of predisaster and postdisaster satellite images, building damage assessment aims at predicting the extent of damage to buildings. With the powerful ability of feature representation, deep neural networks have been successfully applied to building damage assessment. Most existing works simply concatenate predisaster and postdisaster images as input of a deep neural network without considering their correlations. In this article, we propose a novel two-stage convolutional neural network for building damage assessment, called BDANet. In the first stage, a U-Net is used to extract the locations of buildings. Then, the network weights from the first stage are shared in the second stage for building damage assessment. In the second stage, a two-branch multiscale U-Net is employed as the backbone, where predisaster and postdisaster images are fed into the network separately. A cross-directional attention module is proposed to explore the correlations between predisaster and postdisaster images. Moreover, CutMix data augmentation is exploited to tackle the challenge of difficult classes. The proposed method achieves state-of-the-art performance on a large-scale dataset—xBD. The code is available at https://github.com/ShaneShen/BDANet-Building-Damage-Assessment .

17 August 2020 in IEEE Transactions on Geoscience and Remote Sensing
Verified
Journal Article
Closed access
YSYu Shen
Yu Shen
SZSijie Zhu
Sijie Zhu
CCChen Chen
Chen Chen
Qian DuQian Du
Qian Du
LXLiang Xiao
Liang Xiao
DPDelu Pan
Delu Pan
Deep-learning-based methods, such as convolution neural network (CNN), have demonstrated their efficiency in hyperspectral image (HSI) classification. These methods can automatically learn spectral-spatial discriminative features within local patches. However, for each pixel in an HSI, it is not only related to its nearby pixels but also has connections to pixels far away from itself. Therefore, to incorporate the long-range contextual information, a deep fully convolutional network (FCN) with an efficient nonlocal module, named ENL-FCN, is proposed for HSI classification. In the proposed framework, a deep FCN considers an entire HSI as input and extracts spectral-spatial information in a local receptive field. The efficient nonlocal module is embedded in the network as a learning unit to capture the long-range contextual information. Different from the traditional nonlocal neural networks, the long-range contextual information is extracted in a specially designed criss-cross path for computation efficiency. Furthermore, using a recurrent operation, each pixel's response is aggregated from all pixels of HSI. The benefits of our proposed ENL-FCN are threefold: 1) the long-range contextual information is incorporated effectively; 2) the efficient module can be freely embedded in a deep neural network in a plug-and-play fashion; and 3) it has much fewer learning parameters and requires less computational resources. The experiments conducted on three popular HSI data sets demonstrate that the proposed method achieves state-of-the-art classification performance with lower computational cost in comparison with several leading deep neural networks for HSI.

1 September 2021 in IEEE Transactions on Image Processing
Verified
Journal Article
Closed access
SZSijie Zhu
Sijie Zhu
TYTaojiannan Yang
Taojiannan Yang
CCChen Chen
Chen Chen
This work explores the visual explanation for deep metric learning and its applications. As an important problem for learning representation, metric learning has attracted much attention recently, while the interpretation of the metric learning model is not as well-studied as classification. To this end, we propose an intuitive idea to show where contributes the most to the overall similarity of two input images by decomposing the final activation. Instead of only providing the overall activation map of each image, we propose to generate point-to-point activation intensity between two images so that the relationship between different regions is uncovered. We show that the proposed framework can be directly applied to a wide range of metric learning applications and provides valuable information for model understanding. Both theoretical and empirical analyses are provided to demonstrate the superiority of the proposed overall activation map over existing methods. Furthermore, our experiments validate the effectiveness of the proposed point-specific activation map on two applications, i.e. cross-view pattern discovery and interactive retrieval. Code is available at https://github.com/Jeff-Zilence/Explain_Metric_Learning

28 December 2021 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
TYTaojiannan Yang
Taojiannan Yang
SZSijie Zhu
Sijie Zhu
MMMatias Mendieta
Matias Mendieta
Pu WangPu Wang
Pu Wang
RBRavikumar Balakrishnan
Ravikumar Balakrishnan
CCChen Chen
Chen Chen
Most existing deep neural networks are static, which means they can only perform inference at a fixed complexity. But the resource budget can vary substantially across different devices. Even on a single device, the affordable budget can change with different scenarios, and repeatedly training networks for each required budget would be incredibly expensive. Therefore, in this work, we propose a general method called MutualNet to train a single network that can run at a diverse set of resource constraints. Our method trains a cohort of model configurations with various network widths and input resolutions. This mutual learning scheme not only allows the model to run at different width-resolution configurations but also transfers the unique knowledge among these configurations, helping the model to learn stronger representations overall. MutualNet is a general training methodology that can be applied to various network structures (e.g., 2D networks: MobileNets, ResNet, 3D networks: SlowFast, X3D) and various tasks (e.g., image classification, object detection, segmentation, and action recognition), and is demonstrated to achieve consistent improvements on a variety of datasets. Since we only train the model once, it also greatly reduces the training cost compared to independently training several models. Surprisingly, MutualNet can also be used to significantly boost the performance of a single network, if dynamic resource constraints are not a concern. In summary, MutualNet is a unified method for both static and adaptive, 2D and 3D networks. Code and pre-trained models are available at https://github.com/taoyang1122/MutualNet .

30 October 2022
Verified
Book Chapter
Closed access
PDF file
SZSijie Zhu
Sijie Zhu
ZLZhe Lin
Zhe Lin
SCScott Cohen
Scott Cohen
JKJason Kuen
Jason Kuen
ZZZhifei Zhang
Zhifei Zhang
CCChen Chen
Chen Chen