Multispectral pedestrian detection has shown many advantages in a variety of environments, particularly poor illumination conditions, by leveraging visible-thermal modalities. However, in-depth insight into distinguishing the complementary content of multimodal data and exploring the extent of multimodal feature fusion is still lacking. In this paper, we propose a novel multispectral pedestrian detector with multiscale cross-modal homogeneity enhancement and confidence-aware feature fusion. RGB and thermal streams are constructed to extract features and generate candidate proposals. During feature extraction, multiscale cross-modal homogeneity enhancement is proposed to enhance single-modal features using the separated homogeneous features via modal interactions. Homogeneity features encode the semantic information of the scene and are extracted from the RGB-thermal pairs by employing a channel attention mechanism. Proposals from two modalities are united to obtain multimodal proposals. Then, confidence measurement fusion is proposed to achieve multispectral feature fusion in each proposal by measuring the internal confidence of each modality and the interaction confidence between modalities. In addition, a confidence transfer loss function is designed to focus more on hard-to-detect samples during training. Experimental results on two challenging datasets demonstrate that the proposed method achieves better performance compared to existing methods.

Infrared small-dim target detection is an important technology in the fields of infrared guidance, anti-missile, and tracking system. Due to the small size of targets, no obvious structure information, and low image signal-to-noise ratio (SNR), infrared small-dim target detection is still a challenging task. In this letter, a cross-connected bidirectional pyramid network (CBP-Net) is proposed for infrared small-dim target detection. The main body of the CBP-Net is to embed a bottom-up pyramid in the feature pyramid network (FPN), which is designed to provide more comprehensive target information by connecting with the original multi-scale features and the top-down pyramid. The bottom-up pyramid together with the top-down pyramid forms the proposed bidirectional pyramid structure. Then, an region of interest (ROI) feature augment module (RFA) composed of deformable ROI pooling and position attention is designed to fuse multi-scale ROI features and enhance the spatial information of the small-dim target. Besides, a regular constraint loss (RCL) is introduced to restrict multi-scale feature fusion to learn more precise target location information. Experimental results on two challenging datasets show that the performance of the proposed CBP-Net is superior to the state-of-the-art methods.

Remote sensing image captioning aims to describe the content of images using natural language. In contrast with natural images, the scale, distribution, and number of objects generally vary in remote sensing images, making it hard to capture global semantic information and the relationships between objects at different scales. In this paper, in order to improve the accuracy and diversity of captioning, a mask-guided Transformer network with a topic token is proposed. Multi-head attention is introduced to extract features and capture the relationships between objects. On this basis, a topic token is added into the encoder, which represents the scene topic and serves as a prior in the decoder to help us focus better on global semantic information. Moreover, a new Mask-Cross-Entropy strategy is designed in order to improve the diversity of the generated captions, which randomly replaces some input words with a special word (named [Mask]) in the training stage, with the aim of enhancing the model’s learning ability and forcing exploration of uncommon word relations. Experiments on three data sets show that the proposed method can generate captions with high accuracy and diversity, and the experimental results illustrate that the proposed method can outperform state-of-the-art models. Furthermore, the CIDEr score on the RSICD data set increased from 275.49 to 298.39.

Hand pose and shape estimation plays an important role in numerous applications. A cost-effective and practical-friendly approach is to perform accurate hand estimation from a single RGB image, but this task is challenging due to ubiquitous hand self-occlusion and hand-object interaction occlusions. In this paper, we propose a novel SPMHand network to alleviate the effect of occlusions, inspired by the process that humans infer the whole hand when the hand is occluded. The proposed SPMHand consists of two main modules to generate hand segmentations as guidance and conduct hand regressions in a progressive multi-path manner. The segmentation-guided deocclusion module enables the network to “see” the occluded hand by inferring the whole hand segmentation. Specifically, the visible hand segmentation is first obtained and then a hand morphology attention block is introduced to infer the whole hand segmentation by fusing the visible information with the learned hand features. The progressive multi-path regression module is designed to gradually regress the fine hand with intermediate supervisions. Features from deep to shallow are utilized for the hand regressions from coarse to decent. Subsequently, the structure feature, joint heatmaps and segmentations that provide guidance for deocclusion are embedded and fused for the final fine hand regression. Experiments on four challenging datasets illustrate that the proposed SPMHand outperforms the state-of-the-arts in both real-world and synthetic scenes, especially in the present of severe hand-object occlusions.

Developmental Coordination Disorder (DCD) is a type of motor learning difficulty that affects five to six percent of school-aged children, which may have a negative impact on the life of the sufferers. Timely and objective diagnosis of DCD is important for the success of the intervention. The present evaluation methods of DCD rely heavily on observational analysis of occupational therapists and physiotherapists who score the performance when children conduct some designed tasks. However, these methods are expensive, subjective and are not easy to expand to a larger population. A Fine Motor Evaluation System (FMES) is proposed with two views of cameras to record children’s performance when they carry out three fine motor tasks. Automated algorithms are developed to perform automated scoring of fine motor skill. The automated algorithms include task localization and individual task evaluation. The purpose of task localization is to detect each task and extract segments belonging to each task from the original video that includes multiple segments of different tasks. A convolutional neural network with temporal filtering is used to do frame-wise classification and a boundary localization algorithm is proposed to localize each task segment. For individual task evaluation, the extracted video segments of task 1 and task 2 are evaluated based on the proposed feature extraction and time positioning algorithm, and the paper drawings of task 3 are evaluated based on image processing. The proposed methods are validated on a diverse population of children with or without DCD by comparing automated scoring with manual scoring from a professional evaluator. Experimental results suggest that the proposed methods can effectively achieve fine motor evaluation for DCD assessment. Besides, our system is a low-cost solution and the evaluation methods developed are automated, objective and can be suited for large population evaluation and analysis.

Human action recognition based on skeleton has played a key role in various computer vision-related applications, such as smart surveillance, human-computer interaction, and medical rehabilitation. However, due to various viewing angles, diverse body sizes, and occasional noisy data, etc., this remains a challenging task. The existing deep learning-based methods require long time to train the models and may fail to provide an interpretable descriptor to code the temporal-spatial feature of the skeleton sequence. In this paper, a key-segment descriptor and a temporal step matrix model are proposed to semantically present the temporal-spatial skeleton data. First, a skeleton normalization is developed to make the skeleton sequence robust to the absolute body size and initial body orientation. Second, the normalized skeleton data is divided into skeleton segments, which are treated as the action units, combining 3D skeleton pose and the motion. Each skeleton sequence is coded as a meaningful and characteristic key segment sequence based on the key segment dictionary formed by the segments from all the training samples. Third, the temporal structure of the key segment sequence is coded into a step matrix by the proposed temporal step matrix model, and the multiscale temporal information is stored in step matrices with various steps. Experimental results on three challenging datasets demonstrate that the proposed method outperforms all the hand-crafted methods and it is comparable to recent deep learning-based methods.