This study aims at learning deep features from different data to recognise speech emotion. The authors designed a merged convolutional neural network (CNN), which had two branches, one being one‐dimensional (1D) CNN branch and another 2D CNN branch, to learn the high‐level features from raw audio clips and log‐mel spectrograms. The building of the merged deep CNN consists of two steps. First, one 1D CNN and one 2D CNN architectures were designed and evaluated; then, after the deletion of the second dense layers, the two CNN architectures were merged together. To speed up the training of the merged CNN, transfer learning was introduced in the training. The 1D CNN and 2D CNN were trained first. Then, the learned features of the 1D CNN and 2D CNN were repurposed and transferred to the merged CNN. Finally, the merged deep CNN initialised with transferred features was fine‐tuned. Two hyperparameters of the designed architectures were chosen through Bayesian optimisation in the training. The experiments conducted on two benchmark datasets show that the merged deep CNN can improve emotion classification performance significantly.

Remote sensing (RS) scene classification is a challenging task to predict scene categories of RS images. RS images have two main issues: large intraclass variance caused by large resolution variance and confusing information from large geographic covering area. To ease the negative influence from the above two issues. We propose a multigranularity multilevel feature ensemble network (MGML-FENet) to efficiently tackle the RS scene classification task in this article. Specifically, we propose multigranularity multilevel feature fusion branch (MGML-FFB) to extract multigranularity features in different levels of network by channel-separate feature generator (CS-FG). To avoid the interference from confusing information, we propose a multigranularity multilevel feature ensemble module (MGML-FEM), which can provide diverse predictions by full-channel feature generator (FC-FG). Compared to previous methods, our proposed networks have the ability to use structure information and abundant fine-grained features. Furthermore, through the ensemble learning method, our proposed MGML-FENets can obtain more convincing final predictions. Extensive classification experiments on multiple RS datasets (AID, NWPU-RESISC45, UC-Merced, and VGoogle) demonstrate that our proposed networks achieve better performance than previous state-of-the-art (SOTA) networks. The visualization analysis also shows the good interpretability of MGML-FENet.

Remote sensing image classification task is challenging due to the characteristics of complex composition, so different geographic elements in the same image will interfere with each other, resulting in misclassification. To solve this problem, we propose a multibranch ensemble network to enhance the feature representation ability by fusing final output logits and intermediate feature maps. However, simply adding branches will increase the complexity of models and decline the inference efficiency. To reduce the complexity of multibranch network, we make multibranch share more weights and add feature augmentation modules to compensate for the lack of diversity caused by weight sharing. To improve the efficiency of inference, we embed self-distillation (SD) method to transfer knowledge from ensemble network to main branch. Through optimizing with SD, the main branch will have close performance as an ensemble network. In this way, we can cut other branches during inference. In addition, we simplify the process of SD and totally adopt two loss functions to self-distill the logits and feature maps. In this article, we design a compact multibranch ensemble network, which can be trained in an end-to-end manner. Then, we insert an SD method on output logits and feature maps. Our proposed architecture (ESD-MBENet) performs strongly on classification accuracy with compact design. Extensive experiments are applied on three benchmark remote sensing datasets, AID, NWPU-RESISC45, and UC-Merced with three classic baseline models, VGG16, ResNet50, and DenseNet121. Results prove that ESD-MBENet can achieve better accuracy than previous state-of-the-art complex deep learning models. Moreover, abundant visualization analyses make our method more convincing and interpretable.

In the last few years, great efforts have been made to extend the linear projection technique (LPT) for multidimensional data (i.e., tensor), generally referred to as the multilinear projection technique (MPT). The vectorized nature of LPT requires high-dimensional data to be converted into vector, and hence may lose spatial neighborhood information of raw data. MPT well addresses this problem by encoding multidimensional data as general tensors of a second or even higher order. In this paper, we propose a novel multilinear projection technique, called multilinear spatial discriminant analysis (MSDA), to identify the underlying manifold of high-order tensor data. MSDA considers both the nonlocal structure and the local structure of data in the transform domain, seeking to learn the projection matrices from all directions of tensor data that simultaneously maximize the nonlocal structure and minimize the local structure. Different from multilinear principal component analysis (MPCA) that aims to preserve the global structure and tensor locality preserving projection (TLPP) that is in favor of preserving the local structure, MSDA seeks a tradeoff between the nonlocal (global) and local structures so as to drive its discriminant information from the range of the non-local structure and the range of the local structure. This spatial discriminant characteristic makes MSDA have more powerful manifold preserving ability than TLPP and MPCA. Theoretical analysis shows that traditional MPTs, such as multilinear linear discriminant analysis, TLPP, MPCA, and tensor maximum margin criterion, could be derived from the MSDA model by setting different graphs and constraints. Extensive experiments on face databases (ORL, CMU PIE, and the extended Yale-B) and the Weizmann action database demonstrate the effectiveness of the proposed MSDA method.

Diagnostic results can be radically influenced by the quality of 2D ovarian-tumor ultrasound images. However, clinically processed 2D ovarian-tumor ultrasound images contain many artificially recognized symbols, such as fingers, crosses, dashed lines, and letters which assist artificial intelligence (AI) in image recognition. These symbols are widely distributed within the lesion’s boundary, which can also affect the useful feature-extraction-utilizing networks and thus decrease the accuracy of lesion classification and segmentation. Image inpainting techniques are used for noise and object elimination from images. To solve this problem, we observed the MMOTU dataset and built a 2D ovarian-tumor ultrasound image inpainting dataset by finely annotating the various symbols in the images. A novel framework called mask-guided generative adversarial network (MGGAN) is presented in this paper for 2D ovarian-tumor ultrasound images to remove various symbols from the images. The MGGAN performs to a high standard in corrupted regions by using an attention mechanism in the generator to pay more attention to valid information and ignore symbol information, making lesion boundaries more realistic. Moreover, fast Fourier convolutions (FFCs) and residual networks are used to increase the global field of perception; thus, our model can be applied to high-resolution ultrasound images. The greatest benefit of this algorithm is that it achieves pixel-level inpainting of distorted regions without clean images. Compared with other models, our model achieveed better results with only one stage in terms of objective and subjective evaluations. Our model obtained the best results for 256 × 256 and 512 × 512 resolutions. At a resolution of 256 × 256, our model achieved $0.9246$ for SSIM, $22.66$ for FID, and $0.07806$ for LPIPS. At a resolution of 512 × 512, our model achieved $0.9208$ for SSIM, $25.52$ for FID, and $0.08300$ for LPIPS. Our method can considerably improve the accuracy of computerized ovarian tumor diagnosis. The segmentation accuracy was improved from $71.51\%$ to $76.06\%$ for the Unet model and from $61.13\%$ to $66.65\%$ for the PSPnet model in clean images.