Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This article focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, physiological signals, flow, RGB, pose, depth, mesh, and point cloud. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the past five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Last, main issues are highlighted separately for each domain, along with their possible future research directions.

To address the problem of automatically detecting and removing the mask without user interaction, we present a GAN-based automatic approach for face de-occlusion, called Automatic Mask Generation Network for Face De-occlusion Using Stacked Generative Adversarial Networks (AFD-StackGAN). In this approach, we decompose the problem into two primary stages (i.e., Stage-I Network and Stage-II Network) and employ a separate GAN in both stages. Stage-I Network (Binary Mask Generation Network) automatically creates a binary mask for the masked region in the input images (occluded images). Then, Stage-II Network (Face De-occlusion Network) removes the mask object and synthesizes the damaged region with fine details while retaining the restored face’s appearance and structural consistency. Furthermore, we create a paired synthetic face-occluded dataset using the publicly available CelebA face images to train the proposed model. AFD-StackGAN is evaluated using real-world test images gathered from the Internet. Our extensive experimental results confirm the robustness and efficiency of the proposed model in removing complex mask objects from facial images compared to the previous image manipulation approaches. Additionally, we provide ablation studies for performance comparison between the user-defined mask and auto-defined mask and demonstrate the benefits of refiner networks in the generation process.

In this study, the natural frequencies and roots (Eigenvalues) of the transcendental equation in a cantilever steel beam for transverse vibration with clamped free (CF) boundary conditions are estimated using a long short-term memory-recurrent neural network (LSTM-RNN) approach. The finite element method (FEM) package ANSYS is used for dynamic analysis and, with the aid of simulated results, the Euler–Bernoulli beam theory is adopted for the generation of sample datasets. Then, a deep neural network (DNN)-based LSTM-RNN technique is implemented to approximate the roots of the transcendental equation. Datasets are mainly based on the cantilever beam geometry characteristics used for training and testing the proposed LSTM-RNN network. Furthermore, an algorithm using MATLAB platform for numerical solutions is used to cross-validate the dataset results. The network performance is evaluated using the mean square error (MSE) and mean absolute error (MAE). Finally, the numerical and simulated results are compared using the LSTM-RNN methodology to demonstrate the network validity.

A wide variety of applications are used in lossless image compression models, especially in medical, space, and aerial imaging domains. Predictive coding improves the performance of lossless image compression, which highly relies on entropy error. Lower entropy error results in better image compression. The main focus of this research is to improve the prediction process by minimizing the entropy error. This paper proposes a novel idea for improved Median Edge Detection (iMED) predictor for lossless image compression. MED predictor is improved using k-means clustering and finding the local context of pixels using 20-Dimensional Difference (DDx20) for input images and updates the cluster weights using learning rates (µi) to minimize the prediction errors of pixels. The performance of the proposed predictor is evaluated on the standard grey-scale test images dataset and KODAK image dataset. Results are obtained and compared based on entropy error, bits per pixel (bpp), and computational running time in seconds(s) with the MED, GAP, FLIF, and LBP predictors. The performance of the proposed iMED predictor improves significantly in terms of the entropy error, bpp, and computational running time in seconds(s) after comparison with different state-of-the-art predictors.