In this letter, we propose a novel lightweight network HiNet: hybrid inherited feature learning network and its variants: HiNet -ReLU, HiNet -Concat, Large Scale HiNet, 2 stack HiNet and 4 stack HiNet for facial expression recognition. In HiNet, we introduce the hybrid feature (HyFeat) block that follows a two - level hybrid structure to capture the local contextual information of expressive regions. In level -1, HyFeat block uses two different scaled conv filters to preserve the domain knowledge features for expressive regions. Similarly, level -2 captures the enriched features through refine edge variation of the receptive fields by employing another two filters. Thus, HyFeat block allows network to perpetuate prominent features of facial expressions and enhances the discriminability of the HiNet. Moreover, performance of the proposed HiNet is evaluated by conducting person dependent (PD) and person independent (PI) experiments on four datasets: CK+, MUG, AFEW and OULU respectively. Experimental results on CK+ dataset prove the adequacy of the proposed HiNet over its variants in terms of accuracy and computational complexity. Furthermore, the proposed HiNet uses approximately 5 /16, 1/138, 1/144 and 1/31 times less parameters as compared to the MobileNet, VGG -16, VGG - 19 and ResNet, respectively. The experimental and computational complexity analysis demonstrate the significant performance improvement of the proposed method over the state-of-the-art techniques in terms of recognition rate.
Micro-expressions are hard to spot due to fleeting and involuntary moments of facial muscles. Interpretation of micro emotions from video clips is a challenging task. In this paper, we propose affective-motion imaging that cumulates rapid and short-lived variational information of micro-expressions into a single response. Moreover, we have proposed an AffectiveNet: affective-motion feature learning network that can perceive subtle changes and learns the most discriminative dynamic features to describe the emotion classes. The AffectiveNet holds two blocks: MICRoFeat and MFL block. MICRoFeat block conserves the scale-invariant features, which allows the network to capture both coarse and tiny edge variations. While MFL block learns micro-level dynamic variations from two different intermediate convolutional layers. Effectiveness of the proposed network is tested over four datasets by using two experimental setups: person independent (PI) and cross dataset (CD) validation. The experimental results of the proposed network outperform the state-of-the-art approaches with a significant margin for MER approaches.
Facial microexpressions offer useful insights into subtle human emotions. This unpremeditated emotional leakage exhibits the true emotions of a person. However, the minute temporal changes in the video sequences are very difficult to model for accurate classification. In this article, we propose a novel spatiotemporal architecture search algorithm, AutoMER for microexpression recognition (MER). Our main contribution is a new parallelogram design-based search space for efficient architecture search. We introduce a spatiotemporal feature module named 3-D singleton convolution for cell-level analysis. Furthermore, we present four such candidate operators and two 3-D dilated convolution operators to encode the raw video sequences in an end-to-end manner. To the best of our knowledge, this is the first attempt to discover 3-D convolutional neural network (CNN) architectures with a network-level search for MER. The searched models using the proposed AutoMER algorithm are evaluated over five microexpression data sets: CASME-I, SMIC, CASME-II, CAS(ME)2, and SAMM. The proposed generated models quantitatively outperform the existing state-of-the-art approaches. The AutoMER is further validated with different configurations, such as downsampling rate factor, multiscale singleton 3-D convolution, parallelogram, and multiscale kernels. Overall, five ablation experiments were conducted to analyze the operational insights of the proposed AutoMER.
1 September 2023 in IEEE Transactions on Cognitive and Developmental Systems
Micro expression recognition (MER) is a very challenging area of research due to its intrinsic nature and fine-grained changes. In the literature, the problem of MER has been solved through handcrafted/descriptor-based techniques. However, in recent times, deep learning (DL) based techniques have been adopted to gain higher performance for MER. Also, rich survey articles on MER are available by summarizing the datasets, experimental settings, conventional and deep learning methods. In contrast, these studies lack the ability to convey the impact of network design paradigms and experimental setting strategies for DL based MER. Therefore, this paper aims to provide a deep insight into the DL-based MER frameworks with a perspective on promises in network model designing, experimental strategies, challenges, and research needs. Also, the detailed categorization of available MER frameworks is presented in various aspects of model design and technical characteristics. Moreover, an empirical analysis of the experimental and validation protocols adopted by MER methods is presented. The challenges mentioned earlier and network design strategies may assist the affective computing research community in forge ahead in MER research. Finally, we point out the future directions, research needs and draw our conclusions.
Current approaches to breast cancer therapy include neoadjuvant systemic therapy (NST). The efficacy of NST is measured by pathologic complete response (pCR). A patient who attains pCR has significantly enhanced disease-free survival progress. The accurate prediction of pCR in response to a given treatment regimen could increase the likelihood of achieving pCR and prevent toxicities caused by treatments that are not effective. Th early prediction of response to NST can increase the likelihood of survival and help with decisions regarding breast-conserving surgery. An automated NST prediction framework that is able to precisely predict which patient undergoing NST will achieve a pathological complete response (pCR) at an early stage of treatment is needed. Here, we propose an end-to-end efficient multimodal spatiotemporal deep learning framework (deep-NST) framework to predict the outcome of NST prior or at an early stage of treatment. The deep-NST model incorporates imaging data captured at different timestamps of NST regimens, a tumor’s molecular data, and a patient’s demographic data. The efficacy of the proposed work is validated on the publicly available ISPY-1 dataset, in terms of accuracy, area under the curve (AUC), and computational complexity. In addition, seven ablation experiments were carried out to evaluate the impact of each design module in the proposed work. The experimental results show that the proposed framework performs significantly better than other recent methods.