While chatbots are increasingly popular for communication, their effectiveness is limited by their difficulty in understanding users’ emotions. To address this, this study proposes a new hybrid chatbot model called “TEBC-Net” (Text Emotion Bert CNN Network), which combines text and video analysis to interpret user emotions and generate more empathetic responses. At the core of TEBC-Net is a multi-modal emotion analysis system. One component uses Bidirectional Encoder Representations from Transformers (BERT), a well-regarded model in natural language processing (NLP), achieving an 87.21% accuracy rate in detecting emotional cues from text inputs. The second component captures users’ facial expressions through webcam footage. It begins by detecting faces using a pre-trained classifier like Haarcascade. Then, to improve emotion recognition, it preprocesses the image through brightness adjustments and contrast enhancement with Automatic CLAHE and dual gamma correction. This processed image is analyzed by a Convolutional Neural Network (CNN) model trained specifically for emotion recognition, reaching 74.14% accuracy by assigning probabilities to different emotions. By integrating insights from both text and video analysis, TEBC-Net gains a comprehensive understanding of the user’s emotional state and intent. This combined data then informs the chatbot’s response generation module, enabling it to craft responses that are both empathetic and more directly aligned with the user’s emotional needs.

Captioning images is a challenging task at the intersection of Computer Vision (CV) and Natural Language Processing (NLP), that involves generating descriptive text to depict the content of an image. Existing methodologies typically employ Convolutional Neural Networks (CNNs) for feature extraction and Recurrent Neural Networks (RNNs) for generating captions. However, these approaches often suffer from a lack of contextual understanding, inability to capture fine-grained details, and to generate generic captions. This study proposes VisualCaptionNet (VCN), a novel image captioning model that leverages ResNet50 for rich visual feature extraction and a Long Short-Term Memory (LSTM) network for sequential caption generation while retaining context. By incorporating the Bahdanau attention mechanism to focus on relevant image regions and integrating beam search for coherent and contextually relevant descriptions, VCN addresses the limitations of previous methodologies. Extensive experimentation on benchmark datasets such as Flickr30K and Flickr8K demonstrates VCN’s notable improvements of 10% and 12% over baseline models in terms of caption quality, coherence, and relevance. These enhancements emphasize VCN’s effectiveness in advancing image captioning tasks, promising more accurate and contextually relevant descriptions for images.

In this paper, a novel Monogenic Sobel Directional Pattern (MSDP) using fractional order masks is proposed for extracting features. The MSDP uses fractional-order Sobel masks to identify thin edges along with color and texture-based information thereby increasing performance. Other edge-detection methods can identify only thick edges. There are three modules namely feature extraction, dimension reduction via a novel discriminant analysis method, and classification using a Convolutional Neural Network (CNN). The proposed MSDP is insensitive to the rotation and scaling changes existing in the images. The Bat Algorithm-based Optimization (BAO) is used for the selection of the best parameters of MSDP. The best value is modified by the Pearson Mutation (PM) operator in an effort to aid the algorithm in avoiding local optima and achieving a balance between global and local searches. The proposed work uses CNN for classification and achieves higher classification accuracy for six datasets.