With the recent advances in deep learning, different approaches to improving pre-trained language models (PLMs) have been proposed. PLMs have advanced state-of-the-art (SOTA) performance on various natural language processing (NLP) tasks such as machine translation, text classification, question answering, text summarization, information retrieval, recommendation systems, named entity recognition, etc. In this paper, we provide a comprehensive review of prior embedding models as well as current breakthroughs in the field of PLMs. Then, we analyse and contrast the various models and provide an analysis of the way they have been built (number of parameters, compression techniques, etc.). Finally, we discuss the major issues and future directions for each of the main points.

The continuous developments in information technologies have resulted in a significant rise in security concerns, including cybercrimes, unauthorized access, and cyberattacks. Recently, researchers have increasingly turned to social media platforms like X to investigate cyberattacks. Analyzing and collecting news about cyberattacks from tweets can efficiently provide crucial insights into the attacks themselves, including their impacts, occurrence regions, and potential mitigation strategies. However, there is a shortage of labeled datasets related to cyberattacks. This paper describes CybAttT, a dataset of 36,071 English cyberattack-related tweets. These tweets are manually labeled into three classes: high-risk news, normal news, and not news. Our final overall Inner Annotation agreement was 0.99 (Fleiss kappa), which represents high agreement. To ensure dataset reliability and accuracy, we conducted rigorous experiments using different supervised machine learning algorithms and various fine-tuned language models to assess its quality and suitability for its intended purpose. A high F1-score of 87.6% achieved using the CybAttT dataset not only demonstrates the potential of our approach but also validates the high quality and thoroughness of its annotations. We have made our CybAttT dataset accessible to the public for research purposes.

The pervasive reach of social media like the X platform, formerly known as Twitter, offers unique opportunities for real-time analysis of cyberattack developments. By parsing and classifying tweets related to cyberattacks, we can glean valuable insights into their type, location, impact, and potential mitigation strategies. However, with millions of daily tweets, manual analysis is inefficient and time-consuming. This paper proposes an interactive and automated dashboard powered by natural language processing to effectively address this challenge. First, we created the CybAttT dataset, which contains 36,071 manually labeled English cyberattack tweets. We experimented with different classification algorithms. Following that, the best model was deployed and integrated into the streaming pipeline for real-time classification. This dynamic dashboard makes use of four different visualization formats: a geographical map, a data table, informative tiles, and a bar chart. Users can readily access crucial information about attacks, including location, timing, and perpetrators, enabling a swift response and mitigation efforts. Our experimental results demonstrated the dashboard’s promising visualization capabilities, highlighting its potential as a valuable tool for organizations and individuals seeking an intuitive and comprehensive overview of cyberattack events.

Stress is a pervasive global health concern that adversely contributes to morbidity and reduced productivity, yet it often remains unquantified due to its subjective and variant presentation. Although artificial intelligence offers an encouraging path toward automated monitoring of mental states, current state-of-the-art approaches are challenged by the reliance on single-source data, sparsity of labeled samples, and significant class imbalance. This paper proposes NeuroStrainSense, a novel deep multimodal stress detection model that integrates three complementary datasets—WESAD, SWELL-KW, and TILES—through a Transformer-based feature fusion architecture combined with a Variational Autoencoder for generative data augmentation. The Transformer architecture employs four encoder layers with eight multi-head attention heads and a hidden dimension of 512 to capture complex inter-modal dependencies across physiological, audio, and behavioral modalities. Our experiments demonstrate that NeuroStrainSense achieves a state-of-the-art performance with accuracies of 87.1%, 88.5%, and 89.8% on the respective datasets, with F1-scores exceeding 0.85 and AUCs greater than 0.89, representing improvements of 2.6–6.6 percentage points over existing baselines. We propose a robust evaluation framework that quantifies discrimination among stress types through clustering validity metrics, achieving a Silhouette Score of 0.75 and Intraclass Correlation Coefficient of 0.76. Comprehensive ablation experiments confirm the utility of each modality and the VAE augmentation module, with physiological features contributing most significantly (average performance decrease of 5.8% when removed), followed by audio (2.8%) and behavioral features (2.1%). Statistical validation confirms all findings at the p < 0.01 significance level. Beyond binary classification, the model identifies five clinically relevant stress profiles—Cognitive Overload, Burnout, Acute Stress, Psychosomatic, and Low-Grade Chronic—with an expert concordance of Cohen’s κ = 0.71 (p < 0.001), demonstrating the strong ecological validity for personalized well-being and occupational health applications. External validation on the MIT Reality Mining dataset confirms the generalizability with minimal performance degradation (accuracy: 0.785, F1-score: 0.752, AUC: 0.849). This work underlines the potential of integrated multimodal learning and demographically aware generative AI for continuous, precise, and fair stress monitoring across diverse populations and environmental contexts.

The rising incidence of skin cancer necessitates scalable and accurate diagnostic tools. While dermoscopy-based systems have achieved expert-level performance, clinical smartphone images pose challenges due to variability in lighting, resolution, and artifacts. Recent advances in multimodal deep learning have shown promise, yet most approaches rely on simple feature concatenation or single-model classifiers, limiting their ability to capture complex cross-modal interactions. This study aims to bridge the diagnostic gap in resource-limited settings by developing a robust multimodal framework that synergizes clinical smartphone images with structured patient metadata for automated skin cancer classification. We propose a novel hybrid architecture integrating a Swin Transformer V2 (SwinV2-Tiny) for hierarchical visual feature extraction and a Denoising Autoencoder (DAE) with PCA for robust metadata embedding. These heterogeneous modalities are fused via a Gated Attention Mechanism that dynamically weighs feature importance across streams. Classification is performed by a Heterogeneous Meta-Stack Ensemble comprising CatBoost, LightGBM, XGBoost, and Logistic Regression, designed to maximize calibration and generalization across imbalanced classes. Evaluated on the PAD-UFES-20 dataset (2298 clinical smartphone images, six diagnostic classes), the proposed framework achieves state-of-the-art performance with a macro-averaged F1-score of 0.977, accuracy of 0.978, and an AUC of 0.990. It significantly outperforms unimodal baselines and existing multimodal methods, demonstrating superior sensitivity (0.974) and precision (0.981), particularly for underrepresented malignant classes like Melanoma (F1: 0.995) and Squamous Cell Carcinoma (F1: 0.960). The integration of clinical metadata with advanced visual embeddings via gated attention significantly enhances diagnostic reliability. Comprehensive ablation studies confirm the contribution of each architectural component. This framework offers a viable pathway for deploying high-precision, AI-driven dermatological screening tools on standard smartphone devices.

The development of natural language processing (NLP) applications has increasingly focused on dialectal variations of languages. The Tunisian dialect (TD), a widely spoken variant of Arabic, poses unique linguistic challenges due to its lack of standardized writing conventions and influences from multiple languages, including French, Italian, Turkish, and Berber. In this work, we introduce TunDC, a dataset of 20,044 labeled comments designed to advance NLP research on the TD. The dataset covers diverse linguistic forms (Arabic, Latin, and mixed scripts), and each comment was manually annotated for positive or negative sentiment by native speakers, achieving high inter-annotator agreement. To evaluate its effectiveness, we fine-tuned various models on TunDC. The bert-base-arabic-TunDC-mixed model achieved an accuracy of 0.84 and a macro-averaged F1-score of 0.83, demonstrating strong generalization across sentiment categories and writing systems. A stratified data-splitting strategy considering both sentiment and script type further improved accuracy by approximately 8% compared to standard splits. As a publicly available resource, TunDC contributes to the computational linguistics community, fostering advancements in language modeling and applications tailored to the TD.