As mental health (MH) disorders become increasingly prevalent, their multifaceted symptoms and comorbidities with other conditions introduce complexity to diagnosis, posing a risk of underdiagnosis. While machine learning (ML) has been explored to mitigate these challenges, we hypothesized that multiple data modalities support more comprehensive detection and that non-intrusive collection approaches better capture natural behaviors. To understand the current trends, we systematically reviewed 184 studies to assess feature extraction, feature fusion, and ML methodologies applied to detect MH disorders from passively sensed multimodal data, including audio and video recordings, social media, smartphones, and wearable devices. Our findings revealed varying correlations of modality-specific features in individualized contexts, potentially influenced by demographics and personalities. We also observed the growing adoption of neural network architectures for model-level fusion and as ML algorithms, which have demonstrated promising efficacy in handling high-dimensional features while modeling within and cross-modality relationships. This work provides future researchers with a clear taxonomy of methodological approaches to multimodal detection of MH disorders to inspire future methodological advancements. The comprehensive analysis also guides and supports future researchers in making informed decisions to select an optimal data source that aligns with specific use cases based on the MH disorder of interest.

With the advancement of sentiment analysis (SA) models and their incorporation into our daily lives, fairness testing on these models is crucial, since unfair decisions can cause discrimination to a large population. Nevertheless, some challenges in fairness testing include the unknown oracle, the difficulty in generating suitable test inputs, and the lack of a reliable way of fixing the issues. To fill in these gaps, BiasRV, a tool based on metamorphic testing (MT), was introduced and succeeded in uncovering fairness issues in a transformer-based model. However, the extent of unfairness in other SA models has not been thoroughly investigated. Our work conducts a more comprehensive empirical study to reveal the extent of fairness violations, specifically gender fairness, exhibited by other popular word embedding-based SA models. We define fairness violation as the behavior in which an SA model predicts variants created from a text, which merely differ in gender classes, to have different sentiments. Our inspection utilizing BiasRV uncovers at least 30 fairness violations (at BiasRV’s default threshold) in all three SA models. Realizing the importance of addressing such significant violations, we introduce adversarial patches (AP) as a way of patch generation in an automated program repair (APR) system to fix them. We adopt adversarial fine-tuning in AP by retraining SA models using adversarial examples, which are bias-uncovering test cases dynamically generated by a tool named BiasFinder at runtime. Evaluation of the SA models shows that our proposed AP reduces fairness violations by at least 25%.

BACKGROUND The pervasiveness of mental health (MH) disorders, and the shortcomings of current diagnoses, such as dependence on patients' subjective recalls and failure to recognize symptoms due to comorbidity, have motivated the exploration of machine learning (ML) in this domain. The algorithms' ability to model correlations between data and subsequently make deductions is promising to support clinical diagnosis. The interactions among data of different modalities also have great potential that is worth investigating. OBJECTIVE This study systematically reviews existing methodologies that apply ML to multimodal data collected passively from participants to assess features extracted from varying forms of data, fusion techniques adopted to combine them, the subsequent ML algorithms applied to learn from such features, and if any, the findings of the association between modality-specific features and MH disorders. The review focuses on studies adopting passive sensing approaches by hypothesizing that gathering data non-intrusively might be more acceptable and better capture people's natural behaviors. METHODS A systematic search was conducted on the Scopus, ACM Digital Library, PubMed, and IEEE Xplore databases to gather studies from January 2015 to May 2022. Studies were screened for eligibility if they gathered passive data involving 2 or more modalities, adopted non-intrusive data collection approaches, and applied ML approaches to explicitly predict the existence or severity of specific MH disorders. RESULTS A total of 9346 studies were obtained from the search results, and 100 were assessed and included in the current review. The primary data source categories were audio and video recordings (n=48), social media (n=30), and smartphone and wearable devices (n=26). The modalities involved were audio, visual, textual, smartphone sensor, wearable sensor, demographics, and personalities. We found that modality-specific features correlate differently with MH disorders in different contexts. The selection of modality fusion techniques from feature, score/decision, and model levels also relies on specific use cases. In addition, determining an effective data-driven ML model, among supervised learning, neural network, ensemble learning, multi-task learning, and others, are dependent on the nature and information contained within the data and how the algorithms learn the potential relationships between data. CONCLUSIONS All data sources exhibit great potential in detecting MH disorders, but their qualities in terms of reliability, practicality, and ethicality deserve deeper consideration. Such consideration is significant due to the enormous influence the data has on the value and contribution of research, and the subsequent social impacts that the research outcomes can bring to the MH community.