The classification of clinical notes into specific diagnostic categories is critical in healthcare, especially for mental health conditions like anxiety and adjustment disorder. In this study, we compare the performance of various artificial intelligence models, including both traditional machine learning approaches (random forest, support vector machine, K-nearest neighbors, decision tree, and eXtreme Gradient Boost) and deep learning models (DistilBERT and SciBERT), to classify clinical notes into these two diagnoses. Additionally, we implemented three oversampling strategies: No Oversampling, Random Oversampling, and Synthetic Minority Over-sampling Technique (SMOTE), to assess their impact on model performance. Hyperparameter tuning was also applied to optimize model accuracy. Our results indicate that oversampling techniques had minimal impact on model performance overall. The only exception was SMOTE, which showed a positive effect specifically with Bidirectional Encoder Representations from Transformers (BERT)-based models. However, hyperparameter optimization significantly improved accuracy across the models, enhancing their ability to generalize and perform on the dataset. The decision tree and eXtreme Gradient Boost models achieved the highest accuracy among machine learning approaches, both reaching 96%, while the DistilBERT and SciBERT models also attained 96% accuracy in the deep learning category. These findings underscore the importance of hyperparameter tuning in maximizing model performance. This study contributes to the ongoing research on AI-assisted diagnostic tools in mental health by providing insights into the efficacy of different model architectures and data balancing methods.
Objectives: Routinely collected data could support risk stratification after alcohol-related hospitalisation, but inconsistent diagnostic phenotyping and temporally invalid designs threaten model validity. We aimed to develop and temporally evaluate interpretable models for subsequent same-hospital admission recorded in the available extract after alcohol-related admissions. Methods: We analysed anonymised hospital episodes from 2009–2024. Adult alcohol-related admissions were identified by the union of local text labels and ICD-9/10-compatible alcohol codes. At each discharge, models predicted the next same-hospital admission recorded in the extract within 30, 90, and 365 days using only contemporaneous and prior information. Logistic regression, random forest, and histogram gradient boosting were developed using admissions through 2020 and evaluated on later admissions. Discrimination, calibration, Brier scores, and highest-risk-decile performance were estimated with 95% confidence intervals from patient-clustered bootstrap samples. Results: After excluding 58 records without usable admission or discharge dates and 36 records with missing age or age below 18 years, 18,335 alcohol-related index admissions from 10,478 patients were analysed. The text-only rule identified 6,652 patients, compared with 10,478 using text or ICD codes; source use changed completely from text labels in 2009–2019 to ICD codes in 2020–2024. In temporal evaluation, random-forest ROC-AUC was 0.659 (95% CI 0.631–0.683) at 30 days and 0.659 (0.637–0.681) at 90 days. At 365 days, logistic regression and random forest performed similarly (both ROC-AUC 0.659); logistic regression was well calibrated (intercept −0.044, slope 0.906). Its highest-risk decile had 57.4% positive predictive value and captured 18.3% of recorded subsequent admissions. Conclusions: The operational cohort-definition rule materially changed the study population. Temporally evaluated models provided modest risk stratification but no complex algorithm clearly outperformed logistic regression. The observed discrimination and decision-concentration metrics do not support direct clinical deployment. External evaluation and richer clinical and social predictors are required before clinical use.