5 August 2023 in Journal of Biomedical Informatics
With the increasing amount and growing variety of healthcare data, multimodal machine learning supporting integrated modeling of structured and unstructured data is an increasingly important tool for clinical machine learning tasks. However, it is non-trivial to manage the differences in dimensionality, volume, and temporal characteristics of data modalities in the context of a shared target task. Furthermore, patients can have substantial variations in the availability of data, while existing multimodal modeling methods typically assume data completeness and lack a mechanism to handle missing modalities. We propose a Transformer-based fusion model with modality-specific tokens that summarize the corresponding modalities to achieve effective cross-modal interaction accommodating missing modalities in the clinical context. The model is further refined by inter-modal, inter-sample contrastive learning to improve the representations for better predictive performance. We denote the model as Attention-based cRoss-MOdal fUsion with contRast (ARMOUR). We evaluate ARMOUR using two input modalities (structured measurements and unstructured text), six clinical prediction tasks, and two evaluation regimes, either including or excluding samples with missing modalities. Our model shows improved performances over unimodal or multimodal baselines in both evaluation regimes, including or excluding patients with missing modalities in the input. The contrastive learning improves the representation power and is shown to be essential for better results. The simple setup of modality-specific tokens enables ARMOUR to handle patients with missing modalities and allows comparison with existing unimodal benchmark results. We propose a multimodal model for robust clinical prediction to achieve improved performance while accommodating patients with missing modalities. This work lays a foundation for future work to study the effective incorporation of multiple modalities of clinical data into a single model.
1 July 2021 in npj Digital Medicine
As healthcare providers receive fixed amounts of reimbursement for given services under DRG (Diagnosis-Related Groups) payment, DRG codes are valuable for cost monitoring and resource allocation. However, coding is typically performed retrospectively post-discharge. We seek to predict DRGs and DRG-based case mix index (CMI) at early inpatient admission using routine clinical text to estimate hospital cost in an acute setting. We examined a deep learning-based natural language processing (NLP) model to automatically predict per-episode DRGs and corresponding cost-reflecting weights on two cohorts (paid under Medicare Severity (MS) DRG or All Patient Refined (APR) DRG), without human coding efforts. It achieved macro-averaged area under the receiver operating characteristic curve (AUC) scores of 0·871 (SD 0·011) on MS-DRG and 0·884 (0·003) on APR-DRG in fivefold cross-validation experiments on the first day of ICU admission. When extended to simulated patient populations to estimate average cost-reflecting weights, the model increased its accuracy over time and obtained absolute CMI error of 2·40 (1·07%) and 12·79% (2·31%), respectively on the first day. As the model could adapt to variations in admission time, cohort size, and requires no extra manual coding efforts, it shows potential to help estimating costs for active patients to support better operational decision-making in hospitals.
27 May 2025 in IEEE Journal of Biomedical and Health Informatics
Clinical predictive analysis is a crucial task with numerous applications and has been extensively studied using machine learning approaches. Clinical notes, a vital data source, have been employed to develop natural language processing (NLP) models for risk prediction in healthcare with robust performance. However, clinical notes vary considerably in text composition–written by diverse healthcare providers for different purposes–and the impact of these variations on NLP modeling is also underexplored. It also remains uncertain whether the recent Large Language Models (LLMs) with instruction-following capabilities can effectively handle the risk prediction task out-of-the-box, especially when using routinely collected clinical notes instead of polished text. We address these two important research questions in the context of in-hospital mortality prediction within the critical care setting. Specifically, we propose a supervised hierarchical network with note-specific modules to account for variations across different note categories, and provide a detailed comparison with strong supervised baselines and LLMs. We benchmark 34 instruction-following LLMs based on zero-shot, few-shot, and chain-of-thought prompting with diverse prompt templates. Our results demonstrate that the note-specific network delivers improved risk prediction performance compared to established supervised baselines from both measurement-based and text-based modeling. In contrast, LLMs consistently underperform on this critical task, despite their remarkable performances in other domains. This highlights important limitations and raises caution regarding the use of LLMs for risk assessment in the critical setting. Additionally, we show that the proposed model can be leveraged to select informative clinical notes to enhance the training of other models.