Clinicians rely on various data modalities-such as patient history, clinical signs, imaging, and laboratory results-to improve decision making. Multimodal artificial intelligence (AI) systems are emerging as powerful tools to process these diverse data types; however, the clinical adoption of multimodal AI systems is challenging because of data heterogeneity and integration complexities. The 2024 Temerty Centre for AI Research and Education in Medicine symposium, held on June 17, 2024, in Toronto, Canada, explored the potential and challenges of implementing multimodal AI in health care. In this Review, we summarise insights from the symposium. We discuss current applications, such as those used in early diagnosis of sepsis and cardiology, and identify key barriers, including fusion techniques, model selection, generalisation, fairness, safety, security, and international considerations on the responsible deployment of multimodal AI in health care. We outline practical strategies to overcome these obstacles, emphasising technologies such as federated learning to reduce bias and promote equitable health care. By addressing these challenges, multimodal AI can transform clinical practice and improve patient outcomes worldwide.

Short-term acclimation response of individual cells of Thalassiosira weissflogii was monitored by Synchrotron FTIR imaging over the span of 75 minutes. The cells, collected from batch cultures, were maintained in a constant flow of medium, at an irradiance of 120 μmol m⁻² s⁻¹ and at 20 °C. Multiple internal reflections due to the micro fluidic channel were modeled, and showed that fringes are additive sinusoids to the pure absorption of the other components of the system. Preprocessing of the hyperspectral cube (x, y, Abs(λ)) included removing spectral fringe using an EMSC approach. Principal component analysis of the time series of hyperspectral cubes showed macromolecular pool variations (carbohydrates, lipids and DNA/RNA) of less than 2% after fringe correction.

The COVID-19 pandemic has underscored the importance of vaccines, especially for immunocompromised populations like solid organ transplant (SOT) recipients, who often have weaker immune responses. The purpose of this study was to compare deep learning architectures for predicting SARS-CoV-2 vaccine responses 12 months post-vaccination in this high-risk group. Utilizing data from 303 SOT recipients from a Canadian multicenter cohort, models were developed to forecast anti-receptor binding domain (RBD) antibody levels. The study compared traditional machine learning models-logistic regression, epsilon-support vector regression, random forest regressor, and gradient boosting regressor-and deep learning architectures, including long short-term memory (LSTM), recurrent neural networks, and a novel model, routed LSTM. This new model combines capsule networks with LSTM to reduce the need for large datasets. Demographic, clinical, and transplant-specific data, along with longitudinal antibody measurements, were incorporated into the models. The routed LSTM performed best, achieving a mean square error (MSE) of 0.02±0.02 and a Pearson correlation coefficient (PCC) of 0.79±0.24, outperforming all other models. Key factors influencing vaccine response included age, immunosuppression, breakthrough infection, BMI, sex, and transplant type. These findings suggest that AI could be a valuable tool in tailoring vaccine strategies, improving health outcomes for vulnerable transplant recipients.

Transplantation is one of the few areas in medicine in which the definitive treatment is rationed. Subjective decision making poses challenges for the transplantation selection process. Large language models (LLMs), through autonomous artificial intelligence (AI) agents, have implications in objective decision making for complex problems; however, there are gaps in the literature regarding their clinical applications. We aimed to examine the performance of a multidisciplinary committee of AI agents in the liver transplantation selection process as a proof of concept towards improving the objectivity of decision making. This retrospective, hybrid cohort study incorporated both real-world observational data and artificially generated data derived from the Scientific Registry of Transplant Recipients (SRTR) database in the USA. We retrospectively analysed adult patients (aged ≥18 years) who received liver transplants from Jan 1, 2004 to June 1, 2024, and generated a hypothetical cohort of patients with standard absolute contraindications to liver transplantation from a random subset of the cohort. An AI selection committee was configured, consisting of four LLM-based AI agents using GPT-4o-2024-08-06 prompted with specialised roles: transplant hepatologist, transplant surgeon, cardiologist, and social worker. This committee was instructed to assess clinical vignettes generated from the SRTR tabular data (eg, demographics, social determinants of health, end-stage liver disease, comorbidities, and laboratory measurements), placing candidates on the transplantation waitlist if liver transplantation would likely offer a 6-month or 1-year survival benefit or rejecting candidates for transplantation if contraindications were present or if such a survival benefit was unlikely. The primary objective was to assess the accuracy, sensitivity, and specificity of the AI selection committee in predicting 1-year post-transplantation survival. The secondary objectives were the accuracy, sensitivity, and specificity of identifying contraindications to liver transplantation and predicting 6-month post-transplantation survival, and the numbers of false-negative cases (ie, patients who were rejected by the AI selection committee but did receive a transplant) and false-positive cases (ie, patients who were placed on the waitlist by the AI selection committee but had assigned contraindications to transplantation). Of 8412 patients, 7033 (83·6%) were placed on the transplantation waitlist and 1379 (16·4%) were assigned to the hypothetical cohort and had contraindications to liver transplantation. For the prediction of 1-year post-transplantation survival, the AI selection committee had an accuracy of 92·00% (95% CI 91·43-92·58), a sensitivity of 1·00 (0·99-1·00), and a specificity of 0·66 (0·64-0·68). For identifying absolute contraindications to liver transplantation and predicting 6-month survival, accuracies were 98·19% (97·90-98·44) and 94·88% (94·37-95·29), respectively; sensitivities were 1·00 (0·99-1·00) in both cases; and specificities were 0·91 (0·89-0·92) and 0·75 (0·73-0·77), respectively. The most common reason for the AI selection committee to reject patients in false-negative cases was hepatocellular carcinoma that did not meet the Milan criteria (16 [62%] of 26 as a contraindication to transplantation, 16 [62%] of 26 rejected on the basis of 6-month survival prediction, and 14 [61%] of 23 rejected on the basis of 1-year survival prediction); the most common cause of death in false-positive cases was malignancy at 1 year (137 [28%] of 481). LLMs can be leveraged through a multiagent AI system to simulate discussions of a liver transplantation selection committee and provide accurate, objective insights on patients who might or might not benefit from a liver transplant. Lessons learned from this study are a step towards making the liver transplantation selection process more equitable and objective. Transplant AI Initiative, Ajmera Transplant Centre, University Health Network.