Multimodal artificial intelligence (AI) chatbots have been proposed as useful tools to answer patient questions about general medicine.
1 Multimodal AI chatbots, compared with unimodal, text-only AI chatbots, can process complex medical image and text-based information that may further improve their accuracy as a diagnostic and management tool.
2,3 Multimodal chatbots have demonstrated an accuracy of more than 70% when answering questions about general medicine clinical cases based on free-text and image-based input.
4 Coupled with the adoption of electronic health record systems that can access large amounts of digital patient data, computer-assisted clinical decision support systems enabled by multimodal AI can facilitate and potentially optimize clinician decision-making with clinical knowledge and patient data.
5 Pilot studies of AI chatbots have shown that the implementation of generative AI tools in clinical settings may relieve clinician burden and burnout.
6,7 By integrating multimodal AI with clinical decision support systems, clinicians may interact with a chatbot interface by asking open-ended questions about the patient presentation and leverage multimodal data, including imaging, unstructured clinical text in the electronic health record, and laboratory results, to iteratively generate a recommendation for clinician interpretation.