Background: In recent years, ophthalmology has emerged as a new frontier in medical artificial intelligence (AI) with multi-modal AI in ophthalmology garnering significant attention across interdisciplinary research. This integration of various types and data models holds paramount importance as it enables the provision of detailed and precise information for diagnosing eye and vision diseases. By leveraging multi-modal ophthalmology AI techniques, clinicians can enhance the accuracy and efficiency of diagnoses, and thus reduce the risks associated with misdiagnosis and oversight while also enabling more precise management of eye and vision health. However, the widespread adoption of multi-modal ophthalmology poses significant challenges. Main text: In this review, we first summarize comprehensively the concept of modalities in the field of ophthalmology, the forms of fusion between modalities, and the progress of multi-modal ophthalmic AI technology. Finally, we discuss the challenges of current multi-modal AI technology applications in ophthalmology and future feasible research directions. Conclusion: In the field of ophthalmic AI, evidence suggests that when utilizing multi-modal data, deep learning-based multi-modal AI technology exhibits excellent diagnostic efficacy in assisting the diagnosis of various ophthalmic diseases. Particularly, in the current era marked by the proliferation of large-scale models, multi-modal techniques represent the most promising and advantageous solution for addressing the diagnosis of various ophthalmic diseases from a comprehensive perspective. However, it must be acknowledged that there are still numerous challenges associated with the application of multi-modal techniques in ophthalmic AI before they can be effectively employed in the clinical setting.

Background/aims: To improve the accuracy of pterygium screening and detection through smartphones, we established a fusion training model by blending a large number of slit-lamp image data with a small proportion of smartphone data.Method: Two datasets were used, a slit-lamp image dataset containing 20 987 images and a smartphone-based image dataset containing 1094 images. The RFRC (Faster RCNN based on ResNet101) model for the detection model. The SRU-Net (U-Net based on SE-ResNeXt50) for the segmentation models. The open-cv algorithm measured the width, length and area of pterygium in the cornea.Results: The detection model (trained by slit-lamp images) obtained the mean accuracy of 95.24%. The fusion segmentation model (trained by smartphone and slit-lamp images) achieved a microaverage F₁score of 0.8981, sensitivity of 0.8709, specificity of 0.9668 and area under the curve (AUC) of 0.9295. Compared with the same group of patients’ smartphone and slit-lamp images, the fusion model performance in smartphone-based images (F₁score of 0.9313, sensitivity of 0.9360, specificity of 0.9613, AUC of 0.9426, accuracy of 92.38%) is close to the model (trained by slit-lamp images) in slit-lamp images (F₁score of 0.9448, sensitivity of 0.9165, specificity of 0.9689, AUC of 0.9569 and accuracy of 94.29%).Conclusion: Our fusion model method got high pterygium detection and grading accuracy in insufficient smartphone data, and its performance is comparable to experienced ophthalmologists and works well in different smartphone brands.

Purpose: Accurate quantification measurement of tear meniscus is vital for the precise diagnosis of dry eye. In current clinical practice, the measurement of tear meniscus height (TMH) relies on doctors’ manual operation. This study aims to propose a novel automatic artificial intelligence (AI) system to evaluate TMH. Methods: A total of 510 photographs obtained by the oculus camera were labeled. Three thousand and five hundred images were finally attained by data enhancement to train the neural network model parameters, and 60 were used to evaluate the model performance in segmenting the cornea and tear meniscus region. One hundred images were used to test generalization ability of the model. We modified a segmentation model of the cornea and the tear meniscus based on the UNet-like network. The output of the segmentation model is followed by a calculation module that calculates and reports the TMH. Results: Compared with ground truth (GT) manually labeled by clinicians, our modified model achieved a Dice Similarity Coefficient (DSC) and Intersection over union (Iou) of 0.99/0.98 in the corneal segmentation task and 0.92/0.86 for the detection of tear meniscus on the validation set, respectively. On the test set, the TMH automatically measured by our AI system strongly correlates with the results manually calculated by the ophthalmologists. Conclusions: We developed a fully automated and reliable AI system to obtain TMH. After large-scale clinical testing, our method could be used for dry eye screening in clinical practice.