Background: The integration of artificial intelligence (AI) into medicine is growing, with some experts predicting its standalone use soon. However, skepticism remains due to limited positive outcomes from independent validations. This research evaluates AI software’s effectiveness in analyzing chest X-rays (CXR) to identify lung nodules, a possible lung cancer indicator. Methods: This retrospective study analyzed 7,670,212 record pairs from radiological exams conducted between 2020 and 2022 during the Moscow Computer Vision Experiment, focusing on CXR and computed tomography (CT) scans. All images were acquired during clinical routine. The final dataset comprised 100 CXR images (50 with lung nodules, 50 without), selected consecutively and based on inclusion and exclusion criteria, to evaluate the performance of all five AI-based solutions, participating in the Moscow Computer Vision Experiment and analyzing CXR. The evaluation was performed in 3 stages. In the first stage, the probability of a nodule in the lung obtained from AI services was compared with the Ground Truth (1—there is a nodule, 0—there is no nodule). In the second stage, 3 radiologists evaluated the segmentation of nodules performed by the AI services (1—nodule correctly segmented, 0—nodule incorrectly segmented or not segmented at all). In the third stage, the same radiologists additionally evaluated the classification of the nodules (1—nodule correctly segmented and classified, 0—all other cases). The results obtained in stages 2 and 3 were compared with Ground Truth, which was common to all three stages. For each stage, diagnostic accuracy metrics were calculated for each AI service. Results: Three software solutions (Celsus, Lunit INSIGHT CXR, and qXR) demonstrated diagnostic metrics that matched or surpassed the vendor specifications, and achieved the highest area under the receiver operating characteristic curve (AUC) of 0.956 [95% confidence interval (CI): 0.918 to 0.994]. However, when evaluated by three radiologists for accurate nodule segmentation and classification, all solutions performed below the vendor-declared metrics, with the highest AUC reaching 0.812 (95% CI: 0.744 to 0.879). Meanwhile, all AI services demonstrated 100% specificity at stages 2 and 3 of the study. Conclusions: To ensure the reliability and applicability of AI-based software, it is crucial to validate performance metrics using high-quality datasets and engage radiologists in the evaluation process. Developers are recommended to improve the accuracy of the underlying models before allowing the standalone use of the software for lung nodule detection. The dataset created during the study may be accessed at https://mosmed.ai/datasets/mosmeddatargogksnalichiemiotsutstviemlegochnihuzlovtipvii/.

An international reader study was conducted to gauge an average diagnostic accuracy of radiologists interpreting chest X-ray images, including those from fluorography and mammography, and establish requirements for stand-alone radiological artificial intelligence (AI) models. The retrospective studies in the datasets were labelled as containing or not containing target pathological findings based on a consensus of two experienced radiologists, and the results of a laboratory test and follow-up examination, where applicable. A total of 204 radiologists from 11 countries with various experience performed an assessment of the dataset with a 5-point Likert scale via a web platform. Eight commercial radiological AI models analyzed the same dataset. The AI AUROC was 0.87 (95% CI:0.83–0.9) versus 0.96 (95% CI 0.94–0.97) for radiologists. The sensitivity and specificity of AI versus radiologists were 0.71 (95% CI 0.64–0.78) versus 0.91 (95% CI 0.86–0.95) and 0.93 (95% CI 0.89–0.96) versus 0.9 (95% CI 0.85–0.94) for AI. The overall diagnostic accuracy of radiologists was superior to AI for chest X-ray and mammography. However, the accuracy of AI was noninferior to the least experienced radiologists for mammography and fluorography, and to all radiologists for chest X-ray. Therefore, an AI-based first reading could be recommended to reduce the workload burden of radiologists for the most common radiological studies such as chest X-ray and mammography.

Study Design: Retrospective cohort study.Purpose: To evaluate the effectiveness of opportunistic osteoporosis screening using an artificial intelligence (AI) algorithm for detecting vertebral compression deformity (VCD >25%) and reduced bone mineral density (BMD) from routine chest computed tomography (CT) scans.Overview of Literature: Osteoporosis is an insidious metabolic disease that often remains asymptomatic for a long time, and is typically diagnosed due to the occurrence of complications. An approach for diagnosing osteoporosis based on routine CT examinations, including the use of AI services, is being actively studied.Methods: Patients aged >50 years who underwent chest CT using the standard protocol between 09.06.2021 and 30.06.2021 at four inpatient and three outpatient clinics were retrospectively enrolled. The morphometric AI algorithm detected vertebral compression index and vertebrae density in Hounsfield units (HU). The AI algorithm was calibrated for BMD measurements using a phantom. Osteoporotic BMD was defined according to the American College of Radiology criteria (<80 mg/mL). The presence of vertebral fracture (VF) was verified using semiquantitative and algorithm-based qualitative methods by three radiologists, followed by two experts with 15 and 35 years of experience in the field.Results: CT studies of 1,888 patients (mean age, 66.3 years) were included. The AI algorithm detected VCD in 336 patients (17.8%), with 105 (5.5%) having VF. Low BMD values were detected in 501 patients (26.5%; 31.0% of females, 18.6% of males).Conclusions: We observed high diagnostic accuracy of opportunistic osteoporosis screening using AI algorithms for detecting VF and low BMD. AI-based opportunistic screening of osteoporosis and VF in chest CT scans can facilitate the identification of high-risk cohorts.

Introduction. Determining the minimum sample size for solving various tasks is an extremely important and at the same time unexplored problem. There are many methods, but most of them are not applicable for AI-based software validation.Aim: To consider a methodology for determining a balance of classes “norm”/ “abnormality” and propose a statistical approach to determine the data amount necessary for testing AI-based software (validation).Material and Methods. The results of AI-based software were analyzed using dataset of mammograms. Mammograms were classified by the presence of breast cancer (“abnormality”) and the absence of breast cancer (“norm”). The general set contains 123,301 unique studies. The original balance of classes in the study was “norm” 89.3%/“abnormality” 10.7%. As the results of AI-based software (ML-algorithm), a probability of the presence of pathology in the entire study was taken. The following values were used as empirical data (GT): 0 – in case of Bi-RADS classes 1 or 2 diagnosed by a doctor, and 1 – in case of Bi-RADS classes 3, 4, 5. Each data sample is transferred to AI-based software for processing. Quality metrics are calculated based on its results: AUC ROC. All the described actions were repeated 10,000 times for all the studied balances of “norm”/”abnormality”. Based on the results of AUC ROC calculations, mean values were calculated for different random data series with the same balances. Mean AUC ROC values were subjected to analysis.Results. A maximum value of the coefficient of variation of AUC ROC values for 10% “abnormality” share is achieved at the number of studies equal to 190; for the 20% share, it is 80 studies; for the 30% share – 120 studies, for the 40% share – 110 studies, and for the 50% share – 70 studies.Conclusion. Summarizing the conducted study results, it can be concluded that when testing AI-based software, it is necessary to consider that the number of studies reflecting the greatest heterogeneity of AUC ROC values (the largest deviation from the mean value) is different for various class balances. If the purpose of validation is to establish the worst-case behavior of AUC ROC values, then for the studied AI-based software, the “abnormality” share should be 10%, and the number of studies 190. If the validation is carried out under conditions of a limited amount of data, then the “abnormality” share should be 50% and the number of studies equal to 70.

Background: Electronic health records (EHRs) remain a vital source of clinical information, yet processing these heterogeneous data is extremely labor-intensive. Summarization of these data using Large Language Models (LLMs) is considered a promising tool to support practicing physicians. Unbiased, automated quality control is crucial for integrating the tools into routine practice, saving time and labor. This pilot study aimed to assess the potential and constraints of self-contained evaluation of summarization quality (without expert involvement) based on automatic evaluation metrics and LLM-as-a-judge. Methods: The summaries of text data from 30 EHRs were generated by six open-source low-parameter LLMs. The medical summaries were evaluated using standard automatic metrics (BLEU, ROUGE, METEOR, BERTScore) as well as the LLM-as-a-judge approach using the following criteria: relevance, completeness, redundancy, coherence and structure, grammar and terminology, and hallucinations. Expert evaluation was conducted using the same criteria. Results: The results showed that LLMs hold great promise for summarizing medical data. Nevertheless, neither the evaluation metrics nor LLM judges are reliable in detecting factual errors and semantic distortions (hallucinations). In terms of relevance, the Pearson correlation between the summary quality score and the expert opinions was 0.688. Conclusions: Completely automating the evaluation of medical summaries remains challenging. Further research should focus on dedicated methods for detecting hallucinations, along with investigating larger or specialized models trained on medical texts. Additionally, the potential integration of retrieval-augmented generation (RAG) within the LLM-as-a-judge architecture deserves attention. Nevertheless, even now, the combination of LLMs and the automatic evaluation metrics can underpin medical decision support systems by performing initial evaluations and highlighting potential shortcomings for expert review.

Computer vision algorithms based on artificial intelligence for diagnostic radiology are rapidly developing. For effective implementation in clinical practice, it is necessary to determine the capabilities and limitations of these algorithms. Objective. To evaluate the threshold values of the quality metrics of computer vision artificial intelligence-based algorithms for the analysis of mammographic examinations in comparison with the diagnostic accuracy of the radiologist. Materials and methods. The study involving a group of radiologists was performed on the «Web-based Radiology Evaluation Platform» on a labeled dataset from mammography studies. The same dataset was sent to 5 artificial intelligence (AI)-based algorithms. Metrics of diagnostic accuracy of an average radiologist and each AI service were obtained using ROC analysis. Results. The diagnostic accuracy of radiologists (AUC) significantly exceeded the indices of 2 out of 5 AI services. None of the AI services in the study significantly exceeded the AUC of the «average» radiologist. The diagnostic accuracy metrics of the «average» radiologist were: AUC 0.928 (95% CI 0.883—0.976), sensitivity 0.792 (95% CI 0.677—0.907), specificity 0.940 (95% CI 0.874—1.000). Conclusion. When deciding on the implementation of artificial intelligence-based computer vision algorithms for preventive mammography, the minimum AUC value obtained for the “average” radiologist should be used (>0.88).