BACKGROUND: In recent years, the availability of medical datasets and technologies for software development based on artificial intelligence technology has resulted in a growth in the number of solutions for medical diagnostics, particularly mammography. Registered as a medical device, this program can interpret digital mammography, significantly saving time, material, and human resources in healthcare while ensuring the quality of mammary gland preventive studies. AIM: This study aims to justify the possibility and effectiveness of artificial intelligence-based software for the first interpretation of digital mammograms while maintaining the practice of a radiologists second description of X-ray images. MATERIALS AND METHODS: A dataset of 100 digital mammography studies (50 absence of target pathology and 50 ― presence of target pathology, with signs of malignant neoplasms) was processed by software based on artificial intelligence technology that was registered as a medical device in the Russian Federation. Receiver operating characteristic analysis was performed. Limitations of the study include the values of diagnostic accuracy metrics obtained for software based on artificial intelligence technology versions, relevant at the end of 2022. RESULTS: When set to 80.0% sensitivity, artificial intelligence specificity was 90.0% (95% CI, 81.798.3), and accuracy was 85.0% (95% CI, 78.092.0). When set to 100% specificity, artificial intelligence demonstrated 56.0% sensitivity (95% CI, 42.269.8) and 78.0% accuracy (95% CI, 69.986.1). When the sensitivity was set to 100%, the artificial intelligence specificity was 54.0% (95% CI, 40.267.8), and the accuracy was 77.0% (95% CI, 68.885.2). Two approaches have been proposed, providing an autonomous first interpretation of digital mammography using artificial intelligence. The first approach is to evaluate the X-ray image using artificial intelligence with a higher sensitivity than that of the double-reading mammogram by radiologists, with a comparable level of specificity. The second approach implies that artificial intelligence-based software will determine the mammogram category (absence of target pathology or presence of target pathology), indicating the degree of confidence in the obtained result, depending on the corridor into which the predicted value falls. CONCLUSIONS: Both proposed approaches for using artificial intelligence-based software for the autonomous first interpretation of digital mammograms can provide diagnostic quality comparable to, if not superior to, double-image reading by radiologists. The economic benefit from the practical implementation of this approach nationwide can range from 0.6 to 5.5 billion rubles annually.

Background: The integration of artificial intelligence (AI) into medicine is growing, with some experts predicting its standalone use soon. However, skepticism remains due to limited positive outcomes from independent validations. This research evaluates AI software’s effectiveness in analyzing chest X-rays (CXR) to identify lung nodules, a possible lung cancer indicator. Methods: This retrospective study analyzed 7,670,212 record pairs from radiological exams conducted between 2020 and 2022 during the Moscow Computer Vision Experiment, focusing on CXR and computed tomography (CT) scans. All images were acquired during clinical routine. The final dataset comprised 100 CXR images (50 with lung nodules, 50 without), selected consecutively and based on inclusion and exclusion criteria, to evaluate the performance of all five AI-based solutions, participating in the Moscow Computer Vision Experiment and analyzing CXR. The evaluation was performed in 3 stages. In the first stage, the probability of a nodule in the lung obtained from AI services was compared with the Ground Truth (1—there is a nodule, 0—there is no nodule). In the second stage, 3 radiologists evaluated the segmentation of nodules performed by the AI services (1—nodule correctly segmented, 0—nodule incorrectly segmented or not segmented at all). In the third stage, the same radiologists additionally evaluated the classification of the nodules (1—nodule correctly segmented and classified, 0—all other cases). The results obtained in stages 2 and 3 were compared with Ground Truth, which was common to all three stages. For each stage, diagnostic accuracy metrics were calculated for each AI service. Results: Three software solutions (Celsus, Lunit INSIGHT CXR, and qXR) demonstrated diagnostic metrics that matched or surpassed the vendor specifications, and achieved the highest area under the receiver operating characteristic curve (AUC) of 0.956 [95% confidence interval (CI): 0.918 to 0.994]. However, when evaluated by three radiologists for accurate nodule segmentation and classification, all solutions performed below the vendor-declared metrics, with the highest AUC reaching 0.812 (95% CI: 0.744 to 0.879). Meanwhile, all AI services demonstrated 100% specificity at stages 2 and 3 of the study. Conclusions: To ensure the reliability and applicability of AI-based software, it is crucial to validate performance metrics using high-quality datasets and engage radiologists in the evaluation process. Developers are recommended to improve the accuracy of the underlying models before allowing the standalone use of the software for lung nodule detection. The dataset created during the study may be accessed at https://mosmed.ai/datasets/mosmeddatargogksnalichiemiotsutstviemlegochnihuzlovtipvii/.

Introduction: A rapid development of artificial intelligence software, including that for the analysis of digital fluorography images, has been noted recently. Pre-registered as a medical device, this software can be used for stand-alone analysis and triage of test results, allowing radiologists to focus on pathological findings. Objective: To substantiate the possibility and efficiency of using artificial intelligence software for stand-alone analysis and triage of digital fluorography images. Materials and methods: 88,048 digital fluorograms obtained in the first quarter of 2023 were processed using the artificial intelligence software registered in the Russian Federation as a medical device and a ROC analysis of the findings was carried out. Results: We established that default software settings with the sensitivity of 90.4 % (95 % CI: 88.2–92.7) produced specificity of 75.5 % (95 % CI: 75.2–75.8) and accuracy of 75.6 % (95 % CI: 75.3–75.9). At the maximum sensitivity of 100.0 % (95 % CI: 100–100), specificity was 77.4 % (95 % CI: 74.8–80.0) and accuracy was as high as 77.9 % (95 % CI: 75.3–80.5). We have proposed a model of organizing health care which provides for stand-alone sorting of fluorography images by the software, saving normal results without their verification by a radiologist, and sending images with abnormal findings to a radiologist for diagnosis (in the future, as artificial intelligence improves, the latter will be immediately sent to a physician of the clinical specialty). Conclusions: The established optimal scenario includes the use of artificial intelligence software to identify normal findings, which examination by a radiologist is optional when the algorithm is set to maximum sensitivity. Only the findings classified as abnormal will be subject to mandatory revision. The annual economic benefit gained by practical implementation of this approach nationwide can reach 5.6 billion rubles.

Background: The implementation of radiological artificial intelligence (AI) solutions remains challenging due to limitations in existing testing methodologies. This study assesses the efficacy of a comprehensive methodology for performance testing and monitoring of commercial-grade mammographic AI models. Methods: We utilized a combination of retrospective and prospective multicenter approaches to evaluate a neural network based on the Faster R-CNN architecture with a ResNet-50 backbone, trained on a dataset of 3641 mammograms. The methodology encompassed functional and calibration testing, coupled with routine technical and clinical monitoring. Feedback from testers and radiologists was relayed to the developers, who made updates to the AI model. The test dataset comprised 112 medical organizations, representing 10 manufacturers of mammography equipment and encompassing 593,365 studies. The evaluation metrics included the area under the curve (AUC), accuracy, sensitivity, specificity, technical defects, and clinical assessment scores. Results: The results demonstrated significant enhancement in the AI model’s performance through collaborative efforts among developers, testers, and radiologists. Notable improvements included functionality, diagnostic accuracy, and technical stability. Specifically, the AUC rose by 24.7% (from 0.73 to 0.91), the accuracy improved by 15.6% (from 0.77 to 0.89), sensitivity grew by 37.1% (from 0.62 to 0.85), and specificity increased by 10.7% (from 0.84 to 0.93). The average proportion of technical defects declined from 9.0% to 1.0%, while the clinical assessment score improved from 63.4 to 72.0. Following 2 years and 9 months of testing, the AI solution was integrated into the compulsory health insurance system. Conclusions: The multi-stage, lifecycle-based testing methodology demonstrated substantial potential in software enhancement and integration into clinical practice. Key elements of this methodology include robust functional and diagnostic requirements, continuous testing and updates, systematic feedback collection from testers and radiologists, and prospective monitoring.

Aim. To develop an artificial intelligence-based system for the diagnosis of focal liver lesions aimed at supporting clinical decision-making in surgical hepatology.Materials and methods. An artificial intelligence-based technological service was developed for the automatic segmentation and classification of contrast-enhanced computed tomography (CT) images of four types of liver lesions: focal nodular hyperplasia, carcinoma, hemangioma, and simple cyst. The service was trained and tested on datasets comprising 725 CT images using the nnU-Net architecture. Diagnostic performance was evaluated by calculating the AUC-ROC, sensitivity, specificity, and accuracy.Results. The service achieved high performance metrics. The AUC-ROC ranged from 0.847 to 0.928, with a maximum sensitivity of 0.940 for carcinoma and a specificity of 0.900 for focal nodular hyperplasia. Accuracy ranged from 0.883 to 0.922, which demonstrates the algorithm's ability to clearly differentiate between malignant and benign lesions.Conclusion. The machine learning-based service demonstrated high diagnostic performance and shows promise for integration into clinical practice, offering improved detection and classification of liver lesions.