Artificial Intelligence (AI)-based medical computer vision algorithm training and evaluations depend on annotations and labeling. However, variability between expert annotators introduces noise in training data that can adversely impact the performance of AI algorithms. This study aims to assess, illustrate and interpret the inter-annotator agreement among multiple expert annotators when segmenting the same lesion(s)/abnormalities on medical images. We propose the use of three metrics for the qualitative and quantitative assessment of inter-annotator agreement: 1) use of a common agreement heatmap and a ranking agreement heatmap; 2) use of the extended Cohen’s kappa and Fleiss’ kappa coefficients for a quantitative evaluation and interpretation of inter-annotator reliability; and 3) use of the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm, as a parallel step, to generate ground truth for training AI models and compute Intersection over Union (IoU), sensitivity, and specificity to assess the inter-annotator reliability and variability. Experiments are performed on two datasets, namely cervical colposcopy images from 30 patients and chest X-ray images from 336 tuberculosis (TB) patients, to demonstrate the consistency of inter-annotator reliability assessment and the importance of combining different metrics to avoid bias assessment.

Cervical cancer is caused by the persistent infection of certain types of the Human Papillomavirus (HPV) and is a leading cause of female mortality particularly in low and middle-income countries (LMIC). Visual inspection of the cervix with acetic acid (VIA) is a commonly used technique in cervical screening. While this technique is inexpensive, clinical assessment is highly subjective, and relatively poor reproducibility has been reported. A deep learning-based algorithm for automatic visual evaluation (AVE) of aceto-whitened cervical images was shown to be effective in detecting confirmed precancer (i.e. direct precursor to invasive cervical cancer). The images were selected from a large longitudinal study conducted by the National Cancer Institute in the Guanacaste province of Costa Rica. The training of AVE used annotation for cervix boundary, and the data scarcity challenge was dealt with manually optimized data augmentation. In contrast, we present a novel approach for cervical precancer detection using a deep metric learning-based (DML) framework which does not incorporate any effort for cervix boundary marking. The DML is an advanced learning strategy that can deal with data scarcity and bias training due to class imbalance data in a better way. Three different widely-used state-of-the-art DML techniques are evaluated- (a) Contrastive loss minimization, (b) N-pair embedding loss minimization, and, (c) Batch-hard loss minimization. Three popular Deep Convolutional Neural Networks (ResNet-50, MobileNet, NasNet) are configured for training with DML to produce class-separated (i.e. linearly separable) image feature descriptors. Finally, a K-Nearest Neighbor (KNN) classifier is trained with the extracted deep features. Both the feature quality and classification performance are quantitatively evaluated on the same data set as used in AVE. It shows that, unlike AVE, without using any data augmentation, the best model produced from our research improves specificity in disease detection without compromising sensitivity. The present research thus paves the way for new research directions for the related field.

Automated Visual Examination (AVE) is a deep learning algorithm that aims to improve the effectiveness of cervical precancer screening, particularly in low- and medium-resource regions. It was trained on data from a large longitudinal study conducted by the National Cancer Institute (NCI) and has been shown to accurately identify cervices with early stages of cervical neoplasia for clinical evaluation and treatment. The algorithm processes images of the uterine cervix taken with a digital camera and alerts the user if the woman is a candidate for further evaluation. This requires that the algorithm be presented with images of the cervix, which is the object of interest, of acceptable quality, i.e., in sharp focus, with good illumination, without shadows or other occlusions, and showing the entire squamo-columnar transformation zone. Our prior work has addressed some of these constraints to help discard images that do not meet these criteria. In this work, we present a novel algorithm that determines that the image contains the cervix to a sufficient extent. Non-cervix or other inadequate images could lead to suboptimal or wrong results. Manual removal of such images is labor intensive and time-consuming, particularly in working with large retrospective collections acquired with inadequate quality control. In this work, we present a novel ensemble deep learning method to identify cervix images and non-cervix images in a smartphone-acquired cervical image dataset. The ensemble method combined the assessment of three deep learning architectures, RetinaNet, Deep SVDD, and a customized CNN (Convolutional Neural Network), each using a different strategy to arrive at its decision, i.e., object detection, one-class classification, and binary classification. We examined the performance of each individual architecture and an ensemble of all three architectures. An average accuracy and F-1 score of 91.6% and 0.890, respectively, were achieved on a separate test dataset consisting of more than 30,000 smartphone-captured images.

Background: Accelerated global control of cervical cancer would require primary prevention with human papillomavirus (HPV) vaccination in addition to novel screening program strategies that are simple, inexpensive, and effective. We present the feasibility and outcome of a community-based HPV self-sampled screening program. Methods: In Ile Ife, Nigeria, 9406 women aged 30–49 years collected vaginal self-samples, which were tested for HPV in the local study laboratory using Hybrid Capture-2 (HC2) (Qiagen). HPV-positive women were referred to the colposcopy clinic. Gynecologist colposcopic impression dictated immediate management; biopsies were taken when definite acetowhitening was present to produce a histopathologic reference standard of precancer (and to determine final clinical management). Retrospective linkage to the medical records identified 442 of 9406 women living with HIV (WLWH). Results: With self-sampling, it was possible to screen more than 100 women per day per clinic. Following an audio-visual presentation and in-person instructions, overall acceptability of self-sampling was very high (81.2% women preferring self-sampling over clinician collection). HPV positivity was found in 17.3% of women. Intensive follow-up contributed to 85.9% attendance at the colposcopy clinic. Of those referred, 8.2% were initially treated with thermal ablation and 5.6% with large loop excision of transformation zone (LLETZ). Full visibility of the squamocolumnar junction, necessary for optimal visual triage and ablation, declined from 68.5% at age 30 to 35.4% at age 49. CIN2+ and CIN3+ (CIN- Cervical intraepithelial neoplasia), including five cancers, were identified by histology in 5.9 and 3.2% of the HPV-positive women, respectively (0.9 and 0.5% of the total screening population), leading to additional treatment as indicated. The prevalences of HPV infection and CIN2+ were substantially higher (40.5 and 2.5%, respectively) among WLWH. Colposcopic impression led to over- and under-treatment compared to the histopathology reference standard. Conclusion: A cervical cancer screening program using self-sampled HPV testing, with colposcopic immediate management of women positive for HPV, proved feasible in Nigeria. Based on the collected specimens and images, we are now evaluating the use of a combination of partial HPV typing and automated visual evaluation (AVE) of cervical images to improve the accuracy of the screening program.