International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Knowledge of whole heart anatomy is a prerequisite for many clinical applications. Whole heart segmentation (WHS), which delineates substructures of the heart, can be very valuable for modeling and analysis of the anatomy and functions of the heart. However, automating this segmentation can be challenging due to the large variation of the heart shape, and different image qualities of the clinical data. To achieve this goal, an initial set of training data is generally needed for constructing priors or for training. Furthermore, it is difficult to perform comparisons between different methods, largely due to differences in the datasets and evaluation metrics used. This manuscript presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The challenge provided 120 three-dimensional cardiac images covering the whole heart, including 60 CT and 60 MRI volumes, all acquired in clinical environments with manual delineation. Ten algorithms for CT data and eleven algorithms for MRI data, submitted from twelve groups, have been evaluated. The results showed that the performance of CT WHS was generally better than that of MRI WHS. The segmentation of the substructures for different categories of patients could present different levels of challenge due to the difference in imaging and variations of heart shapes. The deep learning (DL)-based methods demonstrated great potential, though several of them reported poor results in the blinded evaluation. Their performance could vary greatly across different network structures and training strategies. The conventional algorithms, mainly based on multi-atlas segmentation, demonstrated good performance, though the accuracy and computational efficiency could be limited. The challenge, including provision of the annotated training data and the blinded evaluation for submitted algorithms on the test data, continues as an ongoing benchmarking resource via its homepage (www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mmwhs/).

Both acetabular undercoverage (hip dysplasia) and overcoverage (pincer-type femoroacetabular impingement) can result in hip osteoarthritis. In contrast to undercoverage, there is a lack of information on radiographic reference values for excessive acetabular coverage. (1) How do common radiographic hip parameters differ in hips with a deficient or an excessive acetabulum in relation to a control group; and (2) what are the reference values determined from these data for acetabular under- and overcoverage? We retrospectively compared 11 radiographic parameters describing the radiographic acetabular anatomy among hip dysplasia (26 hips undergoing periacetabular osteotomy), control hips (21 hips, requiring no rim trimming during surgical hip dislocation), hips with overcoverage (14 hips, requiring rim trimming during surgical hip dislocation), and hips with severe overcoverage (25 hips, defined as having acetabular protrusio). The hips were selected from a patient cohort of a total of 593 hips. Radiographic parameters were assessed with computerized methods on anteroposterior pelvic radiographs and corrected for neutral pelvic orientation with the help of a true lateral radiograph. All parameters except the crossover sign differed among the four study groups. From dysplasia through control and overcoverage, the lateral center-edge angle, acetabular arc, and anteroposterior/craniocaudal coverage increased. In contrast, the medial center-edge angle, extrusion/acetabular index, Sharp angle, and prevalence of the posterior wall sign decreased. The following reference values were found: lateral center-edge angle 23° to 33°, medial center-edge angle 35° to 44°, acetabular arc 61° to 65°, extrusion index 17% to 27%, acetabular index 3° to 13°, Sharp angle 38° to 42°, negative crossover sign, positive posterior wall sign, anterior femoral head coverage 15% to 26%, posterior femoral head coverage 36% to 47%, and craniocaudal coverage 70% to 83%. These acetabular reference values define excessive and deficient coverage. They may be used for radiographic evaluation of symptomatic hips, may offer possible predictors for surgical outcomes, and serve to guide clinical decision-making. Level III, diagnostic study.

Anteroposterior pelvic radiographs are the gold standard of imaging for mechanical hip problems. However, correct interpretation is difficult because the projected morphologic features of the acetabulum and nearly all routinely used hip parameters depend on individual pelvic position, which can vary considerably during acquisition. We developed software that recreates the projected acetabular rim and the measured hip parameters as if obtained in a standardized orientation. The vertical and horizontal distances between two easy identifiable points were used as indicators of tilt and rotation. These points were the middle of the sacrococcygeal joint and the middle of the upper border of the symphyseal gap. Calibration of the indicators was achieved by means of serial pelvic radiographs of 20 cadaver pelves. Validation of tilt indicator in 100 patients and a theoretical error analysis revealed that for accurate tilt prediction an additional one-time lateral radiograph of the pelvis is mandatory. The computer-assisted method allows standardized evaluation of anatomic morphologic differences of femoral coverage (dysplasia, retroversion), making their clinical relevance for development of early osteoarthritis more valuable.

Cephalometric analysis is an essential clinical and research tool in orthodontics for the orthodontic analysis and treatment planning. This paper presents the evaluation of the methods submitted to the Automatic Cephalometric X-Ray Landmark Detection Challenge, held at the IEEE International Symposium on Biomedical Imaging 2014 with an on-site competition. The challenge was set to explore and compare automatic landmark detection methods in application to cephalometric X-ray images. Methods were evaluated on a common database including cephalograms of 300 patients aged six to 60 years, collected from the Dental Department, Tri-Service General Hospital, Taiwan, and manually marked anatomical landmarks as the ground truth data, generated by two experienced medical doctors. Quantitative evaluation was performed to compare the results of a representative selection of current methods submitted to the challenge. Experimental results show that three methods are able to achieve detection rates greater than 80% using the 4 mm precision range, but only one method achieves a detection rate greater than 70% using the 2 mm precision range, which is the acceptable precision range in clinical practice. The study provides insights into the performance of different landmark detection approaches under real-world conditions and highlights achievements and limitations of current image analysis techniques.