Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint—a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

To explore in a prospective trial the prognostic value of hypoxia imaging before and during radiochemotherapy in patients with locally advanced head and neck cancer. Twenty-five patients with stage III/IV head and neck cancer were investigated with [(18)F]-fluoromisonidazole (FMISO) PET/CT at four time points during radiochemotherapy (baseline, 8-10 Gy, 18-20 Gy,50-60 Gy). FMISO PET/CT image parameters were extracted including maximum-tumour-to-background (TBR(max)) and thresholded volume at different TBR ratios. CT volume and baseline FDG-PET/CT image parameters were also included. Parameters at all time points were investigated for their prognostic value with the local-progression-free-survival endpoint (LPFS). Significance was evaluated with multivariate Cox (including clinical parameters) and Log-rank tests. FMISO-image parameters were found to have a strong association with the LPFS endpoint, and were strongest at the week 1 and 2 time points (p = 0.023-0.048 and 0.042-0.061 respectively on multivariate Cox). Parameters measured at baseline were only significant on univariate analysis. None of the clinical parameters, and also FDG- or CT-delineated volumes, were significantly associated with LPFS. This prospective, exploratory study demonstrated that FMISO-PET/CT imaging during the initial phase of treatment carries strong prognostic value. FMISO-PET/CT imaging at 1 or 2 weeks during treatment could be promising way to select patients that would benefit from hypoxia modification or dose-escalated treatment. A validation study is on-going.

Light microscopy enables researchers to observe cellular mechanisms with high spatial and temporal resolution. However, the increasing complexity of current imaging technologies, coupled with financial constraints of potential users, hampers the general accessibility and potential reach of cutting-edge microscopy. Open microscopy can address this issue by making well-designed and well-documented hardware and software solutions openly available to a broad audience. In this Comment, we provide a definition of open microscopy and present recent projects in the field. We discuss current and future challenges of open microscopy and their implications for funders, policymakers, researchers and scientists. We believe that open microscopy requires a holistic approach. Sample preparation, designing and building of hardware components, writing software, data acquisition and data interpretation must go hand in hand to enable interdisciplinary and reproducible science to the benefit of society.

Modern research in the life sciences is unthinkable without computational methods for extracting, quantifying and visualising information derived from microscopy imaging data of biological samples. In the past decade, we observed a dramatic increase in available software packages for these purposes. As it is increasingly difficult to keep track of the number of available image analysis platforms, tool collections, components and emerging technologies, we provide a conservative overview of software that we use in daily routine and give insights into emerging new tools. We give guidance on which aspects to consider when choosing the platform that best suits the user's needs, including aspects such as image data type, skills of the team, infrastructure and community at the institute and availability of time and budget.