PHOTONAI is a high-level Python API designed to simplify and accelerate machine learning model development. It functions as a unifying framework allowing the user to easily access and combine algorithms from different toolboxes into custom algorithm sequences. It is especially designed to support the iterative model development process and automates the repetitive training, hyperparameter optimization and evaluation tasks. Importantly, the workflow ensures unbiased performance estimates while still allowing the user to fully customize the machine learning analysis. PHOTONAI extends existing solutions with a novel pipeline implementation supporting more complex data streams, feature combinations, and algorithm selection. Metrics and results can be conveniently visualized using the PHOTONAI Explorer and predictive models are shareable in a standardized format for further external validation or application. A growing add-on ecosystem allows researchers to offer data modality specific algorithms to the community and enhance machine learning in the areas of the life sciences. Its practical utility is demonstrated on an exemplary medical machine learning problem, achieving a state-of-the-art solution in few lines of code. Source code is publicly available on Github, while examples and documentation can be found at www.photon-ai.com.
The field of neuroimaging has embraced methods from machine learning in a variety of ways. Although an increasing number of initiatives have published open-access neuroimaging datasets, specifically designed benchmarks are rare in the field. In this article, we first describe how benchmarks in computer science and biomedical imaging have fostered methodological progress in machine learning. Second, we identify the special characteristics of neuroimaging data and outline what researchers have to ensure when establishing a neuroimaging benchmark, how datasets should be composed and how adequate evaluation criteria can be chosen. Based on lessons learned from machine learning benchmarks, we argue for an extended evaluation procedure that, next to applying suitable performance metrics, focuses on scientifically relevant aspects such as explainability, robustness, uncertainty, computational efficiency and code quality. Lastly, we envision a collaborative neuroimaging benchmarking platform that combines the discussed aspects in a collaborative and agile framework, allowing researchers across disciplines to work together on the key predictive problems of the field of neuroimaging and psychiatry.
Data-based predictions of individual Cognitive Behavioral Therapy (CBT) treatment response are a fundamental step towards precision medicine. Past studies demonstrated only moderate prediction accuracy (i.e. ability to discriminate between responders and non-responders of a given treatment) when using clinical routine data such as demographic and questionnaire data, while neuroimaging data achieved superior prediction accuracy. However, these studies may be considerably biased due to very limited sample sizes and bias-prone methodology. Adequately powered and cross-validated samples are a prerequisite to evaluate predictive performance and to identify the most promising predictors. We therefore analyzed resting state functional magnet resonance imaging (rs-fMRI) data from two large clinical trials to test whether functional neuroimaging data continues to provide good prediction accuracy in much larger samples. Data came from two distinct German multicenter studies on exposure-based CBT for anxiety disorders, the Protect-AD and SpiderVR studies. We separately and independently preprocessed baseline rs-fMRI data from n = 220 patients (Protect-AD) and n = 190 patients (SpiderVR) and extracted a variety of features, including ROI-to-ROI and edge-functional connectivity, sliding-windows, and graph measures. Including these features in sophisticated machine learning pipelines, we found that predictions of individual outcomes never significantly differed from chance level, even when conducting a range of exploratory post-hoc analyses. Moreover, resting state data never provided prediction accuracy beyond the sociodemographic and clinical data. The analyses were independent of each other in terms of selecting methods to process resting state data for prediction input as well as in the used parameters of the machine learning pipelines, corroborating the external validity of the results. These similar findings in two independent studies, analyzed separately, urge caution regarding the interpretation of promising prediction results based on neuroimaging data from small samples and emphasizes that some of the prediction accuracies from previous studies may result from overestimation due to homogeneous data and weak cross-validation schemes. The promise of resting-state neuroimaging data to play an important role in the prediction of CBT treatment outcomes in patients with anxiety disorders remains yet to be delivered.
Background and Objective: Flow cytometry is a widely used technique for identifying cell populations in patient-derived fluids, such as peripheral blood (PB) or cerebrospinal fluid (CSF). Despite its ubiquity in research and clinical practice, the process of gating, i.e., manually identifying cell types, is labor-intensive and error-prone. The objective of this study is to address this challenge by introducing GateNet, a neural network architecture designed for fully end-to-end automated gating without the need for correcting batch effects. Methods: For this study a unique dataset is used which comprises over 8,000,000 events from N 127 PB and CSF samples which were manually labeled independently by four experts. Applying cross-validation, the classification performance of GateNet is compared to the human experts performance. Additionally, GateNet is applied to a publicly available dataset to evaluate generalization. The classification performance is measured using the F1 score. Results: GateNet achieves F1 scores ranging from 0.910 to 0.997 demonstrating human-level performance on samples unseen during training. In the publicly available dataset, GateNet confirms its generalization capabilities with an F1 score of 0.936. Importantly, we also show that GateNet only requires 10 samples to reach human-level performance. Finally, gating with GateNet only takes 15 microseconds per event utilizing graphics processing units (GPU). Conclusions: GateNet enables fully end-to-end automated gating in flow cytometry, overcoming the labor-intensive and error-prone nature of manual adjustments. The neural network achieves human-level performance on unseen samples and generalizes well to diverse datasets. Notably, its data efficiency, requiring only 10 samples to reach human-level performance, positions GateNet as a widely applicable tool across various domains of flow cytometry.