11 May 2021 in Genome Biology
Most research articles presenting new data analysis methods claim that “the new method performs better than existing methods,” but the veracity of such statements is questionable. Our manuscript discusses and illustrates consequences of the optimistic bias occurring during the evaluation of novel data analysis methods, that is, all biases resulting from, for example, selection of datasets or competing methods, better ability to fix bugs in a preferred method, and selective reporting of method variants. We quantitatively investigate this bias using an example from epigenetic analysis: normalization methods for data generated by the Illumina HumanMethylation450K BeadChip microarray.
15 September 2026 in The Lancet Healthy Longevity
26 December 2025 in PLOS ONE
Background: Individual-level surrogacy (ILS) assesses how well a surrogate endpoint predicts treatment effect at the individual level. The paper discusses mutual information (MI) and the likelihood reduction factor (LRF) as ways to quantify ILS. It also reassesses ILS statements for people with relapsing-remitting multiple sclerosis (RRMS) by using T2 MRI lesions (T2L) as surrogates for disability and disease activity. ILS is often reported using inadequate concepts, e.g. subgroup analysis, correlation, odds ratios, sensitivity/specificity, or metrics based on Prentice’s criteria. Methods: LRF assesses ILS quality by determining prediction quality through shared information (MI) between surrogate(s) and clinical endpoint(s). A simulation study validates LRF as a measure of ILS quality. Individual-level data from ten randomized controlled trials (n = 5673) provided longitudinal information on T2L, T2 MRI lesion volumes (T2V), future disability progression (EDSS), and relapses. LRFs for different scenarios were calculated. Results were compared to those obtained by methods commonly applied in RRMS literature. Results: Simulations confirmed the robustness of LRF as a reliable ILS quality measure. Two of ten trials showed weak ILS between T2V and EDSS (LRF = 0.21, CI95%: 0.16–0.26; LRF = 0.28, CI95%: 0.23–0.34). Other LRFs were below 0.2. A method commonly used in the MS literature also showed no strong ILS. Conclusion: LRF is an important measure to quantify ILS and prediction quality. But it is rarely applied in RRMS research. LRF did not reveal robust surrogacy patterns when applied to data of ten clinical trials. Existing surrogacy claims should be reassessed since ILS assessments in the MS literature may have major limitations.
1 January 2025 in Therapeutic Advances in Neurological Disorders
Background: The course of relapsing-remitting multiple sclerosis (RRMS), frequently preceded by the clinically isolated syndrome (CIS), is variable and challenging to predict. Given many treatment options available, prognostic algorithms are gaining importance in informing initial treatment decisions. However, to date, only a few externally validated exists. External validation, which involves the application of a model to independent data, is essential. Privacy-preserving federated analyses of individual-level data facilitate external validation using clinical datasets that are typically difficult to access. Objectives: Using data from the ProVal-MS study to externally validate the multiple sclerosis treatment decision score (MS-TDS), a predictive algorithm for early RRMS and CIS. The MS-TDS predicts the probability of the occurrence of at least one new or enlarging T2 lesion within 6–24 months following the onset of the disease and supports choosing between initiating platform treatment or a ‘wait-and-see’ approach. A secondary objective is to demonstrate the feasibility of privacy-preserving federated concepts within the Data Integration for Future Medicine (DIFUTURE) consortium. Design: Prospective, multicentric, non-interventional cohort study (ProVal-MS) within DIFUTURE. Methods: The calibrated MS-TDS was evaluated using the area under the receiver operating characteristic curve (AUROC) and the Brier score in both pooled and distributed settings. A decision curve analysis (DCA) was used to evaluate the net benefit of treatment decisions made by the MS-TDS in comparison to those made by treating neurologists. Results: Of the 271 individuals diagnosed with CIS or early RRMS, 202 (78.2%) received platform treatment, while 59 (21.8%) did not receive treatment. The AUROC was 0.561 (95% CI: 0.492–0.630) in the pooled analysis and 0.567 (95% CI: 0.496–0.634) in the distributed analysis. DCA demonstrated a net benefit that was commensurate with that achieved by decisions made by experienced neurologists. Conclusion: The external validation of the MS-TDS demonstrated low, non-significant predictive performance; however, it may serve as a useful complement, particularly for less-experienced neurologists. The distributed validation was found to be both feasible and compliant with data protection regulations.