Computational omics methods packaged as software have become essential to modern biological research. The increasing dependence of scientists on these powerful software tools creates a need for systematic assessment of these methods, known as benchmarking. Adopting a standardized benchmarking practice could help researchers who use omics data to better leverage recent technological innovations. Our review summarizes benchmarking practices from 25 recent studies and discusses the challenges, advantages, and limitations of benchmarking across various domains of biology. We also propose principles that can make computational biology benchmarking studies more sustainable and reproducible, ultimately increasing the transparency of biomedical data and results.

Developing new software tools for analysis of large-scale biological data is a key component of advancing modern biomedical research. Scientific reproduction of published findings requires running computational tools on data generated by such studies, yet little attention is presently allocated to the installability and archival stability of computational software tools. Scientific journals require data and code sharing, but none currently require authors to guarantee the continuing functionality of newly published tools. We have estimated the archival stability of computational biology software tools by performing an empirical analysis of the internet presence for 36,702 omics software resources published from 2005 to 2017. We found that almost 28% of all resources are currently not accessible through uniform resource locators (URLs) published in the paper they first appeared in. Among the 98 software tools selected for our installability test, 51% were deemed “easy to install,” and 28% of the tools failed to be installed at all because of problems in the implementation. Moreover, for papers introducing new software, we found that the number of citations significantly increased when authors provided an easy installation process. We propose for incorporation into journal policy several practical solutions for increasing the widespread installability and archival stability of published bioinformatics software.

Bacillus Calmette–Guerin (BCG) is a live attenuated form of Mycobacterium bovis that was developed 100 years ago as a vaccine against tuberculosis (TB) and has been used ever since to vaccinate children globally. It has also been used as the first-line treatment in patients with nonmuscle invasive bladder cancer (NMIBC), through repeated intravesical applications. Numerous studies have shown that BCG induces off-target immune effects in various pathologies. Accumulating data argue for the critical role of the immune system in the course of neurodegenerative diseases such as Alzheimer’s disease (AD) and Parkinson’s disease (PD). In this study, we tested whether repeated exposure to BCG during the treatment of NMIBC is associated with the risk of developing AD and PD. We presented a multi-center retrospective cohort study with patient data collected between 2000 and 2019 that included 12,185 bladder cancer (BC) patients, of which 2301 BCG-treated patients met all inclusion criteria, with a follow-up of 3.5 to 7 years. We considered the diagnosis date of AD and nonvascular dementia cases for BC patients. The BC patients were partitioned into those who underwent a transurethral resection of the bladder tumor followed by BCG therapy, and a disjoint group that had not received such treatment. By applying Cox proportional hazards (PH) regression and competing for risk analyses, we found that BCG treatment was associated with a significantly reduced risk of developing AD, especially in the population aged 75 years or older. The older population (≥75 years, 1578 BCG treated, and 5147 controls) showed a hazard ratio (HR) of 0.726 (95% CI: 0.529–0.996; p-value = 0.0473). While in a hospital-based cohort, BCG treatment resulted in an HR of 0.416 (95% CI: 0.203–0.853; p-value = 0.017), indicating a 58% lower risk of developing AD. The risk of developing PD showed the same trend with a 28% reduction in BCG-treated patients, while no BCG beneficial effect was observed for other age-related events such as Type 2 diabetes (T2D) and stroke. We attributed BCG’s beneficial effect on neurodegenerative diseases to a possible activation of long-term nonspecific immune effects. We proposed a prospective study in elderly people for testing intradermic BCG inoculation as a potential protective agent against AD and PD.

Worldwide, testing capacity for SARS-CoV-2 is limited and bottlenecks in the scale up of polymerase chain reaction (PCR-based testing exist. Our aim was to develop and evaluate a machine learning algorithm to diagnose COVID-19 in the inpatient setting. The algorithm was based on basic demographic and laboratory features to serve as a screening tool at hospitals where testing is scarce or unavailable. We used retrospectively collected data from the UCLA Health System in Los Angeles, California. We included all emergency room or inpatient cases receiving SARS-CoV-2 PCR testing who also had a set of ancillary laboratory features (n = 1,455) between 1 March 2020 and 24 May 2020. We tested seven machine learning models and used a combination of those models for the final diagnostic classification. In the test set (n = 392), our combined model had an area under the receiver operator curve of 0.91 (95% confidence interval 0.87–0.96). The model achieved a sensitivity of 0.93 (95% CI 0.85–0.98), specificity of 0.64 (95% CI 0.58–0.69). We found that our machine learning algorithm had excellent diagnostic metrics compared to SARS-CoV-2 PCR. This ensemble machine learning algorithm to diagnose COVID-19 has the potential to be used as a screening tool in hospital settings where PCR testing is scarce or unavailable.

Background: Recent advancements in next-generation sequencing have rapidly improved our ability to study genomic material at an unprecedented scale. Despite substantial improvements in sequencing technologies, errors present in the data still risk confounding downstream analysis and limiting the applicability of sequencing technologies in clinical tools. Computational error correction promises to eliminate sequencing errors, but the relative accuracy of error correction algorithms remains unknown.Results: In this paper, we evaluate the ability of error correction algorithms to fix errors across different types of datasets that contain various levels of heterogeneity. We highlight the advantages and limitations of computational error correction techniques across different domains of biology, including immunogenomics and virology. To demonstrate the efficacy of our technique, we apply the UMI-based high-fidelity sequencing protocol to eliminate sequencing errors from both simulated data and the raw reads. We then perform a realistic evaluation of error-correction methods.Conclusions: In terms of accuracy, we find that method performance varies substantially across different types of datasets with no single method performing best on all types of examined data. Finally, we also identify the techniques that offer a good balance between precision and sensitivity.