Background: Machine learning (ML) can be an effective tool to extract information from attribute-rich molecular datasets for the generation of molecular diagnostic tests. However, the way in which the resulting scores or classifications are produced from the input data may not be transparent. Algorithmic explainability or interpretability has become a focus of ML research. Shapley values, first introduced in game theory, can provide explanations of the result generated from a specific set of input data by a complex ML algorithm. Methods: For a multivariate molecular diagnostic test in clinical use (the VeriStrat® test), we calculate and discuss the interpretation of exact Shapley values. We also employ some standard approximation techniques for Shapley value computation (local interpretable model-agnostic explanation (LIME) and Shapley Additive Explanations (SHAP) based methods) and compare the results with exact Shapley values. Results: Exact Shapley values calculated for data collected from a cohort of 256 patients showed that the relative importance of attributes for test classification varied by sample. While all eight features used in the VeriStrat® test contributed equally to classification for some samples, other samples showed more complex patterns of attribute importance for classification generation. Exact Shapley values and Shapley-based interaction metrics were able to provide interpretable classification explanations at the sample or patient level, while patient subgroups could be defined by comparing Shapley value profiles between patients. LIME and SHAP approximation approaches, even those seeking to include correlations between attributes, produced results that were quantitatively and, in some cases qualitatively, different from the exact Shapley values. Conclusions: Shapley values can be used to determine the relative importance of input attributes to the result generated by a multivariate molecular diagnostic test for an individual sample or patient. Patient subgroups defined by Shapley value profiles may motivate translational research. However, correlations inherent in molecular data and the typically small ML training sets available for molecular diagnostic test development may cause some approximation methods to produce approximate Shapley values that differ both qualitatively and quantitatively from exact Shapley values. Hence, caution is advised when using approximate methods to evaluate Shapley explanations of the results of molecular diagnostic tests.

The VeriStrat test is a serum proteomic signature originally discovered in non-responders to second line gefitinib treatment and subsequently used to predict differential benefit from erlotinib versus chemotherapy in previously treated advanced non-small cell lung cancer (NSCLC). Multiple studies highlight the clinical utility of the VeriStrat test, however, the mechanistic connection between VeriStrat-poor classification and poor prognosis in untreated and previously treated patients is still an active area of research. The aim of this study was to correlate VeriStrat status with other circulating biomarkers in advanced NSCLC patients – each with respect to clinical outcomes. Serum samples were prospectively collected from 57 patients receiving salvage chemotherapy and 70 non-EGFR mutated patients receiving erlotinib. Patients were classified as either VeriStrat good or poor based on the VeriStrat test. Luminex immunoassays were used to measure circulating levels of 102 distinct biomarkers implicated in tumor aggressiveness and treatment resistance. A Cox PH model was used to evaluate associations between biomarker levels and clinical outcome, whereas the association of VeriStrat classifications with biomarker levels was assessed via the Mann-Whitney Rank Sum test. VeriStrat was prognostic for outcome within the erlotinib treated patients (HR = 0.29, p < 0.0001) and predictive of differential treatment benefit between erlotinib and chemotherapy ((interaction HR = 0.25; interaction p = 0.0035). A total of 27 biomarkers out of 102 unique analytes were found to be significantly associated with OS (Cox PH p ≤ 0.05), whereas 16 biomarkers were found to be associated with PFS. Thrombospondin-2, C-reactive protein, TNF-receptor I, and placental growth factor were the analytes most highly associated with OS, all with Cox PH p-values ≤0.0001. VeriStrat status was found to be significantly associated with 23 circulating biomarkers (Mann-Whitney Rank Sum p ≤ 0.05), 6 of which had p < 0.001, including C-reactive protein, IL-6, serum amyloid A, CYFRA 21.1, IGF-II, osteopontin, and ferritin. Strong associations were observed between survival and VeriStrat classifications as well as select circulating biomarkers associated with fibrosis, inflammation, and acute phase reactants as part of this study. The associations between these biomarkers and VeriStrat classification might have therapeutic implications for poor prognosis NSCLC patients, particularly with new immunotherapeutic treatment options.

The therapeutic landscape in metastatic melanoma has changed dramatically in the last decade, with the success of immune checkpoint inhibitors resulting in durable responses for a large number of patients. For patients with BRAF mutations, combinations of BRAF and MEK inhibitors demonstrated response rates and benefit comparable to those from immune checkpoint inhibitors, providing the rationale for sequential treatment with targeted and immunotherapies and raising the question of optimal treatment sequencing. Biomarkers for the selection of anti-PD-1 therapy in BRAF wild type (BRAF WT) and in BRAF mutated (BRAF MUT) patients help development of alternative treatments for patients unlikely to benefit, and might lead to better understanding of the interaction of checkpoint inhibition and targeted therapy. In this paper we evaluate the performance of a previously developed serum proteomic test, BDX008, in metastatic melanoma patients treated with anti-PD-1 agents and investigate the role of BRAF mutation status. BDX008, a pre-treatment proteomic test associated with acute phase reactants, wound healing and complement activation, stratifies patients into two groups, BDX008+ and BDX008-, with better and worse outcomes on immunotherapy. Serum samples were available from 71 patients treated with anti-PD1 inhibitors; 25 patients had BRAF mutations, 39 were wild type. Overall, BDX008+ patients had significantly better overall survival (OS) (HR = 0.50, P = 0.016) and a trend for better progression-free survival (PFS) (HR = 0.61, P = 0.060) than BDX008- patients. BDX008 classification was statistically significant in the analyses adjusted for mutation status, LDH, and line of treatment (P = 0.009 for OS and 0.031 for PFS). BRAF WT BDX008+ patients had markedly long median OS of 32.5 months and 53% landmark 2 years survival, with statistically significantly superior OS as compared to BDX008- patients (HR = 0.41, P = 0.032). The difference between BDX008+ and BDX008- in PFS in BRAF WT patients and in OS and PFS in BRAF MUT patients did not reach statistical significance, though numerically was consistent with overall results. The test demonstrated significant interaction with neutrophil-to-lymphocyte ratio (NLR) (PFS P = 0.041, OS P = 0.004). BDX008 as a biomarker selecting for benefit from immune checkpoint blockade, especially in patients with wild type BRAF and in subgroups with low NLR, warrants further evaluation.

Set enrichment methods are commonly used to analyze high-dimensional molecular data and gain biological insight into molecular or clinical phenotypes. One important category of analysis methods employs an enrichment score, which is created from ranked univariate correlations between phenotype and each molecular attribute. Estimates of the significance of the associations are determined via a null distribution generated from phenotype permutation. We investigate some statistical properties of this method and demonstrate how alternative assessments of enrichment can be used to increase the statistical power of such analyses to detect associations between phenotype and biological processes and pathways. For this category of set enrichment analysis, the null distribution is largely independent of the number of samples with available molecular data. Hence, providing the sample cohort is not too small, we show that increased statistical power to identify associations between biological processes and phenotype can be achieved by splitting the cohort into two halves and using the average of the enrichment scores evaluated for each half as an alternative test statistic. Further, we demonstrate that this principle can be extended by averaging over multiple random splits of the cohort into halves. This enables the calculation of an enrichment statistic and associated p value of arbitrary precision, independent of the exact random splits used. It is possible to increase the statistical power of gene set enrichment analyses that employ enrichment scores created from running sums of univariate phenotype-attribute correlations and phenotype-permutation generated null distributions. This increase can be achieved by using alternative test statistics that average enrichment scores calculated for splits of the dataset. Apart from the special case of a close balance between up- and down-regulated genes within a gene set, statistical power can be improved, or at least maintained, by this method down to small sample sizes, where accurate assessment of univariate phenotype-gene correlations becomes unfeasible.

Modern genomic and proteomic profiling methods produce large amounts of data from tissue and blood-based samples that are of potential utility for improving patient care. However, the design of precision medicine tests for unmet clinical needs from this information in the small cohorts available for test discovery remains a challenging task. Obtaining reliable performance assessments at the earliest stages of test development can also be problematic. We describe a novel approach to classifier development designed to create clinically useful tests together with reliable estimates of their performance. The method incorporates elements of traditional and modern machine learning to facilitate the use of cohorts where the number of samples is less than the number of measured patient attributes. It is based on a hierarchy of classification and information abstraction and combines boosting, bagging, and strong dropout regularization. We apply this dropout-regularized combination approach to two clinical problems in oncology using mRNA expression and associated clinical data and compare performance with other methods of classifier generation, including Random Forest. Performance of the new method is similar to or better than the Random Forest in the two classification tasks used for comparison. The dropout-regularized combination method also generates an effective classifier in a classification task with a known confounding variable. Most importantly, it provides a reliable estimate of test performance from a relatively small development set of samples. The flexible dropout-regularized combination approach is able to produce tests tailored to particular clinical questions and mitigate known confounding effects. It allows the design of molecular diagnostic tests addressing particular clinical questions together with reliable assessment of whether test performance is likely to be fit-for-purpose in independent validation at the earliest stages of development.

The remarkable success of immune checkpoint inhibitors (ICIs) has given hope of cure for some patients with advanced cancer; however, the fraction of responding patients is 15–35%, depending on tumor type, and the proportion of durable responses is even smaller. Identification of biomarkers with strong predictive potential remains a priority. Until now most of the efforts were focused on biomarkers associated with the assumed mechanism of action of ICIs, such as levels of expression of programmed death-ligand 1 (PD-L1) and mutation load in tumor tissue, as a proxy of immunogenicity; however, their performance is unsatisfactory. Several assays designed to capture the complexity of the disease by measuring the immune response in tumor microenvironment show promise but still need validation in independent studies. The circulating proteome contains an additional layer of information characterizing tumor–host interactions that can be integrated into multivariate tests using modern machine learning techniques. Here we describe several validated serum-based proteomic tests and their utility in the context of ICIs. We discuss test performances, demonstrate their independence from currently used biomarkers, and discuss various aspects of associated biological mechanisms. We propose that serum-based multivariate proteomic tests add a missing piece to the puzzle of predicting benefit from ICIs.