Background: Cross-validation (CV) is an effective method for estimating the prediction error of a classifier. Some recent articles have proposed methods for optimizing classifiers by choosing classifier parameter values that minimize the CV error estimate. We have evaluated the validity of using the CV error estimate of the optimized classifier as an estimate of the true error expected on independent data. Results: We used CV to optimize the classification parameters for two kinds of classifiers; Shrunken Centroids and Support Vector Machines (SVM). Random training datasets were created, with no difference in the distribution of the features between the two classes. Using these "null" datasets, we selected classifier parameter values that minimized the CV error estimate. 10-fold CV was used for Shrunken Centroids while Leave-One-Out-CV (LOOCV) was used for the SVM. Independent test data was created to estimate the true error. With "null" and "non null" (with differential expression between the classes) data, we also tested a nested CV procedure, where an inner CV loop is used to perform the tuning of the parameters while an outer CV is used to compute an estimate of the error. The CV error estimate for the classifier with the optimal parameters was found to be a substantially biased estimate of the true error that the classifier would incur on independent data. Even though there is no real difference between the two classes for the "null" datasets, the CV error estimate for the Shrunken Centroid with the optimal parameters was less than 30% on 18.5% of simulated training data-sets. For SVM with optimal parameters the estimated error rate was less than 30% on 38% of "null" data-sets. Performance of the optimized classifiers on the independent test set was no better than chance. The nested CV procedure reduces the bias considerably and gives an estimate of the error that is very close to that obtained on the independent testing set for both Shrunken Centroids and SVM classifiers for "null" and "non-null" data distributions. Conclusion: We show that using CV to compute an error estimate for a classifier that has itself been tuned using CV gives a significantly biased estimate of the true error. Proper use of CV for estimating true error of a classifier developed using a well defined algorithm requires that all steps of the algorithm, including classifier parameter tuning, be repeated in each CV loop. A nested CV procedure provides an almost unbiased estimate of the true error.

High-throughput and high-content databases are increasingly important resources in molecular medicine, systems biology, and pharmacology. However, the information usually resides in unwieldy databases, limiting ready data analysis and integration. One resource that offers substantial potential for improvement in this regard is the NCI-60 cell line database compiled by the U.S. National Cancer Institute, which has been extensively characterized across numerous genomic and pharmacologic response platforms. In this report, we introduce a CellMiner (http://discover.nci.nih.gov/cellminer/) web application designed to improve the use of this extensive database. CellMiner tools allowed rapid data retrieval of transcripts for 22,379 genes and 360 microRNAs along with activity reports for 20,503 chemical compounds including 102 drugs approved by the U.S. Food and Drug Administration. Converting these differential levels into quantitative patterns across the NCI-60 clarified data organization and cross-comparisons using a novel pattern match tool. Data queries for potential relationships among parameters can be conducted in an iterative manner specific to user interests and expertise. Examples of the in silico discovery process afforded by CellMiner were provided for multidrug resistance analyses and doxorubicin activity; identification of colon-specific genes, microRNAs, and drugs; microRNAs related to the miR-17-92 cluster; and drug identification patterns matched to erlotinib, gefitinib, afatinib, and lapatinib. CellMiner greatly broadens applications of the extensive NCI-60 database for discovery by creating web-based processes that are rapid, flexible, and readily applied by users without bioinformatics expertise. Cancer Res; 72(14); 3499–511. ©2012 AACR.

Although advances in genomics during the last decade have opened new avenues for translational research and allowed the direct evaluation of clinical samples, there is still a need for reliable preclinical models to test therapeutic strategies. Human cancer-derived cell lines are the most widely used models to study the biology of cancer and to test hypotheses to improve the efficacy of cancer treatment. Since the development of the first cancer cell line, the clinical relevance of these models has been continuously questioned. Based upon recent studies that have fueled the debate, we review the major events in the development of the in vitro models and the emergence of new technologies that have revealed important issues and limitations concerning human cancer cell lines as models. All cancer cell lines do not have equal value as tumor models. Some have been successful, whereas others have failed. However, the success stories should not obscure the growing body of data that motivates us to develop new in vitro preclinical models that would substantially increase the success rate of new in vitro-assessed cancer treatments.

Although in vitro models have been a cornerstone of anti-cancer drug development, their direct applicability to clinical cancer research has been uncertain. Using a state-of-the-art Taqman-based quantitative RT-PCR assay, we investigated the multidrug resistance (MDR) transcriptome of six cancer types, in established cancer cell lines (grown in monolayer, 3D scaffold, or in xenograft) and clinical samples, either containing >75% tumor cells or microdissected. The MDR transcriptome was determined a priori based on an extensive curation of the literature published during the last three decades, which led to the enumeration of 380 genes. No correlation was found between clinical samples and established cancer cell lines. As expected, we found up-regulation of genes that would facilitate survival across all cultured cancer cell lines evaluated. More troubling, however, were data showing that all of the cell lines, grown either in vitro or in vivo, bear more resemblance to each other, regardless of the tissue of origin, than to the clinical samples they are supposed to model. Although cultured cells can be used to study many aspects of cancer biology and response of cells to drugs, this study emphasizes the necessity for new in vitro cancer models and the use of primary tumor models in which gene expression can be manipulated and small molecules tested in a setting that more closely mimics the in vivo cancer microenvironment so as to avoid radical changes in gene expression profiles brought on by extended periods of cell culture.

Advances in the high-throughput omic technologies have made it possible to profile cells in a large number of ways at the DNA, RNA, protein, chromosomal, functional, and pharmacological levels. A persistent problem is that some classes of molecular data are labeled with gene identifiers, others with transcript or protein identifiers, and still others with chromosomal locations. What has lagged behind is the ability to integrate the resulting data to uncover complex relationships and patterns. Those issues are reflected in full form by molecular profile data on the panel of 60 diverse human cancer cell lines (the NCI-60) used since 1990 by the U.S. National Cancer Institute to screen compounds for anticancer activity. To our knowledge, CellMiner is the first online database resource for integration of the diverse molecular types of NCI-60 and related meta data.