Background: While random forests are one of the most successful machine learning methods, it is necessary to optimize their performance for use with datasets resulting from a two-phase sampling design with a small number of cases—a common situation in biomedical studies, which often have rare outcomes and covariates whose measurement is resource-intensive.Methods: Using an immunologic marker dataset from a phase III HIV vaccine efficacy trial, we seek to optimize random forest prediction performance using combinations of variable screening, class balancing, weighting, and hyperparameter tuning.Results: Our experiments show that while class balancing helps improve random forest prediction performance when variable screening is not applied, class balancing has a negative impact on performance in the presence of variable screening. The impact of the weighting similarly depends on whether variable screening is applied. Hyperparameter tuning is ineffective in situations with small sample sizes. We further show that random forests under-perform generalized linear models for some subsets of markers, and prediction performance on this dataset can be improved by stacking random forests and generalized linear models trained on different subsets of predictors, and that the extent of improvement depends critically on the dissimilarities between candidate learner predictions.Conclusion: In small datasets from two-phase sampling design, variable screening and inverse sampling probability weighting are important for achieving good prediction performance of random forests. In addition, stacking random forests and simple linear models can offer improvements over random forests.

In many applications, it is of interest to assess the relative contribution of features (or subsets of features) toward the goal of predicting a response—in other words, to gauge the variable importance of features. Most recent work on variable importance assessment has focused on describing the importance of features within the confines of a given prediction algorithm. However, such assessment does not necessarily characterize the prediction potential of features, and may provide a misleading reflection of the intrinsic value of these features. To address this limitation, we propose a general framework for nonparametric inference on interpretable algorithm-agnostic variable importance. We define variable importance as a population-level contrast between the oracle predictiveness of all available features versus all features except those under consideration. We propose a nonparametric efficient estimation procedure that allows the construction of valid confidence intervals, even when machine learning techniques are used. We also outline a valid strategy for testing the null importance hypothesis. Through simulations, we show that our proposal has good operating characteristics, and we illustrate its use with data from a study of an antibody against HIV-1 infection. Supplementary materials for this article are available online.

HVTN 505 is a preventative vaccine efficacy trial testing DNA followed by recombinant adenovirus serotype 5 (rAd5) in circumcised, Ad5-seronegative men and transgendered persons who have sex with men in the United States. Identified immune correlates of lower HIV-1 risk and a virus sieve analysis revealed that, despite lacking overall efficacy, vaccine-elicited responses exerted pressure on infecting HIV-1 viruses. To interrogate the mechanism of the antibody correlate of HIV-1 risk, we examined antigen-specific antibody recruitment of Fcγ receptors (FcγRs), antibody-dependent cellular phagocytosis (ADCP), and the role of anti-envelope (anti-Env) IgG3. In a prespecified immune correlates analysis, antibody-dependent monocyte phagocytosis and antibody binding to FcγRIIa correlated with decreased HIV-1 risk. Follow-up analyses revealed that anti-Env IgG3 breadth correlated with reduced HIV-1 risk, anti-Env IgA negatively modified infection risk by Fc effector functions, and that vaccine recipients with a specific FcγRIIa single-nucleotide polymorphism locus had a stronger correlation with decreased HIV-1 risk when ADCP, Env-FcγRIIa, and IgG3 binding were high. Additionally, FcγRIIa engagement correlated with decreased viral load setpoint in vaccine recipients who acquired HIV-1. These data support a role for vaccine-elicited anti–HIV-1 Env IgG3, antibody engagement of FcRs, and phagocytosis as potential mechanisms for HIV-1 prevention.

Increasing emphasis on the use of real-world evidence (RWE) to support clinical policy and regulatory decision-making has led to a proliferation of guidance, advice, and frameworks from regulatory agencies, academia, professional societies, and industry. A broad spectrum of studies use real-world data (RWD) to produce RWE, ranging from randomized trials with outcomes assessed using RWD to fully observational studies. Yet, many proposals for generating RWE lack sufficient detail, and many analyses of RWD suffer from implausible assumptions, other methodological flaws, or inappropriate interpretations. TheCausal Roadmapis an explicit, itemized, iterative process that guides investigators to prespecify study design and analysis plans; it addresses a wide range of guidance within a single framework. By supporting the transparent evaluation of causal assumptions and facilitating objective comparisons of design and analysis choices based on prespecified criteria, theRoadmapcan help investigators to evaluate the quality of evidence that a given study is likely to produce, specify a study to generate high-quality RWE, and communicate effectively with regulatory agencies and other stakeholders. This paper aims to disseminate and extend theCausal Roadmapframework for use by clinical and translational researchers; three companion papers demonstrate applications of theCausal Roadmapfor specific use cases.

The best assay or marker to define mRNA-1273 vaccine–induced antibodies as a correlate of protection (CoP) is unclear. In the COVE trial, participants received two doses of the mRNA-1273 COVID-19 vaccine or placebo. We previously assessed IgG binding antibodies to the spike protein (spike IgG) or receptor binding domain (RBD IgG) and pseudovirus neutralizing antibody 50 or 80% inhibitory dilution titer measured on day 29 or day 57, as correlates of risk (CoRs) and CoPs against symptomatic COVID-19 over 4 months after dose. Here, we assessed a new marker, live virus 50% microneutralization titer (LV-MN ₅₀ ), and compared and combined markers in multivariable analyses. LV-MN ₅₀ was an inverse CoR, with a hazard ratio of 0.39 (95% confidence interval, 0.19 to 0.83) at day 29 and 0.51 (95% confidence interval, 0.25 to 1.04) at day 57 per 10-fold increase. In multivariable analyses, pseudovirus neutralization titers and anti-spike binding antibodies performed best as CoRs; combining antibody markers did not improve correlates. Pseudovirus neutralization titer was the strongest independent correlate in a multivariable model. Overall, these results supported pseudovirus neutralizing and binding antibody assays as CoRs and CoPs, with the live virus assay as a weaker correlate in this sample set. Day 29 markers performed as well as day 57 markers as CoPs, which could accelerate immunogenicity and immunobridging studies.