Inverse probability-weighted estimators are widely used in applications where data are missing due to nonresponse or censoring and in the estimation of causal effects from observational studies. Current estimators rely on ignorability assumptions for response indicators or treatment assignment and outcomes being conditional on observed covariates which are assumed to be measured without error. However, measurement error is common for the variables collected in many applications. For example, in studies of educational interventions, student achievement as measured by standardized tests is almost always used as the key covariate for removing hidden biases, but standardized test scores may have substantial measurement errors. We provide several expressions for a weighting function that can yield a consistent estimator for population means using incomplete data and covariates measured with error. We propose a method to estimate the weighting function from data. The results of a simulation study show that the estimator is consistent and has no bias and small variance.
29 December 2014 in Educational Measurement: Issues and Practice
Matching estimators are commonly used to estimate causal effects in nonexperimental settings. Covariate measurement error can be problematic for matching estimators when observational treatment groups differ on latent quantities observed only through error-prone surrogates. We establish necessary and sufficient conditions for matching and weighting with functions of observed covariates to yield unconfounded causal effect estimators, generalizing results from the standard (i.e., no measurement error) case. We establish that in common covariate measurement error settings, including continuous variables with continuous measurement error, discrete variables with misclassification, and factor and item response theory models, no single function of the observed covariates computed for all units in a study is appropriate for matching. However, we demonstrate that in some circumstances, it is possible to create different functions of the observed covariates for treatment and control units to construct a variable appropriate for matching. We also demonstrate the counterintuitive result that in some settings, it is possible to selectively contaminate the covariates with additional measurement error to construct a variable appropriate for matching. We discuss the implications of our results for the choice between matching and weighting estimators with error-prone covariates. Supplementary materials for this article are available online.
12 February 2015 in Educational Measurement: Issues and Practice
This article suggests two alternative statistical approaches for estimating student growth percentiles (SGP). The first is to estimate percentile ranks of current test scores conditional on past test scores directly, by modeling the conditional cumulative distribution functions, rather than indirectly through quantile regressions. This would remove the need for post hoc procedures required to ensure monotonicity of the estimated quantile functions, and for inversion of those functions to obtain SGP. We provide a brief empirical example demonstrating this approach and its potential benefits for handling discreteness of the observed test scores. The second suggestion is to estimate SGP directly from longitudinal item-level data, using multidimensional item response theory models, rather than from test scale scores. This leads to an isomorphism between using item-level data from one test to make inferences about latent student achievement, and using item-level data from multiple tests administered over time to make inferences about latent SGP. This framework can be used to solve the bias problems for current SGP methods caused by measurement error in both the current and past test scores, and provides straightforward assessments of uncertainty in SGP. We note practical problems that need to be addressed to implement our suggestions.
27 August 2018 in Journal of Educational and Behavioral Statistics
This article proposes a flexible extension of the Fay–Herriot model for making inferences from coarsened, group-level achievement data, for example, school-level data consisting of numbers of students falling into various ordinal performance categories. The model builds on the heteroskedastic ordered probit (HETOP) framework advocated by Reardon, Shear, Castellano, and Ho by allowing group parameters to be modeled with regressions on group-level covariates, and residuals modeled using the flexible exponential family of distributions recommended by Efron. We demonstrate that the alternative modeling framework, termed the “Fay–Herriot heteroskedastic ordered probit” (FH-HETOP) model, is useful for mitigating some of the challenges with direct maximum likelihood estimators from the HETOP model. We conduct a simulation study to compare the costs and benefits of several methods for using the FH-HETOP model to estimate group parameters and functions of them, including posterior means, constrained Bayes estimators, and the “triple goal” estimators of Shen and Louis. We also provide an application of the FH-HETOP model to math proficiency data from the Early Childhood Longitudinal Study. Code for estimating the FH-HETOP model and conducting supporting calculations is provided in a new package for the R environment.