1 June 2016 in Annual Review of Statistics and Its Application
Researchers apply sampling weights to take account of unequal sample selection probabilities and to frame coverage errors and nonresponses. If researchers do not weight when appropriate, they risk having biased estimates. Alternatively, when they unnecessarily apply weights, they can create an inefficient estimator without reducing bias. Yet in practice researchers rarely test the necessity of weighting and are sometimes guided more by the current practice in their field than by scientific evidence. In addition, statistical tests for weighting are not widely known or available. This article reviews empirical tests to determine whether weighted analyses are justified. We focus on regression models, though the review's implications extend beyond regression. We find that nearly all weighting tests fall into two categories: difference in coefficients tests and weight association tests. We describe the distinguishing features of each category, present their properties, and explain the close relationship between them. We review the simulation evidence on their sampling properties in finite samples. Finally, we highlight the unanswered theoretical and practical questions that surround these tests and that deserve further research.
1 July 2019 in Children and Youth Services Review
1 June 2020 in Accountability in Research
Survey-based studies on research fraud often feature narrow operationalizations of misbehavior and use limited samples. Such factors potentially hinder the development of strategies aimed at reducing the frequency of wrongdoing among researchers. This study asked full-time faculty members in the natural, social, and applied sciences how frequently six types of research fraud (i.e., data fabrication, plagiarism, data falsification, authorship fraud, publication fraud, and grant fraud) occur in their field of study. These data come from mail and online surveys that were administered to a stratified random sample of tenured and tenure-track faculty members (N = 613) at the top 100 research universities in the United States. Factor-analytic modeling demonstrated that the survey items load on the hypothesized latent constructs and also confirmed the presence of a second-order factor. A specific type of authorship fraud—gift authorship—was perceived to be the most prevalent overall. The least common fraud was a form of data fabrication (i.e., creating data from a study that was never actually conducted). The results were largely consistent with previous studies indicating that serious forms of fraud like data fabrication are relatively rare. Future survey-based studies should pay careful attention to the multidimensional nature of research fraud.
12 September 2019 in Journal of Interpersonal Violence
Many colleges and universities conduct web-based campus climate surveys to understand the prevalence and nature of sexual assault among their students. When designing and fielding a web survey to measure a sensitive topic like sexual assault, methodological decisions, including the length of the field period and the use or amount of an incentive, can affect the representativeness of the respondent sample leading to biased or imprecise estimates. This study uses data from the Campus Climate Survey Validation Study (CCSVS) to assess how the interaction between field period length and survey incentive amount affects nonresponse, sample representativeness, and the precision of survey estimates. Research suggests that using robust incentives gives potential survey respondents a reason to complete the survey beyond their intrinsic motivation to do so. Likewise, extending the field period gives more time to people who may be less intrinsically motivated to complete the survey. Both serve to increase sample size and representativeness, minimize bias, and improve estimate precision. Schools, however, sometimes lack the time and/or resources for both a robust incentive and a lengthy field period, and this study examines the extent to which the potential negative impacts of not using one can be mitigated by the presence of the other. Findings indicate that target response rates can be achieved using a smaller incentive if the field period is lengthy but, even with a lengthy field period, the use of a smaller incentive can result in biased estimates due to a lack of representativeness. Conversely, when a robust incentive is used and weights are developed to adjust for nonresponse, a shorter field period will not have a significant impact on point estimates, but the estimates will be less precise due to fewer respondents participating in the survey.
17 October 2013 in Sociological Methods & Research
For survey methodologists, latent class analysis (LCA) is a powerful tool for assessing the measurement error in survey questions, evaluating survey methods, and estimating the bias in estimates of population prevalence. LCA can be used when gold standard measurements are not available and applied to essentially any set of indicators that meet certain criteria for identifiability. LCA offers quality inference, provided the key threat to model validity—namely, local dependence—can be appropriately addressed either in the study design or in the model-building process. Three potential causes threaten local independence: bivocality, behaviorally correlated error, and latent heterogeneity. In this article, these threats are examined separately to obtain insights regarding (a) questionnaire designs that reduce local dependence, (b) the effects of local dependence on parameter estimation, and (c) modeling strategies to mitigate these effects in statistical inference. The article focuses primarily on the analysis of rare and sensitivity outcomes and proposes a practical approach for diagnosing and mitigating model failures. The proposed approach is empirically tested using real data from a national survey of inmate sexual abuse where measurement errors are a serious concern. Our findings suggest that the proposed modeling strategy was successful in reducing local dependence bias in the estimates, but its success varied by the quality of the indicators available for analysis. With only three indicators, the biasing effects of local dependence can usually be reduced but not always to acceptable levels.