1 November 2022 in Journal of Applied Psychology
This paper systematically revisits prior meta-analytic conclusions about the criterion-related validity of personnel selection procedures, and particularly the effect of range restriction corrections on those validity estimates. Corrections for range restriction in meta-analyses of predictor-criterion relationships in personnel selection contexts typically involve the use of an artifact distribution. After outlining and critiquing five approaches that have commonly been used to create and apply range restriction artifact distributions, we conclude that each has significant issues that often result in substantial overcorrection and that therefore the validity of many selection procedures for predicting job performance has been substantially overestimated. Revisiting prior meta-analytic conclusions produces revised validity estimates. Key findings are that most of the same selection procedures that ranked high in prior summaries remain high in rank, but with mean validity estimates reduced by .10-.20 points. Structured interviews emerged as the top-ranked selection procedure. We also pair validity estimates with information about mean Black-White subgroup differences per selection procedure, providing information about validity-diversity tradeoffs. We conclude that our selection procedures remain useful, but selection predictor-criterion relationships are considerably lower than previously thought. (PsycInfo Database Record (c) 2022 APA, all rights reserved).
1 January 2008 in Annual Review of Psychology
We review developments in personnel selection since the previous review by Hough & Oswald (2000) in the Annual Review of Psychology. We organize the review around a taxonomic structure of possible bases for improved selection, which includes (a) better understanding of the criterion domain and criterion measurement, (b) improved measurement of existing predictor methods or constructs, (c) identification and measurement of new predictor methods or constructs, (d) improved identification of features that moderate or mediate predictor-criterion relationships, (e) clearer understanding of the relationship between predictors or between predictors and criteria (e.g., via meta-analytic synthesis), (f) identification and prediction of new outcome variables, (g) improved ability to determine how well we predict the outcomes of interest, (h) improved understanding of subgroup differences, fairness, bias, and the legal defensibility, (i) improved administrative ease with which selection systems can be used, (j) improved insight into applicant reactions, and (k) improved decision-maker acceptance of selection systems.
1 January 2009 in Psychological Bulletin
Critics of educational admissions tests assert that tests measure nothing more than socioeconomic status (SES) and that their apparent validity in predicting academic performance is an artifact of SES. The authors examined multiple large data sets containing data on admissions and related tests, SES, and grades showing that (a) SES is related to test scores (r = .42 among the population of SAT takers), (b) test scores are predictive of academic performance, and (c) statistically controlling for SES reduces the estimated test?grade correlation from r = .47 to r = .44. Thus, the vast majority of the test?academic performance relationship was independent of SES: The authors concluded that the test?grade relationship is not an artifact of common influences of SES on both test scores and grades.
1 January 2008 in American Psychologist
The authors review criticisms commonly leveled against cognitively loaded tests used for employment and higher education admissions decisions, with a focus on large-scale databases and meta-analytic evidence. They conclude that (a) tests of developed abilities are generally valid for their intended uses in predicting a wide variety of aspects of short-term and long-term academic and job performance, (b) validity is not an artifact of socioeconomic status, (c) coaching is not a major determinant of test performance, (d) tests do not generally exhibit bias by underpredicting the performance of minority group members, and (e) test-taking motivational mechanisms are not major determinants of test performance in these high-stakes settings.
17 September 2014 in Perspectives on Psychological Science
Employees face a variety of work demands that place a premium on personal attributes, such as the degree to which they can be depended on to work independently, deal with stress, and interact positively with coworkers and customers. We examine evidence for the importance of these personality attributes using research strategies intended to answer three fundamental questions, including (a) how well does employees’ standing on these attributes predict job performance?, (b) what types of attributes do employers seek to evaluate in interviews when considering applicants?, and (c) what types of attributes are rated as important for performance in a broad sampling of occupations across the U.S. economy? We summarize and integrate results from these three strategies using the Big Five personality dimensions as our organizing framework. Our findings indicate that personal attributes related to Conscientiousness and Agreeableness are important for success across many jobs, spanning across low to high levels of job complexity, training, and experience necessary to qualify for employment. The strategies lead to differing conclusions about the relative importance of Emotional Stability and Extraversion. We note implications for job seekers, for interventions aimed at changing standing on these attributes, and for employers.
1 January 2007 in Journal of Applied Psychology
The purpose of this research report is to highlight a unique set of issues that arise when considering the effects of range restriction in the context of estimating predictor intercorrelations. Three approaches are used to illustrate the issue: simulation, a concrete applied example, and a reanalysis of a meta-analysis of ability-interview correlations. The general conclusion is that a predictor intercorrelation can differ dramatically from the population value when both predictors are used in a composite that is used operationally for selection. The compensatory nature of a composite means that low scorers on one predictor can only obtain high scores on the composite if they obtain very high scores on the other predictor; this phenomenon distorts the correlation between the predictors.