Reliability is a fundamental problem for measurement in all of science. Although defined in multiple ways, and estimated in even more ways, the basic concepts seem straightforward and need to be understood by practitioners as well as methodologists. Reliability theory is not just for the psychometrician estimating latent variables, it is for everyone who wants to make inferences from measures of individuals or of groups. For the case of a single test administration, we consider multiple measures of reliability, ranging from the worst (β) to average (α, λ3) to best (λ4) split half reliabilities, and consider why model-based estimates (ωh, ωt) should be reported. We also address the utility of test-retest and alternate form reliabilities. The advantages of immediate versus delayed retests to decompose observed score variance into specific, state, and trait scores are discussed. But reliability is not just for test scores, it is also important when evaluating the use of ratings. Estimates that may be applied to continuous data include a set of intraclass correlations while discrete categorical data needs to take advantage of the family of κ statistics. Examples of these various reliability estimates are given using state and trait measures of anxiety given with different delays and under different conditions. An online supplemental materials is provided with more detail and elaboration. The online supplemental materials is also used to demonstrate applications of open source software to examples of real data, and comparisons are made between the many types of reliability. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

In pursuit of a more systematic and comprehensive framework for personality assessment, we introduce procedures for assessing personality traits at the lowest level: nuances. We argue that constructing a personality taxonomy from the bottom up addresses some of the limitations of extant top-down assessment frameworks (e.g., the Big Five), including the opportunity to resolve confusion about the breadth and scope of traits at different levels of the organization, evaluate unique and reliable trait variance at the item level, and clarify jingle/jangle issues in personality assessment. With a focus on applications in survey methodology and transparent documentation, our procedures contain six steps: (1) identification of a highly inclusive pool of candidate items, (2) programmatic evaluation and documentation of item characteristics, (3) test-retest analyses of items with adequate qualitative and quantitative properties, (4) analysis of cross-ratings from multiple raters for items with adequate retest reliability, (5) aggregation of ratings across diverse samples to evaluate generalizability across populations, (6) evaluations of predictive utility in various contexts. We hope these recommendations are the first step in a collaborative effort to identify a comprehensive pool of personality nuances at the lowest level, enabling subsequent construction of a robust hierarchy – from the bottom up.

Recent advances in natural language processing (NLP) have produced general models that can perform complex tasks such as summarizing long passages and translating across languages. Here, we introduce a method to extract adjective similarities from language models as done with survey-based ratings in traditional psycholexical studies but using millions of times more text in a natural setting. The correlational structure produced through this method is highly similar to that of self- and other-ratings of 435 English terms reported by Saucier and Goldberg (1996a). The first three unrotated factors produced using NLP are congruent with those in survey data, with coefficients of 0.89, 0.79, and 0.79. This structure is robust to many modeling decisions: adjective set, including those with 1,710 (Goldberg, 1982) and 18,000 English terms (Allport & Odbert, 1936); the query used to extract correlations; and language model. Notably, Neuroticism and Openness are only weakly and inconsistently recovered. This is a new source of signal that is closer to the original (semantic) vision of the lexical hypothesis. The method can be applied where surveys cannot: in dozens of languages simultaneously, with tens of thousands of items, on historical text, and at extremely large scale for little cost. The code is made public to facilitate reproduction and fast iteration in new directions of research. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The influence of personality on important life outcomes has been widely recognized for thousands of years (Condon, 2014), and the difficulty of its measurement has been vexing for many decades (Galton, 1884; Cattell, 1945; Goldberg, 1981; Ackerman, 2018). The challenge with objective measurement stems from the need for massive amounts of data to account for dynamic interplay between variations in thousands of narrow dispositional traits (aka individual differences in behavior) and the ever-evolving contextual factors inherent to modern living. It is a prototypical “big data” problem. Despite this, dozens of ambitious social scientists have posited a diverse array of personality assessment models. Many of these are heavily imbued with theory, nearly all are focused solely on one domain of personality (e.g., very broad dispositional traits or vocational interests) to the exclusion of others (e.g., cognitive abilities, values, or less generalizable maladaptive behaviors), and most have been derived based on surprisingly small samples drawn from populations that have come to be known as "WEIRD" (Henrich et al., 2010). Simply put, there is widespread need for models that are empirically-grounded in more (and more representative) data.In this manuscript, I demonstrate that it is possible to address the shortcomings of extant theory-driven approaches by combining recent innovations from outside of personality research to empirically derive personality assessment models. This is done by administering a large pool of widely-used public domain items from the International Personality Item Pool (Goldberg et al., 1999) to three large online samples (N > 125,000) using a planned missingness design (Revelle et al., 2016). While the existing "best practices" for developing personality assessment models tends towards several iterative rounds of data collection and analysis guided by theory culminating in publication of only the final product, I have endeavored to make a highly detailed record of all steps followed during the development of the SAPA Personality Inventory in order to encourage feedback regarding critical analytic decisions. This has unfortunately resulted in the production of a book-length manuscript but I hope that this transparency will serve to minimize (even if it does not eliminate) the influence of bias.January 10, 2018: This manuscript remains subject to review and further revision. I welcome additional feedback (by email). I will note the status of these revisions as they occur.