21 May 2018 in Simulation & Gaming
Background. Definitions of gamification tend to vary by person, both in industry and within academia. One particularly popular lay interpretation, introduced and popularized by Ian Bogost, and reiterated by Jan Klabbers, is that gamification is “bullshit” and “exploitationware.” They describe gamification as a marketing term or business practice invented to sell products rather than to represent a real and unique phenomenon relevant to a nascent game science. However, this view is an oversimplification, one which ignores a growing body of theory development and empirical research on gamification within a post-positivist epistemology. In fact, because gamification is so much more outcome-focused than general game design, current gamification research in many ways has a stronger footing in modern social science than much games research does. Aim. In this article, to address common misunderstandings like these, we describe the philosophical underpinnings of modern gamification research, define the relationship between games and gamification, define and situate gamification science as a subdiscipline of game science, and explicate a six-element framework of major concerns within gamification science: predictor constructs, criterion constructs, mediator constructs, moderator constructs, design processes, and research methods. This framework is also presented diagrammatically as a causal path model. Conclusion. Gamification science refers to the development of theories of gamification design and their empirical evaluation within a post-positivist epistemology. The goal of gamification scientist-practitioners should be to understand how to best meet organizational goals through the design of gamification interventions, drawing upon insights derived from both gamification science and games research more broadly.
14 December 2021 in International Journal of Selection and Assessment
1 January 2022 in Technology, Mind, and Behavior
Although it is commonly assumed that implementing unproctored internet testing (UIT) in employee selection systems can result in increased applicant pool diversity, this assumption has not been explicitly tested. Thus, we analyzed the applicant pool composition of a major U.S.-based manufacturing organization across the span of over 8 years (N = 24,963) using an interrupted time series analytic approach. This allowed us to evaluate changes before and after the implementation of unproctored testing as well as changes following subsequent mobile device blocking. The results of this analysis suggested that although adding a UIT option appeared to increase the size of the applicant pool, the magnitude of this effect did not appear to differ between Black and White applicants. Furthermore, removing the option to apply on a mobile device dampened this general effect, but similarly, there were no differences in the magnitude applicant pool reduction between Black and White applicants. This evidence contradicts the common notion that UITs result in more diverse applicant pools, suggesting UIT’s primary value in this regard is increasing access to the application process across groups. Additionally, this study demonstrates the use of interrupted time series analysis as a powerful framework to understand longitudinal effects in real-world employee selection data.
2 October 2022 in International Journal of Testing
Assessment trace data, such as mouse positions and their timing, offer interesting and provocative reflections of individual differences yet are currently underutilized by testing professionals. In this article, we present a 10-step procedure to maximize the probability that a trace data modeling project will be successful: 1) grounding the project in psychometric theory, 2) building technical infrastructure to collect trace data, 3) designing a useful developmental validation study, 4) using a holdout validation approach with collected data, 5) using exploratory analysis to conduct meaningful feature engineering, 6) identifying useful machine learning algorithms to predict a thoughtfully chosen criterion, 7) engineering a machine learning model with meaningful internal cross-validation and hyperparameter selection, 8) conducting model diagnostics to assess if the resulting model is overfitted, underfitted, or within acceptable tolerance, and 9) testing the success of the final model in meeting conceptual, technical, and psychometric goals. If deemed successful, trace data model predictions could then be engineered into decision-making systems. We present this framework within the broader view of psychometrics, exploring the challenges of developing psychometrically valid models using such complex data with much weaker trait signals than assessment developers have typically attempted to model.