Automated scoring is a current hot topic in creativity research. However, most research has focused on the English language and popular verbal creative thinking tasks, such as the alternate uses task. Therefore, in this study, we present a large language model approach for automated scoring of a scientific creative thinking task that assesses divergent ideation in experimental tasks in the German language. Participants are required to generate alternative explanations for an empirical observation. This work analyzed a total of 13,423 unique responses. To predict human ratings of originality, we used XLM‐RoBERTa (Cross‐lingual Language Model‐RoBERTa), a large, multilingual model. The prediction model was trained on 9,400 responses. Results showed a strong correlation between model predictions and human ratings in a held‐out test set (n = 2,682; r = 0.80; CI‐95% [0.79, 0.81]). These promising findings underscore the potential of large language models for automated scoring of scientific creative thinking in the German language. We encourage researchers to further investigate automated scoring of other domain‐specific creative thinking tasks.

The alternate uses task (AUT) is the most popular measure when it comes to the assessment of creative potential. Since their implementation, AUT responses have been rated by humans, which is a laborious task and requires considerable resources. Large language models (LLMs) have shown promising performance in automatically scoring AUT responses in English as well as in other languages, but it is not clear which method works best for German data. Therefore, we investigated the performance of different LLMs for the automated scoring of German AUT responses. We compiled German data across five research groups including ~50,000 responses for 15 different alternate uses objects from eight lab and online survey studies (including ~2300 participants) to examine generalizability across datasets and assessment conditions. Following a pre-registered analysis plan, we compared the performance of two fine-tuned, multilingual LLM-based approaches [Cross-Lingual Alternate Uses Scoring (CLAUS) and the Open Creativity Scoring with Artificial Intelligence (OCSAI)] with the Generative Pre-trained Transformer (GPT-4) in scoring (a) the original German AUT responses and (b) the responses translated to English. We found that the LLM-based scorings were substantially correlated with human ratings, with higher relationships for OCSAI followed by GPT-4 and CLAUS. Response translation, however, had no consistent positive effect. We discuss the generalizability of the results across different items and studies and derive recommendations and future directions.

Creative fluency and originality are pivotal indicators of creative potential. Both have been embedded in hierarchical intelligence models as part of the ability to retrieve information from long‐term memory; an ability that is often measured with indicators of retrieval fluency. Creative fluency and retrieval fluency, both expressed by the count of correct responses, are procedurally highly similar. This raises the question how creative fluency and originality are related with retrieval fluency and how both are predicted by other cognitive abilities. In a multivariate study (N = 320), we found that retrieval fluency is very strongly related with creative fluency (r = .87) and substantially related with originality (r = .59). A combined fluency factor still fitted the data well. Cognitive abilities accounted for 63% variance in fluency and 47% variance in originality. After controlling for established cognitive abilities, latent variables for fluency and originality were unrelated with one another. This suggests that the procedural proximity of the ability to fluently generate either information from long‐term memory or ad‐hoc solutions to unusual tasks and the ability to come up with original ideas needs reconsideration. Locating originality below an overarching retrieval factor is contradicted by the present data.

The present paper questions the content validity of the eight creativity‐related self‐report scales available in PISA 2022's context questionnaire and provides a set of considerations for researchers interested in using these indexes. Specifically, we point out some threats to the content validity of these scales (e.g., creative thinking self‐efficacy, creativity, and openness to intellect) including ambiguity in construct and content definition, construct underrepresentation, or item overlaps between constructs. We then discuss content validity from a theoretical angle, outlining how important is the clarity of content domain definition and corresponding test content, to avoid possible jingle‐jangle fallacies in the item sets (e.g., overlap between same/similar items attributed to different constructs) which undermine the utility of resulting scores. We also present approaches for empirically quantifying content validity. In all, this paper calls for a cautious use and interpretation of creativity self‐report data in PISA 2022 as it stands. It provides guidance for further examination of the content validity of the creativity‐related self‐report scales in PISA 2022. Such future efforts are necessary to enhance the validity of inferences made from these self‐report scale scores, to optimize their comparability across different research teams and nations, toward a more generalizable use in creativity and education research.