Unprecedented opportunities have been brought by advancements in machine learning in the prediction of the economic success of movies. The analysis of movie title keywords is one promising but rarely investigated direction of study. To address this gap, we performed a text mining and exploratory factor analysis (EFA) of the relationships between movie titles and their corresponding movies’ levels of success. Specifically, intragroup and intergroup analyses of 217 top hit movies in the United States and 245 top hit movies in China showed that successful movies in these two major movie markets with outstanding total lifetime grosses featured titles with similar and different patterns of most frequently used words, revealing useful information about viewers’ preferences in these countries. The findings of this study will serve to better inform the movie industry in giving more economically promising names to their products from a machine-learning perspective and inspire further studies.
17 August 2025 in Journal of Educational Measurement
Large language models (LLMs) have shown great potential in automatic scoring. However, due to model characteristics and variation in training materials and pipelines, scoring inconsistency can exist within an LLM and across LLMs when rating the same response multiple times. This study investigates the intra-LLM and inter-LLM consistency in scoring with five LLMs (i.e., Claude, DeepSeek, Gemini, GPT, and Qwen), variability under different temperatures, and their relationship with scoring accuracy. Moreover, a voting strategy that assembles information from different LLMs was proposed to address inconsistent scoring. Using constructed-response items from a science education assessment and open-source data from the Automated Student Assessment Prize (ASAP), we find that: (a) LLMs generally exhibited almost perfect intra-LLM consistency regardless of temperature; (b) inter-LLM consistency was moderate, with higher agreement observed for items that were easier to score; (c) intra-LLM consistency consistently exceeded inter-LLM consistency, supporting the expectation that within-model consistency represents an upper bound for cross-model agreement; (d) intra-LLM consistency was not associated with scoring accuracy, whereas inter-LLM consistency showed a strong positive relationship with accuracy; and (e) majority voting across LLMs improved scoring accuracy by leveraging complementary strengths of different models.