1 March 2023 in Information Systems Research
Analysts, managers, and policymakers are interested in predictive analytics capable of offering better foresight. It is generally accepted that in forecasting scenarios involving organizational policies or consumer decision making, personal characteristics, including personality, may be an important predictor of downstream outcomes. The inclusion of personality features in forecasting models has been hindered by the fact that traditional measurement mechanisms are often infeasible. Text-based personality detection has garnered attention due to the public availability of digital textual traces, however state-of-the-art models proposed by IBM, Google, Facebook, and academic research are not accurate enough to be used for downstream real-world forecasting tasks. We propose a novel text-based personality measurement approach that improves detection of personality dimensions by 10–20 percentage points relative to the best existing methods developed in industry and academia. Using case studies in the finance and health domains, we show that more accurate text-based personality detection can translate into significant improvements in downstream applications such as forecasting future firm performance or predicting pandemic infection rates. Our findings have important implications for managers focused on enabling, producing, or consuming predictive analytics for enhanced agility in decision making.
12 December 2017 in International Journal of Machine Learning and Cybernetics
22 January 2025 in ACM Transactions on Information Systems
The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models (LLMs) such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML models, human- and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that LLMs and transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that LLMs and transformer PLMs tend to score GPT-generated text 10–20% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that even though the LLMs and transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly (in part) due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.
13 March 2025 in ACM Transactions on Management Information Systems
The use of machine learning (ML) to detect depression in online settings has emerged as an important health and wellness use case. In particular, the use of deep learning methods for depression detection from textual content posted on social media has garnered considerable attention. Conversely, there has been relatively limited evaluation of depression detection in clinical environments involving text generated from remote interviews. In this research, we review state-of-the-art feature-based ML, deep learning, and large language models for depression detection. We use a multidimensional analysis framework to benchmark various language models on a novel testbed comprising speech-to-text transcriptions of remote interviews. Our framework considers the impact of different transcription types and interview segments on depression detection performance. Finally, we summarize the key trends and takeaways from the review and benchmark evaluation and provide suggestions to guide the design of future detection methods.