Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Recent advancements in Large Language Models (LLMs) have generated significant interest in their potential for assessing psychological constructs, particularly personality traits. While prior research has explored LLMs’ capabilities in zero-shot or few-shot personality inference, few studies have systematically evaluated LLM embeddings within a psychometric validity framework or examined their correlations with linguistic and emotional markers. Additionally, the comparative efficacy of LLM embeddings against traditional feature engineering methods remains underexplored, leaving gaps in understanding their scalability and interpretability for computational personality assessment. Objective: This study evaluates LLM embeddings for personality trait prediction through four key analyses: (1) performance comparison with zero-shot methods on PANDORA Reddit data, (2) psychometric validation and correlation with LIWC and emotion features, (3) benchmarking against traditional feature engineering approaches, and (4) assessment of model size effects (OpenAI vs BERT vs. RoBERTa). We aim to establish LLM embeddings as a psychometrically valid and efficient alternative for personality assessment. Methods: We conducted a multi-stage analysis using 1 million Reddit posts from the PANDORA Big Five Personality dataset. First, we generated text embeddings using three LLM architectures (RoBERTa, BERT, and OpenAI) and trained a custom BiLSTM model for personality prediction. We compared this approach against zero-shot inference using prompt-based methods. Second, we extracted psycholinguistic features (LIWC categories and NRC emotions) and performed feature engineering to evaluate potential performance enhancements. Third, we assessed the psychometric validity of LLM embeddings; reliability validity using Cronbach's alpha, and convergent validity analysis by examining correlations between embeddings and established linguistic markers. Finally, we performed traditional feature engineering on static psycholinguistic features to assess performance under different settings. Results: LLM embeddings trained using simple deep learning techniques significantly outperform zero-shot approaches on average by 45% across all personality traits. Although, psychometric validation tests indicate moderate reliability, with an average Cronbach’s alpha of 0.63, correlation analyses spark a strong association with key linguistic/emotional markers; such as openness correlates highly with social (r=0.53), conscientiousness with linguistic (r=0.46), extraversion with social (r=0.41), agreeableness with pronoun usage (r=0.40), and neuroticism with politics-related text (r=0.63). Despite adding advanced feature engineering on linguistic features, the performance didn’t improve, suggesting that LLM embeddings inherently capture key linguistic features. Furthermore, our analyses demonstrated efficacy on larger model size with computational cost tradeoff. Conclusions: Our findings demonstrate that LLM embeddings offer a robust alternative to zero-shot methods in personality trait analysis, capturing key linguistic patterns without requiring extensive feature engineering. The correlation with established psycholinguistic markers and the performance tradeoff with computational cost provides a hint for future computational linguistic work who are targeting LLM for personality assessment. Further research should explore fine-tuning strategies to enhance psychometric validity.

Background: The COVID-19 pandemic intensified the challenges associated with mental health and substance use (SU), with societal and economic upheavals leading to heightened stress and increased reliance on drugs as a coping mechanism. Centers for Disease Control and Prevention data from June 2020 showed that 13% of Americans used substances more frequently due to pandemic-related stress, accompanied by an 18% rise in drug overdoses early in the year. Simultaneously, a significant increase in social media engagement provided unique insights into these trends. Our study analyzed social media data from January 2019 to December 2021 to identify changes in SU patterns across the pandemic timeline, aiming to inform effective public health interventions. Objective: This study aims to analyze SU from large-scale social media data during the COVID-19 pandemic, including the prepandemic and postpandemic periods as baseline and consequence periods. The objective was to examine the patterns related to a broader spectrum of drug types with underlying themes, aiming to provide a more comprehensive understanding of SU trends during the COVID-19 pandemic. Methods: We leveraged a deep learning model, Robustly Optimized Bidirectional Encoder Representations from Transformers Pretraining Approach (RoBERTa), to analyze 1.13 billion Twitter (subsequently rebranded X) posts from January 2019 to December 2021, aiming to identify SU posts. The model’s performance was enhanced by a human-in-the-loop strategy that subsequently enriched the annotated data used during the fine-tuning phase. To gain insights into SU trends over the study period, we applied a range of statistical techniques, including trend analysis, k-means clustering, topic modeling, and thematic analysis. In addition, we integrated the system into a real-time application designed for monitoring and preventing SU within specific geographic locations. Results: Our research identified 9 million SU posts in the studied period. Compared to 2019 and 2021, the most substantial display of SU-related posts occurred in 2020, with a sharp 21% increase within 3 days of the global COVID-19 pandemic declaration. Alcohol and cannabinoids remained the most discussed substances throughout the research period. The pandemic particularly influenced the rise in nonillicit substances, such as alcohol, prescription medication, and cannabinoids. In addition, thematic analysis highlighted COVID-19, mental health, and economic stress as the leading issues that contributed to the influx of substance-related posts during the study period. Conclusions: This study demonstrates the potential of leveraging social media data for real-time detection of SU trends during global crises. By uncovering how factors such as mental health and economic stress drive SU spikes, particularly in alcohol and prescription medication, we offer crucial insights for public health strategies. Our approach paves the way for proactive, data-driven interventions that will help mitigate the impact of future crises on vulnerable populations.

Background: User demographics are often hidden in social media data due to privacy concerns. However, demographic information on substance use (SU) can provide valuable insights, allowing public health policy makers to focus on specific cohorts and develop efficient prevention strategies, especially during global crises such as the COVID-19 pandemic. Objective: This study aimed to analyze SU trends at the user level across different demographic dimensions, such as age, gender, race, and ethnicity, with a focus on the COVID-19 pandemic. The study also establishes a baseline for SU trends using social media data. Methods: The study was conducted using large-scale English-language data from Twitter (now known as X) over a 3-year period (2019, 2020, and 2021), comprising 1.13 billion posts. Following preprocessing, the SU posts were identified using our custom-trained deep learning model (Robustly Optimized Bidirectional Encoder Representations From Transformers Pretraining Approach [RoBERTa]), which resulted in the identification of 9 million SU posts. Then, demographic attributes, such as user type, age, gender, race, and ethnicity, as well as sentiments and emotions associated with each post, were extracted via a collection of natural language processing modules. Finally, various qualitative analyses were performed to obtain insight into user behaviors based on demographics. Results: The highest level of user participation in SU discussions was observed in 2020, with a 22.18% increase compared to 2019 and a 25.24% increase compared to 2021. Throughout the study period, male users and teenagers increasingly dominated the SU discussions across all substance types. During the COVID-19 pandemic, user participation in prescription medication discussions was notably higher among female users compared to other substance types. In addition, alcohol use increased by 80% within 2 weeks after the global pandemic declaration in 2020. Conclusions: This study presents a large-scale, fine-grained analysis of SU on social media data, examining trends by age, gender, race, and ethnicity before, during, and after the COVID-19 pandemic. Our findings, contextualized with sociocultural and pandemic-specific factors, provide actionable insights for targeted public health interventions. This study establishes social media data (powered with artificial intelligence and natural language processing tools) as a valuable platform for real-time SU surveillance and prevention during crises.

Large language models (LLMs) have transformed natural language processing, enabling contextually coherent text generation at scale. Although conversational language contains signals associated with personality traits, mapping naturalistic conversation to stable personality-related representations remains challenging. We introduce a real-world benchmark for personality-related inference in LLMs, using 555 semi-structured interviews paired with BFI-10 self-report measures as reference labels. Four recent instruction-tuned models (GPT5-Mini, GPT-4.1-Mini, Meta-LLaMA, and DeepSeek) are evaluated under zero-shot and chain-of-thought (CoT) prompting, predicting individual BFI-10 items and aggregated Big Five trait scores on a 1–5 Likert scale. Across models, repeated inference runs exhibited high internal consistency under fixed settings, but alignment with self-reported personality was weak, with low correlations (r ≤ 0.27), minimal categorical agreement (Cohen’s κ < 0.11), and systematic overestimation toward moderate-to-high trait values. CoT prompting and extended input context provided limited benefits, yielding modest improvements in trait distribution alignment for some models without consistently improving trait-level accuracy or cross-model agreement. Overall, this benchmark characterizes how LLMs map state-rich conversational language to personality-related inferences under constrained observational settings, highlighting current limitations and informing future work on personalization and user-adaptive conversational systems.

UNSTRUCTURED Recent advancements in Large Language Models (LLMs) have sparked interdisciplinary interest in their ability to assess psychological constructs, particularly Personality. While prior machine learning research has focused on evaluating LLMs’ capability to infer personality traits, often via zero-shot or few-shot learning, few studies have systematically examined the applicability of LLM embeddings for Personality Prediction within a robust psychometric validity framework or explored their correlation with psychological and linguistic features. Addressing this gap, we investigate performance of LLM embeddings on a well-labeled PANDORA dataset (Big Five Personality traits from Reddit). Our key contributions are: (1) demonstrating that LLM embeddings significantly outperform zero-shot approaches in personality prediction; (2) validating the reliability of embeddings using psychometric evaluation and analyzing their correlations with linguistic features (e.g., LIWC, emotional markers); (3) contrasting embedding performance against advanced feature engineering methods (joint and independent use of psycholinguistic features); and (4) revealing that model size influences efficacy, OpenAI outperforming RoBERTa. Our findings underscore the potential of LLM embeddings as a scalable and interpretable tool for computational personality assessment.

Background: Personality traits are known predictors of substance use (SU), but their expression and association with SU in digital discourse remain largely unexamined. During the COVID-19 pandemic, the online social engagement heightened and led to an amplification in SU rates, thereby creating a unique natural opportunity to investigate these dynamics through large-scale digital discourse data. In our study, we offer insights beyond traditional self-report methods, which are crucial for developing timely and targeted public health interventions. Objective: We aim to evaluate whether the associations between the Big Five personality traits and SU discourse shifted during the 2019‐2021 period, and to conduct a focused analysis of how these traits predict SU and relate to specific substance types, emotional expression, and demographic factors. Methods: We analyzed a corpus of several hundred million public posts from a major social media platform from 2019 to 2021. Using a pipeline of natural language processing and deep learning models, we identified SU-related posts and subsequently extracted scores for the Big Five personality traits, emotions, and user demographics. We used trend analysis to compare annual shifts in trait-SU associations, while detailed 2020 data underwent rigorous modeling using logistic regression, correlation analysis, and topic modeling to elucidate the core relationships. Results: Our analysis revealed that Extraversion (odds ratio [OR] 3.22, 95% CI 2.98-3.49) and, most strikingly, agreeableness (OR 4.04, 95% CI 3.71-4.41) were the strongest positive predictors of being a substance user. In stark contrast to the conventional self-medication hypothesis, neuroticism emerged as a robust protective factor against SU (OR 0.29, 95% CI 0.26-0.31). This counterintuitive finding was supported by a decreased association between neuroticism and SU posts at the pandemic’s onset in 2020 (Cohen d =−0.13, 95% CI) and a negative correlation with the expression of negative emotions online. Topic modeling further indicated that SU discourse was frequently embedded in social contexts (social drinking and friendly beverage choices) rather than themes of solitary coping. Conclusions: Our findings challenge traditional models by demonstrating that in large-scale online discourse, SU expression is more powerfully linked to social-affiliative traits than to negative emotionality. The paradoxical protective role of neuroticism suggests that established risk profiles may not apply uniformly to digital environments, particularly during a public health crisis. These insights are vital for refining computational methods for public health surveillance and developing interventions that recognize the potent social drivers of SU in the digital age.