Information Systems and Business Analytics Department, College of Business, Florida International University, 11200 SW 8th Street, Miami, FL, 33199, United States, 1 305-348-4610
Auto-generated profile, based on publicly available information
Alfred Castillo
Information Systems and Business Analytics Department, College of Business, Florida International University, 11200 SW 8th Street, Miami, FL, 33199, United States, 1 305-348-4610
Background: The application of large language models (LLMs) in analyzing expert textual online data is a topic of growing importance in computational linguistics and qualitative research within health care settings. Objective: The objective of this study was to understand how LLMs can help analyze expert textual data. Topic modeling enables scaling the thematic analysis of content of a large corpus of data, but it still requires interpretation. We investigate the use of LLMs to help researchers scale this interpretation. Methods: The primary methodological phases of this project were (1) collecting data representing posts to an online nurse forum, as well as cleaning and preprocessing the data; (2) using latent Dirichlet allocation (LDA) to derive topics; (3) using human categorization for topic modeling; and (4) using LLMs to complement and scale the interpretation of thematic analysis. The purpose is to compare the outcomes of human interpretation with those derived from LLMs. Results: There is substantial agreement (247/310, 80%) between LLM and human interpretation. For two-thirds of the topics, human evaluation and LLMs agree on alignment and convergence of themes. Furthermore, LLM subthemes offer depth of analysis within LDA topics, providing detailed explanations that align with and build upon established human themes. Nonetheless, LLMs identify coherence and complementarity where human evaluation does not. Conclusions: LLMs enable the automation of the interpretation task in qualitative research. There are challenges in the use of LLMs for evaluation of the resulting themes.
Background: Web-based forums provide a space for communities of interest to exchange ideas and experiences. Nurse professionals used these forums during the COVID-19 pandemic to share their experiences and concerns. Objective: The objective of this study was to examine the nurse-generated content to capture the evolution of nurses’ work concerns during the COVID-19 pandemic. Methods: We analyzed 14,060 posts related to the COVID-19 pandemic from March 2020 to April 2021. The data analysis stage included unsupervised machine learning and thematic qualitative analysis. We used an unsupervised machine learning approach, latent Dirichlet allocation, to identify salient topics in the collected posts. A human-in-the-loop analysis complemented the machine learning approach, categorizing topics into themes and subthemes. We developed insights into nurses’ evolving perspectives based on temporal changes. Results: We identified themes for biweekly periods and grouped them into 20 major themes based on the work concern inventory framework. Dominant work concerns varied throughout the study period. A detailed analysis of the patterns in how themes evolved over time enabled us to create narratives of work concerns. Conclusions: The analysis demonstrates that professional web-based forums capture nuanced details about nurses’ work concerns and workplace stressors during the COVID-19 pandemic. Monitoring and assessment of web-based discussions could provide useful data for health care organizations to understand how their primary caregivers are affected by external pressures and internal managerial decisions and design more effective responses and planning during crises.