As the use of computational text analysis in the social sciences has increased, topic modeling has emerged as a popular method for identifying latent themes in textual data. Nevertheless, concerns have been raised regarding the validity of the results produced by this method, given that it is largely automated and inductive in nature, and the lack of clear guidelines for validating topic models has been identified by scholars as an area of concern. In response, we conducted a comprehensive systematic review of 789 studies that employ topic modeling. Our goal is to investigate whether the field is moving toward a common framework for validating these models. The findings of our review indicate a notable absence of standardized validation practices and a lack of convergence toward specific methods of validation. This gap may be attributed to the inherent incompatibility between the inductive, qualitative approach of topic modeling and the deductive, quantitative tradition that favors standardized validation. To address this, we advocate for incorporating qualitative validation approaches, emphasizing transparency and detailed reporting to improve the credibility of findings in computational social science research when using topic modeling.
15 July 2023 in Journalism & Mass Communication Quarterly
In today’s hybrid media system, communication professions are connected to media convergence, and this leads to a blurring of boundaries between journalism, public relations (PR), and marketing. Our study demonstrates the blurring boundaries of the communication professions through a longitudinal automated semantic analysis of 336,629 job postings in the fields of journalism, PR, and marketing published between 2015 and 2020 in Austria and Germany. Similarities in skills demanded in job postings in journalism, PR, and marketing in Austria and Germany are greater than differences, but the three job fields are still distinguishable from each other.
Mounting concern surrounds the influence of political actors on journalism, especially as media outlets face increasing financial pressures. These circumstances can give rise to instances of media capture, a mutually corrupting relationship between political actors and media organizations. However, empirical evidence substantiating such mechanisms and their consequences remains limited, particularly in the context of Western democracies. This paper investigates a recent case, in which a former Austrian chancellor allegedly colluded with a tabloid newspaper to receive better news coverage in exchange for increased ad placements by government institutions. Employing automated content analysis of political news articles from six prominent Austrian news outlets spanning 2012 to 2021 (n = 188,203), we adopt a difference-in-differences approach to scrutinize political actors’ visibility and favorability in news coverage for patterns indicative of the alleged serious breach of professional political and journalistic norms. Our findings indicate a substantial increase in the news coverage of the former Austrian chancellor within the news outlet that is alleged to have received bribes. In contrast, several other political actors did not experience similar shifts in visibility nor are similar patterns identified in other media outlets. No such effects have been found for favorability. While these findings do not purport to prove or disprove whether specific political actors or media organizations were involved in unethical or illegal activities, they shed light on potential mechanisms of media capture, highlight contemporary threats to media independence, and underscore the critical importance of safeguarding journalistic integrity and ensuring unbiased information dissemination to the public.
6 August 2025 in Journal of Survey Statistics and Methodology
How we ask questions in surveys significantly impacts the answers. While previous research has examined differences between open-ended and closed-ended survey responses, our understanding remains fragmented, especially considering the growing interest in open-ended questions spurred by advances in automated text analysis. We explore how computational text analysis methods––particularly large language models––can efficiently analyze open-ended responses with promising sentiment and actor identification results, though topic classification remains more challenging. Then, we systematically compare open-ended and closed-ended responses on trust in the Austrian coalition government to assess how different formats shape the expression of political trust. We find significant discrepancies between the trust levels indicated in closed-ended questions and the trust described in open-ended responses, with paired t-tests also revealing statistically significant differences. These findings highlight that question format affects how respondents express their views, raising important considerations about the usefulness and validity of trust measures. They highlight the unique value of open-ended questions in capturing nuanced perspectives often missed by closed-ended formats. This has broader implications for survey design: including open-ended questions could enrich data quality and provide deeper insights into public opinion.