17 May 2024 in Political Analysis
Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.
1 February 2019 in American Political Science Review
Multilevel regression and post-stratification (MRP) is the current gold standard for extrapolating opinion data from nationally representative surveys to smaller geographic units. However, innovations in nonparametric regularization methods can further improve the researcher’s ability to extrapolate opinion data to a geographic unit of interest. I test an ensemble of regularization algorithms and find that there is room for substantial improvement on the multilevel model via more flexible methods of regularization. I propose a modified version of MRP that replaces the multilevel model with a nonparametric approach called Bayesian additive regression trees (BART or, when combined with post-stratification, BARP). I compare both methods across a number of data contexts, demonstrating the benefits of applying more powerful regularization methods to extrapolate opinion data to target geographical units. I provide an R package that implements the BARP method.
13 June 2017 in American Political Science Review
To answer questions about the origins and outcomes of collective action, political scientists increasingly turn to datasets with social network information culled from online sources. However, a fundamental question of external validity remains untested: are the relationships measured between a person and her online peers informative of the kind of offline, “real-world” relationships to which network theories typically speak? This article offers the first direct comparison of the nature and consequences of online and offline social ties, using data collected via a novel network elicitation technique in an experimental setting. We document strong, robust similarity between online and offline relationships. This parity is not driven by sharedidentityof online and offline ties, but a shared nature of relationships in both domains. Our results affirm that online social tie data offer great promise for testing long-standing theories in the social sciences about the role of social networks.
28 July 2021 in American Political Science Review
The relationship between anxiety and investor behavior is well known enough to warrant its own aphorism: a “flight to safety.” We posit that anxiety alters the intensity of voters’ preference for the status quo, inducing a political flight to safety toward establishment candidates. Leveraging the outbreak of the novel coronavirus during the Democratic primary election of 2020, we identify a causal effect of the outbreak on voting, with Biden benefiting between 7 and 15 percentage points at Sanders’s expense. A survey experiment in which participants exposed to an anxiety-inducing prompt choose the less disruptive hypothetical candidate provides further evidence of our theorized flight to safety among US-based respondents. Evidence from 2020 French municipal and US House primary elections suggests a COVID-induced flight to safety generalizes to benefit mainstream candidates across a variety of settings. Our findings suggest an as-yet underappreciated preference for “safe” candidates in times of anxiety.
23 December 2020 in Perspectives on Politics
Knowledge creation is a social enterprise, especially in political science. Sharing new findings widely and quickly is essential for progress. Scholars can now use Twitter to rapidly disseminate ideas, and many do. What are the implications of this new tool? Who uses it, how do they use it, and what are the implications for exacerbating or ameliorating existing inequalities in terms of research dissemination and attention? We construct a novel dataset of all 1,236 political science professors at PhD-granting institutions in the United States who have a Twitter account to answer these questions. We find that female scholars and those on the tenure track are more likely to use Twitter, especially for the dissemination of research. However, we consistently find that research by men shared on Twitter is more likely to be passed along further by men than research by women.
15 October 2024 in PS: Political Science & Politics
Twitter has been a prominent forum for academics communicating online, both among themselves and with policy makers and the broader public. Elon Musk’s takeover of the company brought sweeping changes to many aspects of the platform, including public access to its data; Twitter’s approach to censorship and mis/disinformation; and tweaks to the affordances of the platform. This article addresses a narrower empirical question: What did Elon Musk’s takeover of the platform mean for this academic ecosystem? Using a snowball sample of more than 15,700 academic accounts from the fields of economics, political science, sociology, and psychology, we show that academics in these fields reduced their “engagement” with the platform, measured by either the number of active accounts (i.e., those registering any behavior on a given day) or the number of tweets written (including original tweets, replies, retweets, and quote tweets). We further tested whether this decrease in engagement differed by account type; we found that verified users were significantly more likely to reduce their production of content (i.e., writing new tweets and quoting others’ tweets) but not their engagement with the platform writ large (i.e., retweeting and replying to others’ content).