18 July 2023 in Proceedings of the National Academy of Sciences
Many NLP applications require manual text annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using four samples of tweets and news articles (n= 6,183), we show that ChatGPT outperforms crowd workers for several annotation tasks, including relevance, stance, topics, and frame detection. Across the four datasets, the zero-shot accuracy of ChatGPT exceeds that of crowd workers by about 25 percentage points on average, while ChatGPT’s intercoder agreement exceeds that of both crowd workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003—about thirty times cheaper than MTurk. These results demonstrate the potential of large language models to drastically increase the efficiency of text classification.
18 December 2024 in Journal of Computational Social Science
MZMohammadmasiha Zahedivafa
Mohammadmasiha Zahedivafa
+2
This paper studies the performance of open-source Large Language Models (LLMs) in text classification tasks typical for political science research. By examining tasks like stance, topic, and relevance classification, we aim to guide scholars in making informed decisions about their use of LLMs for text analysis and to establish a baseline performance benchmark that demonstrates the models’ effectiveness. Specifically, we conduct an assessment of both zero-shot and fine-tuned LLMs across a range of text annotation tasks using news articles and tweets datasets. Our analysis shows that fine-tuning improves the performance of open-source LLMs, allowing them to match or even surpass zero-shot GPT $$-$$ 3.5 and GPT-4, though still lagging behind fine-tuned GPT $$-$$ 3.5. We further establish that fine-tuning is preferable to few-shot training with a relatively modest quantity of annotated text. Our findings show that fine-tuned open-source LLMs can be effectively deployed in a broad spectrum of text annotation applications. We provide a Python notebook facilitating the application of LLMs in text annotation for other researchers.
24 July 2020 in Science Advances
Coordinated political influence operations leave a distinct signature in content that machine learning can detect.
14 May 2019 in EPJ Data Science
The recent rise of the political extremism in Western countries has spurred renewed interest in the psychological and moral appeal of political extremism. Empirical support for the psychological explanation using surveys has been limited by lack of access to extremist groups, while field studies have missed psychological measures and failed to compare extremists with contrast groups. We revisit the debate over the psychological and moral appeal of extremism in the U.S. context by analyzing Twitter data of 10,000 political extremists and comparing their text-based psychological constructs with those of 5000 liberal and 5000 conservative users. The results reveal that extremists show a lower positive emotion and a higher negative emotion than partisan users, but their differences in certainty is not significant. In addition, while left-wing extremists express more language indicative of anxiety than liberals, right-wing extremists express lower anxiety than conservatives. Moreover, our results mostly lend support to Moral Foundations Theory for partisan users and extend it to the political extremists. With the exception of ingroup loyalty, we found evidences supporting the Moral Foundations Theory among left- and right-wing extremists. However, we found no evidence for elevated moral foundations among political extremists.
3 September 2016 in Computational and Mathematical Organization Theory
22 August 2023 in Scientific Reports
Some major social media companies are announcing plans to tokenize user engagements, derived from blockchain-based decentralized social media. This would bring financial and reputational incentives for engagement, which might lead users to post more objectionable content. Previous research showed that financial or reputational incentives for accuracy decrease the willingness to share misinformation. However, it is unclear to what extent such outcome would change if engagements instead of accuracy were incentivized, which is a more realistic scenario. To address this question, we conducted a survey experiment to examine the effects of hypothetical token incentives. We find that a simple nudge about the possibility of earning token-based points for the achieved user engagements increases the willingness to share different kinds of news, including misinformation. The presence of penalties for objectionable posts diminishes the positive effect of tokenization rewards on misinformation sharing, but it does not eliminate it. These results have policy implications for content moderation practices if platforms embrace decentralization and engagement tokenization.