Sentiment is central to many studies of communication science, from negativity and polarization in political communication to analyzing product reviews and social media comments in other sub-fields. This study provides an exhaustive comparison of sentiment analysis methods, using a validation set of Dutch economic headlines to compare the performance of manual annotation, crowd coding, numerous dictionaries and machine learning using both traditional and deep learning algorithms. The three main conclusions of this article are that: (1) The best performance is still attained with trained human or crowd coding; (2) None of the used dictionaries come close to acceptable levels of validity; and (3) machine learning, especially deep learning, substantially outperforms dictionary-based methods but falls short of human performance. From these findings, we stress the importance of always validating automatic text analysis methods before usage. Moreover, we provide a recommended step-by-step approach for (automated) text analysis projects to ensure both efficiency and validity.
27 December 2021 in Communication Methods and Measures
We identify three gaps that limit the utility and obstruct the progress of computational text analysis methods (CTAM) for social science research. First, we contend that CTAM development has prioritized technological over validity concerns, giving limited attention to the operationalization of social scientific measurements. Second, we identify a mismatch between CTAMs’ focus on extracting specific contents and document-level patterns, and social science researchers’ need for measuring multiple, often complex contents in the text. Third, we argue that the dominance of English language tools depresses comparative research and inclusivity toward scholarly communities examining languages other than English. We substantiate our claims by drawing upon a broad review of methodological work in the computational social sciences, as well as an inventory of leading research publications using quantitative textual analysis. Subsequently, we discuss implications of these three gaps for social scientists’ uneven uptake of CTAM, as well as the field of computational social science text research as a whole. Finally, we propose a research agenda intended to bridge the identified gaps and improve the validity, utility, and inclusiveness of CTAM.
The process of news consumption has undergone great changes over the past decade: Information is now available in an ever-increasing amount from a plethora of sources. Recent work suggests that most people would favor algorithmic solutions over human editors. This stands in contrast to public and scholarly debate about the pitfalls of algorithmic news selection—i.e., the so-called “filter bubbles.” This study therefore investigates reasons and motivations which might lead people to prefer algorithmic gatekeepers over human ones. We expect that people have more algorithmic appreciation when consuming news to pass time, entertain oneself, or out of escapism than when using news to keep up-to-date with politics (H1). Secondly, we hypothesize the extent to which people are confident in their own cognitive abilities to moderate that relationship: When people are overconfident in their own capabilities to estimate the relevance of information, they are more likely to have higher levels of algorithmic appreciation, due to the third person effect (H2). For testing those two pre-registered hypotheses, we conducted an online survey with a sample of 268 US participants and replicated our study using a sample of 384 Dutch participants. The results show that the first hypothesis cannot be supported by our data. However, a positive interaction between overconfidence and algorithmic appreciation for the gratification of surveillance (i.e., gaining information about the world, society, and politics) was found in both samples. Thereby, our study contributes to our understanding of the underlying reasons people have for choosing different forms of gatekeeping when selecting news.
1 November 2023 in European Journal of Political Research
Populists believe in the sovereignty of the people. Yet, the people can be construed in ethnic terms or in civic terms. Using a novel wording experiment in Germany ( N = 7,034), we examine whether ethnic or civic conceptions of ‘the people’ affect respondents' adherence to key populist attitude items. ‘ Volk ’ in German has an ethno‐nationalist connotation and has been used throughout history to signify ‘ethnic Germans’. The concept ‘ Bürger ’, or citizens, by contrast, lacks an ethnic undercurrent. Similar ethnic connotations of ‘the people’ are also common in other languages. We find that there are statistically significant differences between items framed in an ethnic and a civic manner – and that this differs per item. This relationship is significantly moderated by respondents' degree of exclusive national identity and voting behaviour for the radical right. Our findings suggest that the way in which the people is conceptualized has important implications for the measurement of populist attitudes. When populist attitudes are measured with an ethnic understanding of the ‘the people’, the construct is biased towards right‐wing populism, inhibiting the measurement of populism as a ‘thin ideology’. Moreover, we demonstrate the importance of careful translations in comparative research, since some translated synonyms carry different semantic meanings and thereby change the concept under investigation.
Do parties change their platform in anticipation of electoral losses? Or do parties respond to experienced losses at the previous election? These questions relate to two mechanisms to align public opinion with party platforms: (1) rational anticipation, and (2) electoral performance. While extant work empirically tested, and found support for, the latter mechanism, the effect of rational anticipation has not been put to an empirical test yet. We contribute to the literature on party platform change by theorizing and assessing how party performance motivates parties to change their platform in-between elections. We built a new and unique dataset of >20,000 press releases issued by 15 Dutch national political parties that were in parliament between 1997 and 2014. Utilizing automated text analysis (topic modeling) to measure parties’ platform change, we show that electoral defeat motivates party platform change in-between elections. In line with existing findings, we demonstrate that parties are backward-looking.
8 December 2024 in Journalism & Mass Communication Quarterly
Computational communication science is transitioning from an emerging to an established field within communication research, creating a need for proper guidelines and methodological standards. This forum gathers experienced computational communication science scholars to debate the merits and drawbacks of standardization and discuss the tension between innovation, rigor, and inclusion. The assembled perspectives review current standards for data collection, sharing, and documentation, offering best practices for future research. They argue that high standards and inclusive practices can coexist, enhancing creativity and accessibility. By adopting inclusive guidelines, the computational communication science community can welcome diverse scholars, foster innovation, and advance the field collectively.