Questionnaire design is routinely guided by classic experiments on question form, wording, and context conducted decades ago. This article explores whether two question order effects (one due to the norm of evenhandedness and the other due to subtraction or perceptual contrast) appear in surveys of probability samples in the United States and 11 other countries (Canada, Denmark, Germany, Iceland, Japan, the Netherlands, Norway, Portugal, Sweden, Taiwan, and the United Kingdom; N = 25,640). Advancing theory of question order effects, we propose necessary conditions for each effect to occur, and found that the effects occurred in the nations where these necessary conditions were met. Surprisingly, the abortion question order effect even appeared in some countries in which the necessary condition was not met, suggesting that the question order effect there (and perhaps elsewhere) was not due to subtraction or perceptual contrast. The question order effects were not moderated by education. The strength of the effect due to the norm of evenhandedness was correlated with various cultural characteristics of the nations. Strong support was observed for the form-resistant correlation hypothesis.
This chapter explains the design of the multinational study of questionnaire design (MSQD) and the challenges faced when implementing the project across countries. It then elaborates on the sampling and online implementation of the questionnaire, as well as the questionnaire design experiments selected for the study. The aim of the MSQD was to conduct well-cited question design experiments originally conducted in the United States and assess whether similar results would be observed decades later in the United States and in other countries. The chapter discusses challenges faced in the translation of experiments in which question wording plays a central role. The finding described in the chapter relates to the fact that the rise of online panels employing probability samples in recent years opened new avenues for researchers interested in small-scale substantive or methodological research.
1 March 2022 in The Annals of the American Academy of Political and Social Science
How do people form their attitudes toward complex policy issues? Although there has long been an assumption that people consider the various components of those issues and come to an overall assessment, a growing body of recent work has instead suggested that people may reach summary judgments as a function of heuristic cues and goal-oriented rationalizations. This study examines how well a component-based model fits Americans’ evaluations of the Patient Protection and Affordable Care Act of 2010, an important and highly contentious piece of legislation that contained several constituent parts. Despite strong partisan disagreement about the law, we find that Democrats and Republicans both appear to evaluate the law as a function of their beliefs and what the law would do as well as their confidence in those beliefs. This finding implies that correcting misperceptions and increasing awareness of the components of legislation have the potential to change attitudes.
Why the social character of scientific knowledge makesit trustworthy Are doctors right when they tell usvaccines are safe? Should we take climate experts at the...
This study proposes a novel, certainty‐weighted account of the process by which political beliefs shape political attitudes. Building upon expectancy‐value frameworks, this paper introduces belief certainty as a moderator of belief impact. A Bayesian partial‐pooling approach is used to test a model positing how beliefs about what the Patient Protection and Affordable Care Act does and does not do relate to overall attitudes toward the law. This analytic method manages the complexity of multiple potentially correlated beliefs and evaluations by imposing a structural constraint to avoid multicollinearity and enable accurate estimation of parameters. Data from a nationally representative sample survey of American adults support the model's core propositions. Counterfactual simulations further reveal that belief certainty substantially amplifies the weight of both accurate and inaccurate beliefs, thereby intensifying attitudes and amplifying polarization. These findings highlight the role of belief certainty in shaping political judgments and offer a methodological pathway for researchers to model the interplay of multiple correlated political beliefs in an era of abundant—sometimes erroneous—information.