1 January 2020 in Drug and Alcohol Dependence
7 July 2017 in Educational Psychology
Multiple-choice questions (MCQs) play a key role in standardised testing and in-class assessment. Research into the influence of within-item response order on MCQ characteristics has been mixed. While some researchers have shown preferential selection of response options presented earlier in the answer list, others have failed to replicate these results. This paper investigates a possible explanation for these mixed findings by assessing the influence of distractor strength on MCQ response order effects. A real-world assessment was administered to 232 undergraduates in which the response order within items was systematically varied. A generalised multilevel model was then used to show a significant interaction between distractor strength and response position with regard to response behaviour. Furthermore, the effect was found to be independent of student ability. These findings have implications for MCQ test construction, minimising the negative consequences of MCQ test exposure and promoting improved test-taking strategies.
1 January 2023 in Technology, Mind, and Behavior
As people increasingly communicate via live video—even more since the COVID-19 pandemic—how willing will they be to participate via video in large-scale standardized surveys that inform public policy, many of which have historically been carried out in-person? This registered report tests three potential predictors of willingness to participate in a live video interview: How (a) easy, (b) useful, and (c) enjoyable respondents find live video to use in other contexts. A potential survey-specific moderator of these effects is also tested: The extent to which respondents report that they would be uncomfortable answering a particular question on a sensitive topic via live video relative to other survey modes. In the study, 598 online U.S. respondents rated their willingness to take part in a hypothetical live video survey that might ask about personal information, in the context of also rating their willingness to take part in four other survey modes, two interviewer-administered (in-person and telephone) and two self-administered (a text-only web survey and a “prerecorded video” web survey in which respondents play videos of interviewers reading questions and then enter answers). Findings demonstrate that willingness to participate in a live video interview is significantly predicted by the extent to which respondents perceive live video as useful and enjoyable in other contexts and by their relative discomfort disclosing in live video versus other modes.
22 February 2018 in Public Health Nutrition
Objective: To understand the effects of interviewers on the responses they collect for measures of food security, income and selected survey quality measures (i.e. discrepancy between reported Supplemental Nutrition Assistance Program (SNAP) status and administrative data, length of time between initial and final interview, and missing income data) in the US Department of Agriculture’s National Household Food Acquisition and Purchase Survey (FoodAPS).Design: Using data from FoodAPS, multilevel models with random interviewer effects were fitted to estimate the variance in each outcome measure arising from effects of the interviewers. Covariates describing each household’s socio-economic status, demographics and experience in taking the survey, and interviewer-level experience were included as fixed effects. The variance components in the outcomes due to interviewers were estimated. Outlier interviewers were profiled.Setting: Non-institutionalized households in the continental USA (April 2012–January 2013).Subjects: Individuals (n 14 317) in 4826 households who responded to FoodAPS.Results: There was a substantial amount of variability in the distributions of the outcomes examined (i.e. time between initial and final interview, reported values for food security, individual income, missing income) among the FoodAPS interviewers, even after accounting for the fixed effects of the household- and interviewer-level covariates and removing extreme outlier interviewers.Conclusions: Interviewers may introduce error in food acquisition survey data when they are asked to interact with the respondents. Managers of future surveys with similarly complex data collection procedures could consider using multilevel models to adaptively identify and retrain interviewers who have extreme effects on data collection outcomes.
1 December 2022 in Journal of Official Statistics
Large-scale, nationally representative surveys serve many vital functions, but these surveys can be long and burdensome for respondents. Cutting survey length can help to reduce respondent burden and may improve data quality but removing items from these surveys is not a trivial matter. We propose a method to empirically assess item importance and associated burden in national surveys and guide this decision-making process using different research products produced from such surveys. This method is demonstrated using the Survey of Doctorate Recipients (SDR), a biennial survey administered to individuals with a science, engineering, and health doctorate. We used three main sources of information on the SDR variables: a bibliography of documents using the SDR data as a measure of item use and importance, SDR data table download statistics from the Scientists and Engineers Statistical Data System as an additional measure of item use, and web timing paradata and break-off rates as a measure of burden. Putting this information together, we identified 35 unused items (17% of the survey) and found that the most burdensome items are highly important. We conclude with general recommendations for those hoping to employ similar methodologies in the future.