Cognitive interviewing is a common method used to evaluate survey questions. This study compares traditional cognitive interviewing methods with crowdsourcing, or “tapping into the collective intelligence of the public to complete a task.” Crowdsourcing may provide researchers with access to a diverse pool of potential participants in a very timely and cost-efficient way. Exploratory work found that crowdsourcing participants, with self-administered data collection, may be a viable alternative, or addition, to traditional pretesting methods. Using three crowdsourcing designs (TryMyUI, Amazon Mechanical Turk, and Facebook), we compared the participant characteristics, costs, and quantity and quality of data with traditional laboratory-based cognitive interviews. Results suggest that crowdsourcing and self-administered protocols may be a viable way to collect survey pretesting information, as participants were able to complete the tasks and provide useful information; however, complex tasks may require the skills of an interviewer to administer unscripted probes.
1 January 2009 in American Journal of Public Health
I applied the capture–recapture method to estimate the World Trade Center tower population at the time of the September 11, 2001, terrorist attacks. Available lists helped identify 8965 survivors and 2152 confirmed casualties. The capture–recapture model suggested that an additional 4435 survivors were present, putting the total count of all present at 15 552 (95% confidence interval = 15 216, 15 897). An accurate estimate represents the potential number at risk for trauma as a result of direct exposure to the events of the day.
1 September 2018 in Journal of Official Statistics
This article discusses the critical and complex design decisions associated with transitioning an interviewer-administered survey to a self-administered, postal, web/paper survey. Our approach embeds adaptive, responsive, and tailored (ART) design principles and data visualization during a multi-phased data collection operation to project the outcomes of each phase in preparation for subsequent phases. This requires rapid decision making based upon experimental results using a data visualization system to monitor critical-to-quality (CTQ) metrics and facilitate projections of outcomes from the current phase of data collection to inform the design of the subsequent phase. We describe the objectives of the overall design, the features designed to address these objectives, components of the visual adaptive total design (ATD) system for monitoring quality components and relative costs in real time, and examples of the visualization elements and functionalities that were used in one case study. We also discuss subsequent initiatives to develop an interactive version of the monitoring tool and applications for other studies, including those employing adaptive, responsive, and tailored (ART) designs. Our case study is a series of pilot studies conducted for the Residential Energy Consumption Survey (RECS), sponsored by the U.S. Energy Information Administration (EIA).
This article summarizes the findings of a comprehensive review of current best practices for detecting and preventing interviewer falsification and provides recommendations for mitigating and repairing interviewer errors for surveys of the future. We discuss the potential for interviewer errors and how to avoid them through training, monitoring, verification and remediation strategies. We summarize practices used by a small sample of major survey organizations and offer some insights in terms of best practices and provide recommendations for current and potential future practices for the detection and prevention of interviewer falsification. In particular, we describe what we believe is a state-of-the-art program for interviewer falsification mitigation, including a proposed path forward in the areas of metrics and management.
16 January 2019 in Information, Communication & Society
Social media data are increasingly used by researchers to gain insights on individuals’ behaviors and opinions. Platforms like Twitter provide access to individuals’ postings, networks of friends and followers, and the content to which they are exposed. This article presents the methods and results of an exploratory study to supplement survey data with respondents’ Twitter postings, networks of Twitter friends and followers, and information to which they were exposed about e-cigarettes. Twitter use is important to consider in e-cigarette research and other topics influenced by online information sharing and exposure. Further, Twitter metadata provide direct measures of user’s friends and followers as opposed to survey self-reports. We find that Twitter metadata provide similar information to survey questions on Twitter network size without inducing recall error or other measurement issues. Using sentiment coding and machine learning methods, we find Twitter can elucidate on topics difficult to measure via surveys such as online expressed opinions and network composition. We present and discuss models predicting whether respondents’ tweet positively about e-cigarettes using survey and Twitter data, finding the combined data to provide broader measures than either source alone.