Qualitative coding, the process of assigning labels to text as part of qualitative analysis, is time-consuming and repetitive, especially for large datasets. While available QDAS sometimes allows the semi-automated extension of annotations to unseen data, recent user studies revealed critical issues. In particular, the integration of automated code suggestions into the coding process is not transparent and interactive. In this work, we present Cody, a system for semi-automated qualitative coding that suggests codes based on human-defined coding rules and supervised machine learning (ML). Suggestions and rules can be revised iteratively by users in a lean interface that provides explanations for code suggestions. In a preliminary evaluation, 42% of all documents could be coded automatically based on code rules. Cody is the first coding system to allow users to define query-style code rules in combination with supervised ML. Thereby, users can extend manual annotations to unseen data to improve coding speed and quality.
While qualitative research can produce a rich understanding of peoples’ mind, it requires an essential and strenuous data annotation process known as coding. Coding can be repetitive and time-consuming, particularly for large datasets. Crowdsourcing provides flexible access to workers all around the world, however, researchers remain doubtful about its applicability for coding. In this study, we present an interactive coding system to support crowdsourced deductive coding of semi-structured qualitative data. Through an empirical evaluation on Amazon Mechanical Turk, we assess both the quality and the reliability of crowd-support for coding. Our results show that non-expert coders provide reliable results using our system. The crowd reached a substantial agreement of up to 91% with the coding provided by experts. Our results indicate that crowdsourced coding is an applicable strategy for accelerating a strenuous task. Additionally, we present implications of crowdsourcing to reduce biases in the interpretation of qualitative data.