Recent discussions of annotator agreement have mostly centered around its calculation and interpretation, and the correct choice of indices. Although these discussions are important, they only consider the “back-end” of the story, namely, what to do once the data are collected. Just as important in our opinion is to know how agreement is reached in the first place and what factors influence coder agreement as part of the annotation process or setting, as this knowledge can provide concrete guidelines for the planning and set-up of annotation projects. To investigate whether there are factors that consistently impact annotator agreement we conducted a meta-analytic investigation of annotation studies reporting agreement percentages. Our meta-analysis synthesized factors reported in 96 annotation studies from three domains (word-sense disambiguation, prosodic transcriptions, and phonetic transcriptions) and was based on a total of 346 agreement indices. Our analysis identified seven factors that influence reported agreement values: annotation domain, number of categories in a coding scheme, number of annotators in a project, whether annotators received training, the intensity of annotator training, the annotation purpose, and the method used for the calculation of percentage agreements. Based on our results we develop practical recommendations for the assessment, interpretation, calculation, and reporting of coder agreement. We also briefly discuss theoretical implications for the concept of annotation quality.
It is known that employment is important for psychological health. The present paper claims that the reasons why employed persons report better well-being than unemployed persons might also explain why those in some occupations report better well-being than others. Jahoda's latent deprivation theory (1982) was that employment provides a number of latent beneficial functions, which she identified as time structure, social contact, collective purpose, identity/status, and activity. We argue that this theory can be extended to account for differences in well-being between occupations with different levels of status. Data from two studies, one conducted on a representative German sample (n=565) and the other from a large-scale online study (n=826), largely support this argument. Group differences in well-being and access to latent benefits were found, and mediation was partly supported. The findings are discussed not only in relation to the latent deprivation model but also in relation to common approaches to job satisfaction and mental health.
8 April 2011 in Journal of Occupational and Organizational Psychology