Coder training: Key to enhancing reliability and validity

Lacewell, Onawa, Werner, Annika, Volkens, Andrea, Bara, Judith, Budge, Ian, McDonald, Michael D., Best, Robin and Franzmann, Simon (2013) Coder training: Key to enhancing reliability and validity. In, Mapping policy preferences from texts: statistical solutions for manifesto analysts. Oxford University Press, pp. 169-194. (doi:10.1093/acprof:oso/9780199640041.003.0010).

Abstract

Aside from document collection the major procedure involved in producing the policy estimates is coding of the ‘quasi-sentences’ in the documents into 57 policy categories. Here the concern is that different coders operating across different time periods and countries make the same coding decisions. Difficulties in comparing the coding units identified by different coders has led some outside evaluations to focus on inter-coder agreement on assignment of pre-set coding units, as used in CMP and MARPOR training tests. Properly simulated these show that coders exhibit moderately high reliability at this stage in their training. The simulation also yields the valuable insight that bad practices creep in after coding around 10 documents, so refresher training is required at that point. An independent check-coding experiment in which different coders of manifestos in four countries shows an overall correlation of 0.82 on RILE, confirming the reliability level identified from the final estimates in Chapter 6.

This record has no associated files available for download.

Contributors

Author: Annika Werner

Download statistics