Considering assessor agreement in IR evaluation

Maddalena, Eddy, Roitero, Kevin, Demartini, Gianluca and Mizzaro, Stefano (2017) Considering assessor agreement in IR evaluation. In ICTIR 2017: Proceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval. Association for Computing Machinery. pp. 75-82 . (doi:10.1145/3121050.3121060).

Abstract

The agreement between relevance assessors is an important but understudied topic in the Information Retrieval literature because of the limited data available about documents assessed by multiple judges. This issue has gained even more importance recently in light of crowdsourced relevance judgments, where it is customary to gather many relevance labels for each topic-document pair. In a crowdsourcing setting, agreement is often even used as a proxy for quality, although without any systematic verification of the conjecture that higher agreement corresponds to higher quality. In this paper we address this issue and we study in particular: the effect of topic on assessor agreement; the relationship between assessor agreement and judgment quality; the effect of agreement on ranking systems according to their effectiveness; and the definition of an agreement-aware effectiveness metric that does not discard information about multiple judgments for the same document as it typically happens in a crowdsourcing setting.

This record has no associated files available for download.

Contributors

Download statistics