Bayesian aggregation of categorical distributions with applications in crowdsourcing

Augustin, Alexandry, Venanzi, Matteo, Rogers, Alex and Jennings, Nicholas R. (2017) Bayesian aggregation of categorical distributions with applications in crowdsourcing. In International Joint Conference on Artificial Intelligence. pp. 1411-1417 .

Abstract

A key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregated to form a consensus view of the document's or image's contents. Often, however, these judgments are skewed by workers who provide judgments randomly. Such spammers make the cost of acquiring judgments more expensive and degrade the accuracy of the aggregation. For such cases, we provide a new Bayesian framework for aggregating these responses (expressed in the form of categorical distributions) that for the first time accounts for spammers. We elicit 796 judgments about proportions of objects and colours in images. Experimental results show comparable aggregation accuracy when 60% of the workers are spammers, as other state of the art approaches do when there are no spammers.

This record has no associated files available for download.

Contributors

Author: Alexandry Augustin

Download statistics