The University of Southampton
University of Southampton Institutional Repository

Bayesian aggregation of categorical distributions with applications in crowdsourcing

Bayesian aggregation of categorical distributions with applications in crowdsourcing
Bayesian aggregation of categorical distributions with applications in crowdsourcing
A key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregated to form a consensus view of the document's or image's contents. Often, however, these judgments are skewed by workers who provide judgments randomly. Such spammers make the cost of acquiring judgments more expensive and degrade the accuracy of the aggregation. For such cases, we provide a new Bayesian framework for aggregating these responses (expressed in the form of categorical distributions) that for the first time accounts for spammers. We elicit 796 judgments about proportions of objects and colours in images. Experimental results show comparable aggregation accuracy when 60% of the workers are spammers, as other state of the art approaches do when there are no spammers.
1411-1417
Augustin, Alexandry
dca1be1e-909c-471a-ba63-19da670b095a
Venanzi, Matteo
6c1596de-424e-48ef-8248-09c64a05a9fa
Rogers, Alex
60b99721-b556-4805-ab34-deb808a8666c
Jennings, Nicholas R.
3f6b53c2-4b6d-4b9d-bb51-774898f6f136
Augustin, Alexandry
dca1be1e-909c-471a-ba63-19da670b095a
Venanzi, Matteo
6c1596de-424e-48ef-8248-09c64a05a9fa
Rogers, Alex
60b99721-b556-4805-ab34-deb808a8666c
Jennings, Nicholas R.
3f6b53c2-4b6d-4b9d-bb51-774898f6f136

Augustin, Alexandry, Venanzi, Matteo, Rogers, Alex and Jennings, Nicholas R. (2017) Bayesian aggregation of categorical distributions with applications in crowdsourcing. In International Joint Conference on Artificial Intelligence. pp. 1411-1417 .

Record type: Conference or Workshop Item (Paper)

Abstract

A key problem in crowdsourcing is the aggregation of judgments of proportions. For example, workers might be presented with a news article or an image, and be asked to identify the proportion of each topic, sentiment, object, or colour present in it. These varying judgments then need to be aggregated to form a consensus view of the document's or image's contents. Often, however, these judgments are skewed by workers who provide judgments randomly. Such spammers make the cost of acquiring judgments more expensive and degrade the accuracy of the aggregation. For such cases, we provide a new Bayesian framework for aggregating these responses (expressed in the form of categorical distributions) that for the first time accounts for spammers. We elicit 796 judgments about proportions of objects and colours in images. Experimental results show comparable aggregation accuracy when 60% of the workers are spammers, as other state of the art approaches do when there are no spammers.

Full text not available from this repository.

More information

Published date: 19 August 2017
Venue - Dates: International Joint Conference on Artificial Intelligence, MCEC (Melbourne Convention and Exhibition Center), Melbourne, Australia, 2017-08-19 - 2017-08-25

Identifiers

Local EPrints ID: 444958
URI: http://eprints.soton.ac.uk/id/eprint/444958
PURE UUID: c3c19600-e462-44b1-9305-e4858deff7f7
ORCID for Alexandry Augustin: ORCID iD orcid.org/0000-0003-0285-9444

Catalogue record

Date deposited: 12 Nov 2020 17:34
Last modified: 18 Feb 2021 17:29

Export record

Contributors

Author: Alexandry Augustin ORCID iD
Author: Matteo Venanzi
Author: Alex Rogers
Author: Nicholas R. Jennings

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×