The University of Southampton
University of Southampton Institutional Repository

Aggregating crowdsourced quantitative claims: Additive and multiplicative models

Aggregating crowdsourced quantitative claims: Additive and multiplicative models
Aggregating crowdsourced quantitative claims: Additive and multiplicative models

Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-The-Art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.

Crowdsourcing, probabilistic graphical model, quantitative task, truth discovery
1041-4347
1621-1634
Ouyang, Robin Wentao
a0886331-0eed-46ee-9b72-843ef2bb192f
Kaplan, Lance M.
3812423d-c58a-4896-bd57-7b373505e457
Toniolo, Alice
e54ad578-9232-471a-a5d7-cd3a7bc70872
Srivastava, Mani
77c0ad90-6073-4dd8-908f-05087d856cd6
Norman, Timothy J.
663e522f-807c-4569-9201-dc141c8eb50d
Ouyang, Robin Wentao
a0886331-0eed-46ee-9b72-843ef2bb192f
Kaplan, Lance M.
3812423d-c58a-4896-bd57-7b373505e457
Toniolo, Alice
e54ad578-9232-471a-a5d7-cd3a7bc70872
Srivastava, Mani
77c0ad90-6073-4dd8-908f-05087d856cd6
Norman, Timothy J.
663e522f-807c-4569-9201-dc141c8eb50d

Ouyang, Robin Wentao, Kaplan, Lance M., Toniolo, Alice, Srivastava, Mani and Norman, Timothy J. (2016) Aggregating crowdsourced quantitative claims: Additive and multiplicative models. IEEE Transactions on Knowledge and Data Engineering, 28 (7), 1621-1634, [7422085]. (doi:10.1109/TKDE.2016.2535383).

Record type: Article

Abstract

Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-The-Art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.

This record has no associated files available for download.

More information

Accepted/In Press date: 10 February 2016
e-pub ahead of print date: 29 February 2016
Published date: 1 July 2016
Keywords: Crowdsourcing, probabilistic graphical model, quantitative task, truth discovery

Identifiers

Local EPrints ID: 413293
URI: http://eprints.soton.ac.uk/id/eprint/413293
ISSN: 1041-4347
PURE UUID: dd04ee8f-d83c-471d-8a2a-8e301bfa8b37
ORCID for Timothy J. Norman: ORCID iD orcid.org/0000-0002-6387-4034

Catalogue record

Date deposited: 18 Aug 2017 16:32
Last modified: 16 Mar 2024 04:24

Export record

Altmetrics

Contributors

Author: Robin Wentao Ouyang
Author: Lance M. Kaplan
Author: Alice Toniolo
Author: Mani Srivastava

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×