The University of Southampton
University of Southampton Institutional Repository

Getting by with a little help from the crowd: optimal human computation approaches to social image labeling

Getting by with a little help from the crowd: optimal human computation approaches to social image labeling
Getting by with a little help from the crowd: optimal human computation approaches to social image labeling
Validating user tags helps to refine them, making them more useful for finding images. In the case of interpretation-sensitive tags, however, automatic (i.e., pixel-based) approaches cannot be expected to deliver optimal results. Instead, human input is key. This paper studies how crowdsourcing-based approaches to image tag validation can achieve parsimony in their use of human input from the crowd, in the form of votes collected from workers on a crowdsourcing platform. Experiments in the domain of social fashion images are carried out using the dataset published by the Crowdsourcing Task of the Mediaeval 2013 Multimedia Benchmark. Experimental results reveal that when a larger number of crowd-contributed votes are available, it is difficult to beat a majority vote. However, additional information sources, i.e., crowdworker history and visual image features, allow us to maintain similar validation performance while making use of less crowd-contributed input. Further, investing in “expensive" experts who collaborate to create definitions of interpretation-sensitive concepts does not nec- essarily pay off. Instead, experts can cause interpretations of concepts to drift away from conventional wisdom. In short, validation of interpretation-sensitive user tags for social images is possible, with “just a little help from the crowd."
Loni, Babak
0e9aa358-10c1-47ee-a351-30b82b8c79e1
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Georgescu, Mihai
a335c153-fbee-47d8-aeca-4da36e581524
Riegler, Michael
511433bb-75d0-4591-af3c-4a7ec9d2a835
Morchid, Mohamed
6920e2f8-9bbd-4ee2-81d9-8d28611b2dee
Dufour, Richard
47e4c30f-c50e-408f-bd88-26a6905aa576
Larson, Martha
9ecae1f2-d481-4965-86b1-fe2356bad080
Loni, Babak
0e9aa358-10c1-47ee-a351-30b82b8c79e1
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Georgescu, Mihai
a335c153-fbee-47d8-aeca-4da36e581524
Riegler, Michael
511433bb-75d0-4591-af3c-4a7ec9d2a835
Morchid, Mohamed
6920e2f8-9bbd-4ee2-81d9-8d28611b2dee
Dufour, Richard
47e4c30f-c50e-408f-bd88-26a6905aa576
Larson, Martha
9ecae1f2-d481-4965-86b1-fe2356bad080

Loni, Babak, Hare, Jonathon, Georgescu, Mihai, Riegler, Michael, Morchid, Mohamed, Dufour, Richard and Larson, Martha (2014) Getting by with a little help from the crowd: optimal human computation approaches to social image labeling At CrowdMM 2014, United States. 03 - 07 Nov 2014. 6 pp.

Record type: Conference or Workshop Item (Paper)

Abstract

Validating user tags helps to refine them, making them more useful for finding images. In the case of interpretation-sensitive tags, however, automatic (i.e., pixel-based) approaches cannot be expected to deliver optimal results. Instead, human input is key. This paper studies how crowdsourcing-based approaches to image tag validation can achieve parsimony in their use of human input from the crowd, in the form of votes collected from workers on a crowdsourcing platform. Experiments in the domain of social fashion images are carried out using the dataset published by the Crowdsourcing Task of the Mediaeval 2013 Multimedia Benchmark. Experimental results reveal that when a larger number of crowd-contributed votes are available, it is difficult to beat a majority vote. However, additional information sources, i.e., crowdworker history and visual image features, allow us to maintain similar validation performance while making use of less crowd-contributed input. Further, investing in “expensive" experts who collaborate to create definitions of interpretation-sensitive concepts does not nec- essarily pay off. Instead, experts can cause interpretations of concepts to drift away from conventional wisdom. In short, validation of interpretation-sensitive user tags for social images is possible, with “just a little help from the crowd."

PDF GettingByWithLittleHelp_CrowdMM.pdf - Accepted Manuscript
Download (1MB)

More information

Published date: November 2014
Venue - Dates: CrowdMM 2014, United States, 2014-11-03 - 2014-11-07
Organisations: Web & Internet Science

Identifiers

Local EPrints ID: 370278
URI: http://eprints.soton.ac.uk/id/eprint/370278
PURE UUID: 3563ac04-126a-4cb0-9e74-e1861a69f9f3
ORCID for Jonathon Hare: ORCID iD orcid.org/0000-0003-2921-4283

Catalogue record

Date deposited: 20 Oct 2014 17:50
Last modified: 03 Nov 2017 23:54

Export record

Contributors

Author: Babak Loni
Author: Jonathon Hare ORCID iD
Author: Mihai Georgescu
Author: Michael Riegler
Author: Mohamed Morchid
Author: Richard Dufour
Author: Martha Larson

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×