The University of Southampton
University of Southampton Institutional Repository

How biased is your NLG evaluation?

How biased is your NLG evaluation?
How biased is your NLG evaluation?
Human assessments by either experts or crowdworkers are used extensively for the evaluation of systems employed on a variety of text generative tasks. In this paper, we focus on the human evaluation of textual summaries from knowledge base triple-facts. More specifically, we investigate possible similarities between the evaluation that is performed by experts and crowdworkers. We generate a set of summaries from DBpedia triples using a state-of-the-art neural network architecture. These summaries are evaluated against a set of criteria by both experts and crowdworkers. Our results highlight significant differences between the scores that are provided by the two groups.
1-3
Vougiouklis, Pavlos
4cd0a8f1-c5e2-4ba2-8dcd-753db616b215
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Simperl, Elena
40261ae4-c58c-48e4-b78b-5187b10e4f67
Vougiouklis, Pavlos
4cd0a8f1-c5e2-4ba2-8dcd-753db616b215
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Simperl, Elena
40261ae4-c58c-48e4-b78b-5187b10e4f67

Vougiouklis, Pavlos, Maddalena, Eddy, Hare, Jonathon and Simperl, Elena (2018) How biased is your NLG evaluation? In Proceedings of the 1st CrowdBias Workshop. pp. 1-3 .

Record type: Conference or Workshop Item (Paper)

Abstract

Human assessments by either experts or crowdworkers are used extensively for the evaluation of systems employed on a variety of text generative tasks. In this paper, we focus on the human evaluation of textual summaries from knowledge base triple-facts. More specifically, we investigate possible similarities between the evaluation that is performed by experts and crowdworkers. We generate a set of summaries from DBpedia triples using a state-of-the-art neural network architecture. These summaries are evaluated against a set of criteria by both experts and crowdworkers. Our results highlight significant differences between the scores that are provided by the two groups.

This record has no associated files available for download.

More information

Submitted date: 7 June 2018
Accepted/In Press date: 25 June 2018
Published date: 5 July 2018

Identifiers

Local EPrints ID: 422704
URI: http://eprints.soton.ac.uk/id/eprint/422704
PURE UUID: 054c6da0-5aca-47a2-a16c-15f204941ce8
ORCID for Jonathon Hare: ORCID iD orcid.org/0000-0003-2921-4283
ORCID for Elena Simperl: ORCID iD orcid.org/0000-0003-1722-947X

Catalogue record

Date deposited: 31 Jul 2018 16:30
Last modified: 16 Mar 2024 03:50

Export record

Contributors

Author: Pavlos Vougiouklis
Author: Eddy Maddalena
Author: Jonathon Hare ORCID iD
Author: Elena Simperl ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×