How biased is your NLG evaluation?
How biased is your NLG evaluation?
Human assessments by either experts or crowdworkers are used extensively for the evaluation of systems employed on a variety of text generative tasks. In this paper, we focus on the human evaluation of textual summaries from knowledge base triple-facts. More specifically, we investigate possible similarities between the evaluation that is performed by experts and crowdworkers. We generate a set of summaries from DBpedia triples using a state-of-the-art neural network architecture. These summaries are evaluated against a set of criteria by both experts and crowdworkers. Our results highlight significant differences between the scores that are provided by the two groups.
1-3
Vougiouklis, Pavlos
4cd0a8f1-c5e2-4ba2-8dcd-753db616b215
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Simperl, Elena
40261ae4-c58c-48e4-b78b-5187b10e4f67
5 July 2018
Vougiouklis, Pavlos
4cd0a8f1-c5e2-4ba2-8dcd-753db616b215
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Hare, Jonathon
65ba2cda-eaaf-4767-a325-cd845504e5a9
Simperl, Elena
40261ae4-c58c-48e4-b78b-5187b10e4f67
Vougiouklis, Pavlos, Maddalena, Eddy, Hare, Jonathon and Simperl, Elena
(2018)
How biased is your NLG evaluation?
In Proceedings of the 1st CrowdBias Workshop.
.
Record type:
Conference or Workshop Item
(Paper)
Abstract
Human assessments by either experts or crowdworkers are used extensively for the evaluation of systems employed on a variety of text generative tasks. In this paper, we focus on the human evaluation of textual summaries from knowledge base triple-facts. More specifically, we investigate possible similarities between the evaluation that is performed by experts and crowdworkers. We generate a set of summaries from DBpedia triples using a state-of-the-art neural network architecture. These summaries are evaluated against a set of criteria by both experts and crowdworkers. Our results highlight significant differences between the scores that are provided by the two groups.
This record has no associated files available for download.
More information
Submitted date: 7 June 2018
Accepted/In Press date: 25 June 2018
Published date: 5 July 2018
Identifiers
Local EPrints ID: 422704
URI: http://eprints.soton.ac.uk/id/eprint/422704
PURE UUID: 054c6da0-5aca-47a2-a16c-15f204941ce8
Catalogue record
Date deposited: 31 Jul 2018 16:30
Last modified: 16 Mar 2024 03:50
Export record
Contributors
Author:
Pavlos Vougiouklis
Author:
Jonathon Hare
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics