The University of Southampton
University of Southampton Institutional Repository

Annotator subjectivity in harmony annotations of popular music

Annotator subjectivity in harmony annotations of popular music
Annotator subjectivity in harmony annotations of popular music

Reference annotation datasets containing harmony annotations are at the core of a wide range of studies in music information retrieval (MIR) and related fields. The majority of these datasets contain single reference annotations describing the harmony of each piece. Nevertheless, studies showing differences among annotators in many other MIR tasks make the notion of a single ‘ground-truth’ reference annotation a tenuous one. In this paper, we introduce and analyse the Chordify Annotator Subjectivity Dataset (CASD) containing chord labels for 50 songs from 4 expert annotators in order to gain a better understanding of the differences between annotators in their chord label choice. Our analysis reveals that annotators use distinct chord-label vocabularies, with low chord-label overlap across all annotators. Between annotators, we find only 73 percent overlap on average for the traditional major–minor vocabulary and 54 percent overlap for the most complex chord labels. A factor analysis reveals the relative importance of triads, sevenths, inversions and other musical factors for each annotator on their choice of chord labels and reported difficulty of the songs. Our results further substantiate the existence of a harmonic ‘subjectivity ceiling’: an upper bound for evaluations in computational harmony research. Current state-of-the-art chord-estimation systems perform beyond this subjectivity ceiling by about 10 percent. This suggests that current ACE algorithms are powerful enough to tune themselves to particular annotators' idiosyncrasies. Overall, our results show that annotator subjectivity is an important factor in harmonic transcriptions, which should inform future studies into harmony perception and computational models of harmony.

Annotator subjectivity, harmony, inter-rater agreement
0929-8215
Koops, Hendrik Vincent
ef9577c0-cd7e-4ad2-b6f9-04fe59ce536f
de Haas, W. Bas
d6ace77e-9e3f-487b-9521-305ad5d8bd7c
Burgoyne, John Ashley
8f4363a2-15e4-40dc-b491-f84ff9f05f27
Bransen, Jeroen
c3dd04bf-ea8d-477b-840a-003f0d7a2f49
Kent-Muller, Anna
6385c31d-c386-49ff-8654-189ca689dd9e
Volk, Anja
4eaa5995-80f6-418b-bbff-3ffadd354722
Koops, Hendrik Vincent
ef9577c0-cd7e-4ad2-b6f9-04fe59ce536f
de Haas, W. Bas
d6ace77e-9e3f-487b-9521-305ad5d8bd7c
Burgoyne, John Ashley
8f4363a2-15e4-40dc-b491-f84ff9f05f27
Bransen, Jeroen
c3dd04bf-ea8d-477b-840a-003f0d7a2f49
Kent-Muller, Anna
6385c31d-c386-49ff-8654-189ca689dd9e
Volk, Anja
4eaa5995-80f6-418b-bbff-3ffadd354722

Koops, Hendrik Vincent, de Haas, W. Bas, Burgoyne, John Ashley, Bransen, Jeroen, Kent-Muller, Anna and Volk, Anja (2019) Annotator subjectivity in harmony annotations of popular music. Journal of New Music Research, 48 (3). (doi:10.1080/09298215.2019.1613436).

Record type: Article

Abstract

Reference annotation datasets containing harmony annotations are at the core of a wide range of studies in music information retrieval (MIR) and related fields. The majority of these datasets contain single reference annotations describing the harmony of each piece. Nevertheless, studies showing differences among annotators in many other MIR tasks make the notion of a single ‘ground-truth’ reference annotation a tenuous one. In this paper, we introduce and analyse the Chordify Annotator Subjectivity Dataset (CASD) containing chord labels for 50 songs from 4 expert annotators in order to gain a better understanding of the differences between annotators in their chord label choice. Our analysis reveals that annotators use distinct chord-label vocabularies, with low chord-label overlap across all annotators. Between annotators, we find only 73 percent overlap on average for the traditional major–minor vocabulary and 54 percent overlap for the most complex chord labels. A factor analysis reveals the relative importance of triads, sevenths, inversions and other musical factors for each annotator on their choice of chord labels and reported difficulty of the songs. Our results further substantiate the existence of a harmonic ‘subjectivity ceiling’: an upper bound for evaluations in computational harmony research. Current state-of-the-art chord-estimation systems perform beyond this subjectivity ceiling by about 10 percent. This suggests that current ACE algorithms are powerful enough to tune themselves to particular annotators' idiosyncrasies. Overall, our results show that annotator subjectivity is an important factor in harmonic transcriptions, which should inform future studies into harmony perception and computational models of harmony.

Text
18_06_2019_Annotator - Version of Record
Download (3MB)

More information

Accepted/In Press date: 4 April 2019
e-pub ahead of print date: 20 May 2019
Published date: 2019
Keywords: Annotator subjectivity, harmony, inter-rater agreement

Identifiers

Local EPrints ID: 431806
URI: http://eprints.soton.ac.uk/id/eprint/431806
ISSN: 0929-8215
PURE UUID: 99e0387d-b85d-433c-8f80-0aba5497e75d

Catalogue record

Date deposited: 18 Jun 2019 16:30
Last modified: 16 Mar 2024 02:21

Export record

Altmetrics

Contributors

Author: Hendrik Vincent Koops
Author: W. Bas de Haas
Author: John Ashley Burgoyne
Author: Jeroen Bransen
Author: Anna Kent-Muller
Author: Anja Volk

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×