The University of Southampton
University of Southampton Institutional Repository
Warning ePrints Soton is experiencing an issue with some file downloads not being available. We are working hard to fix this. Please bear with us.

New improved gamma: enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves

New improved gamma: enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves
New improved gamma: enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves
For decades, researchers have debated the relative merits of different measures of people’s ability to discriminate the correctness of their own responses (resolution). The probabilistic approach, primarily led by Nelson, has advocated the Goodman–Kruskal gamma coefficient, an ordinal measure of association. The signal detection approach has advocated parametric measures of distance between the evidence distributions or the area under the receiver operating characteristic (ROC) curve. Here we provide mathematical proof that the indices associated with the two approaches are far more similar than has previously been thought: The true value of gamma is equal to twice the true area under the ROC curve minus one. Using this insight, we report 36 simulations involving 3,600,000 virtual participants that pitted gamma estimated with the original concordance/discordance formula against gamma estimated via ROC curves and the trapezoidal rule. In all but five of our simulations—which system- atically varied resolution, the number of points on the metacognitive scale, and response bias—the ROC-based gamma estimate deviated less from the true value of gamma than did the traditional estimate. Consequently, we recommend using ROC curves to estimate gamma in the future.
1554-351X
108-125
Higham, Philip
4093b28f-7d58-4d18-89d4-021792e418e7
Higham, D. Paul
2c3bfa54-bc11-40ee-a900-90c995b3afac
Higham, Philip
4093b28f-7d58-4d18-89d4-021792e418e7
Higham, D. Paul
2c3bfa54-bc11-40ee-a900-90c995b3afac

Higham, Philip and Higham, D. Paul (2019) New improved gamma: enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves. Behavior Research Methods, 51, 108-125. (doi:10.3758/s13428-018-1125-5).

Record type: Article

Abstract

For decades, researchers have debated the relative merits of different measures of people’s ability to discriminate the correctness of their own responses (resolution). The probabilistic approach, primarily led by Nelson, has advocated the Goodman–Kruskal gamma coefficient, an ordinal measure of association. The signal detection approach has advocated parametric measures of distance between the evidence distributions or the area under the receiver operating characteristic (ROC) curve. Here we provide mathematical proof that the indices associated with the two approaches are far more similar than has previously been thought: The true value of gamma is equal to twice the true area under the ROC curve minus one. Using this insight, we report 36 simulations involving 3,600,000 virtual participants that pitted gamma estimated with the original concordance/discordance formula against gamma estimated via ROC curves and the trapezoidal rule. In all but five of our simulations—which system- atically varied resolution, the number of points on the metacognitive scale, and response bias—the ROC-based gamma estimate deviated less from the true value of gamma than did the traditional estimate. Consequently, we recommend using ROC curves to estimate gamma in the future.

Text
Higham and Higham - 2018 - New improved gamma Enhancing the accuracy of Good - Version of Record
Available under License Creative Commons Attribution.
Download (1MB)

More information

Accepted/In Press date: 1 January 2018
e-pub ahead of print date: 27 September 2018
Published date: 15 February 2019

Identifiers

Local EPrints ID: 424114
URI: http://eprints.soton.ac.uk/id/eprint/424114
ISSN: 1554-351X
PURE UUID: f8fe7062-026b-4ac3-af39-b5f5c205fdd5

Catalogue record

Date deposited: 04 Oct 2018 16:30
Last modified: 25 Nov 2021 19:18

Export record

Altmetrics

Contributors

Author: Philip Higham
Author: D. Paul Higham

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×