New improved gamma: enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves
New improved gamma: enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves
For decades, researchers have debated the relative merits of different measures of people’s ability to discriminate the correctness of their own responses (resolution). The probabilistic approach, primarily led by Nelson, has advocated the Goodman–Kruskal gamma coefficient, an ordinal measure of association. The signal detection approach has advocated parametric measures of distance between the evidence distributions or the area under the receiver operating characteristic (ROC) curve. Here we provide mathematical proof that the indices associated with the two approaches are far more similar than has previously been thought: The true value of gamma is equal to twice the true area under the ROC curve minus one. Using this insight, we report 36 simulations involving 3,600,000 virtual participants that pitted gamma estimated with the original concordance/discordance formula against gamma estimated via ROC curves and the trapezoidal rule. In all but five of our simulations—which system- atically varied resolution, the number of points on the metacognitive scale, and response bias—the ROC-based gamma estimate deviated less from the true value of gamma than did the traditional estimate. Consequently, we recommend using ROC curves to estimate gamma in the future.
108-125
Higham, Philip
4093b28f-7d58-4d18-89d4-021792e418e7
Higham, D. Paul
2c3bfa54-bc11-40ee-a900-90c995b3afac
15 February 2019
Higham, Philip
4093b28f-7d58-4d18-89d4-021792e418e7
Higham, D. Paul
2c3bfa54-bc11-40ee-a900-90c995b3afac
Higham, Philip and Higham, D. Paul
(2019)
New improved gamma: enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves.
Behavior Research Methods, 51, .
(doi:10.3758/s13428-018-1125-5).
Abstract
For decades, researchers have debated the relative merits of different measures of people’s ability to discriminate the correctness of their own responses (resolution). The probabilistic approach, primarily led by Nelson, has advocated the Goodman–Kruskal gamma coefficient, an ordinal measure of association. The signal detection approach has advocated parametric measures of distance between the evidence distributions or the area under the receiver operating characteristic (ROC) curve. Here we provide mathematical proof that the indices associated with the two approaches are far more similar than has previously been thought: The true value of gamma is equal to twice the true area under the ROC curve minus one. Using this insight, we report 36 simulations involving 3,600,000 virtual participants that pitted gamma estimated with the original concordance/discordance formula against gamma estimated via ROC curves and the trapezoidal rule. In all but five of our simulations—which system- atically varied resolution, the number of points on the metacognitive scale, and response bias—the ROC-based gamma estimate deviated less from the true value of gamma than did the traditional estimate. Consequently, we recommend using ROC curves to estimate gamma in the future.
Text
Higham and Higham - 2018 - New improved gamma Enhancing the accuracy of Good
- Version of Record
More information
Accepted/In Press date: 1 January 2018
e-pub ahead of print date: 27 September 2018
Published date: 15 February 2019
Identifiers
Local EPrints ID: 424114
URI: http://eprints.soton.ac.uk/id/eprint/424114
ISSN: 1554-351X
PURE UUID: f8fe7062-026b-4ac3-af39-b5f5c205fdd5
Catalogue record
Date deposited: 04 Oct 2018 16:30
Last modified: 16 Mar 2024 03:18
Export record
Altmetrics
Contributors
Author:
D. Paul Higham
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
