Mean rating difference scores are poor measures of discernment: the role of response criteria
Mean rating difference scores are poor measures of discernment: the role of response criteria
Many interventions aim to protect people from misinformation. Here, we review common measures used to assess their efficacy. Some measures only assess the target behavior (e.g., ability to spot misinformation) and therefore cannot determine whether interventions have overly general effects (e.g., erroneously identifying accurate information as misinformation). Better measures assess discernment, the ability to discriminate target from non-target content. Discernment can determine whether interventions are overly general but is often measured by comparing differences in mean ratings between target and non-target content. We show how this measure is confounded by the configuration of response criteria, leading researchers to incorrectly conclude that an intervention improves discernment. We recommend using measures from signal detection theory, such as the area under the receiver operating characteristic curve, to assess discernment.
Area under the curve, Mean rating difference scores, Misinformation, Receiver operating characteristic analysis, Signal detection theory
Higham, Philip A.
4093b28f-7d58-4d18-89d4-021792e418e7
Modirrousta-Galian, Ariana
5b7bbe48-7221-47e6-bc12-7c8940eb3247
Seabrooke, Tina
bf0d9ea5-8cf7-494b-9707-891762fce6c3
April 2024
Higham, Philip A.
4093b28f-7d58-4d18-89d4-021792e418e7
Modirrousta-Galian, Ariana
5b7bbe48-7221-47e6-bc12-7c8940eb3247
Seabrooke, Tina
bf0d9ea5-8cf7-494b-9707-891762fce6c3
Higham, Philip A., Modirrousta-Galian, Ariana and Seabrooke, Tina
(2024)
Mean rating difference scores are poor measures of discernment: the role of response criteria.
Current Opinion in Psychology, 56, [101785].
(doi:10.1016/j.copsyc.2023.101785).
Abstract
Many interventions aim to protect people from misinformation. Here, we review common measures used to assess their efficacy. Some measures only assess the target behavior (e.g., ability to spot misinformation) and therefore cannot determine whether interventions have overly general effects (e.g., erroneously identifying accurate information as misinformation). Better measures assess discernment, the ability to discriminate target from non-target content. Discernment can determine whether interventions are overly general but is often measured by comparing differences in mean ratings between target and non-target content. We show how this measure is confounded by the configuration of response criteria, leading researchers to incorrectly conclude that an intervention improves discernment. We recommend using measures from signal detection theory, such as the area under the receiver operating characteristic curve, to assess discernment.
Text
1-s2.0-S2352250X23002300-main
- Version of Record
More information
e-pub ahead of print date: 19 December 2023
Published date: April 2024
Additional Information:
Funding Information:
This work was supported in part by an Economic and Social Research Council South Coast Doctoral Training Partnership studentship (ES/P000673/1) awarded to Ariana Modirrousta-Galian.
Publisher Copyright:
© 2023 The Author(s)
Keywords:
Area under the curve, Mean rating difference scores, Misinformation, Receiver operating characteristic analysis, Signal detection theory
Identifiers
Local EPrints ID: 485959
URI: http://eprints.soton.ac.uk/id/eprint/485959
ISSN: 2352-250X
PURE UUID: 09c121b2-0a8a-4647-8bf1-3bdfee3ea6ba
Catalogue record
Date deposited: 04 Jan 2024 06:16
Last modified: 04 May 2024 02:01
Export record
Altmetrics
Contributors
Author:
Ariana Modirrousta-Galian
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics