Gamified inoculation interventions do not improve discrimination between true and fake news: reanalyzing existing research with receiver operating characteristic analysis
Gamified inoculation interventions do not improve discrimination between true and fake news: reanalyzing existing research with receiver operating characteristic analysis
Gamified inoculation interventions designed to improve detection of online misinformation are becoming increasingly prevalent. Two of the most notable interventions of this kind are Bad News and Go Viral!. To assess their efficacy, prior research has typically used pre-post designs in which participants rated the reliability or manipulativeness of true and fake news items before and after playing these games, while most of the time also including a control group who played an irrelevant game (Tetris) or did nothing at all. Mean ratings were then compared between pre-tests and post-tests and/or between the control and experimental conditions. Critically, these prior studies have not separated response bias effects (overall tendency to respond “true” or “fake”) from discrimination (ability to distinguish between true and fake news, commonly dubbed discernment). We reanalyzed the results from five prior studies using receiver operating characteristic (ROC) curves, a method common to signal detection theory (SDT) that allows for discrimination to be measured free from response bias. Across the studies, when comparable true and fake news items were used, Bad News and Go Viral! did not improve discrimination, but rather elicited more “false” responses to all news items (more conservative responding). These novel findings suggest that the current gamified inoculation interventions designed to improve fake news detection are not as effective as previously thought and may even be counterproductive. They also demonstrate the usefulness of ROC analysis, a largely unexploited method in this setting, for assessing the effectiveness of any intervention designed to improve fake news detection.
online misinformation, gamification, fake news games, receiver operating characteristics, signal detection theory
2411–2437
Modirrousta-Galian, Ariana
5b7bbe48-7221-47e6-bc12-7c8940eb3247
Higham, Philip A.
4093b28f-7d58-4d18-89d4-021792e418e7
2023
Modirrousta-Galian, Ariana
5b7bbe48-7221-47e6-bc12-7c8940eb3247
Higham, Philip A.
4093b28f-7d58-4d18-89d4-021792e418e7
Modirrousta-Galian, Ariana and Higham, Philip A.
(2023)
Gamified inoculation interventions do not improve discrimination between true and fake news: reanalyzing existing research with receiver operating characteristic analysis.
Journal of Experimental Psychology: General, 152 (9), .
(doi:10.31234/osf.io/4bgkd).
Abstract
Gamified inoculation interventions designed to improve detection of online misinformation are becoming increasingly prevalent. Two of the most notable interventions of this kind are Bad News and Go Viral!. To assess their efficacy, prior research has typically used pre-post designs in which participants rated the reliability or manipulativeness of true and fake news items before and after playing these games, while most of the time also including a control group who played an irrelevant game (Tetris) or did nothing at all. Mean ratings were then compared between pre-tests and post-tests and/or between the control and experimental conditions. Critically, these prior studies have not separated response bias effects (overall tendency to respond “true” or “fake”) from discrimination (ability to distinguish between true and fake news, commonly dubbed discernment). We reanalyzed the results from five prior studies using receiver operating characteristic (ROC) curves, a method common to signal detection theory (SDT) that allows for discrimination to be measured free from response bias. Across the studies, when comparable true and fake news items were used, Bad News and Go Viral! did not improve discrimination, but rather elicited more “false” responses to all news items (more conservative responding). These novel findings suggest that the current gamified inoculation interventions designed to improve fake news detection are not as effective as previously thought and may even be counterproductive. They also demonstrate the usefulness of ROC analysis, a largely unexploited method in this setting, for assessing the effectiveness of any intervention designed to improve fake news detection.
Text
Accepted_Manuscript_Prepublication_Copy_
- Accepted Manuscript
Restricted to Repository staff only
Request a copy
Text
2023 JEPG Accepted Manuscript
- Accepted Manuscript
More information
Accepted/In Press date: 25 January 2023
Published date: 2023
Keywords:
online misinformation, gamification, fake news games, receiver operating characteristics, signal detection theory
Identifiers
Local EPrints ID: 475360
URI: http://eprints.soton.ac.uk/id/eprint/475360
ISSN: 0096-3445
PURE UUID: 38a1feb7-b7a2-4e16-bf39-405e0e3b1f58
Catalogue record
Date deposited: 16 Mar 2023 17:35
Last modified: 18 Mar 2024 04:03
Export record
Altmetrics
Contributors
Author:
Ariana Modirrousta-Galian
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics