The University of Southampton
University of Southampton Institutional Repository

Gamified inoculation interventions do not improve discrimination between true and fake news: reanalyzing existing research with receiver operating characteristic analysis

Gamified inoculation interventions do not improve discrimination between true and fake news: reanalyzing existing research with receiver operating characteristic analysis
Gamified inoculation interventions do not improve discrimination between true and fake news: reanalyzing existing research with receiver operating characteristic analysis
Gamified inoculation interventions designed to improve detection of online misinformation are becoming increasingly prevalent. Two of the most notable interventions of this kind are Bad News and Go Viral!. To assess their efficacy, prior research has typically used pre-post designs in which participants rated the reliability or manipulativeness of true and fake news items before and after playing these games, while most of the time also including a control group who played an irrelevant game (Tetris) or did nothing at all. Mean ratings were then compared between pre-tests and post-tests and/or between the control and experimental conditions. Critically, these prior studies have not separated response bias effects (overall tendency to respond “true” or “fake”) from discrimination (ability to distinguish between true and fake news, commonly dubbed discernment). We reanalyzed the results from five prior studies using receiver operating characteristic (ROC) curves, a method common to signal detection theory (SDT) that allows for discrimination to be measured free from response bias. Across the studies, when comparable true and fake news items were used, Bad News and Go Viral! did not improve discrimination, but rather elicited more “false” responses to all news items (more conservative responding). These novel findings suggest that the current gamified inoculation interventions designed to improve fake news detection are not as effective as previously thought and may even be counterproductive. They also demonstrate the usefulness of ROC analysis, a largely unexploited method in this setting, for assessing the effectiveness of any intervention designed to improve fake news detection.
online misinformation, gamification, fake news games, receiver operating characteristics, signal detection theory
0096-3445
2411–2437
Modirrousta-Galian, Ariana
5b7bbe48-7221-47e6-bc12-7c8940eb3247
Higham, Philip
4093b28f-7d58-4d18-89d4-021792e418e7
Modirrousta-Galian, Ariana
5b7bbe48-7221-47e6-bc12-7c8940eb3247
Higham, Philip
4093b28f-7d58-4d18-89d4-021792e418e7

Modirrousta-Galian, Ariana and Higham, Philip (2023) Gamified inoculation interventions do not improve discrimination between true and fake news: reanalyzing existing research with receiver operating characteristic analysis. Journal of Experimental Psychology: General, 152 (9), 2411–2437. (doi:10.31234/osf.io/4bgkd).

Record type: Article

Abstract

Gamified inoculation interventions designed to improve detection of online misinformation are becoming increasingly prevalent. Two of the most notable interventions of this kind are Bad News and Go Viral!. To assess their efficacy, prior research has typically used pre-post designs in which participants rated the reliability or manipulativeness of true and fake news items before and after playing these games, while most of the time also including a control group who played an irrelevant game (Tetris) or did nothing at all. Mean ratings were then compared between pre-tests and post-tests and/or between the control and experimental conditions. Critically, these prior studies have not separated response bias effects (overall tendency to respond “true” or “fake”) from discrimination (ability to distinguish between true and fake news, commonly dubbed discernment). We reanalyzed the results from five prior studies using receiver operating characteristic (ROC) curves, a method common to signal detection theory (SDT) that allows for discrimination to be measured free from response bias. Across the studies, when comparable true and fake news items were used, Bad News and Go Viral! did not improve discrimination, but rather elicited more “false” responses to all news items (more conservative responding). These novel findings suggest that the current gamified inoculation interventions designed to improve fake news detection are not as effective as previously thought and may even be counterproductive. They also demonstrate the usefulness of ROC analysis, a largely unexploited method in this setting, for assessing the effectiveness of any intervention designed to improve fake news detection.

Text
Accepted_Manuscript_Prepublication_Copy_ - Accepted Manuscript
Download (1MB)

More information

Accepted/In Press date: 25 January 2023
Published date: 2023
Additional Information: Funded by the Economic and Social Research Council South Coast Doctoral Training Partnership.
Related URLs:
Keywords: online misinformation, gamification, fake news games, receiver operating characteristics, signal detection theory

Identifiers

Local EPrints ID: 475360
URI: http://eprints.soton.ac.uk/id/eprint/475360
ISSN: 0096-3445
PURE UUID: 38a1feb7-b7a2-4e16-bf39-405e0e3b1f58
ORCID for Ariana Modirrousta-Galian: ORCID iD orcid.org/0000-0003-2925-2976
ORCID for Philip Higham: ORCID iD orcid.org/0000-0001-6087-7224

Catalogue record

Date deposited: 16 Mar 2023 17:35
Last modified: 31 Oct 2023 03:10

Export record

Altmetrics

Contributors

Author: Ariana Modirrousta-Galian ORCID iD
Author: Philip Higham ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×