The University of Southampton
University of Southampton Institutional Repository

The role of analytical variability in secondary data replications: a replication of Kim et al. (2014)

The role of analytical variability in secondary data replications: a replication of Kim et al. (2014)
The role of analytical variability in secondary data replications: a replication of Kim et al. (2014)

An article by Kim et al. from 2014 examined individual- and school-level variables affecting the information and communication technology (ICT) literacy level of Korean elementary school students, finding differential gender effects. In this secondary data replication, we used data from the 2018 International Computer and Information Literacy Study, focusing on data from Korea as main replication. As many characteristics of the study as possible, such as variables and analytical strategy, were modelled in the analysis. Additional analyses included 13 countries and jurisdictions, varied centring techniques for variables, and missing data treatment. The replication and analyses were pre-registered via the Open Science Framework. The main analysis did not replicate the main gender finding. However, it was also clear that, despite care taken in a rigorous replication, analytical variability still plays a large role in replications of findings, and with secondary datasets. We discuss the implications of this for secondary data replications.

Replication, analytical variability, international largescale assessment
1380-3611
141-163
Bokhove, Christian
7fc17e5b-9a94-48f3-a387-2ccf60d2d5d8
Bokhove, Christian
7fc17e5b-9a94-48f3-a387-2ccf60d2d5d8

Bokhove, Christian (2022) The role of analytical variability in secondary data replications: a replication of Kim et al. (2014). Educational Research and Evaluation, 27 (1-2), 141-163. (doi:10.1080/13803611.2021.2022319).

Record type: Article

Abstract

An article by Kim et al. from 2014 examined individual- and school-level variables affecting the information and communication technology (ICT) literacy level of Korean elementary school students, finding differential gender effects. In this secondary data replication, we used data from the 2018 International Computer and Information Literacy Study, focusing on data from Korea as main replication. As many characteristics of the study as possible, such as variables and analytical strategy, were modelled in the analysis. Additional analyses included 13 countries and jurisdictions, varied centring techniques for variables, and missing data treatment. The replication and analyses were pre-registered via the Open Science Framework. The main analysis did not replicate the main gender finding. However, it was also clear that, despite care taken in a rigorous replication, analytical variability still plays a large role in replications of findings, and with secondary datasets. We discuss the implications of this for secondary data replications.

Text
kim_article_replication_accepted - Accepted Manuscript
Download (91kB)

More information

Accepted/In Press date: 31 July 2021
Published date: 17 February 2022
Additional Information: Funding Information: The authors approached the “data finder” for this replication, but did not receive a response, so that person cannot be named, though thanked here. The authors thank the SCORE project team at the Center for Open Science and the reviewers of the preregistration. Publisher Copyright: © 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
Keywords: Replication, analytical variability, international largescale assessment

Identifiers

Local EPrints ID: 450731
URI: http://eprints.soton.ac.uk/id/eprint/450731
ISSN: 1380-3611
PURE UUID: 1a41cca2-bf22-4a5d-90be-4fe4da9501ae
ORCID for Christian Bokhove: ORCID iD orcid.org/0000-0002-4860-8723

Catalogue record

Date deposited: 09 Aug 2021 16:31
Last modified: 31 Dec 2022 05:01

Export record

Altmetrics

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×