The role of analytical variability in secondary data replications: a replication of Kim et al. (2014)
The role of analytical variability in secondary data replications: a replication of Kim et al. (2014)
An article by Kim et al. from 2014 examined individual- and school-level variables affecting the information and communication technology (ICT) literacy level of Korean elementary school students, finding differential gender effects. In this secondary data replication, we used data from the 2018 International Computer and Information Literacy Study, focusing on data from Korea as main replication. As many characteristics of the study as possible, such as variables and analytical strategy, were modelled in the analysis. Additional analyses included 13 countries and jurisdictions, varied centring techniques for variables, and missing data treatment. The replication and analyses were pre-registered via the Open Science Framework. The main analysis did not replicate the main gender finding. However, it was also clear that, despite care taken in a rigorous replication, analytical variability still plays a large role in replications of findings, and with secondary datasets. We discuss the implications of this for secondary data replications.
Replication, analytical variability, international largescale assessment
141-163
Bokhove, Christian
7fc17e5b-9a94-48f3-a387-2ccf60d2d5d8
17 February 2022
Bokhove, Christian
7fc17e5b-9a94-48f3-a387-2ccf60d2d5d8
Bokhove, Christian
(2022)
The role of analytical variability in secondary data replications: a replication of Kim et al. (2014).
Educational Research and Evaluation, 27 (1-2), .
(doi:10.1080/13803611.2021.2022319).
Abstract
An article by Kim et al. from 2014 examined individual- and school-level variables affecting the information and communication technology (ICT) literacy level of Korean elementary school students, finding differential gender effects. In this secondary data replication, we used data from the 2018 International Computer and Information Literacy Study, focusing on data from Korea as main replication. As many characteristics of the study as possible, such as variables and analytical strategy, were modelled in the analysis. Additional analyses included 13 countries and jurisdictions, varied centring techniques for variables, and missing data treatment. The replication and analyses were pre-registered via the Open Science Framework. The main analysis did not replicate the main gender finding. However, it was also clear that, despite care taken in a rigorous replication, analytical variability still plays a large role in replications of findings, and with secondary datasets. We discuss the implications of this for secondary data replications.
Text
kim_article_replication_accepted
- Accepted Manuscript
More information
Accepted/In Press date: 31 July 2021
Published date: 17 February 2022
Additional Information:
Funding Information:
The authors approached the “data finder” for this replication, but did not receive a response, so that person cannot be named, though thanked here. The authors thank the SCORE project team at the Center for Open Science and the reviewers of the preregistration.
Publisher Copyright:
© 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
Keywords:
Replication, analytical variability, international largescale assessment
Identifiers
Local EPrints ID: 450731
URI: http://eprints.soton.ac.uk/id/eprint/450731
ISSN: 1380-3611
PURE UUID: 1a41cca2-bf22-4a5d-90be-4fe4da9501ae
Catalogue record
Date deposited: 09 Aug 2021 16:31
Last modified: 17 Mar 2024 06:45
Export record
Altmetrics
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
