The University of Southampton
University of Southampton Institutional Repository

Reproducibility in management science

Reproducibility in management science
Reproducibility in management science
With the help of more than 700 reviewers, we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019. When considering only articles for which data accessibility and hardware and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for 29% of articles, at least part of the data set was not accessible to the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%. These figures represent a significant increase compared with the period before the introduction of the disclosure policy, where only 12% of articles voluntarily provided replication materials, of which 55% could be (largely) reproduced. Substantial het-erogeneity in reproducibility rates across different fields is mainly driven by differences in data set accessibility. Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, and software and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies and suggest potential avenues for enhancing their effectiveness.
reproducibility, replication, crowd science
0025-1909
Fisar, Milos
f90a7a5c-bc33-4526-92e5-dc313b3c9d45
Greiner, Ben
88c745d4-7655-42f4-bf16-a62b8547c859
Huber, Christoph
08ef8324-a4bc-4c12-b207-efe3a9811cec
Katok, Elena
e3ba82d1-6a70-44d2-ada1-1f28b157384d
Ozkes, Ali I.
4bc5e19c-f38e-4ced-aa49-0ceda6a7a398
Lepori, Gabriele
551865b7-2e3a-4de1-aaf9-6c7f23e32e8d
Management Science Reproducibility Collaboration
Fisar, Milos
f90a7a5c-bc33-4526-92e5-dc313b3c9d45
Greiner, Ben
88c745d4-7655-42f4-bf16-a62b8547c859
Huber, Christoph
08ef8324-a4bc-4c12-b207-efe3a9811cec
Katok, Elena
e3ba82d1-6a70-44d2-ada1-1f28b157384d
Ozkes, Ali I.
4bc5e19c-f38e-4ced-aa49-0ceda6a7a398
Lepori, Gabriele
551865b7-2e3a-4de1-aaf9-6c7f23e32e8d

Fisar, Milos, Greiner, Ben, Huber, Christoph, Katok, Elena and Ozkes, Ali I. , Management Science Reproducibility Collaboration (2023) Reproducibility in management science. Management Science. (doi:10.1287/mnsc.2023.03556).

Record type: Article

Abstract

With the help of more than 700 reviewers, we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019. When considering only articles for which data accessibility and hardware and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for 29% of articles, at least part of the data set was not accessible to the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%. These figures represent a significant increase compared with the period before the introduction of the disclosure policy, where only 12% of articles voluntarily provided replication materials, of which 55% could be (largely) reproduced. Substantial het-erogeneity in reproducibility rates across different fields is mainly driven by differences in data set accessibility. Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, and software and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies and suggest potential avenues for enhancing their effectiveness.

Text
AcceptedPaper-6Dec2023 - Accepted Manuscript
Download (1MB)

More information

Accepted/In Press date: 29 November 2023
e-pub ahead of print date: 22 December 2023
Keywords: reproducibility, replication, crowd science

Identifiers

Local EPrints ID: 486033
URI: http://eprints.soton.ac.uk/id/eprint/486033
ISSN: 0025-1909
PURE UUID: 50992dd5-7035-496e-9cb9-d9550165f320
ORCID for Gabriele Lepori: ORCID iD orcid.org/0000-0002-2619-227X

Catalogue record

Date deposited: 05 Jan 2024 18:14
Last modified: 18 Mar 2024 03:54

Export record

Altmetrics

Contributors

Author: Milos Fisar
Author: Ben Greiner
Author: Christoph Huber
Author: Elena Katok
Author: Ali I. Ozkes
Author: Gabriele Lepori ORCID iD
Corporate Author: Management Science Reproducibility Collaboration

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×