Examining the generalizability of research findings from archival data
Examining the generalizability of research findings from archival data
This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
Research reliability, archival data, context sensitivity, generalizability, reproducibility, research reliability
Delios, Andrew
418be60e-1fb4-4509-8c7a-7fa6087e90c0
Clemente, Elena Giulia
9616a5e0-1c28-493a-9fef-885abe857789
Wu, Tao
c30f1ec4-92ee-424b-8883-5881487f2f4b
Tan, Hongbin
d7708e3a-4550-4df3-bc2c-a1d810622858
Wang, Yong
1210cde2-c9aa-47bc-b04f-63f4506133b4
Gordon, Michael
a50bd3f4-22c0-4fbf-9d81-bd2b2a6c0a75
Viganola, Domenico
7797c7b4-8fd4-4935-a218-13fe90692cce
Chen, Zhaowei
5a4bd2e8-0f54-46cb-87a2-40e5b0355a48
Dreber, Anna
98c5d999-5ba7-44f1-a3da-b3903ea89c48
Johannesson, Magnus
4a1049cb-7269-4614-bc08-82e73cb6c013
Pfeiffer, Thomas
607995fd-2389-489e-af00-cabc1d103315
Uhlmann, Eric Luis
7caa5ebf-05f8-49bc-b793-b1429c3e2d47
Dawson, Ian
dff1b440-6c83-4354-92b6-04809460b01a
Generalizability Tests Forecasting Collaboration
19 July 2022
Delios, Andrew
418be60e-1fb4-4509-8c7a-7fa6087e90c0
Clemente, Elena Giulia
9616a5e0-1c28-493a-9fef-885abe857789
Wu, Tao
c30f1ec4-92ee-424b-8883-5881487f2f4b
Tan, Hongbin
d7708e3a-4550-4df3-bc2c-a1d810622858
Wang, Yong
1210cde2-c9aa-47bc-b04f-63f4506133b4
Gordon, Michael
a50bd3f4-22c0-4fbf-9d81-bd2b2a6c0a75
Viganola, Domenico
7797c7b4-8fd4-4935-a218-13fe90692cce
Chen, Zhaowei
5a4bd2e8-0f54-46cb-87a2-40e5b0355a48
Dreber, Anna
98c5d999-5ba7-44f1-a3da-b3903ea89c48
Johannesson, Magnus
4a1049cb-7269-4614-bc08-82e73cb6c013
Pfeiffer, Thomas
607995fd-2389-489e-af00-cabc1d103315
Uhlmann, Eric Luis
7caa5ebf-05f8-49bc-b793-b1429c3e2d47
Dawson, Ian
dff1b440-6c83-4354-92b6-04809460b01a
Delios, Andrew, Clemente, Elena Giulia and Wu, Tao
,
Generalizability Tests Forecasting Collaboration
(2022)
Examining the generalizability of research findings from archival data.
Proceedings of the National Academy of Sciences, 119 (30), [e2120377119].
(doi:10.1073/pnas.2120377119).
Abstract
This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
Text
Examining the Generalizability of Research Findings from Archival Data - Manuscript and Supp Materials
- Accepted Manuscript
More information
e-pub ahead of print date: 19 July 2022
Published date: 19 July 2022
Additional Information:
Funding Information:
ACKNOWLEDGMENTS. This research project benefitted from Ministry of Education (Singapore) Tier 1 Grant R-313-000-131-115 (to A. Delios), National Science Foundation of China Grants 72002158 (to H.T.) and 71810107002 (to H.T.), grants from the Knut and Alice Wallenberg Foundation (to A. Dreber) and the Marianne and Marcus Wallenberg Foundation (through a Wallenberg Scholar grant; to A. Dreber), Austrian Science Fund (FWF) Grant SFB F63 (to A. Dreber), grants from the Jan Wallander and Tom Hedelius Foundation (Svenska Handelsbankens Forskningsstiftelser; to A. Dreber), and an Research & Development (R&D) research grant from Institut Européen d’Administration des Affaires (INSEAD) (to E.L.U.). Dmitrii Dubrov, of the G.T.F.C., was supported by the National Research University Higher School of Economics (HSE University) Basic Research Program.
Publisher Copyright:
© 2022 National Academy of Sciences. All rights reserved.
Keywords:
Research reliability, archival data, context sensitivity, generalizability, reproducibility, research reliability
Identifiers
Local EPrints ID: 468536
URI: http://eprints.soton.ac.uk/id/eprint/468536
ISSN: 0027-8424
PURE UUID: 406b635b-7e28-447a-b19c-3391e40729a5
Catalogue record
Date deposited: 17 Aug 2022 17:03
Last modified: 17 Mar 2024 07:25
Export record
Altmetrics
Contributors
Author:
Andrew Delios
Author:
Elena Giulia Clemente
Author:
Tao Wu
Author:
Hongbin Tan
Author:
Yong Wang
Author:
Michael Gordon
Author:
Domenico Viganola
Author:
Zhaowei Chen
Author:
Anna Dreber
Author:
Magnus Johannesson
Author:
Thomas Pfeiffer
Author:
Eric Luis Uhlmann
Corporate Author: Generalizability Tests Forecasting Collaboration
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
