The University of Southampton
University of Southampton Institutional Repository

Crowdsourcing hypothesis tests: making transparent how design choices shape research results

Crowdsourcing hypothesis tests: making transparent how design choices shape research results
Crowdsourcing hypothesis tests: making transparent how design choices shape research results
To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.
Crowdsourcing, conceptual replications, forecasting, research robustness, scientific transparency, stimulus sampling
0033-2909
451-479
Landy, Justin
f24a63d1-af99-4130-aedc-f36e6a184525
Jia, Miaolei (Liam)
aa9e0400-7b44-4a42-a8e5-03e04fce96b9
Ding, Isabel
c842358c-9edf-4ca6-85a9-e1f8aba6fabe
Viganola, Domenico
06ecfcbc-7ecc-4f41-82fa-d28f23128f7c
Tierney, Warren
ca05ef65-1533-46d0-9c72-16e6e12bc776
Dreber, Anna
e9d4a1bb-7877-4e94-9aed-a9032fe05fff
Johannesson, Magnus
653ae25a-fbeb-440f-8724-fc9e1c889d95
Pfeiffer, Thomas
d3d4dad0-924f-4bc2-9d95-ffc76cb861a2
Ebersole, Charles R.
7cfa869f-8a8a-4bd7-a622-6936fec9fe63
Gronau, Quentin F.
95d1ec22-811a-4523-a2af-f07e31bbac83
Ly, Alexander
a69f27d6-32e5-4ecf-8794-6479a8bd1476
van den Bergh, Don
aba47533-3309-4bf6-a407-a77d2e21f2c3
Marsman, Maarten
01020b50-3ebe-4837-965e-a613ffeed90c
Derks, Koen
94daf768-2c94-4483-8e51-7d79823d5b91
Wagenmakers, Eric-Jan
122aedf6-9c4b-49aa-9a08-00e0d1c47880
Proctor, Andrew
6d5e0834-98a0-4c9f-b097-78279687a1fa
Bartels, Daniel M.
e536b27a-e3a9-41a0-9217-dca2efe18228
Bauman, Christopher W.
6c72c481-84b3-4223-8d40-064fcb924e2e
Brady, William J.
2f68392b-efa1-46e2-a200-6de8d45ab4b1
Cheung, Felix
57fadda3-f7b1-4c5e-b591-b990b78c0875
Cimpian, Andrei
3a359e66-bd04-465a-9106-9c66f061e7d7
Dohle, Simone
fa7225d8-1dd1-454f-8153-f3da8711191c
Donnellan, M. Brent
1c093721-aed1-4cca-8653-af66861fa4fc
Hahn, Adam
89eed56d-f121-4ce8-acf2-54213d838e9c
Hall, Michael P.
a565bd7e-2734-4df7-b58e-3368d6712726
Jiménez-Leal, William
78dbbdce-d0ca-4320-8838-a694bd9e3a5a
Johnson, David J.
1d64e0cb-db98-4477-9c3f-3c1c912c0048
Lucas, Richard E.
2ef59db9-082f-470e-ad9e-706f472807ad
Monin, Benoît
fc06a0ea-e565-4ddf-a65a-70fae368eec0
Montealegre, Andres
a494044c-5028-418f-be91-4959f585d905
Mullen, Elizabeth
a30044c9-4004-4348-98d9-f1f9bd900f50
Pang, Jun
cf863d0d-f09f-45a3-a351-476c6a223a16
Ray, Jennifer
d650f51b-bb34-4bec-afa1-75410a9618bc
Reinero, Diego A.
ac2a98b8-e9c8-4933-a5bb-21808eca8e4c
Reynolds, Jesse
2197072f-bfbe-4d85-8135-32eeac241626
Sowden, Walter
335ef796-d170-4af6-908a-3ff82047db33
Storage, Daniel
647177e4-53b4-42ee-919a-1426410dfef8
Su, Runkun
1c9de5ca-6347-4d56-a1db-b1b4c8445142
Tworek, Christina M.
2f1dae39-28dc-4824-9803-ef1a0315a7ff
Van Bavel, Jay J.
30d54352-9d3e-4e31-93c8-fd167dec9fcc
Walco, Daniel
38508cb9-446a-443a-a3a2-afbf332e9aa4
Wills, Julian
d3727e95-a2fd-4a53-85dc-da0ffa37a31a
Xu, Xiaobing
d5c1b07c-2c18-4478-9293-68537fb4d4f1
Yam, Kai Chi
6ed33b61-ea22-4f5a-ac15-1a2f061a29af
Yang, Xiaoyu
f0f56dc0-ee80-4c4a-8088-c4b405c5d568
Cunningham, William A.
82549c1b-4422-4d2f-a54d-6829c0a61010
Schweinsberg, Martin
8a8293de-de49-459f-b62e-d16dd49064f2
Urwitz, Molly
7e70dad6-9c34-40eb-a9a9-a79159a76557
Uhlmann, Eric L.
d3ab2a35-b4fd-4a7b-bdbc-44b5062ad148
Dawson, Ian
dff1b440-6c83-4354-92b6-04809460b01a
The Crowdsourcing Hypothesis Tests Collaboration
Landy, Justin
f24a63d1-af99-4130-aedc-f36e6a184525
Jia, Miaolei (Liam)
aa9e0400-7b44-4a42-a8e5-03e04fce96b9
Ding, Isabel
c842358c-9edf-4ca6-85a9-e1f8aba6fabe
Viganola, Domenico
06ecfcbc-7ecc-4f41-82fa-d28f23128f7c
Tierney, Warren
ca05ef65-1533-46d0-9c72-16e6e12bc776
Dreber, Anna
e9d4a1bb-7877-4e94-9aed-a9032fe05fff
Johannesson, Magnus
653ae25a-fbeb-440f-8724-fc9e1c889d95
Pfeiffer, Thomas
d3d4dad0-924f-4bc2-9d95-ffc76cb861a2
Ebersole, Charles R.
7cfa869f-8a8a-4bd7-a622-6936fec9fe63
Gronau, Quentin F.
95d1ec22-811a-4523-a2af-f07e31bbac83
Ly, Alexander
a69f27d6-32e5-4ecf-8794-6479a8bd1476
van den Bergh, Don
aba47533-3309-4bf6-a407-a77d2e21f2c3
Marsman, Maarten
01020b50-3ebe-4837-965e-a613ffeed90c
Derks, Koen
94daf768-2c94-4483-8e51-7d79823d5b91
Wagenmakers, Eric-Jan
122aedf6-9c4b-49aa-9a08-00e0d1c47880
Proctor, Andrew
6d5e0834-98a0-4c9f-b097-78279687a1fa
Bartels, Daniel M.
e536b27a-e3a9-41a0-9217-dca2efe18228
Bauman, Christopher W.
6c72c481-84b3-4223-8d40-064fcb924e2e
Brady, William J.
2f68392b-efa1-46e2-a200-6de8d45ab4b1
Cheung, Felix
57fadda3-f7b1-4c5e-b591-b990b78c0875
Cimpian, Andrei
3a359e66-bd04-465a-9106-9c66f061e7d7
Dohle, Simone
fa7225d8-1dd1-454f-8153-f3da8711191c
Donnellan, M. Brent
1c093721-aed1-4cca-8653-af66861fa4fc
Hahn, Adam
89eed56d-f121-4ce8-acf2-54213d838e9c
Hall, Michael P.
a565bd7e-2734-4df7-b58e-3368d6712726
Jiménez-Leal, William
78dbbdce-d0ca-4320-8838-a694bd9e3a5a
Johnson, David J.
1d64e0cb-db98-4477-9c3f-3c1c912c0048
Lucas, Richard E.
2ef59db9-082f-470e-ad9e-706f472807ad
Monin, Benoît
fc06a0ea-e565-4ddf-a65a-70fae368eec0
Montealegre, Andres
a494044c-5028-418f-be91-4959f585d905
Mullen, Elizabeth
a30044c9-4004-4348-98d9-f1f9bd900f50
Pang, Jun
cf863d0d-f09f-45a3-a351-476c6a223a16
Ray, Jennifer
d650f51b-bb34-4bec-afa1-75410a9618bc
Reinero, Diego A.
ac2a98b8-e9c8-4933-a5bb-21808eca8e4c
Reynolds, Jesse
2197072f-bfbe-4d85-8135-32eeac241626
Sowden, Walter
335ef796-d170-4af6-908a-3ff82047db33
Storage, Daniel
647177e4-53b4-42ee-919a-1426410dfef8
Su, Runkun
1c9de5ca-6347-4d56-a1db-b1b4c8445142
Tworek, Christina M.
2f1dae39-28dc-4824-9803-ef1a0315a7ff
Van Bavel, Jay J.
30d54352-9d3e-4e31-93c8-fd167dec9fcc
Walco, Daniel
38508cb9-446a-443a-a3a2-afbf332e9aa4
Wills, Julian
d3727e95-a2fd-4a53-85dc-da0ffa37a31a
Xu, Xiaobing
d5c1b07c-2c18-4478-9293-68537fb4d4f1
Yam, Kai Chi
6ed33b61-ea22-4f5a-ac15-1a2f061a29af
Yang, Xiaoyu
f0f56dc0-ee80-4c4a-8088-c4b405c5d568
Cunningham, William A.
82549c1b-4422-4d2f-a54d-6829c0a61010
Schweinsberg, Martin
8a8293de-de49-459f-b62e-d16dd49064f2
Urwitz, Molly
7e70dad6-9c34-40eb-a9a9-a79159a76557
Uhlmann, Eric L.
d3ab2a35-b4fd-4a7b-bdbc-44b5062ad148
Dawson, Ian
dff1b440-6c83-4354-92b6-04809460b01a

Landy, Justin, Jia, Miaolei (Liam), Ding, Isabel, Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R., Gronau, Quentin F., Ly, Alexander, van den Bergh, Don, Marsman, Maarten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M., Bauman, Christopher W., Brady, William J., Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, M. Brent, Hahn, Adam, Hall, Michael P., Jiménez-Leal, William, Johnson, David J., Lucas, Richard E., Monin, Benoît, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A., Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M., Van Bavel, Jay J., Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Kai Chi, Yang, Xiaoyu, Cunningham, William A., Schweinsberg, Martin, Urwitz, Molly and Uhlmann, Eric L. , The Crowdsourcing Hypothesis Tests Collaboration (2020) Crowdsourcing hypothesis tests: making transparent how design choices shape research results. Psychological Bulletin, 146 (5), 451-479. (doi:10.1037/bul0000220).

Record type: Article

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

Text
Crowdsourcing Hypothesis Tests - Pre-Publication Version - Accepted Manuscript
Download (1MB)

More information

Accepted/In Press date: 29 October 2019
e-pub ahead of print date: 16 January 2020
Published date: May 2020
Keywords: Crowdsourcing, conceptual replications, forecasting, research robustness, scientific transparency, stimulus sampling

Identifiers

Local EPrints ID: 435290
URI: http://eprints.soton.ac.uk/id/eprint/435290
ISSN: 0033-2909
PURE UUID: c0125cf3-e66d-4c09-bf61-1c8fe4ba6984
ORCID for Ian Dawson: ORCID iD orcid.org/0000-0003-0555-9682

Catalogue record

Date deposited: 30 Oct 2019 17:30
Last modified: 28 Apr 2022 02:05

Export record

Altmetrics

Contributors

Author: Justin Landy
Author: Miaolei (Liam) Jia
Author: Isabel Ding
Author: Domenico Viganola
Author: Warren Tierney
Author: Anna Dreber
Author: Magnus Johannesson
Author: Thomas Pfeiffer
Author: Charles R. Ebersole
Author: Quentin F. Gronau
Author: Alexander Ly
Author: Don van den Bergh
Author: Maarten Marsman
Author: Koen Derks
Author: Eric-Jan Wagenmakers
Author: Andrew Proctor
Author: Daniel M. Bartels
Author: Christopher W. Bauman
Author: William J. Brady
Author: Felix Cheung
Author: Andrei Cimpian
Author: Simone Dohle
Author: M. Brent Donnellan
Author: Adam Hahn
Author: Michael P. Hall
Author: William Jiménez-Leal
Author: David J. Johnson
Author: Richard E. Lucas
Author: Benoît Monin
Author: Andres Montealegre
Author: Elizabeth Mullen
Author: Jun Pang
Author: Jennifer Ray
Author: Diego A. Reinero
Author: Jesse Reynolds
Author: Walter Sowden
Author: Daniel Storage
Author: Runkun Su
Author: Christina M. Tworek
Author: Jay J. Van Bavel
Author: Daniel Walco
Author: Julian Wills
Author: Xiaobing Xu
Author: Kai Chi Yam
Author: Xiaoyu Yang
Author: William A. Cunningham
Author: Martin Schweinsberg
Author: Molly Urwitz
Author: Eric L. Uhlmann
Author: Ian Dawson ORCID iD
Corporate Author: The Crowdsourcing Hypothesis Tests Collaboration

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×