The University of Southampton
×
Filter your search:
University of Southampton Institutional Repository

Crowd-sourcing hypothesis tests: Making transparent how design choices shape research results

Crowd-sourcing hypothesis tests: Making transparent how design choices shape research results
Crowd-sourcing hypothesis tests: Making transparent how design choices shape research results
To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowd-sourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.
Social Science Research Network
Landy, J.F.
13e79aaf-9b6a-43d7-b9ca-35a862073f31
Jia, M.
4ebe46ff-ac14-405d-a654-a5020493bd77
Ding, I.L.
42881953-f952-4cab-a30b-bfcf5f65e7f0
Viganola, D.
c4234bcc-1441-4636-8c44-da694edbccb7
Tierney, W.
3f695765-bffe-40a2-8bca-8e1b3604f058
Dreber, A.
98c5d999-5ba7-44f1-a3da-b3903ea89c48
Johannesson, M.
4a1049cb-7269-4614-bc08-82e73cb6c013
Pfeiffer, T.
607995fd-2389-489e-af00-cabc1d103315
Ebersole, C.R.
7cfa869f-8a8a-4bd7-a622-6936fec9fe63
Gronau, Q.F.
558f3d19-c65a-4315-a991-d11b6fd3679e
Ly, A.
eaa29c9c-34b2-493b-b508-e308fb4e848d
van den Bergh, D.
cec2a9cb-d1ef-4619-b32f-ca69ffd7aa8e
Marsman, M.
01020b50-3ebe-4837-965e-a613ffeed90c
Derks, K.
94daf768-2c94-4483-8e51-7d79823d5b91
Wagenmakers, E.-J.
122aedf6-9c4b-49aa-9a08-00e0d1c47880
Proctor, A.
6d5e0834-98a0-4c9f-b097-78279687a1fa
Bartels, D.M.
e536b27a-e3a9-41a0-9217-dca2efe18228
Bauman, C.W.
6c72c481-84b3-4223-8d40-064fcb924e2e
Brady, W.J.
2f68392b-efa1-46e2-a200-6de8d45ab4b1
Cheung, F.
80494bac-30c7-490e-9c4c-4ab80e06374a
Cimpian, A.
3a359e66-bd04-465a-9106-9c66f061e7d7
Dohle, S.
fa7225d8-1dd1-454f-8153-f3da8711191c
Hahn, A.
89eed56d-f121-4ce8-acf2-54213d838e9c
Jiménez-Leal, W.
78dbbdce-d0ca-4320-8838-a694bd9e3a5a
Lucas, R.E.
2ef59db9-082f-470e-ad9e-706f472807ad
Monin, B.
fc06a0ea-e565-4ddf-a65a-70fae368eec0
Montealegre, A.
a494044c-5028-418f-be91-4959f585d905
Mullen, E.
6edc6450-8868-4048-bc34-09d43dff869a
Pang, J.
cf863d0d-f09f-45a3-a351-476c6a223a16
Ray, J.
6fde54e2-6ced-4602-996b-3dbbe20f10eb
Reinero, D.A.
ac2a98b8-e9c8-4933-a5bb-21808eca8e4c
Sowden, W.
335ef796-d170-4af6-908a-3ff82047db33
Storage, D.
647177e4-53b4-42ee-919a-1426410dfef8
Su, R.
1c9de5ca-6347-4d56-a1db-b1b4c8445142
Tworek, C.M.
2f1dae39-28dc-4824-9803-ef1a0315a7ff
van Bavel, J.J.
30d54352-9d3e-4e31-93c8-fd167dec9fcc
Walco, D.
38508cb9-446a-443a-a3a2-afbf332e9aa4
Yam, K.C.
6ed33b61-ea22-4f5a-ac15-1a2f061a29af
Cunningham, W.A.
82549c1b-4422-4d2f-a54d-6829c0a61010
Schweinsberg, M.
8a8293de-de49-459f-b62e-d16dd49064f2
Urwitz, M.
7e70dad6-9c34-40eb-a9a9-a79159a76557
Uhlmann, E.L.
d3ab2a35-b4fd-4a7b-bdbc-44b5062ad148
Adamkovič, M.
5c8a50f2-8ae4-4e37-b669-15a11a1f8796
Alaei, R.
071c6f28-02e3-4b2d-a43d-6eebab4247fc
Albers, C.J.
d620cb5e-6f00-4870-b255-df620e079632
Allard, A.
73f1caec-b202-4134-ad52-5a8697f2e5b5
Andreychik, M.R.
5f04a78e-15f1-4d33-a9d2-8c3ddca5060b
Babinčák, P.
4fe3cc37-9435-43a6-9de1-ea68cafcdd21
Baker, B.J.
97018b3f-0757-4bc5-bb43-71024b34a94d
Conway, P.
765aaaf9-173f-44cf-be9a-c8ffbb51e286
Crowdsource Hypothesis Test Consortium
Landy, J.F.
13e79aaf-9b6a-43d7-b9ca-35a862073f31
Jia, M.
4ebe46ff-ac14-405d-a654-a5020493bd77
Ding, I.L.
42881953-f952-4cab-a30b-bfcf5f65e7f0
Viganola, D.
c4234bcc-1441-4636-8c44-da694edbccb7
Tierney, W.
3f695765-bffe-40a2-8bca-8e1b3604f058
Dreber, A.
98c5d999-5ba7-44f1-a3da-b3903ea89c48
Johannesson, M.
4a1049cb-7269-4614-bc08-82e73cb6c013
Pfeiffer, T.
607995fd-2389-489e-af00-cabc1d103315
Ebersole, C.R.
7cfa869f-8a8a-4bd7-a622-6936fec9fe63
Gronau, Q.F.
558f3d19-c65a-4315-a991-d11b6fd3679e
Ly, A.
eaa29c9c-34b2-493b-b508-e308fb4e848d
van den Bergh, D.
cec2a9cb-d1ef-4619-b32f-ca69ffd7aa8e
Marsman, M.
01020b50-3ebe-4837-965e-a613ffeed90c
Derks, K.
94daf768-2c94-4483-8e51-7d79823d5b91
Wagenmakers, E.-J.
122aedf6-9c4b-49aa-9a08-00e0d1c47880
Proctor, A.
6d5e0834-98a0-4c9f-b097-78279687a1fa
Bartels, D.M.
e536b27a-e3a9-41a0-9217-dca2efe18228
Bauman, C.W.
6c72c481-84b3-4223-8d40-064fcb924e2e
Brady, W.J.
2f68392b-efa1-46e2-a200-6de8d45ab4b1
Cheung, F.
80494bac-30c7-490e-9c4c-4ab80e06374a
Cimpian, A.
3a359e66-bd04-465a-9106-9c66f061e7d7
Dohle, S.
fa7225d8-1dd1-454f-8153-f3da8711191c
Hahn, A.
89eed56d-f121-4ce8-acf2-54213d838e9c
Jiménez-Leal, W.
78dbbdce-d0ca-4320-8838-a694bd9e3a5a
Lucas, R.E.
2ef59db9-082f-470e-ad9e-706f472807ad
Monin, B.
fc06a0ea-e565-4ddf-a65a-70fae368eec0
Montealegre, A.
a494044c-5028-418f-be91-4959f585d905
Mullen, E.
6edc6450-8868-4048-bc34-09d43dff869a
Pang, J.
cf863d0d-f09f-45a3-a351-476c6a223a16
Ray, J.
6fde54e2-6ced-4602-996b-3dbbe20f10eb
Reinero, D.A.
ac2a98b8-e9c8-4933-a5bb-21808eca8e4c
Sowden, W.
335ef796-d170-4af6-908a-3ff82047db33
Storage, D.
647177e4-53b4-42ee-919a-1426410dfef8
Su, R.
1c9de5ca-6347-4d56-a1db-b1b4c8445142
Tworek, C.M.
2f1dae39-28dc-4824-9803-ef1a0315a7ff
van Bavel, J.J.
30d54352-9d3e-4e31-93c8-fd167dec9fcc
Walco, D.
38508cb9-446a-443a-a3a2-afbf332e9aa4
Yam, K.C.
6ed33b61-ea22-4f5a-ac15-1a2f061a29af
Cunningham, W.A.
82549c1b-4422-4d2f-a54d-6829c0a61010
Schweinsberg, M.
8a8293de-de49-459f-b62e-d16dd49064f2
Urwitz, M.
7e70dad6-9c34-40eb-a9a9-a79159a76557
Uhlmann, E.L.
d3ab2a35-b4fd-4a7b-bdbc-44b5062ad148
Adamkovič, M.
5c8a50f2-8ae4-4e37-b669-15a11a1f8796
Alaei, R.
071c6f28-02e3-4b2d-a43d-6eebab4247fc
Albers, C.J.
d620cb5e-6f00-4870-b255-df620e079632
Allard, A.
73f1caec-b202-4134-ad52-5a8697f2e5b5
Andreychik, M.R.
5f04a78e-15f1-4d33-a9d2-8c3ddca5060b
Babinčák, P.
4fe3cc37-9435-43a6-9de1-ea68cafcdd21
Baker, B.J.
97018b3f-0757-4bc5-bb43-71024b34a94d
Conway, P.
765aaaf9-173f-44cf-be9a-c8ffbb51e286

Crowdsource Hypothesis Test Consortium (2020) Crowd-sourcing hypothesis tests: Making transparent how design choices shape research results. (doi:10.2139/ssrn.3654406).

Record type: Other

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowd-sourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

This record has no associated files available for download.

More information

Published date: 9 September 2020
Learn more about School of Psychology research

Identifiers

Local EPrints ID: 500702
URI: http://eprints.soton.ac.uk/id/eprint/500702
DOI: doi:10.2139/ssrn.3654406
PURE UUID: 76825d70-1424-43dd-be57-ced6df5e0896
ORCID for P. Conway: ORCID iD orcid.org/0000-0003-4649-6008

Catalogue record

Date deposited: 09 May 2025 17:19
Last modified: 10 May 2025 02:13

Export record

Altmetrics

Contributors

Author: J.F. Landy
Author: M. Jia
Author: I.L. Ding
Author: D. Viganola
Author: W. Tierney
Author: A. Dreber
Author: M. Johannesson
Author: T. Pfeiffer
Author: C.R. Ebersole
Author: Q.F. Gronau
Author: A. Ly
Author: D. van den Bergh
Author: M. Marsman
Author: K. Derks
Author: E.-J. Wagenmakers
Author: A. Proctor
Author: D.M. Bartels
Author: C.W. Bauman
Author: W.J. Brady
Author: F. Cheung
Author: A. Cimpian
Author: S. Dohle
Author: A. Hahn
Author: W. Jiménez-Leal
Author: R.E. Lucas
Author: B. Monin
Author: A. Montealegre
Author: E. Mullen
Author: J. Pang
Author: J. Ray
Author: D.A. Reinero
Author: W. Sowden
Author: D. Storage
Author: R. Su
Author: C.M. Tworek
Author: J.J. van Bavel
Author: D. Walco
Author: K.C. Yam
Author: W.A. Cunningham
Author: M. Schweinsberg
Author: M. Urwitz
Author: E.L. Uhlmann
Author: M. Adamkovič
Author: R. Alaei
Author: C.J. Albers
Author: A. Allard
Author: M.R. Andreychik
Author: P. Babinčák
Author: B.J. Baker
Author: P. Conway ORCID iD
Corporate Author: Crowdsource Hypothesis Test Consortium

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Library staff additional information
Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×