The University of Southampton
University of Southampton Institutional Repository
Warning ePrints Soton is experiencing an issue with some file downloads not being available. We are working hard to fix this. Please bear with us.

International peer review and decision-making practices, bias, and burden: The funders’ perspective

International peer review and decision-making practices, bias, and burden: The funders’ perspective
International peer review and decision-making practices, bias, and burden: The funders’ perspective
Objective:
Health research funding organisations are responsible for allocating funds for research in a fair, transparent and efficient way. Peer review, and often external peer review, is considered vital for the decision-making process. However, it is well established that peer review can be biased and burdensome for the applicant, peer reviewer and funding organisation. There is limited evidence to suggest which approaches have been considered by funding organisations and whether they have been evaluated to determine their applicability and generalisability. The purpose of this work is to contribute to the evidence gap around the value of alternative and/or innovative approaches to decision-making for grant fund allocation from the perspective of funders’. The aim of this study was to identify and explore decision-making practices used by UK and international funding organisations for the allocation of funding for health related research.

Method:
An online survey was distributed to UK and international health and health-related funding organisations (March/April 2019). The survey collected information about current, past and future approaches to decision-making for grant fund allocation. The survey was sent to 63 funders (targeted list collated by National Institute of Health Research [NIHR]). Social media coverage was used to attract additional funding organisations. The survey focused at the level of a research programme rather than the overall organisation. Therefore, an entry could be provided for every research programme in an organisation. The survey had 3 sections: (1) characteristics of the funding organisation; (2) decision-making approaches currently being used and those used in the past; and (3) approaches that may be considered in the future. Participants were encouraged to provide additional information on what worked well, any potential drawbacks and suggestions for improvements. The survey was active for six weeks (ending 17 April 2019).

Results:
Preliminary descriptive statistics and thematic analysis were conducted on the data available at 3 weeks. Data was quality checked before analysis. From the initial analysis 12 funding organisations responded to the survey providing coverage from charities and research councils in the health sector. The preliminary data shows that funders use triage, external peer review and face-to-face committee meetings. None reported use of open peer review, partial randomisation or sandpits. Key reported benefits of current systems included transparency and fairness, increased quality and relevance of applications, expert opinion, reduced burden through efficient processes, and consistency in assessment. The main drawbacks of current systems included the potential for bias, time and cost burden, and difficulty in securing external peer reviewers. The full analysis and results will be presented along with the recommendations from the study and its contribution to the wider portfolio of work under the NIHR Research on Research in-house programme.

Conclusion:
In the current research environment, funders have a responsibility to demonstrate fair, transparent and efficient decision-making practices, reducing bias and burden to the funder, peer reviewers and the applicant. There is a lack of understanding from the funder’s perspective around what type of approaches work (or not) due to the lack of empirical evidence. This study aims to contribute to and further build on our understanding to provide robust evidence to enhance the decision-making process of grant allocation. Early findings indicate that funders are keen to explore alternative approaches to decision-making that facilitate the process (through reducing bias, time, monetary cost) whilst maintaining transparency, fairness and quality. The findings from this study will be important in the context of determining which approaches may be applicable and generalisable to use within organisations that fund health research.
Meadmore, Katie
4b63707b-4c44-486c-958e-e84645e7ed33
Meadmore, Katie
4b63707b-4c44-486c-958e-e84645e7ed33

Meadmore, Katie (2019) International peer review and decision-making practices, bias, and burden: The funders’ perspective. EMB Live, , Oxford, United Kingdom. 15 - 17 Jul 2019.

Record type: Conference or Workshop Item (Other)

Abstract

Objective:
Health research funding organisations are responsible for allocating funds for research in a fair, transparent and efficient way. Peer review, and often external peer review, is considered vital for the decision-making process. However, it is well established that peer review can be biased and burdensome for the applicant, peer reviewer and funding organisation. There is limited evidence to suggest which approaches have been considered by funding organisations and whether they have been evaluated to determine their applicability and generalisability. The purpose of this work is to contribute to the evidence gap around the value of alternative and/or innovative approaches to decision-making for grant fund allocation from the perspective of funders’. The aim of this study was to identify and explore decision-making practices used by UK and international funding organisations for the allocation of funding for health related research.

Method:
An online survey was distributed to UK and international health and health-related funding organisations (March/April 2019). The survey collected information about current, past and future approaches to decision-making for grant fund allocation. The survey was sent to 63 funders (targeted list collated by National Institute of Health Research [NIHR]). Social media coverage was used to attract additional funding organisations. The survey focused at the level of a research programme rather than the overall organisation. Therefore, an entry could be provided for every research programme in an organisation. The survey had 3 sections: (1) characteristics of the funding organisation; (2) decision-making approaches currently being used and those used in the past; and (3) approaches that may be considered in the future. Participants were encouraged to provide additional information on what worked well, any potential drawbacks and suggestions for improvements. The survey was active for six weeks (ending 17 April 2019).

Results:
Preliminary descriptive statistics and thematic analysis were conducted on the data available at 3 weeks. Data was quality checked before analysis. From the initial analysis 12 funding organisations responded to the survey providing coverage from charities and research councils in the health sector. The preliminary data shows that funders use triage, external peer review and face-to-face committee meetings. None reported use of open peer review, partial randomisation or sandpits. Key reported benefits of current systems included transparency and fairness, increased quality and relevance of applications, expert opinion, reduced burden through efficient processes, and consistency in assessment. The main drawbacks of current systems included the potential for bias, time and cost burden, and difficulty in securing external peer reviewers. The full analysis and results will be presented along with the recommendations from the study and its contribution to the wider portfolio of work under the NIHR Research on Research in-house programme.

Conclusion:
In the current research environment, funders have a responsibility to demonstrate fair, transparent and efficient decision-making practices, reducing bias and burden to the funder, peer reviewers and the applicant. There is a lack of understanding from the funder’s perspective around what type of approaches work (or not) due to the lack of empirical evidence. This study aims to contribute to and further build on our understanding to provide robust evidence to enhance the decision-making process of grant allocation. Early findings indicate that funders are keen to explore alternative approaches to decision-making that facilitate the process (through reducing bias, time, monetary cost) whilst maintaining transparency, fairness and quality. The findings from this study will be important in the context of determining which approaches may be applicable and generalisable to use within organisations that fund health research.

This record has no associated files available for download.

More information

Published date: July 2019
Venue - Dates: EMB Live, , Oxford, United Kingdom, 2019-07-15 - 2019-07-17

Identifiers

Local EPrints ID: 432414
URI: http://eprints.soton.ac.uk/id/eprint/432414
PURE UUID: 8df1ab22-6d9e-4957-b389-7fa8789a881f
ORCID for Katie Meadmore: ORCID iD orcid.org/0000-0001-5378-8370

Catalogue record

Date deposited: 12 Jul 2019 16:30
Last modified: 10 Nov 2021 03:14

Export record

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×