The University of Southampton
University of Southampton Institutional Repository

Influence of external peer reviewer scores for funding applications on funding board decisions: A retrospective analysis of 1561 reviews

Influence of external peer reviewer scores for funding applications on funding board decisions: A retrospective analysis of 1561 reviews
Influence of external peer reviewer scores for funding applications on funding board decisions: A retrospective analysis of 1561 reviews

Objectives To evaluate the influence of external peer reviewer scores on the National Institute for Health Research (NIHR) research funding board decisions by the number of reviewers and type of reviewer expertise Design Retrospective analysis of external peer review scores for shortlisted full applications for funding (280 funding applications, 1236 individual reviewers, 1561 review scores) Setting Four applied health research funding programmes of NIHR, UK Main outcome measures Board decision to fund or not fund research applications Results The mean score of reviewers predicted funding decisions better than individual reviewer scores (area under the receiver operating characteristic (ROC) curve 075, 95% CI 069 to 081 compared with 062, CI 059 to 065) There was no substantial improvement in how accurately mean reviewer scores predicted funding decisions when the number of reviewers increased above 4 (area under ROC curve 075, CI 059 to 091 for four reviewers; 080, CI 067 to 092 for seven or more) Reviewers with differing expertise influenced the board's decision equally, including public and patient reviewers (area under ROC curves from 057, CI 047 to 066 for health economists to 064, CI 057 to 070 for subject-matter experts) The areas under the ROC curves were quite low when using reviewers' scores, confirming that boards do not rely solely on those scores alone to make their funding decisions, which are best predicted by the mean board score Conclusions Boards value scores that originate from a diverse pool of reviewers On the basis of independent reviewer score alone, there is no detectable benefit of using more than four reviewer scores in terms of their influence on board decisions, so to improve efficiency, it may be possible to avoid using larger numbers of reviewers The funding decision is best predicted by the board score

Peer review, Research funding, Score
2044-6055
Sorrell, Lexy
295447c6-703e-4540-b5ea-5e8693e1c9bb
McArdle, Nicola
0ba9d8ac-a669-4eb6-9429-80031143534f
Becque, Taeko
ecd1b4d5-4db8-4442-81c2-04aa291cf2fd
Payne, Helen
b2b5156d-a680-4d07-804b-2e318c90c13d
Stuart, Beth
626862fc-892b-4f6d-9cbb-7a8d7172b209
Turner, Sheila
42f19397-8e9f-435d-a348-2cc1639b5eb4
Wyatt, Jeremy C.
8361be5a-fca9-4acf-b3d2-7ce04126f468
Sorrell, Lexy
295447c6-703e-4540-b5ea-5e8693e1c9bb
McArdle, Nicola
0ba9d8ac-a669-4eb6-9429-80031143534f
Becque, Taeko
ecd1b4d5-4db8-4442-81c2-04aa291cf2fd
Payne, Helen
b2b5156d-a680-4d07-804b-2e318c90c13d
Stuart, Beth
626862fc-892b-4f6d-9cbb-7a8d7172b209
Turner, Sheila
42f19397-8e9f-435d-a348-2cc1639b5eb4
Wyatt, Jeremy C.
8361be5a-fca9-4acf-b3d2-7ce04126f468

Sorrell, Lexy, McArdle, Nicola, Becque, Taeko, Payne, Helen, Stuart, Beth, Turner, Sheila and Wyatt, Jeremy C. (2018) Influence of external peer reviewer scores for funding applications on funding board decisions: A retrospective analysis of 1561 reviews. BMJ Open, 8 (12). (doi:10.1136/bmjopen-2018-022547).

Record type: Article

Abstract

Objectives To evaluate the influence of external peer reviewer scores on the National Institute for Health Research (NIHR) research funding board decisions by the number of reviewers and type of reviewer expertise Design Retrospective analysis of external peer review scores for shortlisted full applications for funding (280 funding applications, 1236 individual reviewers, 1561 review scores) Setting Four applied health research funding programmes of NIHR, UK Main outcome measures Board decision to fund or not fund research applications Results The mean score of reviewers predicted funding decisions better than individual reviewer scores (area under the receiver operating characteristic (ROC) curve 075, 95% CI 069 to 081 compared with 062, CI 059 to 065) There was no substantial improvement in how accurately mean reviewer scores predicted funding decisions when the number of reviewers increased above 4 (area under ROC curve 075, CI 059 to 091 for four reviewers; 080, CI 067 to 092 for seven or more) Reviewers with differing expertise influenced the board's decision equally, including public and patient reviewers (area under ROC curves from 057, CI 047 to 066 for health economists to 064, CI 057 to 070 for subject-matter experts) The areas under the ROC curves were quite low when using reviewers' scores, confirming that boards do not rely solely on those scores alone to make their funding decisions, which are best predicted by the mean board score Conclusions Boards value scores that originate from a diverse pool of reviewers On the basis of independent reviewer score alone, there is no detectable benefit of using more than four reviewer scores in terms of their influence on board decisions, so to improve efficiency, it may be possible to avoid using larger numbers of reviewers The funding decision is best predicted by the board score

Text
e022547.full - Version of Record
Download (514kB)

More information

Accepted/In Press date: 18 October 2018
e-pub ahead of print date: 14 December 2018
Published date: December 2018
Keywords: Peer review, Research funding, Score

Identifiers

Local EPrints ID: 427233
URI: https://eprints.soton.ac.uk/id/eprint/427233
ISSN: 2044-6055
PURE UUID: 23d7be7c-5d66-4555-9ba5-0c37e0f44e4c
ORCID for Jeremy C. Wyatt: ORCID iD orcid.org/0000-0001-7008-1473

Catalogue record

Date deposited: 09 Jan 2019 17:30
Last modified: 17 Sep 2019 00:32

Export record

Altmetrics

Contributors

Author: Lexy Sorrell
Author: Nicola McArdle
Author: Taeko Becque
Author: Helen Payne
Author: Beth Stuart
Author: Sheila Turner
Author: Jeremy C. Wyatt ORCID iD

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×