The University of Southampton
University of Southampton Institutional Repository

Not clinically effective but cost-effective' - Paradoxical conclusions in randomised controlled trials with 'doubly null' results: A cross-sectional study

Not clinically effective but cost-effective' - Paradoxical conclusions in randomised controlled trials with 'doubly null' results: A cross-sectional study
Not clinically effective but cost-effective' - Paradoxical conclusions in randomised controlled trials with 'doubly null' results: A cross-sectional study

Objectives Randomised controlled trials in healthcare increasingly include economic evaluations. Some show small differences which are not statistically significant. Yet these sometimes come to paradoxical conclusions such as: 'the intervention is not clinically effective' but 'is probably cost-effective'. This study aims to quantify the extent of non-significant results and the types of conclusions drawn from them. Design Cross-sectional retrospective analysis of randomised trials published by the UK's National Institute for Health Research (NIHR) Health Technology Assessment programme. We defined as 'doubly null' those trials that found non-statistically significant differences in both primary outcome and cost per patient. Paradoxical was defined as concluding in favour of an intervention, usually compared with placebo or usual care. No human participants were involved. Our sample was 226 randomised trial projects published by the Health Technology Assessment programme 2004 to 2017. All are available free online. Results The 226 projects contained 193 trials with a full economic evaluation. Of these 76 (39%) had at least one 'doubly null' comparison. These 76 trials contained 94 comparisons. In these 30 (32%) drew economic conclusions in favour of an intervention. Overall report conclusions split roughly equally between those favouring the intervention (14), and those favouring either the control (7) or uncertainty (9). Discussion Trials with 'doubly null' results and paradoxical conclusions are not uncommon. The differences observed in cost and quality-adjustedlife year were small and non-statistically significant. Almost all these trials were also published in leading peer-reviewed journals. Although some guidelines for reporting economic results require cost-effectiveness estimates regardless of statistical significance, the interpretability of paradoxical results has nowhere been addressed. Conclusions Reconsideration is required of the interpretation of cost-effectiveness analyses in randomised controlled trials with 'doubly null' results, particularly when economics favours a novel intervention.

conclusions, contradictory, cost effectiveness, divergent, doubly null, economic evaluation, effectiveness, null, paradoxical, randomised controlled trials, RCTs, spin
2044-6055
e029596
Raftery, James
27c2661d-6c4f-448a-bf36-9a89ec72bd6b
Williams, H. C.
28a9bda3-cd46-422b-b5b5-99c33084e712
Clarke, Aileen
b305d25e-bf08-4344-8aee-f101d3d2df2e
Thornton, Jim
ac3af3ba-9b2e-4291-98d5-3618a853123f
Norrie, John
d648d104-39a0-481f-af0f-9a7209d50fb5
Snooks, Helen
7cd8ec11-2196-480d-94c9-e94402cdcba2
Stein, Ken
dba3ca57-81c5-4172-a80e-2b38f61a7cc1
Raftery, James
27c2661d-6c4f-448a-bf36-9a89ec72bd6b
Williams, H. C.
28a9bda3-cd46-422b-b5b5-99c33084e712
Clarke, Aileen
b305d25e-bf08-4344-8aee-f101d3d2df2e
Thornton, Jim
ac3af3ba-9b2e-4291-98d5-3618a853123f
Norrie, John
d648d104-39a0-481f-af0f-9a7209d50fb5
Snooks, Helen
7cd8ec11-2196-480d-94c9-e94402cdcba2
Stein, Ken
dba3ca57-81c5-4172-a80e-2b38f61a7cc1

Raftery, James, Williams, H. C., Clarke, Aileen, Thornton, Jim, Norrie, John, Snooks, Helen and Stein, Ken (2020) Not clinically effective but cost-effective' - Paradoxical conclusions in randomised controlled trials with 'doubly null' results: A cross-sectional study. BMJ Open, 10 (1), e029596, [029596]. (doi:10.1136/bmjopen-2019-029596).

Record type: Article

Abstract

Objectives Randomised controlled trials in healthcare increasingly include economic evaluations. Some show small differences which are not statistically significant. Yet these sometimes come to paradoxical conclusions such as: 'the intervention is not clinically effective' but 'is probably cost-effective'. This study aims to quantify the extent of non-significant results and the types of conclusions drawn from them. Design Cross-sectional retrospective analysis of randomised trials published by the UK's National Institute for Health Research (NIHR) Health Technology Assessment programme. We defined as 'doubly null' those trials that found non-statistically significant differences in both primary outcome and cost per patient. Paradoxical was defined as concluding in favour of an intervention, usually compared with placebo or usual care. No human participants were involved. Our sample was 226 randomised trial projects published by the Health Technology Assessment programme 2004 to 2017. All are available free online. Results The 226 projects contained 193 trials with a full economic evaluation. Of these 76 (39%) had at least one 'doubly null' comparison. These 76 trials contained 94 comparisons. In these 30 (32%) drew economic conclusions in favour of an intervention. Overall report conclusions split roughly equally between those favouring the intervention (14), and those favouring either the control (7) or uncertainty (9). Discussion Trials with 'doubly null' results and paradoxical conclusions are not uncommon. The differences observed in cost and quality-adjustedlife year were small and non-statistically significant. Almost all these trials were also published in leading peer-reviewed journals. Although some guidelines for reporting economic results require cost-effectiveness estimates regardless of statistical significance, the interpretability of paradoxical results has nowhere been addressed. Conclusions Reconsideration is required of the interpretation of cost-effectiveness analyses in randomised controlled trials with 'doubly null' results, particularly when economics favours a novel intervention.

Text
e029596.full - Version of Record
Download (516kB)

More information

Accepted/In Press date: 19 November 2019
Published date: 9 January 2020
Additional Information: © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.
Keywords: conclusions, contradictory, cost effectiveness, divergent, doubly null, economic evaluation, effectiveness, null, paradoxical, randomised controlled trials, RCTs, spin

Identifiers

Local EPrints ID: 437563
URI: http://eprints.soton.ac.uk/id/eprint/437563
ISSN: 2044-6055
PURE UUID: f5dc58de-4e13-4001-abdd-ad2e2f749c04

Catalogue record

Date deposited: 05 Feb 2020 17:32
Last modified: 16 Mar 2024 06:26

Export record

Altmetrics

Contributors

Author: James Raftery
Author: H. C. Williams
Author: Aileen Clarke
Author: Jim Thornton
Author: John Norrie
Author: Helen Snooks
Author: Ken Stein

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×