Not clinically effective but cost-effective' - Paradoxical conclusions in randomised controlled trials with 'doubly null' results: A cross-sectional study
Not clinically effective but cost-effective' - Paradoxical conclusions in randomised controlled trials with 'doubly null' results: A cross-sectional study
Objectives Randomised controlled trials in healthcare increasingly include economic evaluations. Some show small differences which are not statistically significant. Yet these sometimes come to paradoxical conclusions such as: 'the intervention is not clinically effective' but 'is probably cost-effective'. This study aims to quantify the extent of non-significant results and the types of conclusions drawn from them. Design Cross-sectional retrospective analysis of randomised trials published by the UK's National Institute for Health Research (NIHR) Health Technology Assessment programme. We defined as 'doubly null' those trials that found non-statistically significant differences in both primary outcome and cost per patient. Paradoxical was defined as concluding in favour of an intervention, usually compared with placebo or usual care. No human participants were involved. Our sample was 226 randomised trial projects published by the Health Technology Assessment programme 2004 to 2017. All are available free online. Results The 226 projects contained 193 trials with a full economic evaluation. Of these 76 (39%) had at least one 'doubly null' comparison. These 76 trials contained 94 comparisons. In these 30 (32%) drew economic conclusions in favour of an intervention. Overall report conclusions split roughly equally between those favouring the intervention (14), and those favouring either the control (7) or uncertainty (9). Discussion Trials with 'doubly null' results and paradoxical conclusions are not uncommon. The differences observed in cost and quality-adjustedlife year were small and non-statistically significant. Almost all these trials were also published in leading peer-reviewed journals. Although some guidelines for reporting economic results require cost-effectiveness estimates regardless of statistical significance, the interpretability of paradoxical results has nowhere been addressed. Conclusions Reconsideration is required of the interpretation of cost-effectiveness analyses in randomised controlled trials with 'doubly null' results, particularly when economics favours a novel intervention.
conclusions, contradictory, cost effectiveness, divergent, doubly null, economic evaluation, effectiveness, null, paradoxical, randomised controlled trials, RCTs, spin
e029596
Raftery, James
27c2661d-6c4f-448a-bf36-9a89ec72bd6b
Williams, H. C.
28a9bda3-cd46-422b-b5b5-99c33084e712
Clarke, Aileen
b305d25e-bf08-4344-8aee-f101d3d2df2e
Thornton, Jim
ac3af3ba-9b2e-4291-98d5-3618a853123f
Norrie, John
d648d104-39a0-481f-af0f-9a7209d50fb5
Snooks, Helen
7cd8ec11-2196-480d-94c9-e94402cdcba2
Stein, Ken
dba3ca57-81c5-4172-a80e-2b38f61a7cc1
9 January 2020
Raftery, James
27c2661d-6c4f-448a-bf36-9a89ec72bd6b
Williams, H. C.
28a9bda3-cd46-422b-b5b5-99c33084e712
Clarke, Aileen
b305d25e-bf08-4344-8aee-f101d3d2df2e
Thornton, Jim
ac3af3ba-9b2e-4291-98d5-3618a853123f
Norrie, John
d648d104-39a0-481f-af0f-9a7209d50fb5
Snooks, Helen
7cd8ec11-2196-480d-94c9-e94402cdcba2
Stein, Ken
dba3ca57-81c5-4172-a80e-2b38f61a7cc1
Raftery, James, Williams, H. C., Clarke, Aileen, Thornton, Jim, Norrie, John, Snooks, Helen and Stein, Ken
(2020)
Not clinically effective but cost-effective' - Paradoxical conclusions in randomised controlled trials with 'doubly null' results: A cross-sectional study.
BMJ Open, 10 (1), , [029596].
(doi:10.1136/bmjopen-2019-029596).
Abstract
Objectives Randomised controlled trials in healthcare increasingly include economic evaluations. Some show small differences which are not statistically significant. Yet these sometimes come to paradoxical conclusions such as: 'the intervention is not clinically effective' but 'is probably cost-effective'. This study aims to quantify the extent of non-significant results and the types of conclusions drawn from them. Design Cross-sectional retrospective analysis of randomised trials published by the UK's National Institute for Health Research (NIHR) Health Technology Assessment programme. We defined as 'doubly null' those trials that found non-statistically significant differences in both primary outcome and cost per patient. Paradoxical was defined as concluding in favour of an intervention, usually compared with placebo or usual care. No human participants were involved. Our sample was 226 randomised trial projects published by the Health Technology Assessment programme 2004 to 2017. All are available free online. Results The 226 projects contained 193 trials with a full economic evaluation. Of these 76 (39%) had at least one 'doubly null' comparison. These 76 trials contained 94 comparisons. In these 30 (32%) drew economic conclusions in favour of an intervention. Overall report conclusions split roughly equally between those favouring the intervention (14), and those favouring either the control (7) or uncertainty (9). Discussion Trials with 'doubly null' results and paradoxical conclusions are not uncommon. The differences observed in cost and quality-adjustedlife year were small and non-statistically significant. Almost all these trials were also published in leading peer-reviewed journals. Although some guidelines for reporting economic results require cost-effectiveness estimates regardless of statistical significance, the interpretability of paradoxical results has nowhere been addressed. Conclusions Reconsideration is required of the interpretation of cost-effectiveness analyses in randomised controlled trials with 'doubly null' results, particularly when economics favours a novel intervention.
Text
e029596.full
- Version of Record
More information
Accepted/In Press date: 19 November 2019
Published date: 9 January 2020
Additional Information:
© Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.
Keywords:
conclusions, contradictory, cost effectiveness, divergent, doubly null, economic evaluation, effectiveness, null, paradoxical, randomised controlled trials, RCTs, spin
Identifiers
Local EPrints ID: 437563
URI: http://eprints.soton.ac.uk/id/eprint/437563
ISSN: 2044-6055
PURE UUID: f5dc58de-4e13-4001-abdd-ad2e2f749c04
Catalogue record
Date deposited: 05 Feb 2020 17:32
Last modified: 05 Jun 2024 18:52
Export record
Altmetrics
Contributors
Author:
H. C. Williams
Author:
Aileen Clarke
Author:
Jim Thornton
Author:
John Norrie
Author:
Helen Snooks
Author:
Ken Stein
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics