Critical appraisal of nonrandomized studies - a review of recommended and commonly used tools
Critical appraisal of nonrandomized studies - a review of recommended and commonly used tools
RATIONALE, AIMS, AND OBJECTIVES: When randomized controlled trial data are limited or unavailable, or to supplement randomized controlled trial evidence, health technology assessment (HTA) agencies may rely on systematic reviews of nonrandomized studies (NRSs) for evidence of the effectiveness of health care interventions. NRS designs may introduce considerable bias into systematic reviews, and several methodologies by which to evaluate this risk of bias are available. This study aimed to identify tools commonly used to assess bias in NRS and determine those recommended by HTA bodies.
METHODS: Appraisal tools used in NRS were identified through a targeted search of systematic reviews (January 2013-March 2017; MEDLINE and EMBASE [OVID SP]). Recommendations for the critical appraisal of NRS by expert review groups and HTA bodies were reviewed.
RESULTS: From the 686 studies included in the narrative synthesis, 48 critical appraisal tools were identified. Commonly used tools included the Newcastle-Ottawa Scale, the methodological index for NRS, and bespoke appraisal tools. Neither the Cochrane Handbook nor the Centre for Reviews and Dissemination recommends a particular instrument for the assessment of risk of bias in NRS, although Cochrane has recently developed their own NRS critical appraisal tool. Among HTA bodies, only the Canadian Agency for Drugs and Technologies in Health recommends use of a specific critical appraisal tool-SIGN 50 (for cohort or case-control studies). Several criteria including reporting, external validity, confounding, and power were examined.
CONCLUSION: There is no consensus between HTA groups on the preferred appraisal tool. Reviewers should select from a suite of tools on the basis of the design of studies included in their review.
Evaluation Studies as Topic, Evidence-Based Medicine/methods, Humans, Non-Randomized Controlled Trials as Topic/methods, Observer Variation, Systematic Reviews as Topic, Technology Assessment, Biomedical/methods
44-52
Quigley, Joan M.
473a7cea-23e7-4d40-ac5c-e6c3060b5f0f
Thompson, Juliette C.
28a24d4b-8b21-436c-85db-0b6608c46a09
Halfpenny, Nicholas J.
4f36586d-89d9-45d5-a1e1-e350629f483c
Scott, David A.
19b5fd34-9974-4ae4-8be0-27a693639e20
February 2019
Quigley, Joan M.
473a7cea-23e7-4d40-ac5c-e6c3060b5f0f
Thompson, Juliette C.
28a24d4b-8b21-436c-85db-0b6608c46a09
Halfpenny, Nicholas J.
4f36586d-89d9-45d5-a1e1-e350629f483c
Scott, David A.
19b5fd34-9974-4ae4-8be0-27a693639e20
Quigley, Joan M., Thompson, Juliette C., Halfpenny, Nicholas J. and Scott, David A.
(2019)
Critical appraisal of nonrandomized studies - a review of recommended and commonly used tools.
Journal of Evaluation in Clinical Practice, 25 (1), .
(doi:10.1111/jep.12889).
Abstract
RATIONALE, AIMS, AND OBJECTIVES: When randomized controlled trial data are limited or unavailable, or to supplement randomized controlled trial evidence, health technology assessment (HTA) agencies may rely on systematic reviews of nonrandomized studies (NRSs) for evidence of the effectiveness of health care interventions. NRS designs may introduce considerable bias into systematic reviews, and several methodologies by which to evaluate this risk of bias are available. This study aimed to identify tools commonly used to assess bias in NRS and determine those recommended by HTA bodies.
METHODS: Appraisal tools used in NRS were identified through a targeted search of systematic reviews (January 2013-March 2017; MEDLINE and EMBASE [OVID SP]). Recommendations for the critical appraisal of NRS by expert review groups and HTA bodies were reviewed.
RESULTS: From the 686 studies included in the narrative synthesis, 48 critical appraisal tools were identified. Commonly used tools included the Newcastle-Ottawa Scale, the methodological index for NRS, and bespoke appraisal tools. Neither the Cochrane Handbook nor the Centre for Reviews and Dissemination recommends a particular instrument for the assessment of risk of bias in NRS, although Cochrane has recently developed their own NRS critical appraisal tool. Among HTA bodies, only the Canadian Agency for Drugs and Technologies in Health recommends use of a specific critical appraisal tool-SIGN 50 (for cohort or case-control studies). Several criteria including reporting, external validity, confounding, and power were examined.
CONCLUSION: There is no consensus between HTA groups on the preferred appraisal tool. Reviewers should select from a suite of tools on the basis of the design of studies included in their review.
This record has no associated files available for download.
More information
Accepted/In Press date: 22 January 2018
e-pub ahead of print date: 27 February 2019
Published date: February 2019
Keywords:
Evaluation Studies as Topic, Evidence-Based Medicine/methods, Humans, Non-Randomized Controlled Trials as Topic/methods, Observer Variation, Systematic Reviews as Topic, Technology Assessment, Biomedical/methods
Identifiers
Local EPrints ID: 441446
URI: http://eprints.soton.ac.uk/id/eprint/441446
ISSN: 1356-1294
PURE UUID: bc51357a-b221-4014-a195-646e19c04d64
Catalogue record
Date deposited: 12 Jun 2020 16:30
Last modified: 17 Mar 2024 04:02
Export record
Altmetrics
Contributors
Author:
Joan M. Quigley
Author:
Juliette C. Thompson
Author:
Nicholas J. Halfpenny
Author:
David A. Scott
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics