The University of Southampton
University of Southampton Institutional Repository

The influence of spectrum (case-mix) on diagnostic test accuracy

The influence of spectrum (case-mix) on diagnostic test accuracy
The influence of spectrum (case-mix) on diagnostic test accuracy

This thesis assessed the degree to which the technique of meta-analysis can provide insight into spectrum effects through comparing study results between studies (or between subgroups within studies). Chapter 1 introduced the concept of diagnostic accuracy as the means by which diagnostic tests can be evaluated and also introduced the idea that diagnostic tests can operate differently according to spectrum-related characteristics. It was hypothesised that meta-analysis may provide the best available tool to identify the extent to which various the sources of heterogeneity, including spectrum, can affect test accuracy. Chapter 2 explained four methods of meta-analysis that allow for variability in threshold and for variation in DOR with threshold. Only the so-called 'advanced' models possess the characteristics of an 'optimal' meta-analytic method. Chapter 3 reported a methodological review of how heterogeneity has been examined in existing systematic reviews of diagnostic test accuracy. Less than optimal methods of meta analysis that do not allow for threshold effects have been commonly employed. Spectrum related variables were commonly investigated as potential sources of heterogeneity and 'statistically significant' results often reported. The few reviews using the advanced models of meta-analysis showed overall improved systematic review methods and were more likely to have considered spectrum-related characteristics. Chapter 4 reported a detailed case study comparing the four meta-analytic methods on a large dataset of tests for the detection of tuberculosis. The main observations arsing from these analyses were further explored in Chapter 5 using data obtained from a large sample of previously published systematic reviews of diagnostic tests and using only spectrum-related covariates. The main findings were as follows: 1. On average, weighting the Moses model by the inverse variance of the log of the DOR (SE(lnDOR)) underestimated the results of the unweighted Moses model by around 30%, with considerable disagreement between models. This underestimation is likely due to bias in the SE(lnDOR) and hence it is likely that the weighted model results are misleading. The circumstances that lead to biased SE(lnDOR) are common in diagnostic test meta-analyses therefore this form of weighting is not recommended. 2. The unweighted Moses model results were more similar to those of the HSROC model than those of the weighted Moses model, however it cannot be relied upon to approximate the results of the 'optimal' HSROC model. 3. The BVN model and the HSROC model produce almost identical results for the primary data analyses (this was investigated only in Chapter 4) 4. For the HSROC model, allowing for differences in the distribution of test results between diseased and nondiseased by covariate (shape differences) sometimes affects the conclusions that would be drawn from an analysis and sometimes not. Although the magnitude of differences between groups may vary between models, the inclusion of a shape interaction term does not necessarily change the strength of evidence for differences in accuracy. It is not clear whether potential differences in the distributions of test results (differences in shape) should be routinely modelled or whether the more simple parallel curve approach will generally suffice. The optimal approach for the investigation of heterogeneity requires further investigation 5. Finally, strong evidence of effects from spectrum-related characteristics on at least one model parameter were identified by the parallel or crossing curve HSROC model for over half of the investigations conducted in Chapter 5 (32/50). This could have considerable implications for the use of tests in practice. The advanced methods of meta-analysis show promise in enabling the detection of clinically important spectrum effects. However, one of the ongoing challenges in the investigation of heterogeneity, and especially spectrum, in systematic reviews are limitations in the primary study data.

University of Southampton
Dinnes, Jacqueline
ce211091-1ed2-4f86-b7ae-81b88b35263b
Dinnes, Jacqueline
ce211091-1ed2-4f86-b7ae-81b88b35263b

Dinnes, Jacqueline (2008) The influence of spectrum (case-mix) on diagnostic test accuracy. University of Southampton, Doctoral Thesis.

Record type: Thesis (Doctoral)

Abstract

This thesis assessed the degree to which the technique of meta-analysis can provide insight into spectrum effects through comparing study results between studies (or between subgroups within studies). Chapter 1 introduced the concept of diagnostic accuracy as the means by which diagnostic tests can be evaluated and also introduced the idea that diagnostic tests can operate differently according to spectrum-related characteristics. It was hypothesised that meta-analysis may provide the best available tool to identify the extent to which various the sources of heterogeneity, including spectrum, can affect test accuracy. Chapter 2 explained four methods of meta-analysis that allow for variability in threshold and for variation in DOR with threshold. Only the so-called 'advanced' models possess the characteristics of an 'optimal' meta-analytic method. Chapter 3 reported a methodological review of how heterogeneity has been examined in existing systematic reviews of diagnostic test accuracy. Less than optimal methods of meta analysis that do not allow for threshold effects have been commonly employed. Spectrum related variables were commonly investigated as potential sources of heterogeneity and 'statistically significant' results often reported. The few reviews using the advanced models of meta-analysis showed overall improved systematic review methods and were more likely to have considered spectrum-related characteristics. Chapter 4 reported a detailed case study comparing the four meta-analytic methods on a large dataset of tests for the detection of tuberculosis. The main observations arsing from these analyses were further explored in Chapter 5 using data obtained from a large sample of previously published systematic reviews of diagnostic tests and using only spectrum-related covariates. The main findings were as follows: 1. On average, weighting the Moses model by the inverse variance of the log of the DOR (SE(lnDOR)) underestimated the results of the unweighted Moses model by around 30%, with considerable disagreement between models. This underestimation is likely due to bias in the SE(lnDOR) and hence it is likely that the weighted model results are misleading. The circumstances that lead to biased SE(lnDOR) are common in diagnostic test meta-analyses therefore this form of weighting is not recommended. 2. The unweighted Moses model results were more similar to those of the HSROC model than those of the weighted Moses model, however it cannot be relied upon to approximate the results of the 'optimal' HSROC model. 3. The BVN model and the HSROC model produce almost identical results for the primary data analyses (this was investigated only in Chapter 4) 4. For the HSROC model, allowing for differences in the distribution of test results between diseased and nondiseased by covariate (shape differences) sometimes affects the conclusions that would be drawn from an analysis and sometimes not. Although the magnitude of differences between groups may vary between models, the inclusion of a shape interaction term does not necessarily change the strength of evidence for differences in accuracy. It is not clear whether potential differences in the distributions of test results (differences in shape) should be routinely modelled or whether the more simple parallel curve approach will generally suffice. The optimal approach for the investigation of heterogeneity requires further investigation 5. Finally, strong evidence of effects from spectrum-related characteristics on at least one model parameter were identified by the parallel or crossing curve HSROC model for over half of the investigations conducted in Chapter 5 (32/50). This could have considerable implications for the use of tests in practice. The advanced methods of meta-analysis show promise in enabling the detection of clinically important spectrum effects. However, one of the ongoing challenges in the investigation of heterogeneity, and especially spectrum, in systematic reviews are limitations in the primary study data.

Text
1297228.pdf - Version of Record
Available under License University of Southampton Thesis Licence.
Download (5MB)

More information

Published date: 2008

Identifiers

Local EPrints ID: 466711
URI: http://eprints.soton.ac.uk/id/eprint/466711
PURE UUID: 8f1f5d2f-b9fb-4149-907d-aa8445b68167

Catalogue record

Date deposited: 05 Jul 2022 06:26
Last modified: 23 Jul 2022 02:21

Export record

Contributors

Author: Jacqueline Dinnes

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×