Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes
Purpose: Where multiple in silico tools are concordant, the American College of Medical Genetics and Genomics/Association for Molecular Pathology (ACMG/AMP) framework affords supporting evidence toward pathogenicity or benignity, equivalent to a likelihood ratio of ~2. However, limited availability of “clinical truth sets” and prior use in tool training limits their utility for evaluation of tool performance. Methods: We created a truth set of 9,436 missense variants classified as deleterious or tolerated in clinically validated high-throughput functional assays for BRCA1, BRCA2, MSH2, PTEN, and TP53 to evaluate predictive performance for 44 recommended/commonly used in silico tools. Results: Over two-thirds of the tool–threshold combinations examined had specificity of <50%, thus substantially overcalling deleteriousness. REVEL scores of 0.8–1.0 had a Positive Likelihood Ratio (PLR) of 6.74 (5.24–8.82) compared to scores <0.7 and scores of 0–0.4 had a Negative Likelihood Ratio (NLR) of 34.3 (31.5–37.3) compared to scores of >0.7. For Meta-SNP, the equivalent PLR = 42.9 (14.4–406) and NLR = 19.4 (15.6–24.9). Conclusion: Against these clinically validated “functional truth sets," there was wide variation in the predictive performance of commonly used in silico tools. Overall, REVEL and Meta-SNP had best balanced accuracy and might potentially be used at stronger evidence weighting than current ACMG/AMP prescription, in particular for predictions of benignity.
2096-2104
Cubuk, Cankut
9a916072-4100-4c45-ab8e-8a67cf6e2170
Garrett, Alice
d8fcdea3-8231-4d5e-8ada-4f932ca1a0c3
Choi, S.
5b29fa89-bafc-4e07-91ad-6eab03601a48
et al.,
96c90377-641f-4276-9d09-6968e3f36258
Eccles, Diana
5b59bc73-11c9-4cf0-a9d5-7a8e523eee23
Turnbull, Clare
63408861-754b-4f55-a010-29d1bea914e2
Callaway, Alison
07ee9b43-9249-4a56-9582-4dc41d613101
November 2021
Cubuk, Cankut
9a916072-4100-4c45-ab8e-8a67cf6e2170
Garrett, Alice
d8fcdea3-8231-4d5e-8ada-4f932ca1a0c3
Choi, S.
5b29fa89-bafc-4e07-91ad-6eab03601a48
et al.,
96c90377-641f-4276-9d09-6968e3f36258
Eccles, Diana
5b59bc73-11c9-4cf0-a9d5-7a8e523eee23
Turnbull, Clare
63408861-754b-4f55-a010-29d1bea914e2
Callaway, Alison
07ee9b43-9249-4a56-9582-4dc41d613101
Cubuk, Cankut, Garrett, Alice, Choi, S., et al., and Turnbull, Clare
,
CanVIG-UK
(2021)
Clinical likelihood ratios and balanced accuracy for 44 in silico tools against multiple large-scale functional assays of cancer susceptibility genes.
Genetics in Medicine, 23 (11), .
(doi:10.1038/s41436-021-01265-z).
Abstract
Purpose: Where multiple in silico tools are concordant, the American College of Medical Genetics and Genomics/Association for Molecular Pathology (ACMG/AMP) framework affords supporting evidence toward pathogenicity or benignity, equivalent to a likelihood ratio of ~2. However, limited availability of “clinical truth sets” and prior use in tool training limits their utility for evaluation of tool performance. Methods: We created a truth set of 9,436 missense variants classified as deleterious or tolerated in clinically validated high-throughput functional assays for BRCA1, BRCA2, MSH2, PTEN, and TP53 to evaluate predictive performance for 44 recommended/commonly used in silico tools. Results: Over two-thirds of the tool–threshold combinations examined had specificity of <50%, thus substantially overcalling deleteriousness. REVEL scores of 0.8–1.0 had a Positive Likelihood Ratio (PLR) of 6.74 (5.24–8.82) compared to scores <0.7 and scores of 0–0.4 had a Negative Likelihood Ratio (NLR) of 34.3 (31.5–37.3) compared to scores of >0.7. For Meta-SNP, the equivalent PLR = 42.9 (14.4–406) and NLR = 19.4 (15.6–24.9). Conclusion: Against these clinically validated “functional truth sets," there was wide variation in the predictive performance of commonly used in silico tools. Overall, REVEL and Meta-SNP had best balanced accuracy and might potentially be used at stronger evidence weighting than current ACMG/AMP prescription, in particular for predictions of benignity.
Text
In_silico_Cubuk_MANUSCRIPT_clean_140621
- Accepted Manuscript
Spreadsheet
In_silico_Cubuk_Supplementary_Tables_140621
Restricted to Repository staff only
Request a copy
Text
In_silico_Cubuk_TABLES_140621
Restricted to Repository staff only
Request a copy
Text
In_silico_Cubuk_Fig1_150621
Restricted to Repository staff only
Request a copy
More information
Accepted/In Press date: 17 June 2021
Published date: November 2021
Additional Information:
Funding Information:
C.C., A.G., S.C., L.K., B.T., and H.H. are supported by CRUK Catalyst Award CanGene-CanVar (C61296/A27223). N.W. is currently supported by a Sir Henry Dale Fellowship jointly funded by the Wellcome Trust and the Royal Society (grant number 220134/Z/20/Z). J.W. is funded by Wellcome Trust [107469/Z/15/Z], Medical Research Council (UK), British Heart Foundation [RE/18/4/34215], and the NIHR Imperial College Biomedical Research Centre
Publisher Copyright:
© 2021, The Author(s).
Identifiers
Local EPrints ID: 449890
URI: http://eprints.soton.ac.uk/id/eprint/449890
ISSN: 1098-3600
PURE UUID: 20d43fa8-58f2-407c-8335-31006ebb88c4
Catalogue record
Date deposited: 23 Jun 2021 16:32
Last modified: 17 Mar 2024 06:39
Export record
Altmetrics
Contributors
Author:
Cankut Cubuk
Author:
Alice Garrett
Author:
S. Choi
Author:
et al.
Author:
Clare Turnbull
Author:
Alison Callaway
Corporate Author: CanVIG-UK
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
