The University of Southampton
×
Filter your search:
University of Southampton Institutional Repository

Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data

Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data
Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data

Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.

Accuracy estimates, Bias, Cherry-picking, Data-driven methods, Depression, Optimal cutoff
0895-4356
137-147
Bhandari, Parash Mani
8e074bf6-af10-4dbe-8737-ecd899de36a9
Levis, Brooke
3b07dd93-2fba-481f-89b5-557430db5dca
Neupane, Dipika
2f74531b-6979-4801-ab67-c3f96e8b4287
Patten, Scott B.
e57d52d9-9413-424f-8952-e6e763b17ca4
Shrier, Ian
ff201922-20fe-4839-a0eb-36b361f4883c
Thombs, Brett D.
5f4b5a36-d7ea-490b-b2de-de640d1df1e9
Benedetti, Andrea
a3c86900-ef15-438a-9577-e8fd82281bd6
Sun, Ying
f661a990-09c7-417d-b1dd-fd649d65f275
He, Chen
32c5edb0-8858-49af-bca6-e3234f2a66bc
Rice, Danielle B.
cb4b00b2-84e8-4ffc-961e-5e5dc1702d28
Krishnan, Ankur
0265d277-4393-4d98-a535-a7de8a4b3a1d
Wu, Yin
7c62914b-8572-4128-a350-e73027abdf83
Azar, Marleine
3eb6d432-bf00-41f8-902b-1fae4775cb14
Sanchez, Tatiana A.
ca000a1f-f1d8-4821-a1a0-0ab02ee27894
Chiovitti, Matthew J.
c149bbf7-3aa0-40bc-9a4a-d9695dfd954f
Saadat, Nazanin
e138d956-9ecd-4e11-a6f7-e58c1d4c1dd3
Riehm, Kira E.
97395ccb-4153-487c-b30e-b53d16581a59
Imran, Mahrukh
43bd5edb-09b3-4685-b3b3-921186e99bed
Negeri, Zelalem
c81d03a2-976c-41d4-bbc6-dea8145b1b42
Boruff, Jill T.
7f8bec38-eeb3-4d40-8a9e-d3ee585631b0
Cuijpers, Pim
1ba10453-df47-4e99-ae9e-22f5dad50c7b
Gilbody, Simon
6d135882-a8df-444c-85aa-980dddef45a5
Ioannidis, John P.A.
7f1c53f4-117c-47a9-89b2-cb1e381b0552
Kloda, Lorie A.
1d5ebd55-6668-429b-b352-6f52c7119888
Ziegelstein, Roy C.
486f6e94-a278-490c-8ce7-0c5569fb210f
Comeau, Liane
2c56a722-1fb8-4ac2-9ab0-f7f0e3aaf486
Mitchell, Nicholas D.
a116aa9e-da78-4e7b-99d9-b10f63877d14
Tonelli, Marcello
0a490b3d-ff37-41bb-ae9e-33eb0eeeaaf5
Vigod, Simone N.
87840887-98d7-4057-8889-5e74be3cd7ec
Aceti, Franca
aab40c95-91d3-4cab-a72e-0b3f293c7129
Alvarado, Rubén
448e638d-b12e-40e3-b8d9-1ed87cffc80a
Alvarado-Esquivel, Cosme
1972b5db-4b7d-4299-94fb-ef1f76cb94ac
Bakare, Muideen O.
89cdc3bf-9156-41ee-b005-225944cdc96d
Barnes, Jacqueline
85b99f2f-7320-4f92-bebe-0aedca094517
Bavle, Amar D.
34cbedd8-ae7f-4a2b-b7b6-0c5204193299
Beck, Cheryl Tatano
de4f9ef0-7bb5-409a-a204-431569530b01
Bindt, Carola
28b42316-c3ae-4c07-a514-571f89c5cbb1
Boyce, Philip M.
d0264efb-def0-46b8-955c-6978d8b89b9d
Bunevicius, Adomas
ee984811-c9f6-476c-ac94-583a5172e696
Castro e Couto, Tiago
08025e25-fba0-454d-975c-046c592b9959
Chaudron, Linda H.
dca244bb-6830-45cd-9157-23539aba3c16
Correa, Humberto
8b8e9f64-e82f-41b9-804b-1c30ddc14564
de Figueiredo, Felipe Pinheiro
6463ff8a-79aa-454d-8822-aa265bc6812c
Eapen, Valsamma
7b1f1500-57e7-42e3-abef-cce2997ef4a5
Favez, Nicolas
aa8ef88e-f195-464a-87e5-4522b786c99d
Felice, Ethel
842126eb-e04a-47e6-9bac-1614a9568229
Fernandes, Michelle
c01982fe-7650-409e-9936-8eba93f103af
Figueiredo, Barbara
0fd356ec-a114-4e49-9748-8b80dadd155c
Fisher, Jane R.W.
c52d0b51-491a-45e3-8899-d15f05a05eee
Garcia-Esteve, Lluïsa
832debfe-0c73-445c-a418-51479fe38258
the Depression Screening Data (DEPRESSD) EPDS Group
Bhandari, Parash Mani
8e074bf6-af10-4dbe-8737-ecd899de36a9
Levis, Brooke
3b07dd93-2fba-481f-89b5-557430db5dca
Neupane, Dipika
2f74531b-6979-4801-ab67-c3f96e8b4287
Patten, Scott B.
e57d52d9-9413-424f-8952-e6e763b17ca4
Shrier, Ian
ff201922-20fe-4839-a0eb-36b361f4883c
Thombs, Brett D.
5f4b5a36-d7ea-490b-b2de-de640d1df1e9
Benedetti, Andrea
a3c86900-ef15-438a-9577-e8fd82281bd6
Sun, Ying
f661a990-09c7-417d-b1dd-fd649d65f275
He, Chen
32c5edb0-8858-49af-bca6-e3234f2a66bc
Rice, Danielle B.
cb4b00b2-84e8-4ffc-961e-5e5dc1702d28
Krishnan, Ankur
0265d277-4393-4d98-a535-a7de8a4b3a1d
Wu, Yin
7c62914b-8572-4128-a350-e73027abdf83
Azar, Marleine
3eb6d432-bf00-41f8-902b-1fae4775cb14
Sanchez, Tatiana A.
ca000a1f-f1d8-4821-a1a0-0ab02ee27894
Chiovitti, Matthew J.
c149bbf7-3aa0-40bc-9a4a-d9695dfd954f
Saadat, Nazanin
e138d956-9ecd-4e11-a6f7-e58c1d4c1dd3
Riehm, Kira E.
97395ccb-4153-487c-b30e-b53d16581a59
Imran, Mahrukh
43bd5edb-09b3-4685-b3b3-921186e99bed
Negeri, Zelalem
c81d03a2-976c-41d4-bbc6-dea8145b1b42
Boruff, Jill T.
7f8bec38-eeb3-4d40-8a9e-d3ee585631b0
Cuijpers, Pim
1ba10453-df47-4e99-ae9e-22f5dad50c7b
Gilbody, Simon
6d135882-a8df-444c-85aa-980dddef45a5
Ioannidis, John P.A.
7f1c53f4-117c-47a9-89b2-cb1e381b0552
Kloda, Lorie A.
1d5ebd55-6668-429b-b352-6f52c7119888
Ziegelstein, Roy C.
486f6e94-a278-490c-8ce7-0c5569fb210f
Comeau, Liane
2c56a722-1fb8-4ac2-9ab0-f7f0e3aaf486
Mitchell, Nicholas D.
a116aa9e-da78-4e7b-99d9-b10f63877d14
Tonelli, Marcello
0a490b3d-ff37-41bb-ae9e-33eb0eeeaaf5
Vigod, Simone N.
87840887-98d7-4057-8889-5e74be3cd7ec
Aceti, Franca
aab40c95-91d3-4cab-a72e-0b3f293c7129
Alvarado, Rubén
448e638d-b12e-40e3-b8d9-1ed87cffc80a
Alvarado-Esquivel, Cosme
1972b5db-4b7d-4299-94fb-ef1f76cb94ac
Bakare, Muideen O.
89cdc3bf-9156-41ee-b005-225944cdc96d
Barnes, Jacqueline
85b99f2f-7320-4f92-bebe-0aedca094517
Bavle, Amar D.
34cbedd8-ae7f-4a2b-b7b6-0c5204193299
Beck, Cheryl Tatano
de4f9ef0-7bb5-409a-a204-431569530b01
Bindt, Carola
28b42316-c3ae-4c07-a514-571f89c5cbb1
Boyce, Philip M.
d0264efb-def0-46b8-955c-6978d8b89b9d
Bunevicius, Adomas
ee984811-c9f6-476c-ac94-583a5172e696
Castro e Couto, Tiago
08025e25-fba0-454d-975c-046c592b9959
Chaudron, Linda H.
dca244bb-6830-45cd-9157-23539aba3c16
Correa, Humberto
8b8e9f64-e82f-41b9-804b-1c30ddc14564
de Figueiredo, Felipe Pinheiro
6463ff8a-79aa-454d-8822-aa265bc6812c
Eapen, Valsamma
7b1f1500-57e7-42e3-abef-cce2997ef4a5
Favez, Nicolas
aa8ef88e-f195-464a-87e5-4522b786c99d
Felice, Ethel
842126eb-e04a-47e6-9bac-1614a9568229
Fernandes, Michelle
c01982fe-7650-409e-9936-8eba93f103af
Figueiredo, Barbara
0fd356ec-a114-4e49-9748-8b80dadd155c
Fisher, Jane R.W.
c52d0b51-491a-45e3-8899-d15f05a05eee
Garcia-Esteve, Lluïsa
832debfe-0c73-445c-a418-51479fe38258

Bhandari, Parash Mani, Levis, Brooke, Neupane, Dipika, Patten, Scott B., Shrier, Ian and Thombs, Brett D. , the Depression Screening Data (DEPRESSD) EPDS Group (2021) Data-driven methods distort optimal cutoffs and accuracy estimates of depression screening tools: a simulation study using individual participant data. Journal of Clinical Epidemiology, 137, 137-147. (doi:10.1016/j.jclinepi.2021.03.031).

Record type: Article

Abstract

Objective: To evaluate, across multiple sample sizes, the degree that data-driven methods result in (1) optimal cutoffs different from population optimal cutoff and (2) bias in accuracy estimates. Study design and setting: A total of 1,000 samples of sample size 100, 200, 500 and 1,000 each were randomly drawn to simulate studies of different sample sizes from a database (n = 13,255) synthesized to assess Edinburgh Postnatal Depression Scale (EPDS) screening accuracy. Optimal cutoffs were selected by maximizing Youden's J (sensitivity+specificity–1). Optimal cutoffs and accuracy estimates in simulated samples were compared to population values. Results: Optimal cutoffs in simulated samples ranged from ≥ 5 to ≥ 17 for n = 100, ≥ 6 to ≥ 16 for n = 200, ≥ 6 to ≥ 14 for n = 500, and ≥ 8 to ≥ 13 for n = 1,000. Percentage of simulated samples identifying the population optimal cutoff (≥ 11) was 30% for n = 100, 35% for n = 200, 53% for n = 500, and 71% for n = 1,000. Mean overestimation of sensitivity and underestimation of specificity were 6.5 percentage point (pp) and -1.3 pp for n = 100, 4.2 pp and -1.1 pp for n = 200, 1.8 pp and -1.0 pp for n = 500, and 1.4 pp and -1.0 pp for n = 1,000. Conclusions: Small accuracy studies may identify inaccurate optimal cutoff and overstate accuracy estimates with data-driven methods.

This record has no associated files available for download.

More information

Accepted/In Press date: 29 March 2021
e-pub ahead of print date: 8 April 2021
Published date: 1 September 2021
Keywords: Accuracy estimates, Bias, Cherry-picking, Data-driven methods, Depression, Optimal cutoff

Identifiers

Local EPrints ID: 453412
URI: http://eprints.soton.ac.uk/id/eprint/453412
DOI: doi:10.1016/j.jclinepi.2021.03.031
ISSN: 0895-4356
PURE UUID: 367bce64-8c64-4eca-848f-542a77bd00f7

Catalogue record

Date deposited: 13 Jan 2022 18:24
Last modified: 17 Mar 2024 12:51

Export record

Altmetrics

Contributors

Author: Parash Mani Bhandari
Author: Brooke Levis
Author: Dipika Neupane
Author: Scott B. Patten
Author: Ian Shrier
Author: Brett D. Thombs
Author: Andrea Benedetti
Author: Ying Sun
Author: Chen He
Author: Danielle B. Rice
Author: Ankur Krishnan
Author: Yin Wu
Author: Marleine Azar
Author: Tatiana A. Sanchez
Author: Matthew J. Chiovitti
Author: Nazanin Saadat
Author: Kira E. Riehm
Author: Mahrukh Imran
Author: Zelalem Negeri
Author: Jill T. Boruff
Author: Pim Cuijpers
Author: Simon Gilbody
Author: John P.A. Ioannidis
Author: Lorie A. Kloda
Author: Roy C. Ziegelstein
Author: Liane Comeau
Author: Nicholas D. Mitchell
Author: Marcello Tonelli
Author: Simone N. Vigod
Author: Franca Aceti
Author: Rubén Alvarado
Author: Cosme Alvarado-Esquivel
Author: Muideen O. Bakare
Author: Jacqueline Barnes
Author: Amar D. Bavle
Author: Cheryl Tatano Beck
Author: Carola Bindt
Author: Philip M. Boyce
Author: Adomas Bunevicius
Author: Tiago Castro e Couto
Author: Linda H. Chaudron
Author: Humberto Correa
Author: Felipe Pinheiro de Figueiredo
Author: Valsamma Eapen
Author: Nicolas Favez
Author: Ethel Felice
Author: Michelle Fernandes
Author: Barbara Figueiredo
Author: Jane R.W. Fisher
Author: Lluïsa Garcia-Esteve
Corporate Author: the Depression Screening Data (DEPRESSD) EPDS Group

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Library staff additional information
Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×