Using Chao’s estimator as a stopping criterion for technology-assisted review
Using Chao’s estimator as a stopping criterion for technology-assisted review
Technology-Assisted Review (TAR) aims to reduce the human effort required for screening processes such as abstract screening for systematic literature reviews. Human reviewers label documents as relevant or irrelevant during this process, while the system incrementally updates a prediction model based on the reviewers' previous decisions. After each model update, the system proposes new documents it deems relevant, to prioritize relevant documentsover irrelevant ones. A stopping criterion is necessary to guide users in stopping the review process to minimize the number of missed relevant documents and the number of read irrelevant documents. In this paper, we propose and evaluate a new ensemble-based Active Learning strategy and a stopping criterion based on Chao's Population Size Estimator that estimates the prevalence of relevant documents in the dataset. Our simulation study demonstrates that this criterion performs well on several datasets and is compared to other methods presented in the literature.
Bron, M.P.
aeb572d7-c422-4aeb-b9f6-15c5f46994e1
van der Heijden, P.G.M.
85157917-3b33-4683-81be-713f987fd612
Feelders, A.J.
349af62f-7b86-4ace-97db-809d16267d18
Siebes, A.P.J.M.
1d916263-10a8-4e0c-be79-a36843223bae
Bron, M.P.
aeb572d7-c422-4aeb-b9f6-15c5f46994e1
van der Heijden, P.G.M.
85157917-3b33-4683-81be-713f987fd612
Feelders, A.J.
349af62f-7b86-4ace-97db-809d16267d18
Siebes, A.P.J.M.
1d916263-10a8-4e0c-be79-a36843223bae
Bron, M.P., van der Heijden, P.G.M., Feelders, A.J. and Siebes, A.P.J.M.
(2024)
Using Chao’s estimator as a stopping criterion for technology-assisted review.
ACM Transactions on Information Systems.
(doi:10.48550/arXiv.2404.01176).
Abstract
Technology-Assisted Review (TAR) aims to reduce the human effort required for screening processes such as abstract screening for systematic literature reviews. Human reviewers label documents as relevant or irrelevant during this process, while the system incrementally updates a prediction model based on the reviewers' previous decisions. After each model update, the system proposes new documents it deems relevant, to prioritize relevant documentsover irrelevant ones. A stopping criterion is necessary to guide users in stopping the review process to minimize the number of missed relevant documents and the number of read irrelevant documents. In this paper, we propose and evaluate a new ensemble-based Active Learning strategy and a stopping criterion based on Chao's Population Size Estimator that estimates the prevalence of relevant documents in the dataset. Our simulation study demonstrates that this criterion performs well on several datasets and is compared to other methods presented in the literature.
Text
Bron et al. (2024) archiv 2404.01176
- Author's Original
Text
3724116
- Version of Record
More information
Accepted/In Press date: 26 February 2024
e-pub ahead of print date: 17 March 2024
Identifiers
Local EPrints ID: 504632
URI: http://eprints.soton.ac.uk/id/eprint/504632
ISSN: 1046-8188
PURE UUID: dad3e74e-e31f-4de8-8aef-49f8e69510e4
Catalogue record
Date deposited: 16 Sep 2025 17:03
Last modified: 17 Sep 2025 01:49
Export record
Altmetrics
Contributors
Author:
M.P. Bron
Author:
A.J. Feelders
Author:
A.P.J.M. Siebes
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
