The University of Southampton
University of Southampton Institutional Repository

Crowdsourcing the general public for large scale molecular pathology studies in cancer

Crowdsourcing the general public for large scale molecular pathology studies in cancer
Crowdsourcing the general public for large scale molecular pathology studies in cancer
Background

Citizen science, scientific research conducted by non-specialists, has the potential to facilitate biomedical research using available large-scale data, however validating the results is challenging. The Cell Slider is a citizen science project that intends to share images from tumors with the general public, enabling them to score tumor markers independently through an internet-based interface.

Methods

From October 2012 to June 2014, 98,293 Citizen Scientists accessed the Cell Slider web page and scored 180,172 sub-images derived from images of 12,326 tissue microarray cores labeled for estrogen receptor (ER). We evaluated the accuracy of Citizen Scientist's ER classification, and the association between ER status and prognosis by comparing their test performance against trained pathologists.

Findings

The area under ROC curve was 0.95 (95% CI 0.94 to 0.96) for cancer cell identification and 0.97 (95% CI 0.96 to 0.97) for ER status. ER positive tumors scored by Citizen Scientists were associated with survival in a similar way to that scored by trained pathologists. Survival probability at 15 years were 0.78 (95% CI 0.76 to 0.80) for ER-positive and 0.72 (95% CI 0.68 to 0.77) for ER-negative tumors based on Citizen Scientists classification. Based on pathologist classification, survival probability was 0.79 (95% CI 0.77 to 0.81) for ER-positive and 0.71(95% CI 0.67 to 0.74) for ER-negative tumors. The hazard ratio for death was 0.26 (95% CI 0.18 to 0.37) at diagnosis and became greater than one after 6.5 years of follow-up for ER scored by Citizen Scientists, and 0.24 (95% CI 0.18 to 0.33) at diagnosis increasing thereafter to one after 6.7 (95% CI 4.1 to 10.9) years of follow-up for ER scored by pathologists.

Interpretation

Crowdsourcing of the general public to classify cancer pathology data for research is viable, engages the public and provides accurate ER data. Crowdsourced classification of research data may offer a valid solution to problems of throughput requiring human input.
citizen science, crowd science, crowdsourcing, breast cancer
1-22
Candido do Reis, Francisco J.
047ffdc4-a750-4161-94f4-60e48a4ed551
Lynn, Stuart
ed890572-88f5-4ab0-9519-53d47a2b8078
Ali, H. Raza
4249d1bb-f687-4b7b-b20e-28d97b6c3909
Eccles, Diana
5b59bc73-11c9-4cf0-a9d5-7a8e523eee23
Hanby, Andrew
fa07f79e-3501-4e77-812d-7865aeb5daf7
Provenzano, Elena
82947ec0-92a3-4ef3-8269-7bf313a1ba92
Caldas, Carlos
96032f49-a8e5-45e0-a5f0-128b15f91944
Howat, William J.
3e5963d4-8992-4768-9b9d-0e3ec317510b
McDuffus, Leigh-Anne
494eadb5-bb52-4cb6-bb7a-4aa3f3833653
Liu, Bin
cee507f3-c3f9-4ee7-95c8-8f40a293d90f
Daley, Frances
581867f3-1aff-494e-ab6c-937a02f2ba0c
Coulson, Penny
c0caffd3-d612-43a1-affb-0bdd5e87e1e6
Vyas, Rupesh J.
18e0f67f-a08e-4baa-8279-ea090fa8681e
Harris, Leslie M.
c2880127-78ef-49e4-889f-ce5c619b9b17
Owens, Joanna M.
07db9e84-b1d5-4def-8273-c2823ce1001d
Carton, Amy F.M.
1e68aa48-6303-4b9c-9a28-0189df3e0eca
McQuillan, Janette P.
9b2cc186-81be-4584-a07f-7095e23505d9
Paterson, Andy M.
383565b9-1c3e-496b-936a-90f68caab763
Hirji, Zohra
7e074cf9-1d9f-44dc-ae06-f7e804353469
Christie, Sarah K.
272e01c1-d278-4d4a-9963-b4466a94e5e7
Holmes, Amber R.
ffef6dcb-6673-4c1c-9929-b01cad6a18d9
Schmidt, Marjanka K.
ced57cb8-8b44-4563-b910-d4f2cf62784a
Garcia-Closas, Montserrat
bd1054d7-f127-4fa8-b52e-67a352d29009
Easton, Douglas F.
2661cf5e-8fc6-4f1d-b27a-e60cac8c8819
Bolla, Manjeet K.
48251b35-e658-4a6c-b83e-0765d3ec5fed
Wang, Qin
f0bdc716-b02a-4d97-b9b5-c6ade087c2f3
Benitez, Javier
aaf768e0-3166-4656-a787-89512ca1828c
Milne, Roger L.
387456b7-771b-4533-85c8-006c67db7e36
Mannermaa, Arto
57d1eeb0-ece8-4e4c-8dff-7225cf7ad243
Couch, Fergus
151cb536-4895-4c73-967e-7eb42f51de8c
Devilee, Peter
70de16eb-ec65-4e3e-b8f7-6333896e213f
Tollenaar, Robert A.E.M.
e3970361-64a3-462d-a117-d47619625f08
Seynaeve, Caroline
1e8ad8ea-5121-4eee-88d6-45e34614f7af
Cox, Angela
41fd94c3-79a2-49d4-bda0-616488b147da
Cross, Simon S.
e2d76f64-fd98-475f-aa54-96be07847c1e
Blows, Fiona M.
59d07595-d255-4d87-8d66-b176d68dc64a
Sanders, Joyce
deb06a8d-5210-4fe3-be3f-4e04c7e87d4c
de Groot, Renate
d23c9405-b203-4f21-b385-8be63c4ce86a
Figueroa, Jonine
d6e4a2fa-e692-4b7d-bd66-3798fc9f7858
Sherman, Mark
b902cc3b-75cb-42b5-a618-0390d842987c
Hooning, Maartje
7c14aa74-b056-43ec-8e27-428e8a281b5f
Brenner, Hermann
5611e29d-2940-4994-a5e4-726f79a56de6
Holleczek, Bernd
f98e319e-3ac9-415e-b208-1b8ddc41fa08
Stegmaier, Christa
b7941cb7-24ec-4af2-8962-288259e2f8fd
Lintott, Chris
d386ecf3-a365-49c8-bc8a-8530eb0197c1
Pharoah, Paul D.P.
75503648-dfdc-4da1-93c9-26fa44dec680
Candido do Reis, Francisco J.
047ffdc4-a750-4161-94f4-60e48a4ed551
Lynn, Stuart
ed890572-88f5-4ab0-9519-53d47a2b8078
Ali, H. Raza
4249d1bb-f687-4b7b-b20e-28d97b6c3909
Eccles, Diana
5b59bc73-11c9-4cf0-a9d5-7a8e523eee23
Hanby, Andrew
fa07f79e-3501-4e77-812d-7865aeb5daf7
Provenzano, Elena
82947ec0-92a3-4ef3-8269-7bf313a1ba92
Caldas, Carlos
96032f49-a8e5-45e0-a5f0-128b15f91944
Howat, William J.
3e5963d4-8992-4768-9b9d-0e3ec317510b
McDuffus, Leigh-Anne
494eadb5-bb52-4cb6-bb7a-4aa3f3833653
Liu, Bin
cee507f3-c3f9-4ee7-95c8-8f40a293d90f
Daley, Frances
581867f3-1aff-494e-ab6c-937a02f2ba0c
Coulson, Penny
c0caffd3-d612-43a1-affb-0bdd5e87e1e6
Vyas, Rupesh J.
18e0f67f-a08e-4baa-8279-ea090fa8681e
Harris, Leslie M.
c2880127-78ef-49e4-889f-ce5c619b9b17
Owens, Joanna M.
07db9e84-b1d5-4def-8273-c2823ce1001d
Carton, Amy F.M.
1e68aa48-6303-4b9c-9a28-0189df3e0eca
McQuillan, Janette P.
9b2cc186-81be-4584-a07f-7095e23505d9
Paterson, Andy M.
383565b9-1c3e-496b-936a-90f68caab763
Hirji, Zohra
7e074cf9-1d9f-44dc-ae06-f7e804353469
Christie, Sarah K.
272e01c1-d278-4d4a-9963-b4466a94e5e7
Holmes, Amber R.
ffef6dcb-6673-4c1c-9929-b01cad6a18d9
Schmidt, Marjanka K.
ced57cb8-8b44-4563-b910-d4f2cf62784a
Garcia-Closas, Montserrat
bd1054d7-f127-4fa8-b52e-67a352d29009
Easton, Douglas F.
2661cf5e-8fc6-4f1d-b27a-e60cac8c8819
Bolla, Manjeet K.
48251b35-e658-4a6c-b83e-0765d3ec5fed
Wang, Qin
f0bdc716-b02a-4d97-b9b5-c6ade087c2f3
Benitez, Javier
aaf768e0-3166-4656-a787-89512ca1828c
Milne, Roger L.
387456b7-771b-4533-85c8-006c67db7e36
Mannermaa, Arto
57d1eeb0-ece8-4e4c-8dff-7225cf7ad243
Couch, Fergus
151cb536-4895-4c73-967e-7eb42f51de8c
Devilee, Peter
70de16eb-ec65-4e3e-b8f7-6333896e213f
Tollenaar, Robert A.E.M.
e3970361-64a3-462d-a117-d47619625f08
Seynaeve, Caroline
1e8ad8ea-5121-4eee-88d6-45e34614f7af
Cox, Angela
41fd94c3-79a2-49d4-bda0-616488b147da
Cross, Simon S.
e2d76f64-fd98-475f-aa54-96be07847c1e
Blows, Fiona M.
59d07595-d255-4d87-8d66-b176d68dc64a
Sanders, Joyce
deb06a8d-5210-4fe3-be3f-4e04c7e87d4c
de Groot, Renate
d23c9405-b203-4f21-b385-8be63c4ce86a
Figueroa, Jonine
d6e4a2fa-e692-4b7d-bd66-3798fc9f7858
Sherman, Mark
b902cc3b-75cb-42b5-a618-0390d842987c
Hooning, Maartje
7c14aa74-b056-43ec-8e27-428e8a281b5f
Brenner, Hermann
5611e29d-2940-4994-a5e4-726f79a56de6
Holleczek, Bernd
f98e319e-3ac9-415e-b208-1b8ddc41fa08
Stegmaier, Christa
b7941cb7-24ec-4af2-8962-288259e2f8fd
Lintott, Chris
d386ecf3-a365-49c8-bc8a-8530eb0197c1
Pharoah, Paul D.P.
75503648-dfdc-4da1-93c9-26fa44dec680

Candido do Reis, Francisco J., Lynn, Stuart, Ali, H. Raza, Eccles, Diana, Hanby, Andrew, Provenzano, Elena, Caldas, Carlos, Howat, William J., McDuffus, Leigh-Anne, Liu, Bin, Daley, Frances, Coulson, Penny, Vyas, Rupesh J., Harris, Leslie M., Owens, Joanna M., Carton, Amy F.M., McQuillan, Janette P., Paterson, Andy M., Hirji, Zohra, Christie, Sarah K., Holmes, Amber R., Schmidt, Marjanka K., Garcia-Closas, Montserrat, Easton, Douglas F., Bolla, Manjeet K., Wang, Qin, Benitez, Javier, Milne, Roger L., Mannermaa, Arto, Couch, Fergus, Devilee, Peter, Tollenaar, Robert A.E.M., Seynaeve, Caroline, Cox, Angela, Cross, Simon S., Blows, Fiona M., Sanders, Joyce, de Groot, Renate, Figueroa, Jonine, Sherman, Mark, Hooning, Maartje, Brenner, Hermann, Holleczek, Bernd, Stegmaier, Christa, Lintott, Chris and Pharoah, Paul D.P. (2015) Crowdsourcing the general public for large scale molecular pathology studies in cancer. EBioMedicine, 1-22. (doi:10.1016/j.ebiom.2015.05.009).

Record type: Article

Abstract

Background

Citizen science, scientific research conducted by non-specialists, has the potential to facilitate biomedical research using available large-scale data, however validating the results is challenging. The Cell Slider is a citizen science project that intends to share images from tumors with the general public, enabling them to score tumor markers independently through an internet-based interface.

Methods

From October 2012 to June 2014, 98,293 Citizen Scientists accessed the Cell Slider web page and scored 180,172 sub-images derived from images of 12,326 tissue microarray cores labeled for estrogen receptor (ER). We evaluated the accuracy of Citizen Scientist's ER classification, and the association between ER status and prognosis by comparing their test performance against trained pathologists.

Findings

The area under ROC curve was 0.95 (95% CI 0.94 to 0.96) for cancer cell identification and 0.97 (95% CI 0.96 to 0.97) for ER status. ER positive tumors scored by Citizen Scientists were associated with survival in a similar way to that scored by trained pathologists. Survival probability at 15 years were 0.78 (95% CI 0.76 to 0.80) for ER-positive and 0.72 (95% CI 0.68 to 0.77) for ER-negative tumors based on Citizen Scientists classification. Based on pathologist classification, survival probability was 0.79 (95% CI 0.77 to 0.81) for ER-positive and 0.71(95% CI 0.67 to 0.74) for ER-negative tumors. The hazard ratio for death was 0.26 (95% CI 0.18 to 0.37) at diagnosis and became greater than one after 6.5 years of follow-up for ER scored by Citizen Scientists, and 0.24 (95% CI 0.18 to 0.33) at diagnosis increasing thereafter to one after 6.7 (95% CI 4.1 to 10.9) years of follow-up for ER scored by pathologists.

Interpretation

Crowdsourcing of the general public to classify cancer pathology data for research is viable, engages the public and provides accurate ER data. Crowdsourced classification of research data may offer a valid solution to problems of throughput requiring human input.

PDF
CandidoDoReis_Crowdsourcing.pdf - Accepted Manuscript
Restricted to Repository staff only
Request a copy

More information

Accepted/In Press date: 8 May 2015
e-pub ahead of print date: 9 May 2015
Keywords: citizen science, crowd science, crowdsourcing, breast cancer
Organisations: Cancer Sciences

Identifiers

Local EPrints ID: 376992
URI: https://eprints.soton.ac.uk/id/eprint/376992
PURE UUID: a0ea2f45-1a12-4c05-a042-46cb7e5eb70d

Catalogue record

Date deposited: 13 May 2015 11:07
Last modified: 17 Jul 2017 21:04

Export record

Altmetrics

Contributors

Author: Francisco J. Candido do Reis
Author: Stuart Lynn
Author: H. Raza Ali
Author: Diana Eccles
Author: Andrew Hanby
Author: Elena Provenzano
Author: Carlos Caldas
Author: William J. Howat
Author: Leigh-Anne McDuffus
Author: Bin Liu
Author: Frances Daley
Author: Penny Coulson
Author: Rupesh J. Vyas
Author: Leslie M. Harris
Author: Joanna M. Owens
Author: Amy F.M. Carton
Author: Janette P. McQuillan
Author: Andy M. Paterson
Author: Zohra Hirji
Author: Sarah K. Christie
Author: Amber R. Holmes
Author: Marjanka K. Schmidt
Author: Montserrat Garcia-Closas
Author: Douglas F. Easton
Author: Manjeet K. Bolla
Author: Qin Wang
Author: Javier Benitez
Author: Roger L. Milne
Author: Arto Mannermaa
Author: Fergus Couch
Author: Peter Devilee
Author: Robert A.E.M. Tollenaar
Author: Caroline Seynaeve
Author: Angela Cox
Author: Simon S. Cross
Author: Fiona M. Blows
Author: Joyce Sanders
Author: Renate de Groot
Author: Jonine Figueroa
Author: Mark Sherman
Author: Maartje Hooning
Author: Hermann Brenner
Author: Bernd Holleczek
Author: Christa Stegmaier
Author: Chris Lintott
Author: Paul D.P. Pharoah

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×