Crowdsourcing airway annotations in chest computed tomography images
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Crowdsourcing airway annotations in chest computed tomography images. / Cheplygina, Veronika; Perez-Rovira, Adria; Kuo, Wieying; Tiddens, Harm A.W.M.; de Bruijne, Marleen.
In: PLoS ONE, Vol. 16, e0249580, 2021.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Crowdsourcing airway annotations in chest computed tomography images
AU - Cheplygina, Veronika
AU - Perez-Rovira, Adria
AU - Kuo, Wieying
AU - Tiddens, Harm A.W.M.
AU - de Bruijne, Marleen
N1 - Publisher Copyright: © 2021 Cheplygina et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
PY - 2021
Y1 - 2021
N2 - Measuring airways in chest computed tomography (CT) scans is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated scans for good performance. We investigate whether crowdsourcing can be used to gather airway annotations. We generate image slices at known locations of airways in 24 subjects and request the crowd workers to outline the airway lumen and airway wall. After combining multiple crowd workers, we compare the measurements to those made by the experts in the original scans. Similar to our preliminary study, a large portion of the annotations were excluded, possibly due to workers misunderstanding the instructions. After excluding such annotations, moderate to strong correlations with the expert can be observed, although these correlations are slightly lower than inter-expert correlations. Furthermore, the results across subjects in this study are quite variable. Although the crowd has potential in annotating airways, further development is needed for it to be robust enough for gathering annotations in practice. For reproducibility, data and code are available online: http://github.com/adriapr/crowdairway.git.
AB - Measuring airways in chest computed tomography (CT) scans is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated scans for good performance. We investigate whether crowdsourcing can be used to gather airway annotations. We generate image slices at known locations of airways in 24 subjects and request the crowd workers to outline the airway lumen and airway wall. After combining multiple crowd workers, we compare the measurements to those made by the experts in the original scans. Similar to our preliminary study, a large portion of the annotations were excluded, possibly due to workers misunderstanding the instructions. After excluding such annotations, moderate to strong correlations with the expert can be observed, although these correlations are slightly lower than inter-expert correlations. Furthermore, the results across subjects in this study are quite variable. Although the crowd has potential in annotating airways, further development is needed for it to be robust enough for gathering annotations in practice. For reproducibility, data and code are available online: http://github.com/adriapr/crowdairway.git.
UR - http://www.scopus.com/inward/record.url?scp=85104720439&partnerID=8YFLogxK
U2 - 10.1371/journal.pone.0249580
DO - 10.1371/journal.pone.0249580
M3 - Journal article
C2 - 33886587
AN - SCOPUS:85104720439
VL - 16
JO - PLoS ONE
JF - PLoS ONE
SN - 1932-6203
M1 - e0249580
ER -
ID: 262898423