Biomedical image analysis competitions: The state of current participation practice
Publikation: Working paper › Preprint › Forskning
Standard
Biomedical image analysis competitions : The state of current participation practice. / Eisenmann, Matthias; Reinke, Annika; Weru, Vivienn; Tizabi, Minu Dietlinde; Isensee, Fabian; Adler, Tim J.; Godau, Patrick; Cheplygina, Veronika; Kozubek, Michal; Ali, Sharib; Gupta, Anubha; Kybic, Jan; Noble, Alison; Solórzano, Carlos Ortiz de; Pachade, Samiksha; Petitjean, Caroline; Sage, Daniel; Wei, Donglai; Wilden, Elizabeth; Alapatt, Deepak; Andrearczyk, Vincent; Baid, Ujjwal; Bakas, Spyridon; Balu, Niranjan; Bano, Sophia; Bawa, Vivek Singh; Bernal, Jorge; Bodenstedt, Sebastian; Casella, Alessandro; Choi, Jinwook; Commowick, Olivier; Daum, Marie; Depeursinge, Adrien; Dorent, Reuben; Egger, Jan; Eichhorn, Hannah; Engelhardt, Sandy; Ganz, Melanie; Girard, Gabriel; Hansen, Lasse; Heinrich, Mattias; Heller, Nicholas; Hering, Alessa; Huaulmé, Arnaud; Kim, Hyunjeong; Thambawita, Vajira; Zhao, Xin; Lund, Christina B.; Ren, Jintao; Yang, Lin; MICCAI challenge collaboration.
arXiv.org, 2022. s. 1_30.Publikation: Working paper › Preprint › Forskning
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - UNPB
T1 - Biomedical image analysis competitions
T2 - The state of current participation practice
AU - Eisenmann, Matthias
AU - Reinke, Annika
AU - Weru, Vivienn
AU - Tizabi, Minu Dietlinde
AU - Isensee, Fabian
AU - Adler, Tim J.
AU - Godau, Patrick
AU - Cheplygina, Veronika
AU - Kozubek, Michal
AU - Ali, Sharib
AU - Gupta, Anubha
AU - Kybic, Jan
AU - Noble, Alison
AU - Solórzano, Carlos Ortiz de
AU - Pachade, Samiksha
AU - Petitjean, Caroline
AU - Sage, Daniel
AU - Wei, Donglai
AU - Wilden, Elizabeth
AU - Alapatt, Deepak
AU - Andrearczyk, Vincent
AU - Baid, Ujjwal
AU - Bakas, Spyridon
AU - Balu, Niranjan
AU - Bano, Sophia
AU - Bawa, Vivek Singh
AU - Bernal, Jorge
AU - Bodenstedt, Sebastian
AU - Casella, Alessandro
AU - Choi, Jinwook
AU - Commowick, Olivier
AU - Daum, Marie
AU - Depeursinge, Adrien
AU - Dorent, Reuben
AU - Egger, Jan
AU - Eichhorn, Hannah
AU - Engelhardt, Sandy
AU - Ganz, Melanie
AU - Girard, Gabriel
AU - Hansen, Lasse
AU - Heinrich, Mattias
AU - Heller, Nicholas
AU - Hering, Alessa
AU - Huaulmé, Arnaud
AU - Kim, Hyunjeong
AU - Thambawita, Vajira
AU - Zhao, Xin
AU - Lund, Christina B.
AU - Ren, Jintao
AU - Yang, Lin
AU - MICCAI challenge collaboration
PY - 2022
Y1 - 2022
N2 - The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.
AB - The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.
KW - cs.CV
KW - cs.LG
M3 - Preprint
SP - 1_30
BT - Biomedical image analysis competitions
PB - arXiv.org
ER -
ID: 331486503