Standard
Why is the winner the best? / Eisenmann, Matthias; Reinke, Annika; Weru, Vivienn; Tizabi, Minu Dietlinde; Isensee, Fabian; Adler, Tim J.; Ali, Sharib; Andrearczyk, Vincent; Aubreville, Marc; Baid, Ujjwal; Bakas, Spyridon; Balu, Niranjan; Bano, Sophia; Bernal, Jorge; Bodenstedt, Sebastian; Casella, Alessandro; Cheplygina, Veronika; Daum, Marie; Bruijne, Marleen de; Depeursinge, Adrien; Dorent, Reuben; Egger, Jan; Ellis, David G.; Engelhardt, Sandy; Ganz, Melanie; Ghatwary, Noha; Girard, Gabriel; Godau, Patrick; Gupta, Anubha; Hansen, Lasse; Harada, Kanako; Heinrich, Mattias; Heller, Nicholas; Hering, Alessa; Huaulmé, Arnaud; Jannin, Pierre; Kavur, Ali Emre; Kodym, Oldřich; Kozubek, Michal; Li, Jianning; Li, Hongwei; Ma, Jun; Martín-Isla, Carlos; Menze, Bjoern; Noble, Alison; Oreiller, Valentin; Padoy, Nicolas; Pati, Sarthak; Payette, Kelly; Rädsch, Tim; Rafael-Patiño, Jonathan; Bawa, Vivek Singh; Speidel, Stefanie; Sudre, Carole H.; Wijnen, Kimberlin van; Wagner, Martin; Wei, Donglai; Yamlahi, Amine; Yap, Moi Hoon; Yuan, Chun; Zenk, Maximilian; Zia, Aneeq; Zimmerer, David; Aydogan, Dogu Baran; Bhattarai, Binod; Bloch, Louise; Brüngel, Raphael; Cho, Jihoon; Choi, Chanyeol; Dou, Qi; Ezhov, Ivan; Friedrich, Christoph M.; Fuller, Clifton; Gaire, Rebati Raman; Galdran, Adrian; Faura, Álvaro García; Grammatikopoulou, Maria; Hong, SeulGi; Jahanifar, Mostafa; Jang, Ikbeom; Kadkhodamohammadi, Abdolrahim; Kang, Inha; Kofler, Florian; Kondo, Satoshi; Kuijf, Hugo; Li, Mingxing; Luu, Minh Huan; Martinčič, Tomaž; Morais, Pedro; Naser, Mohamed A.; Oliveira, Bruno; Owen, David; Pang, Subeen; Park, Jinah; Park, Sung-Hong; Płotka, Szymon; Puybareau, Elodie; Rajpoot, Nasir; Ryu, Kanghyun; Saeed, Numan; Shephard, Adam; Shi, Pengcheng; Štepec, Dejan; Subedi, Ronast; Tochon, Guillaume; Torres, Helena R.; Urien, Helene; Vilaça, João L.; Wahid, Kareem Abdul; Wang, Haojie; Wang, Jiacheng; Wang, Liansheng; Wang, Xiyue; Wiestler, Benedikt; Wodzinski, Marek; Xia, Fangfang; Xie, Juanying; Xiong, Zhiwei; Yang, Sen; Yang, Yanwu; Zhao, Zixuan; Maier-Hein, Klaus; Jäger, Paul F.; Kopp-Schneider, Annette; Maier-Hein, Lena.
Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023. IEEE Computer Society Press, 2023. s. 19955-19966.
Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Harvard
Eisenmann, M, Reinke, A, Weru, V, Tizabi, MD, Isensee, F, Adler, TJ, Ali, S, Andrearczyk, V, Aubreville, M, Baid, U, Bakas, S, Balu, N, Bano, S, Bernal, J, Bodenstedt, S, Casella, A, Cheplygina, V, Daum, M
, Bruijne, MD, Depeursinge, A, Dorent, R, Egger, J, Ellis, DG, Engelhardt, S
, Ganz, M, Ghatwary, N, Girard, G, Godau, P, Gupta, A, Hansen, L, Harada, K, Heinrich, M, Heller, N, Hering, A, Huaulmé, A, Jannin, P, Kavur, AE, Kodym, O, Kozubek, M, Li, J, Li, H, Ma, J, Martín-Isla, C, Menze, B, Noble, A, Oreiller, V, Padoy, N, Pati, S, Payette, K, Rädsch, T, Rafael-Patiño, J, Bawa, VS, Speidel, S, Sudre, CH, Wijnen, KV, Wagner, M, Wei, D, Yamlahi, A, Yap, MH, Yuan, C, Zenk, M, Zia, A, Zimmerer, D, Aydogan, DB, Bhattarai, B, Bloch, L, Brüngel, R, Cho, J, Choi, C, Dou, Q, Ezhov, I, Friedrich, CM, Fuller, C, Gaire, RR, Galdran, A, Faura, ÁG, Grammatikopoulou, M, Hong, S, Jahanifar, M, Jang, I, Kadkhodamohammadi, A, Kang, I, Kofler, F, Kondo, S, Kuijf, H, Li, M, Luu, MH, Martinčič, T, Morais, P, Naser, MA, Oliveira, B, Owen, D, Pang, S, Park, J, Park, S-H, Płotka, S, Puybareau, E, Rajpoot, N, Ryu, K, Saeed, N, Shephard, A, Shi, P, Štepec, D, Subedi, R, Tochon, G, Torres, HR, Urien, H, Vilaça, JL, Wahid, KA, Wang, H, Wang, J, Wang, L, Wang, X, Wiestler, B, Wodzinski, M, Xia, F, Xie, J, Xiong, Z, Yang, S, Yang, Y, Zhao, Z, Maier-Hein, K, Jäger, PF, Kopp-Schneider, A & Maier-Hein, L 2023,
Why is the winner the best? i
Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023. IEEE Computer Society Press, s. 19955-19966, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouve, Canada,
18/06/2023.
https://doi.org/10.1109/CVPR52729.2023.01911APA
Eisenmann, M., Reinke, A., Weru, V., Tizabi, M. D., Isensee, F., Adler, T. J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., Bakas, S., Balu, N., Bano, S., Bernal, J., Bodenstedt, S., Casella, A., Cheplygina, V., Daum, M.
, Bruijne, M. D., ... Maier-Hein, L. (2023).
Why is the winner the best? I
Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023 (s. 19955-19966). IEEE Computer Society Press.
https://doi.org/10.1109/CVPR52729.2023.01911Vancouver
Eisenmann M, Reinke A, Weru V, Tizabi MD, Isensee F, Adler TJ o.a.
Why is the winner the best? I Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023. IEEE Computer Society Press. 2023. s. 19955-19966
https://doi.org/10.1109/CVPR52729.2023.01911Author
Eisenmann, Matthias ; Reinke, Annika ; Weru, Vivienn ; Tizabi, Minu Dietlinde ; Isensee, Fabian ; Adler, Tim J. ; Ali, Sharib ; Andrearczyk, Vincent ; Aubreville, Marc ; Baid, Ujjwal ; Bakas, Spyridon ; Balu, Niranjan ; Bano, Sophia ; Bernal, Jorge ; Bodenstedt, Sebastian ; Casella, Alessandro ; Cheplygina, Veronika ; Daum, Marie ; Bruijne, Marleen de ; Depeursinge, Adrien ; Dorent, Reuben ; Egger, Jan ; Ellis, David G. ; Engelhardt, Sandy ; Ganz, Melanie ; Ghatwary, Noha ; Girard, Gabriel ; Godau, Patrick ; Gupta, Anubha ; Hansen, Lasse ; Harada, Kanako ; Heinrich, Mattias ; Heller, Nicholas ; Hering, Alessa ; Huaulmé, Arnaud ; Jannin, Pierre ; Kavur, Ali Emre ; Kodym, Oldřich ; Kozubek, Michal ; Li, Jianning ; Li, Hongwei ; Ma, Jun ; Martín-Isla, Carlos ; Menze, Bjoern ; Noble, Alison ; Oreiller, Valentin ; Padoy, Nicolas ; Pati, Sarthak ; Payette, Kelly ; Rädsch, Tim ; Rafael-Patiño, Jonathan ; Bawa, Vivek Singh ; Speidel, Stefanie ; Sudre, Carole H. ; Wijnen, Kimberlin van ; Wagner, Martin ; Wei, Donglai ; Yamlahi, Amine ; Yap, Moi Hoon ; Yuan, Chun ; Zenk, Maximilian ; Zia, Aneeq ; Zimmerer, David ; Aydogan, Dogu Baran ; Bhattarai, Binod ; Bloch, Louise ; Brüngel, Raphael ; Cho, Jihoon ; Choi, Chanyeol ; Dou, Qi ; Ezhov, Ivan ; Friedrich, Christoph M. ; Fuller, Clifton ; Gaire, Rebati Raman ; Galdran, Adrian ; Faura, Álvaro García ; Grammatikopoulou, Maria ; Hong, SeulGi ; Jahanifar, Mostafa ; Jang, Ikbeom ; Kadkhodamohammadi, Abdolrahim ; Kang, Inha ; Kofler, Florian ; Kondo, Satoshi ; Kuijf, Hugo ; Li, Mingxing ; Luu, Minh Huan ; Martinčič, Tomaž ; Morais, Pedro ; Naser, Mohamed A. ; Oliveira, Bruno ; Owen, David ; Pang, Subeen ; Park, Jinah ; Park, Sung-Hong ; Płotka, Szymon ; Puybareau, Elodie ; Rajpoot, Nasir ; Ryu, Kanghyun ; Saeed, Numan ; Shephard, Adam ; Shi, Pengcheng ; Štepec, Dejan ; Subedi, Ronast ; Tochon, Guillaume ; Torres, Helena R. ; Urien, Helene ; Vilaça, João L. ; Wahid, Kareem Abdul ; Wang, Haojie ; Wang, Jiacheng ; Wang, Liansheng ; Wang, Xiyue ; Wiestler, Benedikt ; Wodzinski, Marek ; Xia, Fangfang ; Xie, Juanying ; Xiong, Zhiwei ; Yang, Sen ; Yang, Yanwu ; Zhao, Zixuan ; Maier-Hein, Klaus ; Jäger, Paul F. ; Kopp-Schneider, Annette ; Maier-Hein, Lena. / Why is the winner the best?. Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023. IEEE Computer Society Press, 2023. s. 19955-19966
Bibtex
@inproceedings{28648e06535a4e93aae208386ddf4ee7,
title = "Why is the winner the best?",
abstract = "International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The {"}typical{"} lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.",
keywords = "cs.CV, cs.LG",
author = "Matthias Eisenmann and Annika Reinke and Vivienn Weru and Tizabi, {Minu Dietlinde} and Fabian Isensee and Adler, {Tim J.} and Sharib Ali and Vincent Andrearczyk and Marc Aubreville and Ujjwal Baid and Spyridon Bakas and Niranjan Balu and Sophia Bano and Jorge Bernal and Sebastian Bodenstedt and Alessandro Casella and Veronika Cheplygina and Marie Daum and Bruijne, {Marleen de} and Adrien Depeursinge and Reuben Dorent and Jan Egger and Ellis, {David G.} and Sandy Engelhardt and Melanie Ganz and Noha Ghatwary and Gabriel Girard and Patrick Godau and Anubha Gupta and Lasse Hansen and Kanako Harada and Mattias Heinrich and Nicholas Heller and Alessa Hering and Arnaud Huaulm{\'e} and Pierre Jannin and Kavur, {Ali Emre} and Old{\v r}ich Kodym and Michal Kozubek and Jianning Li and Hongwei Li and Jun Ma and Carlos Mart{\'i}n-Isla and Bjoern Menze and Alison Noble and Valentin Oreiller and Nicolas Padoy and Sarthak Pati and Kelly Payette and Tim R{\"a}dsch and Jonathan Rafael-Pati{\~n}o and Bawa, {Vivek Singh} and Stefanie Speidel and Sudre, {Carole H.} and Wijnen, {Kimberlin van} and Martin Wagner and Donglai Wei and Amine Yamlahi and Yap, {Moi Hoon} and Chun Yuan and Maximilian Zenk and Aneeq Zia and David Zimmerer and Aydogan, {Dogu Baran} and Binod Bhattarai and Louise Bloch and Raphael Br{\"u}ngel and Jihoon Cho and Chanyeol Choi and Qi Dou and Ivan Ezhov and Friedrich, {Christoph M.} and Clifton Fuller and Gaire, {Rebati Raman} and Adrian Galdran and Faura, {{\'A}lvaro Garc{\'i}a} and Maria Grammatikopoulou and SeulGi Hong and Mostafa Jahanifar and Ikbeom Jang and Abdolrahim Kadkhodamohammadi and Inha Kang and Florian Kofler and Satoshi Kondo and Hugo Kuijf and Mingxing Li and Luu, {Minh Huan} and Toma{\v z} Martin{\v c}i{\v c} and Pedro Morais and Naser, {Mohamed A.} and Bruno Oliveira and David Owen and Subeen Pang and Jinah Park and Sung-Hong Park and Szymon P{\l}otka and Elodie Puybareau and Nasir Rajpoot and Kanghyun Ryu and Numan Saeed and Adam Shephard and Pengcheng Shi and Dejan {\v S}tepec and Ronast Subedi and Guillaume Tochon and Torres, {Helena R.} and Helene Urien and Vila{\c c}a, {Jo{\~a}o L.} and Wahid, {Kareem Abdul} and Haojie Wang and Jiacheng Wang and Liansheng Wang and Xiyue Wang and Benedikt Wiestler and Marek Wodzinski and Fangfang Xia and Juanying Xie and Zhiwei Xiong and Sen Yang and Yanwu Yang and Zixuan Zhao and Klaus Maier-Hein and J{\"a}ger, {Paul F.} and Annette Kopp-Schneider and Lena Maier-Hein",
year = "2023",
doi = "10.1109/CVPR52729.2023.01911",
language = "English",
pages = "19955--19966",
booktitle = "Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023",
publisher = "IEEE Computer Society Press",
note = "null ; Conference date: 18-06-2023 Through 22-06-2023",
}
RIS
TY - GEN
T1 - Why is the winner the best?
AU - Eisenmann, Matthias
AU - Reinke, Annika
AU - Weru, Vivienn
AU - Tizabi, Minu Dietlinde
AU - Isensee, Fabian
AU - Adler, Tim J.
AU - Ali, Sharib
AU - Andrearczyk, Vincent
AU - Aubreville, Marc
AU - Baid, Ujjwal
AU - Bakas, Spyridon
AU - Balu, Niranjan
AU - Bano, Sophia
AU - Bernal, Jorge
AU - Bodenstedt, Sebastian
AU - Casella, Alessandro
AU - Cheplygina, Veronika
AU - Daum, Marie
AU - Bruijne, Marleen de
AU - Depeursinge, Adrien
AU - Dorent, Reuben
AU - Egger, Jan
AU - Ellis, David G.
AU - Engelhardt, Sandy
AU - Ganz, Melanie
AU - Ghatwary, Noha
AU - Girard, Gabriel
AU - Godau, Patrick
AU - Gupta, Anubha
AU - Hansen, Lasse
AU - Harada, Kanako
AU - Heinrich, Mattias
AU - Heller, Nicholas
AU - Hering, Alessa
AU - Huaulmé, Arnaud
AU - Jannin, Pierre
AU - Kavur, Ali Emre
AU - Kodym, Oldřich
AU - Kozubek, Michal
AU - Li, Jianning
AU - Li, Hongwei
AU - Ma, Jun
AU - Martín-Isla, Carlos
AU - Menze, Bjoern
AU - Noble, Alison
AU - Oreiller, Valentin
AU - Padoy, Nicolas
AU - Pati, Sarthak
AU - Payette, Kelly
AU - Rädsch, Tim
AU - Rafael-Patiño, Jonathan
AU - Bawa, Vivek Singh
AU - Speidel, Stefanie
AU - Sudre, Carole H.
AU - Wijnen, Kimberlin van
AU - Wagner, Martin
AU - Wei, Donglai
AU - Yamlahi, Amine
AU - Yap, Moi Hoon
AU - Yuan, Chun
AU - Zenk, Maximilian
AU - Zia, Aneeq
AU - Zimmerer, David
AU - Aydogan, Dogu Baran
AU - Bhattarai, Binod
AU - Bloch, Louise
AU - Brüngel, Raphael
AU - Cho, Jihoon
AU - Choi, Chanyeol
AU - Dou, Qi
AU - Ezhov, Ivan
AU - Friedrich, Christoph M.
AU - Fuller, Clifton
AU - Gaire, Rebati Raman
AU - Galdran, Adrian
AU - Faura, Álvaro García
AU - Grammatikopoulou, Maria
AU - Hong, SeulGi
AU - Jahanifar, Mostafa
AU - Jang, Ikbeom
AU - Kadkhodamohammadi, Abdolrahim
AU - Kang, Inha
AU - Kofler, Florian
AU - Kondo, Satoshi
AU - Kuijf, Hugo
AU - Li, Mingxing
AU - Luu, Minh Huan
AU - Martinčič, Tomaž
AU - Morais, Pedro
AU - Naser, Mohamed A.
AU - Oliveira, Bruno
AU - Owen, David
AU - Pang, Subeen
AU - Park, Jinah
AU - Park, Sung-Hong
AU - Płotka, Szymon
AU - Puybareau, Elodie
AU - Rajpoot, Nasir
AU - Ryu, Kanghyun
AU - Saeed, Numan
AU - Shephard, Adam
AU - Shi, Pengcheng
AU - Štepec, Dejan
AU - Subedi, Ronast
AU - Tochon, Guillaume
AU - Torres, Helena R.
AU - Urien, Helene
AU - Vilaça, João L.
AU - Wahid, Kareem Abdul
AU - Wang, Haojie
AU - Wang, Jiacheng
AU - Wang, Liansheng
AU - Wang, Xiyue
AU - Wiestler, Benedikt
AU - Wodzinski, Marek
AU - Xia, Fangfang
AU - Xie, Juanying
AU - Xiong, Zhiwei
AU - Yang, Sen
AU - Yang, Yanwu
AU - Zhao, Zixuan
AU - Maier-Hein, Klaus
AU - Jäger, Paul F.
AU - Kopp-Schneider, Annette
AU - Maier-Hein, Lena
PY - 2023
Y1 - 2023
N2 - International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.
AB - International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multi-center study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and postprocessing (66%). The "typical" lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.
KW - cs.CV
KW - cs.LG
U2 - 10.1109/CVPR52729.2023.01911
DO - 10.1109/CVPR52729.2023.01911
M3 - Article in proceedings
SP - 19955
EP - 19966
BT - Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023
PB - IEEE Computer Society Press
Y2 - 18 June 2023 through 22 June 2023
ER -
