Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities; Results of the WMH Segmentation Challenge.

Kuijf, Hugo J; Biesbroek, J Matthijs; de Bresser, Jeroen; Heinen, Rutger; Andermatt, Simon; Bento, Mariana; Berseth, Matt; Belyaev, Mikhail; Cardoso, M Jorge; Casamitjana, Adria; Collins, D Louis; Dadar, Mahsa; Georgiou, Achilleas; Ghafoorian, Mohsen; Jin, Dakai; Khademi, April; Knight, Jesse; Li, Hongwei; Llado, Xavier; Luna, Miguel; ... (2019). Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities; Results of the WMH Segmentation Challenge. IEEE transactions on medical imaging, 38(11), pp. 2556-2568. Institute of Electrical and Electronics Engineers IEEE 10.1109/TMI.2019.2905770

[img] Text
08669968.pdf - Published Version
Restricted to registered users only
Available under License Publisher holds Copyright.

Download (13MB) | Request a copy

Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. Automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their method on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge ( Sixty T1+FLAIR images from three MR scanners were released with manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. Segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: (1) Dice similarity coefficient, (2) modified Hausdorff distance (95th percentile), (3) absolute log-transformed volume difference, (4) sensitivity for detecting individual lesions, and (5) F1-score for individual lesions. Additionally, methods were ranked on their inter-scanner robustness. Twenty participants submitted their method for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation.

Item Type:

Journal Article (Original Article)


04 Faculty of Medicine > Department of Radiology, Neuroradiology and Nuclear Medicine (DRNN) > Institute of Diagnostic and Interventional Neuroradiology

UniBE Contributor:

McKinley, Richard, Wiest, Roland Gerhard Rudi


600 Technology > 610 Medicine & health




Institute of Electrical and Electronics Engineers IEEE




Martin Zbinden

Date Deposited:

15 Jul 2019 15:10

Last Modified:

02 Mar 2023 23:32

Publisher DOI:


PubMed ID:





Actions (login required)

Edit item Edit item
Provide Feedback