Concept-Centric Visual Turing Tests for Method Validation

Fountoukidou, Tatiana; Sznitman, Raphael (10 October 2019). Concept-Centric Visual Turing Tests for Method Validation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Lecture Notes in Computer Science: Vol. 11768 (pp. 254-262). Springer, Cham 10.1007/978-3-030-32254-0_29

[img] Text
Fountoukidou-Sznitman2019_Chapter_Concept-CentricVisualTuringTes.pdf - Published Version
Restricted to registered users only
Available under License Publisher holds Copyright.

Download (2MB) | Request a copy

Recent advances in machine learning for medical imaging have led to impressive increases in model complexity and overall capabilities. However, the ability to discern the precise information a machine learning method is using to make decisions has lagged behind and it is often unclear how these performances are in fact achieved. Conventional evaluation metrics that reduce method performance to a single number or a curve only provide limited insights. Yet, systems used in clinical practice demand thorough validation that such crude characterizations miss. To this end, we present a framework to evaluate classification methods based on a number of interpretable concepts that are crucial for a clinical task. Our approach is inspired by the Turing Test concept and how to devise a test that adaptively questions a method for its ability to interpret medical images. To do this, we make use of a Twenty Questions paradigm whereby we use a probabilistic model to characterize the method’s capacity to grasp task-specific concepts, and we introduce a strategy to sequentially query the method according to its previous answers. The results show that the probabilistic model is able to expose both the dataset’s and the method’s biases, and can be used to reduce the number of queries needed for confident performance evaluation.

Item Type:

Conference or Workshop Item (Paper)


10 Strategic Research Centers > ARTORG Center for Biomedical Engineering Research

Graduate School:

Graduate School for Cellular and Biomedical Sciences (GCB)

UniBE Contributor:

Fountoukidou, Tatiana, Sznitman, Raphael


600 Technology > 610 Medicine & health
600 Technology > 620 Engineering




Lecture Notes in Computer Science


Springer, Cham




Tatiana Fountoukidou

Date Deposited:

05 Nov 2019 13:21

Last Modified:

05 Dec 2022 15:29

Publisher DOI:





Actions (login required)

Edit item Edit item
Provide Feedback