Concept-Centric Visual Turing Tests for Method Validation

Fountoukidou, Tatiana; Sznitman, Raphael (10 October 2019). Concept-Centric Visual Turing Tests for Method Validation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Lecture Notes in Computer Science: Vol. 11768 (pp. 254-262). Springer, Cham 10.1007/978-3-030-32254-0_29

Text Fountoukidou-Sznitman2019_Chapter_Concept-CentricVisualTuringTes.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (2MB)

Recent advances in machine learning for medical imaging have led to impressive increases in model complexity and overall capabilities. However, the ability to discern the precise information a machine learning method is using to make decisions has lagged behind and it is often unclear how these performances are in fact achieved. Conventional evaluation metrics that reduce method performance to a single number or a curve only provide limited insights. Yet, systems used in clinical practice demand thorough validation that such crude characterizations miss. To this end, we present a framework to evaluate classification methods based on a number of interpretable concepts that are crucial for a clinical task. Our approach is inspired by the Turing Test concept and how to devise a test that adaptively questions a method for its ability to interpret medical images. To do this, we make use of a Twenty Questions paradigm whereby we use a probabilistic model to characterize the method’s capacity to grasp task-specific concepts, and we introduce a strategy to sequentially query the method according to its previous answers. The results show that the probabilistic model is able to expose both the dataset’s and the method’s biases, and can be used to reduce the number of queries needed for confident performance evaluation.

Interest & Impact

Downloads

3 since deposited on 05 Nov 2019
0 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item