Tascon Morales, Sergio; Márquez Neila, Pablo; Sznitman, Raphael (2024). Targeted Visual Prompting for Medical Visual Question Answering (Submitted). In: The Third Workshop on Applications of Medical AI (AMAI). Marrakesh, Morocco. October 6 , 2024.
Full text not available from this repository.With growing interest in recent years, medical visual question answering (Med-VQA) has rapidly evolved, with multimodal large language models (MLLMs) emerging as an alternative to classical model architectures. Specifically, their ability to add visual information to the input of pre-trained LLMs brings new capabilities for image interpretation. However, simple visual errors cast doubt on the actual visual understanding abilities of these models. To address this, region-based questions have been proposed as a means to assess and enhance actual visual understanding through compositional evaluation. To combine these two perspectives, this paper introduces targeted visual prompting to equip MLLMs with region-based questioning capabilities. By presenting the model with both the isolated region and the region in its context in a customized visual prompt, we show the effectiveness of our method across multiple datasets while comparing it to several baseline models.
Item Type: |
Conference or Workshop Item (Paper) |
|---|---|
Division/Institute: |
10 Strategic Research Centers > ARTORG Center for Biomedical Engineering Research 10 Strategic Research Centers > ARTORG Center for Biomedical Engineering Research > ARTORG Center - AI in Medical Imaging Laboratory |
Graduate School: |
Graduate School for Cellular and Biomedical Sciences (GCB) |
UniBE Contributor: |
Tascon Morales, Sergio, Márquez Neila, Pablo, Sznitman, Raphael |
Subjects: |
500 Science > 570 Life sciences; biology 600 Technology > 610 Medicine & health 000 Computer science, knowledge & systems |
Language: |
English |
Submitter: |
Sergio Tascon Morales |
Date Deposited: |
16 Jul 2024 10:10 |
Last Modified: |
16 Jul 2024 10:10 |
URI: |
https://boris.unibe.ch/id/eprint/199034 |
Actions (login required)
 |
Edit item |