Reichenpfader, Daniel; Rösslhuemer, Philipp; Denecke, Kerstin (2024). Large Language Model-Based Evaluation of Medical Question Answering Systems: Algorithm Development and Case Study. Studies in health technology and informatics, 313, pp. 22-27. IOS Press 10.3233/SHTI240006
|
Text
SHTI-313-SHTI240006.pdf - Published Version Available under License Creative Commons: Attribution-Noncommercial (CC-BY-NC). Download (483kB) | Preview |
BACKGROUND
Healthcare systems are increasingly resource constrained, leaving less time for important patient-provider interactions. Conversational agents (CAs) could be used to support the provision of information and to answer patients' questions. However, information must be accessible to a variety of patient populations, which requires understanding questions expressed at different language levels.
METHODS
This study describes the use of Large Language Models (LLMs) to evaluate predefined medical content in CAs across patient populations. These simulated populations are characterized by a range of health literacy. The evaluation framework includes both fully automated and semi-automated procedures to assess the performance of a CA.
RESULTS
A case study in the domain of mammography shows that LLMs can simulate questions from different patient populations. However, the accuracy of the answers provided varies depending on the level of health literacy.
CONCLUSIONS
Our scalable evaluation framework enables the simulation of patient populations with different health literacy levels and helps to evaluate domain specific CAs, thus promoting their integration into clinical practice. Future research aims to extend the framework to CAs without predefined content and to apply LLMs to adapt medical information to the specific (health) literacy level of the user.
Item Type: |
Journal Article (Original Article) |
|---|---|
Division/Institute: |
04 Faculty of Medicine > Department of Radiology, Neuroradiology and Nuclear Medicine (DRNN) > Institute of Diagnostic, Interventional and Paediatric Radiology |
UniBE Contributor: |
Rösslhuemer, Philipp |
Subjects: |
600 Technology > 610 Medicine & health |
ISSN: |
0926-9630 |
Publisher: |
IOS Press |
Language: |
English |
Submitter: |
Pubmed Import |
Date Deposited: |
30 Apr 2024 12:44 |
Last Modified: |
01 May 2024 07:49 |
Publisher DOI: |
10.3233/SHTI240006 |
PubMed ID: |
38682499 |
Uncontrolled Keywords: |
Algorithms Consumer Health Information Conversational Agents Large Language Model Natural Language Processing |
BORIS DOI: |
10.48350/196367 |
URI: |
https://boris.unibe.ch/id/eprint/196367 |
Actions (login required)
 |
Edit item |