Measuring the performance of prediction models to personalize treatment choice.

Efthimiou, Orestis; Hoogland, Jeroen; Debray, Thomas P A; Seo, Michael; Furukawa, Toshiaki A; Egger, Matthias; White, Ian R (2023). Measuring the performance of prediction models to personalize treatment choice. Statistics in medicine, 42(8), pp. 1188-1206. Wiley-Blackwell 10.1002/sim.9665

Preview

Text Efthimiou_StatMed_2023.pdf - Published Version Available under License Creative Commons: Attribution (CC-BY). Download (2MB) | Preview

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

Interest & Impact

Downloads

159 since deposited on 31 Jan 2023
61 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item