A Meta-Learning Approach to Predicting Performance and Data Requirements

Jain, Achin; Swaminathan, Gurumurthy; Favaro, Paolo; Yang, Hao; Ravichandran, Avinash; Harutyunyan, Hrayr; Achille, Alessandro; Dabeer, Onkar; Schiele, Bernt; Swaminathan, Ashwin; Soatto, Stefano (2023). A Meta-Learning Approach to Predicting Performance and Data Requirements. In: International Conference on Computer Vision.

Text Jain_A_Meta-Learning_Approach_to_Predicting_Performance_and_Data_Requirements_CVPR_2023_paper.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (5MB)

We propose an approach to estimate the number of samples required for a model to reach a target performance.
We find that the power law, the de facto principle to estimate model performance, leads to a large error when using
a small dataset (e.g., 5 samples per class) for extrapolation. This is because the log-performance error against the
log-dataset size follows a nonlinear progression in the fewshot regime followed by a linear progression in the highshot regime. We introduce a novel piecewise power law
(PPL) that handles the two data regimes differently. To
estimate the parameters of the PPL, we introduce a random forest regressor trained via meta learning that generalizes across classification/detection tasks, ResNet/ViT based
architectures, and random/pre-trained initializations. The
PPL improves the performance estimation on average by
37% across 16 classification and 33% across 10 detection
datasets, compared to the power law. We further extend the
PPL to provide a confidence bound and use it to limit the
prediction horizon that reduces over-estimation of data by
76% on classification and 91% on detection datasets

Interest & Impact

Downloads

1 since deposited on 23 May 2024
1 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item