Boosting Self-Supervised Learning via Knowledge Transfer

Noroozi, Mehdi; Vinjimoor, Ananth; Favaro, Paolo; Pirsiavash, Hamed (June 2018). Boosting Self-Supervised Learning via Knowledge Transfer. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, Utah, United States. June 18 - June 22, 2018.

[img] Text
Noroozi_Boosting_Self-Supervised_Learning_CVPR_2018_paper.pdf - Published Version
Restricted to registered users only
Available under License Publisher holds Copyright.

Download (1MB)

In self-supervised learning, one trains a model to solve a so-called pretext task on a dataset without the need for human annotation. The main objective, however, is to transfer this model to a target domain and task. Currently, the most effective transfer strategy is fine-tuning, which restricts one to use the same model or parts thereof for both pretext and target tasks. In this paper, we present a novel framework for self-supervised learning that overcomes limitations in designing and comparing different tasks, models, and data domains. In particular, our framework decouples the structure of the self-supervised model from the final task-specific fine-tuned model. This allows us to: 1) quantitatively assess previously incompatible models including handcrafted features; 2) show that deeper neural network models can learn better representations from the same pretext task; 3) transfer knowledge learned with a deep model to a shallower one and thus boost its learning. We use this framework to design a novel self-supervised task, which achieves state-of-the-art performance on the common benchmarks in PASCAL VOC 2007, ILSVRC12 and Places by a significant margin. Our learned features shrink the mAP gap between models trained via self supervised learning and supervised learning from 5.9% to 2.6% in object detection on PASCAL VOC 2007.

Item Type:

Conference or Workshop Item (Paper)

Division/Institute:

08 Faculty of Science > Institute of Computer Science (INF)

UniBE Contributor:

Noroozi, Mehdi, Favaro, Paolo

Subjects:

000 Computer science, knowledge & systems
500 Science > 510 Mathematics

Language:

English

Submitter:

Xiaochen Wang

Date Deposited:

28 May 2019 15:38

Last Modified:

05 Dec 2022 15:26

BORIS DOI:

10.7892/boris.126520

URI:

https://boris.unibe.ch/id/eprint/126520

Actions (login required)

Edit item Edit item
Provide Feedback