Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

Xu, Shuqi; Mariani, Manuel Sebastian; Lü, Linyuan; Medo, Matúš (2020). Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data. Journal of informetrics, 14(1), p. 101005. Elsevier 10.1016/j.joi.2019.101005

[img] Text
1-s2.0-S1751157719301646-main.pdf - Published Version
Restricted to registered users only
Available under License Publisher holds Copyright.

Download (4MB) | Request a copy

Despite the increasing use of citation-based metrics for research evaluation purposes, we do not know yet which metrics best deliver on their promise to gauge the significance of a scientific paper or a patent. We assess 17 network-based metrics by their ability to identify milestone papers and patents in three large citation datasets. We find that traditional information-retrieval evaluation metrics are strongly affected by the interplay between the age distribution of the milestone items and age biases of the evaluated metrics. Outcomes of these metrics are therefore not representative of the metrics’ ranking ability. We argue in favor of a modified evaluation procedure that explicitly penalizes biased metrics and allows us to reveal metrics’ performance patterns that are consistent across the datasets. PageRank and LeaderRank turn out to be the best-performing ranking metrics when their age bias is suppressed by a simple transformation of the scores that they produce, whereas other popular metrics, including citation count, HITS and Collective Influence, produce significantly worse ranking results.

Item Type:

Journal Article (Original Article)

Division/Institute:

04 Faculty of Medicine > Pre-clinic Human Medicine > BioMedical Research (DBMR) > DBMR Forschung Mu35 > Forschungsgruppe Radio-Onkologie
04 Faculty of Medicine > Pre-clinic Human Medicine > BioMedical Research (DBMR) > DBMR Forschung Mu35 > Forschungsgruppe Radio-Onkologie

04 Faculty of Medicine > Department of Haematology, Oncology, Infectious Diseases, Laboratory Medicine and Hospital Pharmacy (DOLS) > Clinic of Radiation Oncology

UniBE Contributor:

Medo, Matúš

Subjects:

600 Technology > 610 Medicine & health

ISSN:

1751-1577

Publisher:

Elsevier

Language:

English

Submitter:

Beatrice Scheidegger

Date Deposited:

09 Dec 2020 18:04

Last Modified:

13 Dec 2020 02:51

Publisher DOI:

10.1016/j.joi.2019.101005

BORIS DOI:

10.7892/boris.148445

URI:

https://boris.unibe.ch/id/eprint/148445

Actions (login required)

Edit item Edit item
Provide Feedback