Setwise comparison: efficient fine-grained rating of movement videos using algorithmic support - a proof of concept study.

Steinheimer, Saskia; Dorn, Jonas F; Morrison, Cecily; Sarkar, Advait; D'Souza, Marcus; Boisvert, Jacques; Bedi, Rishi; Burggraaff, Jessica; Kontschieder, Peter; Dahlke, Frank; Sellen, Abigail; Uitdehaag, Bernard M J; Kappos, Ludwig; Kamm, Christian P. (2020). Setwise comparison: efficient fine-grained rating of movement videos using algorithmic support - a proof of concept study. Disability and Rehabilitation, 42(18), pp. 2640-2646. Taylor & Francis 10.1080/09638288.2018.1563832

Full text not available from this repository. (Request a copy)


Clinical ordinal rating scales of movements, e.g., the Expanded Disability Status Scale, have poor intra- and interrater reliability, are insensitive to subtle differences and result in coarse-grained ratings compared to relative comparative rating methods. We therefore established video-based setwise comparison as a fine-grained, reliable and efficient rating method of motor dysfunction using algorithmic support.


Eight neurologists rated a set of 40 multiple sclerosis patient videos of the Finger-to-Nose-Test using both the newly developed setwise comparison and the established pairwise comparison techniques, which result in a continuous rating scale. Reliability was assessed by the intra-class correlation coefficient. Construct validity was estimated as Pearson's correlation between the continuous scale and severity ratings according to the Neurostatus scale for upper-extremity tremor/dysmetria and the Nine-hole-peg-test. Comparing the time needed for ratings assessed efficiency.


Intra-class correlation coefficient was 0.83 for setwise and 0.7 for pairwise comparison. Correlation to the tremor/dysmetria score of the Neurostatus was 0.86 for both rating procedures and correlation to the Nine-hole-peg-test was 0.64 (setwise) and 0.66 (pairwise). The time needed to rate 40 videos was 22.9 ± 6.9 minutes (setwise) and 77.8 ± 14.5 minutes (pairwise).


Setwise comparison is an efficient, valid and reliable method for fine-grained rating of motor dysfunction that can be applied to larger datasets. It is substantially more efficient than pairwise comparison. Implications for rehabilitation Disability rating is crucial in clinical neurorehabilitation and in clinical trials. Humans are naturally inconsistent in rating items on ordinal scales leading to poor intra- and interrater reliability, insensitivity to subtle differences and coarse-grained ratings. Video-based setwise comparison is a new rating method enabling fine-grained, reliable and efficient ratings of motor dysfunction using algorithmic support.

Item Type:

Journal Article (Original Article)


04 Faculty of Medicine > Department of Head Organs and Neurology (DKNS) > Clinic of Neurology

UniBE Contributor:

Steinheimer, Saskia Marie and Kamm, Christian Philipp


600 Technology > 610 Medicine & health




Taylor & Francis




Chantal Kottler

Date Deposited:

19 Dec 2019 11:26

Last Modified:

04 Sep 2020 01:30

Publisher DOI:


PubMed ID:


Uncontrolled Keywords:

Multiple sclerosis disability disease progression motor dysfunction setwise comparison video rating


Actions (login required)

Edit item Edit item
Provide Feedback