Setwise comparison: efficient fine-grained rating of movement videos using algorithmic support - a proof of concept study.

Steinheimer, Saskia; Dorn, Jonas F; Morrison, Cecily; Sarkar, Advait; D'Souza, Marcus; Boisvert, Jacques; Bedi, Rishi; Burggraaff, Jessica; Kontschieder, Peter; Dahlke, Frank; Sellen, Abigail; Uitdehaag, Bernard M J; Kappos, Ludwig; Kamm, Christian P. (2020). Setwise comparison: efficient fine-grained rating of movement videos using algorithmic support - a proof of concept study. Disability and Rehabilitation, 42(18), pp. 2640-2646. Taylor & Francis 10.1080/09638288.2018.1563832

Full text not available from this repository.

PURPOSE

Clinical ordinal rating scales of movements, e.g., the Expanded Disability Status Scale, have poor intra- and interrater reliability, are insensitive to subtle differences and result in coarse-grained ratings compared to relative comparative rating methods. We therefore established video-based setwise comparison as a fine-grained, reliable and efficient rating method of motor dysfunction using algorithmic support.

MATERIALS AND METHODS

Eight neurologists rated a set of 40 multiple sclerosis patient videos of the Finger-to-Nose-Test using both the newly developed setwise comparison and the established pairwise comparison techniques, which result in a continuous rating scale. Reliability was assessed by the intra-class correlation coefficient. Construct validity was estimated as Pearson's correlation between the continuous scale and severity ratings according to the Neurostatus scale for upper-extremity tremor/dysmetria and the Nine-hole-peg-test. Comparing the time needed for ratings assessed efficiency.

RESULTS

Intra-class correlation coefficient was 0.83 for setwise and 0.7 for pairwise comparison. Correlation to the tremor/dysmetria score of the Neurostatus was 0.86 for both rating procedures and correlation to the Nine-hole-peg-test was 0.64 (setwise) and 0.66 (pairwise). The time needed to rate 40 videos was 22.9 ± 6.9 minutes (setwise) and 77.8 ± 14.5 minutes (pairwise).

CONCLUSIONS

Setwise comparison is an efficient, valid and reliable method for fine-grained rating of motor dysfunction that can be applied to larger datasets. It is substantially more efficient than pairwise comparison. Implications for rehabilitation Disability rating is crucial in clinical neurorehabilitation and in clinical trials. Humans are naturally inconsistent in rating items on ordinal scales leading to poor intra- and interrater reliability, insensitivity to subtle differences and coarse-grained ratings. Video-based setwise comparison is a new rating method enabling fine-grained, reliable and efficient ratings of motor dysfunction using algorithmic support.

Item Type:	Journal Article (Original Article)
Division/Institute:	04 Faculty of Medicine > Department of Head Organs and Neurology (DKNS) > Clinic of Neurology
UniBE Contributor:	Steinheimer, Saskia Marie, Kamm, Christian Philipp
Subjects:	600 Technology > 610 Medicine & health
ISSN:	0963-8288
Publisher:	Taylor & Francis
Language:	English
Submitter:	Chantal Kottler
Date Deposited:	19 Dec 2019 11:26
Last Modified:	05 Dec 2022 15:33
Publisher DOI:	10.1080/09638288.2018.1563832
PubMed ID:	30782055
Uncontrolled Keywords:	Multiple sclerosis disability disease progression motor dysfunction setwise comparison video rating
URI:	https://boris.unibe.ch/id/eprint/136485