A Systematic Literature Review of Software Visualization Evaluation

Merino, Leonel; Ghafari, Mohammad; Anslow, Craig; Nierstrasz, Oscar (2018). A Systematic Literature Review of Software Visualization Evaluation. Journal of systems and software, 144, pp. 165-180. Elsevier 10.1016/j.jss.2018.06.027

	Text 1-s2.0-S0164121218301237-main.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (779kB) | Request a copy
Preview	Text Meri18a-postprint.pdf - Accepted Version Available under License Creative Commons: Attribution-Noncommercial-No Derivative Works (CC-BY-NC-ND). Download (1MB) | Preview

Abstract Context: Software visualizations can help developers to analyze multiple aspects of complex software systems, but their effectiveness is often uncertain due to the lack of evaluation guidelines.
Objective: We identify common problems in the evaluation of software visualizations with the goal of formulating guidelines to improve future evaluations.
Method: We review the complete literature body of 387 full papers published in the SOFTVIS/VISSOFT conferences, and study 181 of those from which we could extract evaluation strategies, data collection methods, and other aspects of the evaluation.
Results: Of the proposed software visualization approaches, 62 lack a strong evaluation. We argue that an effective software visualization should not only boost time and correctness but also recollection, usability, engagement, and other emotions.
Conclusion: We call on researchers proposing new software visualizations to provide evidence of their effectiveness by conducting thorough (i) case studies for approaches that must be studied in situ, and when variables can be controlled, (ii) experiments with randomly selected participants of the target audience and real-world open source software systems to promote reproducibility and replicability. We present guidelines to increase the evidence of the effectiveness of software visualization approaches, thus improving their adoption rate.

Interest & Impact

Downloads

78 since deposited on 05 Jun 2019
50 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item