A Systematic Literature Review of Software Visualization Evaluation

Merino, Leonel; Ghafari, Mohammad; Anslow, Craig; Nierstrasz, Oscar (2018). A Systematic Literature Review of Software Visualization Evaluation. Journal of systems and software, 144, pp. 165-180. Elsevier 10.1016/j.jss.2018.06.027

[img] Text
1-s2.0-S0164121218301237-main.pdf - Published Version
Restricted to registered users only
Available under License Publisher holds Copyright.

Download (779kB) | Request a copy
Meri18a-postprint.pdf - Accepted Version
Available under License Creative Commons: Attribution-Noncommercial-No Derivative Works (CC-BY-NC-ND).

Download (1MB) | Preview

Abstract Context: Software visualizations can help developers to analyze multiple aspects of complex software systems, but their effectiveness is often uncertain due to the lack of evaluation guidelines.
Objective: We identify common problems in the evaluation of software visualizations with the goal of formulating guidelines to improve future evaluations.
Method: We review the complete literature body of 387 full papers published in the SOFTVIS/VISSOFT conferences, and study 181 of those from which we could extract evaluation strategies, data collection methods, and other aspects of the evaluation.
Results: Of the proposed software visualization approaches, 62 lack a strong evaluation. We argue that an effective software visualization should not only boost time and correctness but also recollection, usability, engagement, and other emotions.
Conclusion: We call on researchers proposing new software visualizations to provide evidence of their effectiveness by conducting thorough (i) case studies for approaches that must be studied in situ, and when variables can be controlled, (ii) experiments with randomly selected participants of the target audience and real-world open source software systems to promote reproducibility and replicability. We present guidelines to increase the evidence of the effectiveness of software visualization approaches, thus improving their adoption rate.

Item Type:

Journal Article (Original Article)


08 Faculty of Science > Institute of Computer Science (INF)
08 Faculty of Science > Institute of Computer Science (INF) > Software Composition Group (SCG)

UniBE Contributor:

Merino, Leonel; Ghafari, Mohammad and Nierstrasz, Oscar


000 Computer science, knowledge & systems








Oscar Nierstrasz

Date Deposited:

05 Jun 2019 10:19

Last Modified:

01 Oct 2020 02:30

Publisher DOI:


Uncontrolled Keywords:

scg-pub snf-asa2 scg18 jb18





Actions (login required)

Edit item Edit item
Provide Feedback