An Exploratory Study on the Usage of Gherkin Features in Open-Source Projects

Chandorkar, Adwait; Patkar, Nitish; Sorbo, Andrea Di; Nierstrasz, Oscar (March 2022). An Exploratory Study on the Usage of Gherkin Features in Open-Source Projects. In: Workshop on Validation, Analysis and Evolution of Software Tests (VST 2022, co-located with SANER 2022). 10.1109/SANER53432.2022.00134

Preview

Text Patk22b.pdf - Accepted Version Available under License Publisher holds Copyright. Download (403kB) | Preview

With behavior-driven development (BDD), domain experts describe system behavior and desired outcomes through natural language-like sentences, e.g., using the Gherkin language. BDD frameworks partially convert the content of Gherkin specifications into executable test code. Previous studies have reported several issues with the current BDD practice, for example long repetitive Gherkin specifications and slow-running test suites. Data tables and additional features were added to the Gherkin syntax to express compactly test inputs (e.g., provide different combinations of input values and desired outputs to run tests multiple times) and also to improve the readability of Gherkin files (henceforth called spec files). However, there is no empirical evidence about the actual usage of these Gherkin features. To fill this gap, we analyzed the content of 1,572 spec files extracted from 23 open-source projects. For each spec file, we collected a set of metrics modeling the structure and the usage of the different Gherkin features. We found that only a minority of the considered spec files (i.e., 590) used data tables that contain two rows, on average. We also used statistical tests to compare the contents of spec files with and without data tables and found significant differences between the two populations, especially for what concerns the number of lines of code (LoC). On the one hand, our results shed some light on the discrepancies between the recommendations for defining Gherkin specifications and their actual adoption in practice. On the other hand, our findings demonstrate that the adoption of additional features, such as data tables, might only partially help to reduce the length of Gherkin specifications.

Interest & Impact

Downloads

54 since deposited on 15 Aug 2022
44 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item