Quotation Cerqueira Revoredo, Kate, Djurica, Djordje, Mendling, Jan. 2021. A study into the practice of reporting software engineering experiments. Empirical Software Engineering. 26 (113)




It has been argued that reporting software engineering experiments in a standardized way helps researchers find relevant information, understand how experiments were conducted and assess the validity of their results. Various guidelines have been proposed specifically for software engineering experiments. The benefits of such guidelines have often been emphasized, but the actual uptake and practice of reporting have not yet been investigated since the introduction of many of the more recent guidelines. In this research, we utilize a mixed-method study design including sequence analysis techniques for evaluating to which extent papers follow such guidelines. Our study focuses on the four most prominent software engineering journals and the time period from 2000 to 2020. Our results show that many experimental papers miss information suggested by guidelines, that no de facto standard sequence for reporting exists, and that many papers do not cite any guidelines. We discuss these findings and implications for the discipline of experimental software engineering focusing on the review process and the potential to refine and extend guidelines, among others, to account for theory explicitly.


Press 'enter' for creating the tag

Publication's profile

Status of publication Published
Affiliation WU
Type of publication Journal article
Journal Empirical Software Engineering
Citation Index SCI
WU-Journal-Rating new INF-A
Language English
Title A study into the practice of reporting software engineering experiments
Volume 26
Number 113
Year 2021
URL https://link.springer.com/content/pdf/10.1007/s10664-021-10007-3.pdf
DOI http://dx.doi.org/10.1007/s10664-021-10007-3
Open Access Y
Open Access Link https://doi.org/10.1007/s10664-021-10007-3


Causal Process Mining: Concepts and Tool
Cerqueira Revoredo, Kate (Former researcher)
Djurica, Djordje (Former researcher)
Mendling, Jan (Details)
Institute for Data, Process and Knowledge Management (AE Sabou) (Details)
Google Scholar: Search