Navigate, Understand, Communicate: How Developers Locate Performance Bugs

Abstract

Background: Performance bugs can lead to severe issues regarding computation efficiency, power consumption, and user experience. Locating these bugs is a difficult task because developers have to judge for every costly operation whether runtime is consumed necessarily or unnecessarily. Objective: We wanted to investigate how developers, when locating performance bugs, navigate through the code, understand the program, and communicate the detected issues.

Method: We performed a qualitative user study observing twelve developers trying to fix documented performance bugs in two open source projects. The developers worked with a profiling and analysis tool that visually depicts runtime information in a list representation and embedded into the source code view.

Results: We identified typical navigation strategies developers used for pinpointing the bug, for instance, following method calls based on runtime consumption. The integration of visualization and code helped developers to understand the bug. Sketches visualizing data structures and algorithms turned out to be valuable for externalizing and communicating the comprehension process for complex bugs.

Conclusion: Fixing a performance bug is a code comprehension and navigation problem. Flexible navigation features based on executed methods and a close integration of source code and performance information support the process.

Supplementary material

  1. Tutorial:
    Here, you can find the slides (PDF) and the video (MP4) we used in the tutorial phase of our study.

  2. Locating Bugs:
    We also provide supplementary material for each research question. The advices we prepared for each bug in case a team got stuck can be found here (ZIP). The questions we asked after each bug fixing session can be found on the slides (PDF).

    RQ1: Navigating and Understanding

    RQ1.1 How was information from the profiling tool or other parts of the IDE used to locate the performance bug?
    Cross-case analysis (in German) (XLSX)

    RQ1.2 Is the in-situ visualization of the profiling data beneficial compared to a traditional list representation?
    Cross-case analysis (in German) (XLSX)

    RQ1.3 What navigation strategies do developers pursue to locate a specific performance bug?
    Interaction logs (ZIP)
    Navigation visualizations (ZIP)
    Screen recordings for Bug 3 (without audio because of confidentiality) (ZIP)

    RQ2: Understanding and Communicating

    RQ2.1 How do developers communicate with each other when locating a performance bug?
    Coding (XLSX)
    Sketches (ZIP)
    Screen recordings for Bug 3 (without audio because of confidentiality) (ZIP)

    RQ2.2 Could sketches help to understand and communicate a performance bug?
    Coding (XLSX)
    Sketches (ZIP)
    Cross-case analysis (in German) (XLSX)
    Sketching videos for Bug 3 (without audio because of confidentiality) (ZIP)

  3. Questionnaire:
    The questionnaire that the participants filled out at the end of the study can be found here (PDF).

Bugs used in the study

Apache Commons Collections:

COLLECTIONS-416 (ListUtils.removeAll() is very slow)
COLLECTIONS-427
(Performance problem in SetUniqueList.retainAll())
COLLECTIONS-429 (Performance problem in MultiValueMap)

Guava: Google Core Libraries for Java

#1155 (Performance of ImmutableSet.contains)

Publications

Sebastian Baltes, Oliver Moseler, Fabian Beck, and Stephan Diehl: Navigate, Understand, Communicate: How Developers Locate Performance Bugs. In: 9th International Symposium on Empirical Software Engineering and Measurement (ESEM'15). To appear. Preprint Slides

Fabian Beck, Oliver Moseler, Stephan Diehl, and Günter Daniel Rey: In situ understanding of performance bottlenecks through visually augmented code. In: 21st International Conference on Program Comprehension (ICPC'13). Preprint

Copyright Notice

The documents distributed by this server have been provided by the contributing authors by means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.