The Cost of Build Tool Downgrades: An Empirical Study of the Kubernetes Project

Abstract

Software build tools automate the transformation of source code into deliverables. Since developers invoke builds multiple times per day, the performance of a build tool directly impacts developer productivity. Motivated by the potential of productivity improvements, artifact-based build tools have emerged to accelerate builds. Despite their advantages, recent work shows that a considerable proportion of projects that adopt these tools later downgrade to others. While prior work has explained the rationale of build tool downgrades, the cost of these downgrades in terms of build duration and computational resource usage is not well understood. Without such an understanding, software teams may make under-informed decisions about adopting or abandoning build tools.

In this thesis, we conduct an empirical study of the performance penalties associated with the downgrade of build tools in the Kubernetes project, focusing on its downgrade from an artifact-based build tool (Bazel) to a language-specific solution (Go Build). We reproduce and analyze full and incremental builds of change sets during the period leading up to the downgrade event. Our results show that, on the one hand, Bazel builds are significantly shorter than Go Build ones, specifically 23.06–38.66 incremental builds.

On the other hand, Bazel builds impose a larger memory footprint than Go Build (81.42–351.07 builds) and greater CPU load at parallelism settings beyond eight for full builds and beyond one for incremental builds. To understand the financial impact of abandoning Bazel, we further analyze the costs of resource consumption during continuous integration builds. We find that Bazel tends to be less costly than Go Build, with the estimated additional financial costs of downgrading ranging from 22.62–39.14 reaching up to 75.92 our findings by replicating our Kubernetes study on smaller projects, we observe that build tool downgrades have a smaller impact on build durations for codebases smaller than Kubernetes; however, artifact-based build tools consistently impose a larger memory footprint than their replacements.

We conclude that abandoning artifact-based build tools, despite perceived maintainability benefits, tends to incur considerable performance costs for large projects. This trade-off between tool complexity and efficiency highlights the need for informed decision-making and lays the groundwork for future research to develop tools that support balancing these trade-offs and more pragmatic decision-making about build tool adoption.