Measuring and Predicting Computer Software Performance: Tools and Approaches

Abstract

Measurement-based software performance evaluation is essential to computer science and industry alike, yet despite its widespread adoption, the current level of statistical rigor is inadequate, putting published results into question: for example, the majority of publications fail to report any dispersion metric at all. To foster widespread adoption of statistically rigorous performance evaluation, the first part of this thesis proposes the use of formal experiment design and non-parametric analysis techniques, and presents a distributed infrastructure that lowers the cost of rigorous experimentation by automating the experiment design and execution process, while minimizing the variability in computer performance response metrics. Then, to address cases where rigorous performance experimentation is infeasible, either due to infrastructure costs or unavailability of target platforms, the second part of this thesis builds on the previously discussed techniques and infrastructure to introduce two performance prediction techniques: one to predict when code changes will cause performance changes during software development, and another to predict performance metrics on unavailable platforms using benchmark-based statistical models.