Estimating and Improving the Performance of Prediction Models for Regression Test Selection

Abstract

Researchers have proposed models to predict the percentage of the selected test cases when a Regression Test Selection (RTS) technique is used. One of the most successful and best performing RTS predictors is the Rosenblum and Weyuker (RW) coverage-based prediction model. However, previous evaluation results on RW predictor show that although it performs well on some subject programs, it deviates from actual percentage significantly on others. To understand the reason impacting RW predictor's performance, this research work presents a set of experiments on four factors that can potentially impact the RTS prediction performance. We setup two different set of experiments on several Java open-source test subjects and three RTS techniques. Our study on the effect of each factor on the RW performance reveals that large amount of code changes and significant code coverage overlaps between test cases are the two factors contributing to RW predictor's prediction error. Based on the experimental results and through regression analysis of the impacting factors, we propose a RW error estimator that can help testers and developers to gain a better understanding of RW predictor's confidence level and get insight into the applicability of the RW predictor to different organizations products and processes. To further improve RW predictor's performance, we propose an improved RW prediction model utilizing the error estimator to compensate prediction error. We also design a specific RTS improvement technique while presenting Harrold et al's improvement which also incorporates change history. Our experiments on these improved RW predictors demonstrate that they can reduce RW prediction error and improve performance.