On Fast Computation of Nested Cross-Validation

Abstract

Cross-validation is a resampling procedure that provides a point estimate of prediction error. Uncertainty in this estimate is challenging to quantify. Nested Cross-validation addresses this by providing a prediction interval. However, computing this interval can be infeasible as it requires running the cross-validation procedure and refitting the model an extraordinary number of times. Under penalized regression, we provide a fast and efficient way to compute the prediction interval with a single model fit. We give a thorough analysis of our proposed method, from the theoretical property of its core computation technique to its implementation and complexity. Our method performs best when the number of folds scales with sample size and reduce the computation by O(n), where n is the sample size. For application, we evaluate nested cross-validation on tuning parameters for signal regression models, comparing results with restricted maximum likelihood and other cross-validation based procedures.