Estimating Risk-adjusted Process Performance with a Bias/Variance Trade-off

Abstract

Decision makers responsible for managing the performance of a process commonly base their decisions on an estimate of present performance, a comparison of estimates across multiple streams, and the trend in performance estimates over time. Their decisions are well-informed when the risk-adjusted estimates of the performance measure (or parameter) are accurate and precise. The work is motivated by three applications to estimate a parameter at the present time from a stream of data where the parameter drifts slowly in an unpredictable way over time. It is common practice to estimate its value using either present time data only or using present and historical data. When sample sizes by time period are small, an estimate based on present time data is imprecise and can lead to uninformative or misleading conclusions. We can choose to estimate the parameter using an aggregate of historical and present time data but this choice trades more bias for less variability when the parameter is drifting over time. We propose to regulate the bias/variance trade-off using estimating equations that down-weight past data. We derive approximations for the variance of the estimator and the distribution of a hypothesis test statistic involving the estimator through known asymptotic properties of the estimating functions. We study the proposed approach relative to current practices with real or realistic data from each application. We offer simulations and analytic examples to generalize the comparisons and validate the approximations. We explore considerations related to implementing the proposed approach. We suggest future work to extend the applicability of this work.