Dimensionality reduction methods in multivariate prediction

Abstract

The estimation of a predictive model for multivariate responses requires the estimation of several parameters. In the presence of several correlated explanatory variables it may be necessary to consider the predictions from a sub-space of the space spanned by the whole set of predictors. This dimensionality reduction decreases the number of parameter to be estimated and may increase the precision of the predictions. In many situations the sample based optimal Least Squares solutions have proved to yield worse predictions that those obtained with heuristic Dimensionality Reduction Methods (DRMs). In this thesis we give a thorough discussion of various DRMs giving novel interpretations. These DRMs include Principal Component Regression (PCR), Partial least Squares (PLS). Reduced Rank Regression (RRR) and Canonical Correlation Regression (CCR). We also discuss the different algorithms proposed for PLS and suggest a modified one for more efficient computation. We introduce a common objective function from which the various DRMs can be obtained as special cases. The common objective function shows that methods like PCR and PLS include in their objective the variance of the predictive space retained by the latent sub-space. We suggest determining the predictive latent space by maximizing simultaneously the variance explained in both the predictive and the explanatory spaces.We call this method Maximum Overall Redundancy (MOR) and introduce a weighted version of it. WMOR. The matrix solution of this method is made up of the convex sum of the matrix that generates the principal components and the atrix that generates the RRR solutions. Letting the weights vary we obtain a continuum of solutions from PCA to RRR. The weight can be determined by optimizing a measure of distance between the latest sub-space and the original spaces. Another way of obtaining latent spaces that retain a good portion of the variance of the original X space is to assign weights iteratively to the RRR solution matrix and deflating the X space of the latent directions previously determined. This method gives good results, similar to those of PLS, in fact, but it is costly in terms of computation. The classical MLE approach based on multinormal assumptions does not seem to provide estimates that are more useful than the sample based ones. In fact, for the Reduced Rank Regression model, these turn out to be the Canonical Correlation solutions and the RRR solutions, which have been out-performed in many applications by other methods. However, we obtain the MLE estimates for the joint reduction of the predictive and response spaces, that is for MOR. By choosing the variances of the errors to have special forms, we can obtain the CCA and RRR solutions and also the sample based MOR and WMOR solutions.

We apply the various DRMs to two data sets. The first one was published as an application of PLS to a Poly-Ethylene reaction. The second was obtained from a simulator of a copolymer reactor. We find that the WMOR methods perform well and are comparable to PLS. We also perform a simulation study to understand the performance of the various DRMs. In the study we consider different data structures for which we compare the performance of the DRMs with respect to the objective function and the sum of squared prediction errors. The study confirms that MOR and WMOR with different weights all perform well in predicting the responses and its results are comparable to those of PLS.