Towards Explainable Neural Networks for Mathematical Programming

Abstract

Mathematical programming has primarily been a study of numerical optimization where the solution is obtained following a procedure recursively until convergence. Recent applications of Neural Networks (NN) as surrogate models have challenged this view by treating optimization problems as approximating unobserved functions that map input parameters to optimal solutions, which is referred to as \textit{solution functions}. In this thesis, we investigate properties of the solution function, and develop NN based methods for deriving explicit formulae of this mapping. Drawing from the principles of the Universal Approximation Theorem, we investigate the sources of NN approximation errors that arise when training models on dataset consists of input parameters and optimal solutions for previously solved problems.

Drawing on insights from multiparametric programming (MP), which explores piecewise-linear (PWL) properties of quadratic programs with linear constraints (QP), we demonstrate that a NN with ReLU activation functions is a general form of the solution function of QPs. Despite this fact, we show that achieving perfect representations of this function proves difficult when employing black-box models and standard NN training methods. We propose a semi-supervised NN model that learns explicit parameters of each linear segment of the QP solution function from problem coefficients analytically, but is only trained for a few number of model parameters to construct the solution function. Using IEEE DC optimal power flow test sets, we show that this approach accurately learns to represent the QP solution function with minimal error, ensuring optimal solutions for any right-hand side (RHS) parameters.

Building on the semi-supervised model, we further propose a closed-form NN model and a learning via discovery algorithm that learns an exact representation of the solution function without training, by applying algebraic operations to derive all model weights from problem coefficients without the need for training data. Our proposed learning algorithm begins with an optimal solution to initiate the NN model, then discovers each linear segment of the PWL solution function and expands the model parameters iteratively without using solvers. The proposed procedure yields a NN model that acts as the closed-form solution to QP with linear constraints, ensuring that optimal solutions for the discovered regions are both feasible and optimal. It is shown that the proposed CF-NN model offer significant advantages. The model takes seconds to learn the solution function and can seamlessly generalize outside the training distribution. Results on IEEE DC-OPF test cases significantly outperforms traditional deep NN training methods, achieving optimal solutions with near-zero calculation errors in all discovered critical regions.

In this thesis, we primarily focused on methodologies for uncertain parameters added to the right hand side of equality constraints. Further work is needed to examine the effect of uncertainty on inequality constraints, improve the scalability of the approach to larger systems by ensuring the learning algorithm to visit all critical regions of the feasible domain, reducing floating point precision errors, and addressing the issues related to degeneracy.