The determination of structured Hessian matrices via automatic differentiation

Abstract

In using automatic differentiation (AD) for Hessian computation, efficiency can be achieved by exploiting the sparsity existing in the derivative matrix. However, in the case where the Hessian is dense, this cannot be done and the space requirements to compute the Hessian can become very large. But if the underlying function can be expressed in a structured form, a “deeper” sparsity can be exploited to minimize the space requirement. In this thesis, we provide a summary of automatic differentiation (AD) techniques, as applied to Jacobian and Hessian matrix determination, as well as the graph coloring techniques involved in exploiting their sparsity. We then discuss how structure in the underlying function can be used to greatly improve efficiency in gradient/Jacobian computation. We then propose structured methods for Hessian computation that substantially reduce the space required. Finally, we propose a method for Hessian computation where the structure of the function is not provided.