Rigorous Validation of Hydrologic Models in Support of Decision-Making

Abstract

Hydrologic models are often relied upon to inform decisions in hydrology and water resources applications. Typically, hydrologic models are validated (i.e., deemed fit-for-purpose) using the split-sample test introduced by Klemeš in 1986, where a model is shown to reproduce historical data that was not used in training the model. However, simple history matching is a necessary but insufficient condition to show reliability in decision-making, and there are many examples in the literature of models with excellent diagnostic metrics but insufficient skill for informing decisions. Furthermore, the current methods of model validation and uncertainty estimation are not easily understood by non-modellers, and decision-makers are often handed model outputs that are not in a readily usable form to inform decision-making. There exist many calls in literature for both the development of rigorous model validation methodologies, and techniques to address the gap between scientists and decision-makers.

As a response to the gap in literature for new model validation methodologies and ways of improving communication between scientists and decision-makers, this thesis introduces Decision Crash Testing (DCT), which is capable of directly evaluating a model’s performance in a decision-making context. The DCT framework uses a series of synthetic reality experiments to recreate the model-building process and test whether the model correctly informs the decision in a set of hypothetical realities. These provide a baseline estimate of the difficulty of the decision and the probability that the model is capable of informing the correct decision, which is much more easily interpretable by decision-makers.

The DCT framework is demonstrated using two case studies derived from reservoir management applications in an Ontario watershed. The development of the hydrologic models that underlie both case studies is a deviation from the typical approach of using level-pool routed inflow estimates for model calibration; instead, the observed stage data is calibrated to directly, and the reservoirs are explicitly represented in the hydrologic model. This method avoids some of the pitfalls of calibrating to estimated inflows with known numerical artefacts. The two case studies illustrate the ability of the DCT framework to assess the decision-making ability of the evaluated model, to provide a framework within which to meaningfully assess improvements the evaluated model for specific decision-making applications, and to test the impacts of various decision formulations on the ability of the model to inform decision-making.