Comparing Distributions with the Probability of Agreement

Abstract

In this thesis we adapt the probability of agreement (PoA) methodology for the comparison of distributions. Most of the commonly used methods for comparing distributions are rooted in hypothesis testing where decisions are made using p-values. The proposed methodology, however, provides a more context-driven comparison by accounting for practically important differences. Two situations are considered: first, the one-sample comparison problem in which we have observed one sample and interest lies in determining whether the sample comes from a given known distribution. Second, we consider the two-sample comparison of distributions in which we have observed two independent samples and interest lies in determining whether these samples have the same distribution. The Horvitz-Thompson estimator is used to estimate the cumulative distribution function(s) corresponding the sample(s) under comparison and the asymptotic normality of the Horvitz-Thompson estimator is used to estimate the PoA. Confidence intervals (CIs) are also determined for the estimated PoA so as to quantify estimation uncertainty. We develop two methods for calculating CIs: one based on asymptotic normality and the delta-method and the other based on the bootstrap. To illustrate the application and interpretation of the methodology, we consider both real world and simulated examples. We also conduct a simulation study that evaluates the bias and variance of the PoA estimator as well as the coverage of the associated CIs. Finally we propose the relative density methodology as a graphical supplement that provides further information about the similarities and differences between the distributions under comparison. In summary, the contributions of this thesis are (1), the generalization of the PoA methodology to the one- and two-sample comparison of distributions, and (2), the suggestion of using the relative density and the PoA methodologies in tandem to gain more thorough information about the similarities and differences between the distributions under comparison.