Categories as a Foundation for both Learning and Reasoning

Abstract

This thesis explores two distinct research topics, both applying category theory to machine learning.

The first topic discusses Vector Symbolic Architectures (VSAs). I present the first attempt at formalising VSAs with category theory. VSAs are built to perform symbolic reasoning in high-dimensional vector spaces. I present a brief literature survey demonstrating that the topic is currently completely unexplored. I discuss some desiderata for VSA models, then describe an initial formalisation that covers two of the three desiderata. My formalisation focuses on two of the three primary components of a VSA: binding and bundling, and presents a proof of why element-wise operations constitute the ideal means of performing binding and bundling. The work extends beyond vectors, to any co-presheaves with the desired properties. For example, GHRR representations are captured by this generalisation.

The second line of work discusses, and expands upon, recent work by Milewski in the construction of "pre-lenses." This work is motivated by pre-established formalisations of supervised machine learning. From the perspective of category theory, pre-lenses are interesting because they unify the category Para, or Learn, with its dual co-Para, or co-Learn. From a computer science perspective, pre-lenses are interesting because they enable programmers to build neural networks with vanilla function composition, and they unify various network features by leveraging the fact that they are profunctors. I replicate Milewski's code, extend it to the non-synthetic data, MNIST, implement re-parameterisations, and describe generative models as dual to discriminative models by way of pre-lenses. This work involved creating a simple dataloader to read in external files, randomising the order that inputs are presented during learning, and fixing some bugs that didn't manifest when training occurred on the very small dataset used by Milewski.