Continuous Spatial and Temporal Representations in Machine Vision

Abstract

This thesis explores continuous spatial and temporal representations in machine vision. For spatial representations, we explore the Spatial Semantic Pointer as a biologically plausible representation of continuous space its use in performing spatial memory and reasoning tasks. We show that SSPs can be used to encode visual images into high dimensional memory vectors. These vectors can be used to store, retrieve, and manipulate spatial information, as well as perform search and scanning tasks within the vector algebra space. We also demonstrate the psychological plausibility of these representations by qualitatively reproducing Kosslyn's famous map scanning experiment.

For temporal representations, we extend the original 1D Legendre Memory Unit to take multi-dimensional input signals and compare its ability to store temporal information against the Long Short-Term Memory Unit on the task of video action recognition. We show that the multi-dimensional LMU is able to match the LSTM in representing visual data over time. In particular, we demonstrate that the LMU is able to achieve much better performance when the total number of parameters is limited and that the LMU architecture allows it to continue operating at with fewer parameters than the LSTM.