Biological Plausibility in Modern Hopfield Networks
| dc.contributor.author | Snow, Mallory | |
| dc.date.accessioned | 2022-12-19T21:23:24Z | |
| dc.date.available | 2022-12-19T21:23:24Z | |
| dc.date.issued | 2022-12-19 | |
| dc.date.submitted | 2022-12-07 | |
| dc.description.abstract | Modern Hopfield Networks (HNs) have the ability to store a large number of target memories (e.g. binary patterns) and then recall a memory in its entirety when prompted by a sub-set or perturbed version of it; in this sense, these networks demonstrate properties of a Content Addressable Memory (CAM). Associative memory operates in a similar manner, with memories being elicited by cues of partial or noisy information. Hence, these models provide a basis for modelling associative memory in humans. However, these modern HNs often use functions that rely on many-body synapses and are thus not accurate to neurobiology. More biologically realistic versions of modern HNs have been proposed, although these implementations often still utilize the softmax function. Computing the softmax for a single node in a layer requires the knowledge of all other nodes in that layer, resulting in non-local computations. The softmax function also causes each target memory stored in the network to correspond to the activation of a single hidden node. Such one- hot representations are not robust, and not observed in biological neural networks. In this thesis, we investigate more biologically plausible versions of a modern HN that uses the softmax function. Specifically, we introduce a more biological version of this network by using a local softmax function – the locLSE network. To ensure each target memory has a distributed hidden representation, we propose a network that applies population coding to the hidden layer via the Neural Engineering Framework (NEF) – the Distributed locLSE network. Both proposed networks can learn the connection weights using a local learning rule derived from gradient descent on the energy function. It is found that the proposed, more biologically accurate, versions of the original modern HN still demonstrate capabilities of a CAM at similar rates to the original. Lastly, the NEF is applied to the entire locLSE network by implementing it using a software tool called Nengo. The locLSE network in Nengo is found to behave as a CAM for small datasets of target memories, and provided there are enough neurons in the network. | en |
| dc.identifier.uri | http://hdl.handle.net/10012/18965 | |
| dc.language.iso | en | en |
| dc.pending | false | |
| dc.publisher | University of Waterloo | en |
| dc.subject | associative memory | en |
| dc.subject | Hopfield networks | en |
| dc.subject | biological plausibility | en |
| dc.subject | dynamical systems | en |
| dc.title | Biological Plausibility in Modern Hopfield Networks | en |
| dc.type | Master Thesis | en |
| uws-etd.degree | Master of Mathematics | en |
| uws-etd.degree.department | David R. Cheriton School of Computer Science | en |
| uws-etd.degree.discipline | Computer Science | en |
| uws-etd.degree.grantor | University of Waterloo | en |
| uws-etd.embargo.terms | 0 | en |
| uws.contributor.advisor | Orchard, Jeff | |
| uws.contributor.affiliation1 | Faculty of Mathematics | en |
| uws.peerReviewStatus | Unreviewed | en |
| uws.published.city | Waterloo | en |
| uws.published.country | Canada | en |
| uws.published.province | Ontario | en |
| uws.scholarLevel | Graduate | en |
| uws.typeOfResource | Text | en |