Learning Visual and Barrel Cortex Representations from Multimodal Convergence
Loading...
Files
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Waterloo
Abstract
Multisensory perception emerges from interactions between sensory pathways, but biologically inspired computational models often examine sensory modalities in isolation. This thesis presents a computational model of mouse visual and tactile processing through whiskers that incorporates anatomically motivated cortical pathways and modulation between sensory modalities. The visual pathway was based on MouseNet, a convolutional neural network derived from the organization of the mouse visual cortex. The whisker pathway was designed to reflect the hierarchical and temporal processing of signals through the somatosensory cortex and higher cortical regions. Representations produced from whisker input were used to modulate activity in the superficial layers of the primary visual cortex through a learned inhibitory gate motivated by experimentally observed somatosensory influences on visual processing. The visual and whisker encoders were trained jointly using a contrastive learning objective that aligned representations from paired simulated sensory observations. Training produced substantially greater similarity and retrieval performance for matched visual and whisker samples than for untrained representations, indicating that the model learned a shared embedding space across modalities. To evaluate how multimodal training shaped the individual sensory pathways, visual and whisker representations were separately compared with modality-specific neural recordings using representational similarity analysis (RSA). The learned visual representations exhibited measurable correspondence with recorded visual cortical activity. Enabling the inhibitory gate modestly increased neural alignment relative to disabling it within the same trained models, particularly in VISp layer 5, but gated models did not reliably outperform models trained without the gate. Orientation selectivity increased in most evaluated visual layers despite not being explicitly included in the training objective, and trained VISp values were broadly comparable to physiological measurements from mouse visual cortex. Whisker representations retained condition-dependent structure but did not show consistent positive correspondence with the neural geometry, with animal-level RSA varying in sign and magnitude. These results demonstrate a framework for integrating visual and tactile processing within a biologically grounded neural network and evaluating the resulting representations directly against experimental neural data. This work provides a foundation for investigating how interactions between sensory modalities and anatomical constraints shape cortical representations.