Learning Visual and Barrel Cortex Representations from Multimodal Convergence

dc.contributor.authorNg, Candace
dc.date.accessioned2026-08-31T18:12:33Z
dc.date.issued2026-08-31
dc.date.submitted2026-08-27
dc.description.abstractMultisensory perception emerges from interactions between sensory pathways, but biologically inspired computational models often examine sensory modalities in isolation. This thesis presents a computational model of mouse visual and tactile processing through whiskers that incorporates anatomically motivated cortical pathways and modulation between sensory modalities. The visual pathway was based on MouseNet, a convolutional neural network derived from the organization of the mouse visual cortex. The whisker pathway was designed to reflect the hierarchical and temporal processing of signals through the somatosensory cortex and higher cortical regions. Representations produced from whisker input were used to modulate activity in the superficial layers of the primary visual cortex through a learned inhibitory gate motivated by experimentally observed somatosensory influences on visual processing. The visual and whisker encoders were trained jointly using a contrastive learning objective that aligned representations from paired simulated sensory observations. Training produced substantially greater similarity and retrieval performance for matched visual and whisker samples than for untrained representations, indicating that the model learned a shared embedding space across modalities. To evaluate how multimodal training shaped the individual sensory pathways, visual and whisker representations were separately compared with modality-specific neural recordings using representational similarity analysis (RSA). The learned visual representations exhibited measurable correspondence with recorded visual cortical activity. Enabling the inhibitory gate modestly increased neural alignment relative to disabling it within the same trained models, particularly in VISp layer 5, but gated models did not reliably outperform models trained without the gate. Orientation selectivity increased in most evaluated visual layers despite not being explicitly included in the training objective, and trained VISp values were broadly comparable to physiological measurements from mouse visual cortex. Whisker representations retained condition-dependent structure but did not show consistent positive correspondence with the neural geometry, with animal-level RSA varying in sign and magnitude. These results demonstrate a framework for integrating visual and tactile processing within a biologically grounded neural network and evaluating the resulting representations directly against experimental neural data. This work provides a foundation for investigating how interactions between sensory modalities and anatomical constraints shape cortical representations.
dc.identifier.urihttps://hdl.handle.net/10012/24139
dc.language.isoen
dc.pendingfalse
dc.publisherUniversity of Waterlooen
dc.subjectmultimodal representation learning
dc.subjectbiologically inspired neural networks
dc.subjectrepresentational similarity analysis
dc.titleLearning Visual and Barrel Cortex Representations from Multimodal Convergence
dc.typeMaster Thesis
uws-etd.degreeMaster of Applied Science
uws-etd.degree.departmentSystems Design Engineering
uws-etd.degree.disciplineSystem Design Engineering
uws-etd.degree.grantorUniversity of Waterlooen
uws-etd.embargo.terms0
uws.contributor.advisorTripp, Bryan
uws.contributor.affiliation1Faculty of Engineering
uws.peerReviewStatusUnrevieweden
uws.published.cityWaterlooen
uws.published.countryCanadaen
uws.published.provinceOntarioen
uws.scholarLevelGraduateen
uws.typeOfResourceTexten

Files

Original bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
Ng_Candace.pdf
Size:
3.32 MB
Format:
Adobe Portable Document Format

License bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
license.txt
Size:
6.4 KB
Format:
Item-specific license agreed upon to submission
Description: