dc.contributor.author | Sahu, Gaurav | |
dc.date.accessioned | 2020-08-28 19:40:04 (GMT) | |
dc.date.available | 2020-08-28 19:40:04 (GMT) | |
dc.date.issued | 2020-08-28 | |
dc.date.submitted | 2020-08-18 | |
dc.identifier.uri | http://hdl.handle.net/10012/16194 | |
dc.description.abstract | Effective fusion of data from multiple modalities, such as video, speech, and text, is a challenging task due to the heterogeneous nature of multimodal data. In this work, we propose fusion techniques that aim to model context from different modalities effectively. Instead of defining a deterministic fusion operation, such as concatenation, for the network, we let the network decide “how” to combine given multimodal features more effectively. We propose two networks: 1) Auto-Fusion network, which aims to compress information from different modalities while preserving the context, and 2) GAN-Fusion, which regularizes the learned latent space given context from complementing modalities. A quantitative evaluation on the tasks of multimodal machine translation and emotion recognition suggests that our adaptive networks can better model context from other modalities than all existing methods, many of which employ massive transformer-based networks. | en |
dc.language.iso | en | en |
dc.publisher | University of Waterloo | en |
dc.subject | multimodal deep learning | en |
dc.subject | multimodal fusion | en |
dc.subject | generative adversarial networks | en |
dc.subject | multimodal machine translation | en |
dc.subject | speech emotion recognition | en |
dc.title | Adaptive Fusion Techniques for Effective Multimodal Deep Learning | en |
dc.type | Master Thesis | en |
dc.pending | false | |
uws-etd.degree.department | David R. Cheriton School of Computer Science | en |
uws-etd.degree.discipline | Computer Science | en |
uws-etd.degree.grantor | University of Waterloo | en |
uws-etd.degree | Master of Mathematics | en |
uws.contributor.advisor | Vechtomova, Olga | |
uws.contributor.affiliation1 | Faculty of Mathematics | en |
uws.published.city | Waterloo | en |
uws.published.country | Canada | en |
uws.published.province | Ontario | en |
uws.typeOfResource | Text | en |
uws.peerReviewStatus | Unreviewed | en |
uws.scholarLevel | Graduate | en |