Gated Multimodal Units for Information Fusion
Abstract
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
Cite
Text
Ovalle et al. "Gated Multimodal Units for Information Fusion." International Conference on Learning Representations, 2017.Markdown
[Ovalle et al. "Gated Multimodal Units for Information Fusion." International Conference on Learning Representations, 2017.](https://mlanthology.org/iclr/2017/ovalle2017iclr-gated/)BibTeX
@inproceedings{ovalle2017iclr-gated,
title = {{Gated Multimodal Units for Information Fusion}},
author = {Ovalle, John Edison Arevalo and Solorio, Thamar and Montes-y-Gómez, Manuel and González, Fabio A.},
booktitle = {International Conference on Learning Representations},
year = {2017},
url = {https://mlanthology.org/iclr/2017/ovalle2017iclr-gated/}
}