Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards
Abstract
Generating accurate descriptions for online fashion items is important not only for enhancing customers' shopping experiences, but also for the increase of online sales. Besides the need of correctly presenting the attributes of items, the expressions in an enchanting style could better attract customer interests. The goal of this work is to develop a novel learning framework for accurate and expressive fashion captioning. Different from popular work on image captioning, it is hard to identify and describe the rich attributes of fashion items. We seed the description of an item by first identifying its attributes, and introduce attribute-level semantic (ALS) reward and sentence-level semantic (SLS) reward as metrics to improve the quality of text descriptions. We further integrate the training of our model with maximum likelihood estimation (MLE), attribute embedding, and Reinforcement Learning (RL). To facilitate the learning, we build a new FAshion CAptioning Dataset (FACAD), which contains 800K images and 120K corresponding enchanting and diverse descriptions. Experiments on FACAD demonstrate the effectiveness of our model.
Cite
Text
Yang et al. "Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards." Proceedings of the European Conference on Computer Vision (ECCV), 2020. doi:10.1007/978-3-030-58601-0_1Markdown
[Yang et al. "Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards." Proceedings of the European Conference on Computer Vision (ECCV), 2020.](https://mlanthology.org/eccv/2020/yang2020eccv-fashion/) doi:10.1007/978-3-030-58601-0_1BibTeX
@inproceedings{yang2020eccv-fashion,
title = {{Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards}},
author = {Yang, Xuewen and Zhang, Heming and Jin, Di and Liu, Yingru and Wu, Chi-Hao and Tan, Jianchao and Xie, Dongliang and Wang, Jue and Wang, Xin},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2020},
doi = {10.1007/978-3-030-58601-0_1},
url = {https://mlanthology.org/eccv/2020/yang2020eccv-fashion/}
}