Scene Designer: A Unified Model for Scene Search and Synthesis from Sketch

Leo Sampaio Ferraz Ribeiro, Tu Bui, John P. Collomosse, Moacir Ponti

ICCVW 2021 pp. 2424-2433

doi:10.1109/ICCVW54120.2021.00275 /iccvw/2021/ribeiro2021iccvw-scene/

Abstract

Scene Designer is a novel method for searching and generating images using free-hand sketches of scene compositions; i.e. drawings that describe both the appearance and relative positions of objects. Our core contribution is a single unified model to learn both a cross-modal search embedding for matching sketched compositions to images, and an object embedding for layout synthesis. We show that a graph neural network (GNN) followed by Transformer under our novel contrastive learning setting is required to allow learning correlations between object type, appearance and arrangement, driving a mask generation module that synthesizes coherent scene layouts, whilst also delivering state of the art sketch based visual search of scenes.

PDF ICCVW Semantic Scholar

Cite

Text

Ribeiro et al. "Scene Designer: A Unified Model for Scene Search and Synthesis from Sketch." IEEE/CVF International Conference on Computer Vision Workshops, 2021. doi:10.1109/ICCVW54120.2021.00275

Markdown

[Ribeiro et al. "Scene Designer: A Unified Model for Scene Search and Synthesis from Sketch." IEEE/CVF International Conference on Computer Vision Workshops, 2021.](https://mlanthology.org/iccvw/2021/ribeiro2021iccvw-scene/) doi:10.1109/ICCVW54120.2021.00275

BibTeX

@inproceedings{ribeiro2021iccvw-scene,
  title     = {{Scene Designer: A Unified Model for Scene Search and Synthesis from Sketch}},
  author    = {Ribeiro, Leo Sampaio Ferraz and Bui, Tu and Collomosse, John P. and Ponti, Moacir},
  booktitle = {IEEE/CVF International Conference on Computer Vision Workshops},
  year      = {2021},
  pages     = {2424-2433},
  doi       = {10.1109/ICCVW54120.2021.00275},
  url       = {https://mlanthology.org/iccvw/2021/ribeiro2021iccvw-scene/}
}