Magic Layouts: Structural Prior for Component Detection in User Interface Designs

Dipu Manandhar, Hailin Jin, John Collomosse

CVPR 2021 pp. 15809-15818

doi:10.1109/CVPR46437.2021.01555 /cvpr/2021/manandhar2021cvpr-magic/

Abstract

We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar. Specifically we learn a prior over mobile UI layouts, encoding common spatial co-occurrence relationships between different UI components. Conditioning region proposals using this prior leads to performance gains on UI layout parsing for both hand-drawn UIs and app screenshots, which we demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.

PDF CVPR Semantic Scholar

Cite

Text

Manandhar et al. "Magic Layouts: Structural Prior for Component Detection in User Interface Designs." Conference on Computer Vision and Pattern Recognition, 2021. doi:10.1109/CVPR46437.2021.01555

Markdown

[Manandhar et al. "Magic Layouts: Structural Prior for Component Detection in User Interface Designs." Conference on Computer Vision and Pattern Recognition, 2021.](https://mlanthology.org/cvpr/2021/manandhar2021cvpr-magic/) doi:10.1109/CVPR46437.2021.01555

BibTeX

@inproceedings{manandhar2021cvpr-magic,
  title     = {{Magic Layouts: Structural Prior for Component Detection in User Interface Designs}},
  author    = {Manandhar, Dipu and Jin, Hailin and Collomosse, John},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2021},
  pages     = {15809-15818},
  doi       = {10.1109/CVPR46437.2021.01555},
  url       = {https://mlanthology.org/cvpr/2021/manandhar2021cvpr-magic/}
}