Style Adaptive Semantic Image Editing with Transformers

Abstract

The goal of semantic image editing is to modify an image based on an input semantic label map, to carry out the necessary image manipulation. Existing approaches typically lack control over the style of the editing, resulting in insufficient flexibility to support the desired level of customization, e.g. , to turn an object into a particular style or to pick a specific instance. In this work, we propose Style Adaptive Semantic Image Editing (SASIE), where a reference image is used as an additional input about style, to guide the image manipulation process in a more adaptive manner. Moreover, we propose a new transformer-based architecture for SASIE, in which intra-/inter-image multi-head self-attention blocks transfer intra-/inter-knowledge. The content of the edited areas is synthesized according to the given semantic label, while the style of the edited areas is inherited from the reference image. Extensive experiments on multiple datasets suggest that our method is highly effective and enables customizable image manipulation.