Towards Quantitative Evaluation Metrics for Image Editing Approaches

Abstract

In the rapidly evolving field of Generative AI, this work takes initial steps towards establishing a systematic approach for comparing image editing methods. Currently, there is a lack of quantitative metrics for evaluating image editing tasks, with new methods being evaluated mostly qualitatively. Our methodology involves three key components: 1) The creation of a large synthetic dataset using GAN-Control, which enables the generation of ground-truth images for consistent edits across different facial identities; 2) A matching procedure that pairs the edited images with their corresponding ground-truth; and 3) Application of the Perceptual Distance metric to matched pairs. We assessed the effectiveness of our proposed framework through a user study and a set of simulation experiments. Our results indicate that our approach can rank image-editing methods in a way that aligns with human judgment. This research seeks to lay the foundation for a comprehensive evaluation framework for image editing techniques in subsequent studies, initiating a dialogue on this topic.