Adapting Blackbox Generative Models via Inversion

Abstract

Adapting large-scale generative AI tools to differ- ent end uses continues to be challenging, as many industry grade image generator models are not publicly available. Thus, to finetune an industry grade image generator is not currently feasible in the classical sense of finetuning certain layers of a given deep-network. Instead, we present an alternative perspective for the problem of adapt- ing large-scale generative models that does not require access to the full model. Recognizing the expense of storing and fine-tuning generative models, as well as the restricted access to weights and gradients (often limited to API calls only), we introduce AdvIN (Adapting via Inversion). This approach advocates the use of inversion methods, followed by training a latent generative model as being equivalent to adaptation. We evaluate the feasibility of such a framework on StyleGANs with real distribution shifts, and outline some open research questions. Even with simple in- version and latent generation strategies, AdvIN is surprisingly competitive to fine-tuning based methods, making it a promising alternative for end-to-end fine-tuning