A Practitioner's Guide to Continual Multimodal Pretraining

Abstract

Multimodal foundation models, despite being extensively pretrained, become outdated over time. Research into continual pretraining mainly explores (1) infrequent, indiscriminate updates on large-scale new data, or (2) frequent, sample-level updates. However, practical model deployment often operates in the gap between these limit cases, as real-world applications require continual adaptation to specific subdomains or tasks. In this work, we complement current approaches through a new, continual multimodal pretraining test bed with realistic compute constraints and practical deployment requirements (\texttt{FoMo-in-Flux}), and provide \textit{comprehensive practical guidance} for effective continual model updates---investigating different method choices, pipeline design and data-centric deployment scenarios.