DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting

Abstract

We propose a novel joint deblurring and multi-frame interpolation (DeMFI) framework in a two-stage manner, called DeMFINet, which converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate based on flow-guided attentive-correlation-based feature bolstering (FAC-FB) module and recursive boosting (RB), in terms of multi-frame interpolation (MFI). Its baseline version performs featureflow-based warping with FAC-FB module to obtain a sharp-interpolated frame as well to deblur two center-input frames. Its extended version further improves the joint performance based on pixel-flow-based warping with GRU-based RB. Our FAC-FB module effectively gathers the distributed blurry pixel information over blurry input frames in featuredomain to improve the joint performances. RB trained with recursive boosting loss enables DeMFI-Net to adequately select smaller RB iterations for a faster runtime during inference, even after the training is finished. As a result, our DeMFI-Net achieves state-of-the-art (SOTA) performances for diverse datasets with significant margins compared to recent joint methods. All source codes, including pretrained DeMFI-Net, are publicly available at https://github.com/JihyongOh/DeMFI.