Test-Time Training for Image Inpainting

Abstract

Image inpainting is the task of filling missing regions in images with plausible and coherent content. The usual process involves training a CNN on a large collection of examples that it can learn from, to later apply this knowledge on new, unseen images with areas to complete. One important consideration is that each training dataset has its own domain, and domains in the image space are extremely diverse. For optimal results, the user image at inference time should belong to the same domain as the training examples, which greatly limits the range of inputs a trained network can be used for. Moreover, collecting new data is hard, and training on different datasets requires a great amount of computation power and time. In this thesis, we propose a test-time learning approach for inpainting. More specifically, we train a CNN like described above, but in addition, we use the user image with holes to build a new dataset, on which we continue training the network for a small number of iterations during test-time training. To the best of our knowledge, test-time training has never been done for inpainting. With this technique, our hope is that the model will learn to fill holes that are specific to the user image better. It also facilitates domain adaptation, which means that a wider range of input domains at inference time can produce acceptable results when using the same pretrained model. We obtain results demonstrating that even for a state-of-the-art model, our method can achieve significant improvements, in particular for perceptual scores. Moreover, if used in a software, it has the advantage of being optional, meaning that the user can choose to keep the original result if there is no significant improvement, which makes our framework beneficial in all situations.