Leveraging stable diffusion for animation inbetweening

Inbetweening is a process in animation that involves creating intermediate frames between two keyframes. The intended result is to create the illusion of movement by smoothly transitioning one image into another. It is a fundamental part of the animation creative process, but it is also both one of the least creative and of the most laborious, since the aforementioned illusion of movement requires frames to have a strictly consistent style and structure. This project aims at employing the impressive generative capabilities of the Stable Diffusion XL model to automate the intermediate step of inbetweening, learning an illustrator's personal style and applying it to a video of actors interpreting a scene. The model would also need to be flexible enough to work for multiple scenes with minimal, ideally none, additional training needed beyond the existing dataset. Key findings include the effective use of ControlNets and the FreeU technique, and employing masking techniques to address visual inconsistencies. Future work will focus on further refining these methods and exploring additional techniques to enhance the robustness and efficiency of the animation pipeline, as well as automating the generation of additional training data.