Creating Videos from Images

GANs can generate novel photorealistic images after being trained on a group of example photos. This technique can also be used to create variations of an image, either by applying “filters” or new poses or angles of the base image.

Limitations

Could we push this approach to its utmost potential and generate a talking head from just one or a few images? This task presents a significant challenge. Traditional methods, including deep learning techniques, often introduce noticeable artifacts when applying “warping” transformationsYaroslav Ganin, Daniil Kononenko, Diana Sungatullina, and Victor Lempitsky. 2016. “DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation.” ArXiv (Cornell University), July. https://doi.org/10.48550/arxiv.1607.07215. to a set of images, thereby diminishing the realism of the outputAverbuch-Elor, Hadar, Daniel Cohen-Or, Johannes Kopf, and Michael F. Cohen. 2017. “Bringing Portraits to Life.” ACM Transactions on Graphics 36 (6): 1–13. https://doi.org/10.1145/3130800.3130818..

Despite recent advances in convolutional neural networks (CNNs) capable of generating highly realistic human head images, these models typically necessitate training on extensive datasets portraying a single individual to establish a personalized talking head model. However, practical scenarios frequently require the creation of such models with limited input, possibly just a few image views or even a single ...