Generative Rotoscoping: A First-Person Autobiographical Exploration on Generative Video-to-Video Practices

This paper contributes a first-person exploration on AI video-to-video technologies, which I call "Generative Rotoscoping". This includes: insights on the creation process, a set of prototype explorations, and an integrated workflow for multi-modal video generation. Generative video is rapidly evolving and delivering higher quality outputs. While video generation models have potential for film-making and content creation, they lack controllability for creative expression: viable videos can require hundreds of unsuccessful attempts. To understand this emergent practice, and due to the constant evolution of models and limited number of early adopters, I explored Generative Rotoscoping over 12 months and created AI workflows leading to over 40,000 video/image files examining a variety of models and techniques including: structural guidance, frame consistency, image referencing and masks, compositing, among others. Insights from this work can serve as a starting point for designing the next generation of video authoring tools.