"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy

Modern systems for language-guided human-robot interaction require two key components for broad adoption: adaptivity and learning efficiency. Unfortunately, existing data-driven approaches for learning instruction-following agents cannot adapt, failing to incorporate additional natural language supervision, and even if they could, require hundreds of demonstrations to learn even simple policies. In this work, we address these problems by presenting a framework for incorporating and adapting to natural language corrections -- ''to the right'', or ''no, towards the book'' -- as the robot executes. To focus on rich manipulation domains where the sample efficiency of existing methods is prohibitive, we work within a shared autonomy paradigm: instead of discrete turn-taking between a human and robot, our shared autonomy paradigm splits agency between the human and robot. In our approach, natural language is an input to a learned model that produces a meaningful, low-dimensional control space that the human can use to guide the robot. Each real-time correction refines the human's control space, enabling the execution of precise, extended behaviors -- with the added benefit of requiring only a handful of demonstrations to learn. We evaluate our approach via a user study, where users work with a Franka Emika Panda manipulator to complete complex manipulation tasks. Compared to existing learned baselines covering both open-loop instruction following and single-turn shared autonomy, we show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users because of its reliability, precision, and ease of use.