SIGCHI Lifetime Research Award Talk—Seeing Past Looking Forward

I have long been interested in how computers can help people perform skilled tasks—a theme that underlies essentially all of my research. As the intentionally convoluted title of this talk implies, I will take a look back at some of this research to explain what came later, and to speculate on what might be next. My earliest work as a graduate student was as part of a team developing novel hypermedia editing and presentation tools for technical documentation of equipment maintenance procedures. Understanding the effort involved in using these tools manually led to my dissertation, which explored rule-based techniques for the automated generation of 3D graphics that could ultimately replace the pictures found in maintenance manuals. Based on information about the task to be depicted, I designed a system that chose the objects to include, determined the level of detail at which to render them (e.g., adding more detail to disambiguate objects that could otherwise be confused with each other), highlighted important objects for emphasis, and created additional “metaobjects” (e.g., arrows to show actions). When I started as a faculty member, several of my students built on this research direction in both 3D and 2D domains, and we were soon collaborating with colleagues to develop generation approaches for coordinated multimedia explanations that combined graphics with text. At the same time, I was becoming excited by the potential of virtual reality (VR) to render objects stereoscopically and interact with them in 3D, in abstract domains as well as physical ones. And it was an easy jump from there to augmented reality (AR), cobbling together a simple monoscopic optical see- through head-worn display. Our first AR system was a “hybrid” desktop window manager that embedded windows displayed on a flat panel display within a much larger virtual surround presented on the AR display. With that AR display now available as a tool, it became stunningly clear that so much of the effort we’d put into carefully crafting stand-alone pictures to explain tasks was unnecessary. If the user could instead look directly at the actual task domain, then a simple highlight or arrow, overlaid in situ, could show which button to push or knob to turn on a panel of controls. Inspired by that observation, much of my lab’s research has since addressed the design of AR user interfaces, for users indoors and outdoors, working alone and together. While AR does make some problems easier, it introduces difficult challenges! In classical user interfaces, designers and programmers control essentially all of what users experience; for example, determining the color at every pixel, accounting in advance for a limited set of variations. In contrast, AR celebrates and must work in perceptual context with the physical world. And that world is supremely fluid, in ways that are often unpredictable, as users and things in their environment move and change relative to each other. In response, the virtual user interface must change dynamically, as needed, to complement rather than interfere with the physical world; for example, a less important virtual object shouldn’t obscure a more important object, whether physical or virtual. Bearing this in mind and working not only in AR and VR, but in many kinds of desktop, mobile, and wearable user interfaces, my lab has tackled user interface design issues in domains such as healthcare, games, graphical editing, navigation, and equipment assembly. What comes next? I will conclude this talk by hazarding a few guesses (it is a dangerous sport!) about where some of this might lead.