Authoring LLM-Based Assistance for Real-World Contexts and Tasks
Authors
Advances in AI hold the possibility of assisting users with highly varied and individual needs, but the breadth of assistance that these systems could provide creates a challenge for how users specify their goals to the system. To support the authoring of AI assistance for real-world tasks, we propose the concept of Contextually-Driven Prompts (CDPs) that define how an AI assistant should respond to real-world context. We implemented a prototype system for authoring and executing CDPs, which provides suggestions to assist users with finding the right level of assistance for their goal. We also conducted a user study (N=10) to investigate how participants express and refine their goals for real-world tasks. Results revealed a number of strategies for initiating and refining CDPs with suggestions, and implications for the design of future authoring interfaces.
Research Questions / Practical Problems
Question signals indexed for this paper.
Research Questions
3- How can users easily define complex task goals while enabling systems to understand and execute context-based tasks?Category: Multimodal Perception and Cross-Channel UnderstandingSimilar questionsarrow_forward
- Can context-driven structured natural language prompts (CDPs) improve users' ability to design intelligent assistant tasks?Category: Multimodal Perception and Cross-Channel UnderstandingSimilar questionsarrow_forward
- How can vision-language models (VLMs) help intelligent assistants better understand visual information in users' environments to support task design?Category: Multimodal Perception and Cross-Channel UnderstandingSimilar questionsarrow_forward
Practical Problems
1- Users struggle to express complex goals efficiently, and intelligent assistants fail to adequately support everyday tasks.Category: Multimodal Perception and Cross-Channel UnderstandingSimilar questionsarrow_forward
- 67%
VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
UIST '24· Human-LLM Collaboration +2
- 60%
Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns
CHI '18· Biosensors & Physiological Monitoring +1
- 60%
Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition
CHI '20· Human Pose & Activity Recognition +1
- 60%
Trade-offs for Substituting a Human with an Agent in a Pair Programming Context: The Good, the Bad, and the Ugly
CHI '21· Human-LLM Collaboration +1
- 60%
Visualizing Examples of Deep Neural Networks at Scale
CHI '21· Human-LLM Collaboration +1
- 60%
Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models
CHI '22· Generative AI (Text, Image, Music, Video) +1
- 60%
AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts
CHI '22· Human-LLM Collaboration +1
- 60%
Tracing and Visualizing Human-ML/AI Collaborative Processes through Artifacts of Data Work
CHI '23· Human-LLM Collaboration +1
- 60%
Comparing Sentence-Level Suggestions to Message-Level Suggestions in AI-Mediated Communication
CHI '23· Human-LLM Collaboration +1
- 60%
"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
CHI '23· Generative AI (Text, Image, Music, Video) +1
Based on Jaccard similarity of research subtopics & professions (≥60%)