Vajra: Step-by-step programming with natural language
Authors
Building natural language programming systems that are geared towards end-users requires the abstraction of formalisms inherently introduced by programming languages, capturing the intent of natural language inputs and mapping it to existing programming language constructs. We present a novel end-user programming paradigm for Python, which maps natural language commands into Python code. The proposed semantic parsing model aims to reduce the barriers for producing well-formed code (syntactic gap) and for exploring third-party APIs (lexico-semantic gap). The proposed method was imple-mented in a supporting system and evaluated in a usability study involving programmers as well as non-programmers. The results show that both groups are able to produce code with or without prior programming experience.
Research Questions / Practical Problems
Question signals indexed for this paper.
- 100%
Content-Driven Local Response: Supporting Sentence-Level and Message-Level Mobile Email Replies With and Without AI
CHI '25· Voice User Interface (VUI) Design +1
- 100%
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
IUI '25· Voice User Interface (VUI) Design +1
- 80%
Enabling Conversational Interaction with Mobile UI using Large Language Models
CHI '23· Voice User Interface (VUI) Design +1
- 80%
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
UIST '21· Voice User Interface (VUI) Design +1
- 75%
Tap&Say: Touch Location-Informed Large Language Model for Multimodal Text Correction on Smartphones
CHI '25· Human-LLM Collaboration
- 67%
ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models
CHI '24· Voice User Interface (VUI) Design +2
- 67%
Multimodal Error Correction for Speech-to-Text in a Mobile Office Automated Vehicle: Results From a Remote Study
IUI '22· Automated Driving Interface & Takeover Design +2
- 67%
VoiceAlign: A Shimming Layer for Enhancing the Usability of Legacy Voice User Interface Systems
IUI '26· Voice User Interface (VUI) Design +2
- 67%
ReMap: Lowering the Barrier to Help-Seeking with Multimodal Search
UIST '20· Voice User Interface (VUI) Design +2
- 60%
GestAKey: Touch Interaction on Individual Keycaps
CHI '18· Hand Gesture Recognition +1
Based on Jaccard similarity of research subtopics & professions (≥60%)