Empirical Evidence on Conversational Control of GUI in Semantic Automation

This research explores integration of a Large Language Model (LLM) fine-tuned to conversationally control the user interface (UI) for a Semantic Automation Layer (SAL). We condense SAL capabilities from prior work and prioritize with business analysts and data engineers via a Kano model, before implementing a prototypical UI. We augment the UI with our conversational engine and propose In-situ Prompt Engineering and learn from Human Feedback to smoothen the interaction and manipulation of UI through natural language commands. To evaluate the efficacy and usability of conversational control in various use-case scenarios, we conduct and report on an empirical interaction design user study. Our findings provide evidence supporting enhanced user engagement and satisfaction. We also observe significant increase of trust in AI after working with our conversational UI. This work generates areas for further refinement and research towards more intelligent, highly-integrated conversational UIs even beyond our application within Semantic Automation. We discuss our findings and point out next steps paving the way for future research and development in creating more intuitive and adaptive user interfaces.