Designing the Conversational Agent: Asking Follow-up Questions for Information ElicitationConversational Agents (CAs) can facilitate information elicitation in various scenarios, such as semi-structured interviews. Current CAs can ask predetermined questions but lack skills for asking follow-up questions. Thus, we designed three approaches for CAs to automatically ask follow-up questions, i.e., follow-ups on concepts, follow-ups on related concepts, and general follow-ups. To investigate their effects, we conducted a user study (N=26) in which a CA interviewer asked follow-up questions generated by algorithms and crafted by human wizards. Our results showed that the CA's follow-up questions were readable and effective in information elicitation. The follow-ups on concepts and related concepts achieved a lower drop rate and better relevance, while the general follow-ups elicited more informative responses. Further qualitative analysis of the human-CA interview data revealed algorithm drawbacks and identified follow-up question techniques used by the human wizards. We provided design implications for improving information elicitation of future CAs based on the results.2024JHJiaxiong Hu et al.Session 3f: Conversations with MachinesCSCW
What makes virtual intimacy...intimate? Understanding the Phenomenon and Practice of Computer-Mediated Paid CompanionshipVirtual romance service (VRS), as a notable commodification of intimacy, is currently emerging in China. Such service is not similar to the kind of intimacy that fans and idols generate through parasocial relationships, but behaves as the direct dyadic intimacy between service providers (\emph{virtual lovers}) and buyers (\emph{customers}). To gain a deep understanding of computer-mediated paid companionship, we study emerging user behaviors in VRS through a mixed-method study, including a survey ($N = 178$) and a follow-up semi-structured interview ($N = 22$) with both virtual lovers and customers to learn about their motivations, perceptions, and how virtual lovers provide online paid companionship to meet customers' emotional needs. We found three behavioral strategies of virtual lovers and the fact that they provide service in surface and deep acting and real feeling. Customers also see VRS as a way to obtain affective benefits with reduced affective costs. We also found that VRS customers paid for the tangible benefits of an idealized romantic partner, rather than long-term commitment and emotional investment, and identified key characteristics that VRS reduces from intimate relationships that fit its pay-per-use feature. We conclude by discussing the nature of virtual lovers and design implications for computer-mediated paid companionship.2023WLWeijun Li et al.Social ConnectionsCSCW
Layout Generation for Various Scenarios in Mobile Shopping AppsLayout is essential for the product listing pages (PLPs) in mobile shopping applications. To clearly convey the information that consumers require and to achieve specific functions, PLPs layouts often have many variations driven by scenarios. In this work, we study the PLPs layout design for different scenarios and propose a design space to guide the large-scale creation of PLPs. We propose LayoutVQ-VAE, a novel model specialized in generating layouts with internal and external constraints. LayoutVQ-VAE differs from previous methods as it learns a discrete latent representation of layout and can model the relationship between layout representation and scenarios without applying heuristics. Experiments on publicly available benchmarks for different layout types validate that our method performs comparably or favorably against the state-of-the-art methods. Case studies show that the proposed approach including the design space and model is effective in producing large-scale high-quality PLPs layouts for mobile shopping platforms.2023QJQianzhi Jing et al.College of Computer Science and TechnologyRecommender System UXCHI
Auto-Icon: An Automated Code Generation Tool for Icon Designs Assisting in UI DevelopmentApproximately 50% of development resources are devoted to UI development tasks [62]. Occupied a large proportion of development resources, developing icons can be a time-consuming task, because developers need to consider not only effective implementation methods but also easy-to-understand descriptions. In this study, we define 100 icon classes through an iterative open coding for the existing icon design sharing website. Based on a deep learning model and computer vision methods, we propose an approach to automatically convert icon images to fonts with descriptive labels, thereby reducing the laborious manual effort for developers and facilitating UI development. We quantitatively evaluate the quality of our method in the real world UI development environment and demonstrate that our method offers developers accurate, efficient, readable, and usable code for icon designs, in terms of saving 65.2% developing time.2021SFSidong Feng et al.AutoML InterfacesIUI
Shing: A Conversational Agent to Alert Customers of Suspected Online-payment Fraud with Empathetical Communication SkillsAlerting customers on suspected online-payment fraud and persuade them to terminate transactions is increasingly requested with the rapid growth of digital finance worldwide. We explored the feasibility of using a conversational agent (CA) to fulfill this request. Shing, a voice-based CA, proactively initializes and repairs the conversation with empathetical communication skills in order to alert customers when a suspected online-payment fraud is detected, collects important information for fraud scrutiny and persuades customers to terminate the transaction once the fraud is confirmed. We evaluated our system by comparing it with a rule-based CA with regards to customer response and perceptions in a real-world context where our systems took 144,795 phone calls in total in which 83,019 (57.3%) natural breakdowns happened. Results showed that more customers stopped risky transactions after conversing with Shing. They seemed more willing to converse with Shing for more dialogue turns and provide transaction details. Our work presents practical implications for the design of proactive CA.2021JGJingya Guo et al.Alibaba GroupConversational ChatbotsAgent Personality & AnthropomorphismPrivacy by Design & User ControlCHI
Scene-Aware Behavior Synthesis for Virtual Pets in Mixed RealityVirtual pets are an alternative to real pets, providing a substitute for people with allergies or preparing people for adopting a real pet. Recent advancements in mixed reality pave the way for virtual pets to provide a more natural and seamless experience for users. However, one key challenge is embedding environmental awareness into the virtual pet (e.g., identifying the food bowl's location) so that they can behave naturally in the real world. We propose a novel approach to synthesize virtual pet behaviors by considering scene semantics, enabling a virtual pet to behave naturally in mixed reality. Given a scene captured from the real world, our approach synthesizes a sequence of pet behaviors (e.g., resting after eating). Then, we assign each behavior in the sequence to a location in the real scene. We conducted user studies to evaluate our approach, which showed the efficacy of our approach in synthesizing natural virtual pet behaviors.2021WLWei Liang et al.Beijing Institute of TechnologyMixed Reality WorkspacesDigital Art Installations & Interactive PerformanceCHI
Redefining Natural User InterfaceThis SIG focuses on new definitions of Natural User Interface (NUI). With the adoption of wearable devices, VR & AR displays, affective computing, and voice user interface, we think it’s necessary to review our understanding and definition of NUI. This SIG aims to expand discussion and development related to NUI in two areas: first, what experience should NUIs achieve today? How can we build UIs to leverage other senses besides vision and hearing such as tactility, olfaction and gustation? Second, how can we detect, capture and compute people’s behavioral signals in a natural way and provide output accordingly? What are the current available technologies to achieve NUIs, and what new technologies should be invented to achieve it?2018LFLimin Paul Fu et al.Alibaba DAMO AcademyFull-Body Interaction & Embodied InputBrain-Computer Interface (BCI) & NeurofeedbackMixed Reality WorkspacesCHI