Situated Live Programming for Human-Robot Collaboration

Document Title

Situated Live Programming for Human-Robot Collaboration

Document Information

• Subject Area: Human-Robot Collaboration and Robot Programmability
• Keywords: Human-Computer Interaction, Human-Robot Collaboration, Trigger-Action Programming, End-User Programming, Live Programming, Augmented Reality, Robot Programming

Research Background and Issues

• Problems and Challenges:
  • Current collaborative robots (cobots) are often limited to isolated single tasks in practical applications, failing to achieve complex human-robot interactions.
  • Existing programming methods (such as teach programming and task scheduling algorithms) are not user-friendly for non-programmers, limiting widespread adoption by small and medium-sized enterprises.
  • Collaborative tasks require precise coordination between robots and humans or other systems in the environment, which remains a challenge for non-experts.
• Significance:
  • Enhancing the accessibility of robot programming will make it easier for small and medium-sized enterprises to "re-task" robots to adapt to different production environments and needs.
  • Allowing end-users to autonomously program robots is crucial for lowering technical barriers and promoting technology adoption.
• Research Motivation:
  • There is a need for an easy-to-use programming method that enables non-expert users to accomplish complex human-robot collaboration tasks.
  • The study proposes a framework combining live programming and Trigger-Action Programming (TAP).

Solution

• Method or Solution:
  • A novel live programming approach for human-robot collaboration, called "Situated Live Programming" (SLP), is proposed, integrating Trigger-Action Programming.
  • Within the framework, users can define robot behaviors by incrementally constructing trigger-action pairs.
  • SLP applies TAP to complex dynamic environments while supporting incremental and interactive programming.
  • A lightweight manufacturing operation prototype system based on SLP has been implemented, allowing users to program using an augmented video interface installed on the robot.
• Innovations:
  • Situating and making Trigger-Action Programming "live," significantly reducing users' cognitive load regarding the overall task model.
  • Providing an incremental development process that allows users to iteratively adjust and optimize rules during robot behavior execution.
  • Offering a simple graphical interface that supports the creation of triggers and actions directly within the task environment using dynamic video streams.
• Implementation Steps and Key Technologies:
  1. Users can annotate task areas and define trigger-action pairs through the video interface.
  2. The system monitors the task environment in real-time (via robot cameras, 3D depth cameras, etc.) and triggers actions when conditions are met.
  3. The ROS (Robot Operating System) architecture is used to coordinate information flow and hardware operations.
  4. Users can incrementally build programs while testing and adjusting them in real-time.

Research Outcomes

• Specific Results:
  • Developed an open-source SLP prototype tool and validated it through experiments.
  • In the study, users successfully completed three types of lightweight manufacturing tasks (sorting, packaging, collaborative assembly) using the developed system.
• Experiments and Evaluation:
  • Participants (N=10) included university students and graduate students with some programming background, all of whom successfully completed the assigned tasks.
  • Three main usage strategies were observed during the experiments: fully autonomous programming, incremental rule addition (trial and error), and exploratory programming.
  • Users rated the system's usability highly (SUS score of 75.8), finding the interface intuitive and operable without extensive programming experience.
• Advantages:
  • More flexible than traditional programming methods, allowing users to gradually create coordinated robot behaviors through "trial and error."
  • Simplifies the decomposition of complex tasks and rule design, supporting personalized programming strategies.
• Limitations and Future Directions:
  • The current framework does not guarantee task output completeness or optimality, and its applicability is limited to predefined high-level actions.
  • Scalability in complex dynamic environments, multi-task scenarios, or multi-robot collaboration in real industrial settings requires further validation.
  • The user interface needs improvement to enhance rule visualization, debugging functionality, and priority conflict management.