Rediscovering Affordance: A Reinforcement Learning Perspective

Document Title

Rediscovering Affordance: A Reinforcement Learning Perspective

Document Information

• Subject Area: Human-Computer Interaction (HCI), Reinforcement Learning, Perception and Action
• Keywords: Affordance, Reinforcement Learning, Perception, Action, Interaction, Robotics, Machine Learning, Adaptation, Motion Planning, HCI

Research Background and Issues

• Identified Problems or Challenges:

  • Existing theories on "Affordance" fail to clearly explain its formation mechanisms, specifically how humans discover and adapt to the affordances of different objects through interaction.
  • Current theories (e.g., ecological perspectives and cognitive perspectives) cannot simultaneously explain the processes of observation, learning, and adaptation to new objects or interfaces.
  • There is a lack of a unified theory to describe the learning and adaptation processes of affordances.

• Importance of the Problem:

  • Affordance is a core concept in the field of Human-Computer Interaction (HCI) and is widely applied in design principles. Understanding how humans form accurate interaction cognition with new objects is crucial for designing intuitive and usable interfaces.
  • Although methods such as deep learning have been applied to object detection, they still fail to explain how to "dynamically learn and adapt" interactions.

• Research Motivation and Related Work:

  • The authors aim to propose a systematic framework for affordance formation through the theory of Reinforcement Learning (RL).
  • This study draws on related work in psychology, computer vision, and reinforcement learning but seeks to integrate these approaches to explain the dynamic formation and adaptation of affordance perception.

Solution

• Proposed Method or Solution:

  • The authors propose a reinforcement learning-based "affordance formation theory": users learn to associate possible movements with corresponding perceptual features by exploring interaction signals in the environment (e.g., success/failure feedback).
  • Two cognitive processes are proposed: 1) utility association between behavioral actions and perceptual signals; 2) induction and naming of interaction action categories.
  • A virtual robot model is constructed to simulate the human process of learning affordances on unfamiliar interactive components (widgets).

• Innovations:

  • Integration of the "ecological perspective" and "cognitive categorization perspective" from psychology with reinforcement learning theory in machine learning to provide a biologically plausible and highly generalizable dynamic affordance learning framework.
  • Validation of the theory through computer simulations, extending it into a computational model applicable for simulation and practice.
  • Proposal and experimental validation of "motion planning" in reinforcement learning as a key mechanism for affordance discovery and adaptation.

• Implementation Steps and Key Techniques:

  1. User Studies: Two user experiments were conducted to investigate how humans perceive and adapt to the affordances of interactive components:
     • Experiment 1: Examined how different cognitive mechanisms influence the formation of interaction affordances.
     • Experiment 2: Studied how humans adjust cognition through feedback signals when interactive components behave unexpectedly.
  2. Robot Simulation Model:
     • Developed a virtual robot trained with reinforcement learning algorithms (e.g., Proximal Policy Optimization, PPO) to interact with different types of interactive components (buttons, sliders, deceptive components).
     • Evaluated the robot's adaptive performance on both common components and new components ("deceptive" components).
  3. Reinforcement Learning Modeling:
     • Tasks were modeled as Markov Decision Processes (MDP), allowing the robot to learn optimal interaction strategies through trial-and-error processes.
     • Implemented an action classifier to correctly label learned movements as specific action types (e.g., "press" or "slide").

Research Outcomes

• Specific Results:

  • Quantitative and qualitative data from user experiments indicate that humans can flexibly switch between three mechanisms—"feature comparison," "recognition," and "motion planning"—to learn and adapt affordances during interaction.
  • The virtual robot model successfully demonstrated the use of reinforcement learning to learn and adapt interactions with both common components and "deceptive components."

• Advantages Compared to Existing Solutions:

  • Provides the first detailed theoretical framework for affordance learning, guided by reinforcement learning, explaining how users learn to interact with new interfaces through "direct interaction."
  • Addresses the limitations of existing deep learning models that mechanically classify interactive objects, offering a dynamic learning perspective through "continuous exploration and adjustment."
  • Offers a generative computational model, which has been relatively scarce in past affordance research.

• Experimental and Evaluation Results:

  • User studies show that motion planning plays a critical role in behavioral adjustments after initial failures, particularly as humans gradually adapt to "unfamiliar" or "misleading" designs.
  • The virtual robot achieved a 91.3% success rate in button interactions and a 94.5% success rate in slider interactions; for "deceptive components," while early performance was poor, the robot successfully adapted and exhibited adjustment trends similar to humans.

• Limitations and Future Directions:

  • Limitations:
    • Although the model exhibits learning trends similar to humans, its adaptation efficiency is significantly lower than that of humans.
    • The current model primarily relies on motion planning and does not simultaneously integrate all three mechanisms (e.g., feature comparison, recognition).
  • Future Directions:
    • Introduce advanced techniques such as meta-learning to improve the robot's learning efficiency with fewer necessary interaction samples.
    • Develop a more comprehensive model that integrates feature comparison and recognition mechanisms into a single framework.
    • Design a computational tool based on the current model to predict the affordance of new interface designs—for example, analyzing whether a particular interface design is "intuitive and easy to use" for the target user group.

Supplementary Information

• Open Science Data:
  • The authors will provide anonymized user study data, virtual robot simulation models (MuJoCo), and reinforcement learning models (Python) on the project page (http://userinterfaces.aalto.fi/affordance).
• Acknowledgments:
  • The research was supported by the Department of Communications and Networking at Aalto University, the Finnish Center for Artificial Intelligence, and related funding projects.