HapticGen: Generative Text-to-Vibration Model for Streamlining Haptic Design

Research Background and Issues

• Issues and Challenges:

  • Vibration haptic feedback can enhance user experiences in virtual reality (VR), gaming, and assistive technologies, but designing such feedback requires specialized skills, complex tools, and significant time investment.
  • Haptic signal datasets are limited, typically containing only a few hundred examples, which are insufficient for comprehensive machine learning training.
  • Existing design tools lack support for natural language input, making it difficult for designers to translate nuanced real-world sensations into haptic signals.

• Importance:

  • Vibration haptic feedback has become critical for enhancing user experiences, but current design complexities and data scarcity severely limit its widespread adoption.
  • Developing automated solutions for haptic design can lower the barrier to entry, enabling non-expert users to create haptic feedback.

• Related Work:

  • Generative AI has made significant progress in image and audio generation, with models like DALL-E and AudioGen.
  • Few precedents exist for applying generative AI to the haptic domain, and there is currently a lack of large-scale datasets and standardized methods for generating haptic signals.

Solution

• Proposed Method:

  • Introduce HapticGen, a generative text-to-vibration model designed to generate haptic signals from textual input.
  • The model is based on an autoregressive Transformer, leveraging a retrained EnCodec encoder to optimize haptic data representation.
  • Employ an automated audio-to-haptic signal conversion method to generate large-scale initial haptic datasets from existing audio datasets.

• Core Innovations:

  1. Generating Haptic Signals from Text: HapticGen is the first generative AI model capable of producing vibration haptic signals from textual descriptions.
  2. Data Augmentation: Utilize large language models (LLMs) to enhance initial textual descriptions, creating diverse text labels with haptic characteristics.
  3. Alignment with Human Preferences:
     • Manually refine haptic signals and collect user ratings to construct an expert-annotated dataset.
     • Further adjust the model using Direct Preference Optimization (DPO) to better reflect user design needs.

• Implementation Steps:

  • Data Preprocessing:
    1. Remove speech-related audio data that cannot be represented in vibration feedback.
    2. Use LLMs to enhance text labels, generating multiple haptic description variants.
    3. Convert audio signals into vibration signals by controlling frequency and dynamic settings to create signals suitable for haptic devices.
  • Model Architecture:
    • Employ an autoregressive Transformer model based on MusicGen, adjusting the encoder and decoder to process haptic data.
    • Train the model using filtered initial data and augmented content.
  • Interface Development:
    • Provide a graphical user interface supporting text input, vibration signal generation, and signal playback, allowing users to directly verify vibration signals via Meta Quest controllers.

Research Outcomes

• Specific Achievements:

  1. Developed the HapticGen model, capable of quickly generating diverse vibration haptic signals from textual input.
  2. Established two large-scale haptic datasets:
     • Expert-voted dataset containing 1,297 pairs of [text prompts, vibration signals, user ratings].
     • User-voted dataset containing 3,229 similar records.
  3. Compared to baseline methods (e.g., audio-based generation models), HapticGen significantly improved user experience and haptic signal quality.

• Experimental Results:

  • In A/B testing against baseline models, HapticGen excelled in haptic experience (authenticity, expressiveness) and system usability (workload, future usage intention).
  • User satisfaction with HapticGen-generated signals was higher than with baseline models, with some users noting the signals felt more natural and aligned with their intentions.

• Advantages:

  1. Improved Design Efficiency: Rapid generation of vibration haptic signals significantly reduces design time.
  2. Content Diversity: Generated haptic signals exhibit higher expressiveness, with users noting their ability to capture dynamic and detailed scenarios.
  3. Accessibility: The tool simplifies the haptic design process, enabling non-expert users to participate in creating haptic feedback.

• Limitations and Future Directions:

  • Limitations:
    1. The model struggles to effectively handle highly complex or multi-stage haptic descriptions (e.g., "running followed by walking").
    2. Some users reported that signal intensity was insufficiently pronounced on hardware devices.
    3. The lack of editing tools limits users' ability to fine-tune generated signals.
  • Future Work:
    1. Explore new audio-to-haptic mapping methods to enhance signal detail representation.
    2. Investigate improved optimization methods (e.g., KTO or BCO) to enhance model training.
    3. Develop additional features allowing users to directly edit generated haptic signals (e.g., adjusting timing or frequency characteristics).

The success of HapticGen not only demonstrates the cross-domain potential of generative AI but also establishes a new paradigm for haptic design. By enabling text-to-haptic signal generation, it paves the way for more intuitive and scalable design workflows.