Effective 2D Stroke-based Gesture Augmentation for RNNs

Title of the Paper

Effective 2D Stroke-Based Gesture Augmentation for RNNs

Paper Information

• Subject Area: Gesture Recognition, Data Augmentation, Deep Learning
• Keywords: Dataset, Neural Networks, Gesture Recognition and Customization, Data Augmentation, Sequential Data, Training Optimization

Research Background and Problem

• Identified Problems or Challenges:

  1. Recurrent Neural Networks (RNNs) typically require large amounts of training data, but in user-customized gesture applications, each gesture class may only have one or two data points.
  2. Even with data augmentation strategies to generate more data, the training time and model optimization capabilities of RNNs remain constrained.
  3. Current gesture recognition methods have the following limitations: traditional k-NN-based methods require manual feature extraction, while RNNs face challenges with limited data and complex scenarios.

• Necessity of the Research:

  • Improving the flexibility and accuracy of gesture recognition systems is key to enhancing user experience.
  • Customized gesture systems need to reduce the high demand for raw data to meet real-time recognition requirements.
  • Current research on gesture data augmentation has not fully explored how to combine augmentation methods to optimize classification performance.

• Research Motivation and Related Work:

  • Inspired by augmentation techniques based on data variation in the gesture recognition field, various existing methods such as Generative Adversarial Networks (GANs) and hybrid generation strategies have improved gesture classification accuracy to some extent, but issues like long training times and reliance on large amounts of real data persist.

Solution

• Proposed Method or Solution:

  1. Design a multi-step "augmentation chaining" strategy that combines multiple data augmentation methods, sequentially stacking variations of different gesture classes to expand the training data.
  2. Apply limited preprocessing steps, such as initial offset normalization, for real-time (mid-gesture) gesture classification with RNN models.
  3. Independently test various augmentation techniques and explore performance changes when combining these techniques.

• Innovations:

  • Propose the "All Variability Chain" (AVC): combining multiple categories of variations (e.g., Gaussian noise, spatial resampling, perspective transformation, translation) within the augmentation chain.
  • Conduct experiments using various public datasets (e.g., $1-GDS, EDS1, EDS2, Lemarchand datasets) to evaluate the generalization ability and performance of the method.
  • Test the effectiveness of the augmentation chain in constrained scenarios with very few training samples, such as generating synthetic validation sets to address data scarcity.

• Implementation Steps and Key Techniques:

  1. Data Preprocessing: Normalize the initial point of gesture data to the origin while retaining high-frequency noise and differences in the number of points to introduce more feature variations.
  2. Evaluation of Individual Augmentation Methods:
     • Test twelve augmentation methods, including Gaussian noise, temporal jittering, rotation, and shear.
  3. Augmentation Chain Design and Experimentation:
     • Categorize augmentation methods by variation type (e.g., spatial transformations, temporal transformations, shape changes).
     • Sequentially stack multiple augmentation techniques and optimize the order of steps in the chain based on experimental results.

Research Outcomes

• Specific Results:

  1. With only one training sample provided, the AVC augmentation chain significantly improved RNN classification accuracy (from a baseline of 70.29% to 96.98% on the $1-GDS dataset).
  2. Experiments demonstrated that the AVC augmentation chain performed better across four different public datasets, significantly outperforming common generation methods (e.g., Gestures à Go Go and GPSR).
  3. Even with smaller models and training set sizes, the method achieved competitive performance compared to existing approaches.

• Comparison with Existing Solutions:

  1. Compared to traditional single augmentation methods or pretraining-based generation methods (e.g., Gestures à Go Go), the AVC augmentation chain exhibited stronger generalization capabilities.
  2. The augmentation chain method allows RNN models to function effectively without relying on large-scale real data, making it suitable for user-customized gesture scenarios.
  3. In synthetic data scenarios, the AVC chain method effectively reduced overfitting issues while significantly improving training efficiency for gesture recognition.

• Experimental or Evaluation Results:

  • Compared different augmentation methods (single methods vs. chained combinations) in terms of augmented data volume, training time, and classification accuracy.
  • Found that synthetic data generated by the AVC augmentation chain exhibited more representative variations, complementing the feature limitations of original gesture data.
  • As the volume of synthetic data increased, accuracy improvements gradually diminished, reaching an optimal point when 300 synthetic samples per gesture class were generated (training time was only 50 seconds).

• Limitations and Future Directions:

  1. Limitations:
     • The optimization of the augmentation chain was primarily based on the $1-GDS dataset, and its robustness across datasets requires further validation.
     • The interaction between the augmentation chain and factors such as model size and the number of gesture classes has not been fully explored.
  2. Future Directions:
     • Automate the optimization of augmentation chain methods, for example, by using genetic algorithms or reinforcement learning to further enhance combined augmentation strategies.
     • Extend the applicability of the method to multimodal datasets (e.g., 3D gestures or complex symbol data).
     • Explore generating higher-quality, more realistic synthetic data to improve the generalization performance of models.