I see an IC: A Mixed-Methods Approach to Study Human Problem-Solving Processes in Hardware Reverse Engineering

Title of the Paper

I see an IC: A Mixed-Methods Approach to Study Human Problem-Solving Processes in Hardware Reverse Engineering

Paper Information

• Subject Area: Hardware Reverse Engineering (HRE) and Human-Computer Interaction (HCI)
• Keywords: Hardware Reverse Engineering, Integrated Circuits, Eye Tracking, Think Aloud, Mixed-Methods Research, Problem Solving, Semiconductor Industry

Research Background and Problem

• Identified Problems or Challenges:

  1. Hardware Reverse Engineering (HRE) requires close collaboration between tools and analysts, but there is a lack of detailed studies on how human analysts solve problems.
  2. Eye tracking and think-aloud methods have been applied to study cognitive and behavioral processes in other fields, but their application in HRE remains unexplored.
  3. Existing literature shows inconsistent conclusions regarding the combination of think-aloud (TA) and eye-tracking data, lacking methodological consistency.

• Significance of the Research:

  • Hardware trust and security require effective reverse engineering research to identify counterfeit and malicious hardware components.
  • A better understanding of human analysts' behavior can facilitate the development of educational tools and hardware protection schemes.

• Research Motivation and Related Work:

  • Lee and Johnson-Laird describe HRE as a complex problem-solving scenario requiring frequent hypothesis reevaluation.
  • HCI research, eye-tracking technology, and think-aloud techniques have been successfully used in other fields to explore cognitive and behavioral characteristics of problem-solving.

Solution

• Proposed Methods or Solutions:

  • Develop a hybrid methodology combining eye tracking and think-aloud (TA) to study problem-solving processes in HRE.
  • Conduct experiments with 41 participants performing hardware reverse engineering tasks, capturing their visual attention and thought processes through eye tracking and think-aloud data.

• Innovative Contributions:

  1. First application of the combined think-aloud and eye-tracking methods in the HRE domain.
  2. Comparative evaluation of two think-aloud methods (Concurrent Think Aloud (CTA) and Retrospective Think Aloud (RTA)) and integration with eye-tracking data to provide a more granular understanding of problem-solving processes.
  3. Experimental design to test how task complexity (regular tasks vs. camouflaged gate tasks) affects problem-solving.

• Implementation Steps and Key Techniques:

  1. Design a game-based reverse engineering task (ReverSim) and integrate it with eye-tracking equipment (Tobii Pro Nano).
  2. Record participants' behavior using think-aloud methods (CTA recorded in real-time or RTA based on post-task reflection).
  3. Analyze Areas of Interest (AOI) and quantify gaze distribution characteristics using eye-tracking data to identify task-critical points.
  4. Conduct in-depth behavioral analysis using a code segmentation and content analysis framework for think-aloud data.

Research Findings

• Specific Findings:

  1. Feasibility of Eye Tracking:
     • Eye tracking revealed visual distribution patterns across different circuit components and could distinguish research depth on various logic gates even in complex scenarios.
  2. Comparison of Think-Aloud Methods:
     • RTA was more advantageous for generating high-level reflections, while CTA excelled in capturing fine-grained behaviors and real-time synchronization.
  3. Potential of Combined Methods:
     • Combining TA data with eye-tracking heatmaps provided insights into participants' navigation patterns during circuit analysis.
     • Eye-tracking data could identify participants' problem-solving strategies, reducing reliance on manual annotations.
  4. Task Case Analysis:
     • Using highly disruptive "camouflaged gates" to divert attackers' attention demonstrates a potential "cognitive interference" protection method.

• Comparative Advantages Over Existing Solutions:

  • Unlike methods relying on tool outputs, the proposed approach directly studies analysts' cognitive processes, offering a new perspective for developing more effective hardware protection and educational measures.

• Experimental and Evaluation Results:

  1. Eye-tracking metrics (e.g., fixation time and fixation rate) indicated that camouflaged gates significantly diverted visual attention.
  2. TA methods did not significantly affect participants' task performance or user experience.
  3. The hybrid analysis method demonstrated advantages over single-method approaches, accelerating the implementation of behavioral analysis models.

• Limitations and Future Directions:

  1. Limitations:
     • Participants were predominantly young, well-educated students, and the sample's limitations may affect external validity.
     • Non-native English speakers might struggle to fully articulate their thoughts.
     • Cognitive factors like working memory and stress were not comprehensively covered in the study of HRE processes.
  2. Future Directions:
     • Automating AOI definition and leveraging machine learning for behavioral analysis.
     • Exploring the combined potential of eye tracking and other physiological features (e.g., pupil dilation, electrophysiological signals).
     • Expanding to more complex circuit tasks and quantitative analysis of higher-level cognitive factors.

Through this study, the authors not only validated the feasibility of the proposed method but also provided methodological guidance for the hardware security field and other visually intensive problem-solving domains.