Perfection Not Required? Human-AI Partnerships in Code Translation

Title of the Paper

Perfection Not Required? Human-AI Partnerships in Code Translation

Paper Information

• Domain: Human-AI Collaboration and the Use of Generative AI in Application Modernization
• Keywords: Neural Machine Translation (NMT), Generative AI, Imperfect AI, Code Translation, Application Modernization, Human-AI Collaboration

Research Background and Problem

• Identified Problems or Challenges:
  • Recent attempts in software engineering to use generative NMT models for translating code between programming languages have shown that the generated code may contain compilation or logical errors.
  • It remains unclear whether software engineers can accept flawed AI outputs and how they can assist in detecting and correcting these errors.
• Significance:
  • Many organizations face a shortage of technical resources when migrating legacy systems to modern platforms such as cloud computing. Generative AI models can help address this challenge.
  • The reliability and acceptability of code translation directly impact the productivity and quality of modernization migration projects.
• Research Motivation and Related Work:
  • Investigated the potential of generative AI in software engineering and its interaction design in code translation tasks.
  • Related work includes studies on using generative AI for code repair, API recommendations, and type inference.

Solution

• Proposed Solution:
  • Conducted scenario-based design interviews with 11 software engineers to explore methods for accepting NMT model-generated code in the context of application modernization.
  • Designed three multi-stage user interface variants to demonstrate how AI can assist in code translation, support users in identifying low-confidence code segments, and provide alternative translation suggestions.
• Innovations:
  • UI features such as confidence markers and alternative translations integrate generative model outputs with user needs, helping engineers understand and improve AI-generated code.
  • Positioned generative AI as a collaborative partner rather than a tool, emphasizing human-AI collaboration.
• Implementation Steps and Key Techniques:
  • Used the TransCoder model for generating code translations.
  • Developed partial functional prototypes to help users explore effective utilization of NMT models through three user interface variants.
  • Collected feedback and suggestions from engineers on AI-translated code through interviews.

Research Findings

• Specific Findings:

  1. Software engineers expressed concerns about the quality of AI-generated code but believed that good development practices (e.g., code reviews and unit testing) could make such code acceptable.
  2. Confidence markers and alternative translations helped engineers better understand the model and identify errors in translations.
  3. Highlighted the importance of improving human-AI collaboration models, where humans and AI systems can play complementary roles in application modernization.
  4. Proposed future research directions, including further optimization of generative AI-user interaction design.

• Advantages Compared to Existing Solutions:

  • Offered unique user interface designs that integrate AI confidence, alternative translations, and human-AI collaboration tasks, effectively complementing engineers’ existing workflows and validation mechanisms.
  • Demonstrated the practical value of generative AI in scenarios with objective quality requirements, emphasizing its applicability to real-world problems.

• Experimental or Evaluation Results:

  • The three user interface variants were widely accepted and effectively guided engineers in completing code review tasks.
  • Even erroneous code outputs contributed to task efficiency, provided the errors were easy to detect and fix.

• Limitations and Future Directions:

  • Limitations: The study focused solely on code translation tasks and has not been extended to other domains or generative AI scenarios.
  • Future Directions: Design more diverse interaction modes, such as real-time AI-assisted code writing; enhance capabilities for generating documentation and understanding architectures; explore the impact of feedback mechanisms on model training.

Conclusion and Insights

By investigating software engineers' acceptance of AI-translated code, this study found that generative AI is practical for application modernization tasks, even when the generated code contains defects. The research also demonstrated how intelligent user interface design can help engineers streamline the code review process and enhance human-AI interaction. The insights from this work provide valuable references for future applications of generative AI technology in other fields.