Copying style, Extracting value: Illustrators’ Perception of AI Style Transfer and its Impact on Creative Labor

Research Background and Issues

• Identified Problems or Challenges:

  • Generative text-to-image models (e.g., style transfer technologies) pose a threat to artists, particularly illustrators, in their professional careers. These models can extract and replicate specific styles of illustrators, often without their consent.
  • Although style is crucial to artistic creation, computer science research often lacks a profound understanding of the concept of style and fails to incorporate interpretations from art history and industry practices.
  • Whether style transfer outputs can accurately reproduce an artist's unique style, as well as the associated economic and ethical issues, remains insufficiently explored.

• Significance:

  • Illustration style is not only a hallmark of an artist's personal creation but also a source of economic value. The replication and commodification of style raise critical issues such as artist displacement and copyright disputes.
  • Understanding artists' perceptions and evaluations of style transfer is essential for designing technologies that support creativity while respecting artists.

• Research Motivation and Related Work:

  • This paper builds on existing style transfer research, emphasizing illustrators' perceptions and evaluations of the technology, addressing a gap in human-computer interaction research regarding the perspectives of artistic creators.
  • It proposes two main research questions (RQs): How do artists perceive the results of style transfer? Who benefits from these results within the political economy of creative labor?

Solution

• Proposed Methods or Solutions:

  • The authors recruited four professional illustrators and fine-tuned a generative diffusion model (Stable Diffusion 1.5) for their works.
  • Using LoRA (Low-Rank Adaptation) technology, the model was trained to learn the specific styles of the participants while retaining its original capabilities.
  • Semi-structured interviews and thematic analysis were conducted to evaluate the model's performance.

• Innovations:

  • The authors systematically explored the successes and shortcomings of style transfer from the perspective of illustrators for the first time.
  • They introduced the concept of "boundary objects" to analyze style transfer, revealing the differing experiences of various stakeholders (e.g., artists, clients) regarding the same style transfer results.
  • Style transfer was redefined as a supply chain optimization tool rather than merely a creative support tool.

• Implementation Steps and Key Technologies:

  1. Collect 30 works from illustrators and use BLIP to automatically generate image descriptions.
  2. Apply LoRA technology to fine-tune model parameters within the Stable Diffusion framework.
  3. Gather illustrators' feedback on the performance of style transfer through generated images and interviews, analyzing successes and shortcomings.
  4. Discuss experimental results in relation to the broader political economy of the creative industry.

Research Outcomes

• Specific Findings:

  • While style transfer technology can successfully simulate certain isolated elements (e.g., color, texture, and lighting), it fails to transcend these local features to capture the complete style of the artist.
  • Illustrators pointed out that style transfer neglects the close integration of style with content and its dynamic generative process, failing to reflect the "semantic" significance of style.
  • Different participants and observers evaluated style transfer differently based on their familiarity with the style, demonstrating the subjectivity of style perception and assessment.

• Comparison with Existing Solutions and Advantages:

  • Previous technical research primarily focused on model performance or style transfer optimization algorithms, whereas this study highlights the real experiences and evaluations of illustrators.
  • Through theoretical and empirical analysis, the study expands the understanding of style transfer, uncovering its boundary characteristics and socio-economic impacts.

• Experimental or Evaluation Results:

  • Illustrators generally believed that the model outputs could not fully express their style, particularly in terms of semantic elements (e.g., specific thematic objects) and the absence of creative nuances.
  • Illustrators tended to evaluate the style transfer results of other participants as closer to the original works, while being more critical of the outputs of their own styles.

• Limitations and Future Directions:

  • Due to the small sample size of only four illustrators, the results may lack generalizability.
  • Future research could explore more diverse artistic forms (e.g., animation, music) or investigate the perceptions of clients and managers regarding style transfer.
  • Examine how style transfer can balance the interests of various stakeholders and design technologies that better meet practical needs.

In summary, this study provides a novel perspective on the evaluation and optimization of style transfer, emphasizing that the design and application of technology should be grounded in the experiences and interests of artists. It also reveals the role of such technologies in optimizing the supply chain within the creative industry.