A Framework to Characterize Reporting on Generative AI Use

Paper Title

A Framework to Characterize Reporting on Generative AI Use

Publication Info

• Topic area: Reporting practices and frameworks for understanding generative AI use.
• Keywords: Generative AI, reporting framework, transparency, use cases, user studies, empirical methods, industry practices, adoption patterns, methodological gaps, AI narratives.

Background and Problem

• Problem / challenge: Current reporting on generative AI use is fragmented, incomplete, and often ambiguous, lacking methodological rigor and specificity. This hinders a comprehensive understanding of how generative AI is used in practice.
• Significance: Understanding generative AI use is critical for researchers, policymakers, and developers to address real-world implications, improve transparency, and guide responsible AI development and regulation.
• Motivation and related work: While prior work has focused on AI capabilities and benchmark evaluations, these do not reflect real-world use. Efforts like user studies, surveys, and log analyses have emerged but remain isolated and inconsistent. This paper seeks to systematically map the space of reporting practices and address these gaps.

Solution

• Proposed approach: A multi-dimensional framework to systematically characterize and analyze reporting on generative AI use, covering research questions, methods, and study populations.
• Novelty:
  1. Development of an empirically-informed framework that specifies dimensions of reporting on generative AI use.
  2. Application of the framework to analyze 111 industry documents, revealing patterns and gaps in current reporting practices.
  3. Critical reflection on the narratives advanced by industry reporting and their implications.
  4. Creation of a publicly available annotated dataset for further research.
• Procedure and key techniques:
  1. Framework development: Conducted an integrative review of 40 documents to identify dimensions of reporting (e.g., "who uses AI," "what is it used for") and methods (e.g., empirical, axiomatic).
  2. Framework validation: Iteratively refined the framework using additional documents and broader literature on technology use.
  3. Framework application: Analyzed 111 documents from six major AI providers (e.g., OpenAI, Google) using the framework, annotating use-related information and methods.
  4. Analysis: Identified trends, gaps, and mismatches in reporting practices, and evaluated the co-occurrence of use questions and methods.

Results

• Concrete findings:
  • 61% of documents report on potential use, 51% on actual use, and only 6% on projected use.
  • Key use questions like "where" and "when" are under-reported, and temporal patterns are rarely addressed.
  • Many documents lack methodological transparency, with 34% not reporting any method and 18% ambiguously describing methods.
  • Use information is often vague, over-generalized, or misaligned with the underlying evidence.
• Advantage over baselines:
  • The framework systematically captures diverse dimensions of reporting, enabling a more structured and critical analysis compared to ad hoc or fragmented approaches.
  • Reveals gaps and inconsistencies in industry reporting that were previously unexamined.
• Experiments / evaluation:
  • Corpus: 111 documents from six major AI providers (OpenAI, Google, Microsoft, Meta, Anthropic, AI2).
  • Metrics: Coverage of use questions, co-occurrence of questions, methods employed, and alignment between claims and evidence.
  • Analysis: Heatmaps, co-occurrence matrices, and thematic analysis of reporting patterns.
• Limitations and future work:
  • Limited to English-language documents and specific providers.
  • Does not include independent or non-industry reporting.
  • Future work could explore user needs for reporting, expand the framework to other contexts, and investigate global reporting practices.

Summary

This paper introduces a framework to systematically characterize reporting on generative AI use, addressing gaps in transparency and methodological rigor. By applying the framework to 111 industry documents, the study reveals patterns, omissions, and inconsistencies in current reporting practices, such as the overemphasis on potential use and the lack of contextual and temporal information. The framework and annotated dataset provide tools for researchers, policymakers, and practitioners to improve reporting practices and foster a more nuanced understanding of generative AI use.