Statslator: Interactive Translation of NHST and Estimation Statistics Reporting Styles in Scientific Documents

Title of the Paper

Statslator: Interactive Translation of NHST and Estimation Statistics Reporting Styles in Scientific Documents

Paper Information

• Domain: Human-Computer Interaction (HCI), transformation and interaction of statistical reporting styles
• Keywords: statistics, interactive systems, reading interface, estimation statistics, NHST, transparent statistics, exploratory explanation

Research Background and Problem Statement

• Identified Problems or Challenges:

  1. Statistical reporting styles in scientific literature are diverse, often difficult to understand, incomplete, or even misleading.
  2. Reporting p-values in NHST (e.g., significance testing) is prone to misinterpretation and fails to effectively capture the uncertainty of results.
  3. Inconsistencies in symbolic representation and numerical precision of statistical reports across documents, along with graphical representations, may lead to misunderstandings.

• Why It Matters:

  • Statistical reporting is a core component of scientific research, directly influencing readers' assessment of result reliability and cross-document comparisons.
  • The coexistence of multiple statistical reporting styles fails to meet all reader preferences, and many documents lack sufficient information for comprehensive interpretation.

• Motivation and Related Work:

  • The HCI research community has begun advocating for the use of estimation statistics (e.g., effect sizes and confidence intervals) instead of traditional NHST, but this shift has limited coverage, with many documents still adhering to traditional reporting methods.
  • While there are methods to improve personalized reading experiences of documents, there is still a lack of tools capable of dynamically transforming statistical reports.

Proposed Solution

• Method or Solution:

  • Statslator System: An interactive tool capable of transforming NHST or estimation statistical reports in scientific documents into the reader’s preferred style, including formats such as confidence intervals, p-values, standardized effect sizes, and more.
  • The system extracts and transforms statistical information from text using formulas and generates graphical and interactive explanations to help readers better understand the results.

• Innovative Features:

  • Developed transformation methods to support bidirectional conversion between different reporting styles, addressing deficiencies in existing literature.
  • Operates without requiring raw data, effectively transforming information based solely on existing document content.
  • Combines personalized reading with transparency, enabling users to verify the accuracy of statistical reports.

• Implementation Steps:

  1. Statistical Information Extraction: Use OpenAI GPT-3.5 or regular expressions to extract reported statistical values and related information, marking sources and verifying accuracy.
  2. Formula Transformation: Propose and validate a series of mathematical formulas to convert statistical information into formats such as confidence intervals, p-values, and standardized effect sizes.
  3. Visualization: Support interactive charts, dynamic hypothetical outcome plots (HOPs), and allow users to customize chart styles.
  4. Error Detection and Feature Expansion: Provide functionality to check report consistency, preventing errors or inconsistencies in author-reported data.

Research Outcomes

• Specific Results:

  1. Document analysis revealed that over 65% of CHI papers contain sufficient statistical information for conversion between different styles.
  2. Conversion between different reporting formats (e.g., t-values, means, standard deviations, or p-values) achieved accuracy and confidence interval coverage comparable to results calculated from raw data.
  3. Developed the Statslator PDF viewer, offering real-time interactive conversion and report validation functionality, demonstrated through three specific application cases.

• Advantages Over Existing Methods:

  • Compatible with traditional static documents (e.g., PDF format) without requiring authors to revise their papers.
  • Provides reader-driven flexibility in statistical style selection, supporting graphical and interactive content.
  • Helps readers assess report accuracy, supplement missing information, and compare across papers.

• Experimental or Evaluation Results:

  • Multiple formula transformation strategies were validated for accuracy under common experimental conditions, including methods suitable for small sample experiments.
  • The system detected actual errors in reports and flagged them, helping users identify mistakes or inconsistencies.

• Limitations and Future Directions:

  1. Some statistical reports may lack sufficient information, leading to conversion failures, such as reports containing only p-values that cannot derive confidence intervals.
  2. System accuracy depends on the quality of author-reported content and cannot handle data reported using incorrect methods.
  3. Currently supports a limited range of statistical tests (e.g., t-tests), requiring future expansion to include more non-parametric tests and Bayesian statistical reports.

Summary: The Statslator system demonstrates the feasibility of instant transformation of existing statistical reporting styles, addressing comprehension barriers for scientific readers while freeing authors to adopt better statistical practices. It also offers a dynamic upgrade path for traditional publication formats that do not support interactivity.