Linting Style and Substance in READMEs

Paper Title

Linting Style and Substance in READMEs

Publication Info

• Topic area: Improving README quality through customizable linting tools.
• Keywords: README, linting, documentation quality, domain-specific language, large language models, accessibility, customization, user study, automation, software documentation.

Background and Problem

• Problem / challenge: Existing linters for documentation focus primarily on superficial aspects like style, grammar, and formatting. They fail to address deeper, domain-specific, and semantic issues such as accessibility, tone, and content completeness.
• Significance: READMEs are critical for software adoption, usability, and collaboration. Poorly written or incomplete READMEs can create barriers for users and developers, reducing project success and accessibility.
• Motivation and related work: Prior work has explored documentation quality and linting tools, but these are limited to predefined, rigid rules. Tools like markdownlint and Proselint focus on syntax and style but lack flexibility for domain-specific needs. This paper builds on these efforts by integrating programmatic and LLM-based approaches to address both style and substance.

Solution

• Proposed approach: LintMe, a customizable markdown linter that uses a domain-specific language (DSL) to define linting rules. It combines programmatic operators and LLM-based evaluations to address a wide range of README quality issues.
• Novelty:
  1. Integration of programmatic logic and LLMs to evaluate both syntactic and semantic aspects of READMEs.
  2. A lightweight DSL for creating customizable linting rules tailored to specific domains and user needs.
  3. Support for user-defined rules, enabling flexibility and adaptability to diverse documentation standards.
  4. Application of linting to non-traditional domains like culinary recipes, demonstrating the system's expressiveness.
• Procedure and key techniques:
  • Users create linting rules using a DSL that supports 21 operators, including programmatic checks (e.g., link validation) and LLM-based evaluations (e.g., tone analysis).
  • Rules can be authored, customized, and executed via a web-based playground or CLI.
  • LintMe provides presets for different domains (e.g., software libraries, datasets) and supports iterative refinement of rules.
  • Evaluations include user studies, LLM comparisons, and application to alternate domains like recipes.

Results

• Concrete findings:
  • LintMe flagged more issues than naive LLM usage, identifying an average of 25.4 issues per README compared to 9.6 (free-prompt) and 7.25 (rules-provided).
  • User study (N=11) participants found LintMe useful for identifying gaps, maintaining standards, and customizing rules, though some noted a learning curve.
  • Application to recipes showed that LintMe could enforce style guides and detect systematic rule violations across domains.
• Advantage over baselines:
  • LintMe outperformed naive LLM usage in identifying issues, particularly in nuanced areas like tone, objectivity, and accessibility.
  • Unlike traditional linters, LintMe supports deep customization and evaluates both style and substance.
• Experiments / evaluation:
  • User study: Participants used LintMe to lint and customize rules for their own READMEs.
  • LLM comparison: Tested LintMe against GPT-based models with and without predefined rules.
  • Expressiveness case study: Applied LintMe to culinary recipes, demonstrating its adaptability to non-README domains.
• Limitations and future work:
  • Vulnerability to malicious use due to operators like customCode and execute.
  • Dependence on LLMs introduces risks of non-determinism and hallucination.
  • Limited evaluation scope (e.g., small sample size for LLM comparison, specific domains).
  • Future work includes expanding to full documentation websites, improving usability, and exploring hierarchical error bundling.

Summary

LintMe is a novel linting tool that addresses both style and substance in READMEs by integrating programmatic logic with LLM-based evaluations. It enables users to create and customize rules using a lightweight DSL, making it adaptable to diverse domains and community standards. Evaluations show that LintMe outperforms naive LLM usage and traditional linters in identifying nuanced issues. While limitations like LLM reliance and usability challenges remain, LintMe demonstrates the potential for extending linting to complex documentation and culturally mediated text-based domains.