PolicyCraft: Supporting Collaborative and Participatory Policy Design through Case-Grounded Deliberation

Research Background and Issues

• What problems or challenges did the authors identify?
  Current community and organizational policy-making often adopts a top-down centralized approach, limiting the participation of affected stakeholders. This can result in policies that do not align with community needs and are perceived as illegitimate, leading to dissatisfaction or rejection. This issue affects not only online communities (e.g., content moderation policies) but also local governments (e.g., urban planning regulations).

• Why is this issue important?
  The illegitimacy of policies may lead community members to reject them, organize protests, or even leave the community. Such phenomena pose significant obstacles to community stability, trust, and the achievement of shared goals.

• Research Motivation and Related Work
  To address the above issues, the authors explore how to support more collaborative and participatory policy design methods. Drawing on prior research, they propose that concrete cases in group discussions are essential tools for building shared understanding and identifying mismatches between abstract policies and real-world scenarios. However, the connection between these cases and policy design remains fragmented and unsystematic. Therefore, it is necessary to develop tools to systematically support case-driven policy design.

Solution

• What methods or solutions did the authors propose?
  The authors designed a system called PolicyCraft to support communities in collaboratively designing policies based on case studies. By enabling users to propose, evaluate, revise, and vote on specific cases, the system helps community members identify disagreements in policy design (whether at the textual level or deeper conceptual differences).

• What is innovative about this solution?
  PolicyCraft systematically links concrete cases with abstract policy design, providing an efficient tool to guide community discussions and collaboration. This structured approach helps identify the sources of disagreement (whether at the case level or policy level) and facilitates effective policy iteration driven by consensus.

• What are the implementation steps and key technologies used?

  1. Initial Input: Organizers provide several seed policies and related cases to address the cold-start problem in discussions.
  2. Iterative Process: Community members vote on and discuss specific cases, propose improvements to policies, and annotate existing cases (e.g., whether the policy clearly addresses the case).
  3. Finalization: Members vote on the proposed policies and provide feedback to help select the final policy to be implemented.
     The system also incorporates a built-in AI assistant (based on LLM) to support case generation, policy revision, and proposals.

Research Outcomes

• What specific outcomes were achieved?

  1. Policies designed using PolicyCraft received stronger community support and greater consensus in voting, showing significant advantages compared to baseline systems without case scaffolding.
  2. Compared to baseline conditions, PolicyCraft more frequently encouraged members to modify policies based on cases proposed by others, demonstrating stronger collaboration.
  3. Course instructors evaluated policies generated through PolicyCraft as more comprehensive and creative, though occasionally verbose in certain scenarios.

• What advantages does it have over existing solutions?
  PolicyCraft significantly outperforms existing tools: it not only supports the collection of opinions but also explicitly helps communities collaboratively draft and revise policies in a case-driven manner. This integrated process enhances the systematic nature of policy design.

• What were the experimental or evaluation results?
  Experiments showed that groups using PolicyCraft:

  • Achieved advantages in balancing disagreement and consensus in policy design, with higher support rates and lower disagreement (low entropy) in voting.
  • Better helped members understand each other's perspectives, enhancing collaboration. The proportion of policy designs based on specific cases increased significantly.
    External evaluations indicated that policies designed with PolicyCraft were clearer, though occasionally verbose in certain scenarios.

• Limitations and Future Directions
  Limitations:

  • Due to its case-driven nature, policies generated by PolicyCraft may be overly specific, leading to redundancy and complexity in implementation. This aligns with feedback from instructors and external evaluations.
  • The limited usage of the AI assistant indicates a need to further optimize its design to enhance user experience.

  Future Directions:

  1. Support for balancing policy specificity and abstraction: Develop tools to help members summarize case-driven details in a timely manner, ensuring policy conciseness.
  2. Strengthen global review and synthesis capabilities to reduce redundant policies.
  3. Enhance collective sense-making through visual interfaces and LLM-generated quick summaries.
  4. Explore better ways to coordinate different levels of power structures in communities, particularly in environments with strong centralized decision-making.

Through the design and application of PolicyCraft, the authors propose a novel approach to addressing issues such as significant disagreements and limited participation in traditional policy design. This work provides important insights for future tool design and methods of public participation.