Aligning Data with the Goals of an Organization and Its Workers: Designing Data Labeling for Social Service Case Notes

Title of the Paper

Aligning Data with the Goals of an Organization and Its Workers: Designing Data Labeling for Social Service Case Notes

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Social Services, and Data Labeling Optimization
• Keywords: Social Work, Nonprofit Organizations, Case Notes, Data Collection Practices, Data Labeling, Design Principles

Research Background and Issues

• Issues and Challenges:

  • In the field of social services, challenges in data collection practices include misalignment between data objectives and social workers' goals, inadequate data labeling design, and system usability issues.
  • While data labeling is central to project evaluation and funding reports, it offers limited support for actual case management and service delivery.
  • Current data collection primarily relies on manual recording by social workers, which often results in incomplete or inaccurate data and imposes an additional burden on their work.

• Significance:

  • Data-driven practices have become essential tools for nonprofit organizations to evaluate performance and secure funding, influencing organizational strategic planning and service optimization.
  • Proper design can balance organizational goals with individual social workers' motivations, reducing the burden of data recording while ensuring data quality.

• Research Motivation and Related Work:

  • Current HCI research largely focuses on the challenges of data collection rather than solutions.
  • Previous studies have highlighted the misalignment between data labeling and performance evaluation but have rarely explored how to make data collection more meaningful and efficient.

Solution

• Methods and Innovations:

  • The authors propose a series of design principles to improve data labeling systems through collaboration between social workers and organizational managers.
  • The study employs semi-structured interviews and the speed dating methodology to generate and explore 15 design possibilities.
  • Core innovations include redesigning the data labeling system to enhance goal alignment, improve the visibility of data label usage, increase portability, and enhance labeling accuracy.

• Implementation Steps and Key Technologies:

  • Integrating User Needs into Solutions: Interviews were conducted to identify social workers' pain points and needs, including unclear label definitions, insufficient detail, and lack of transparency in data usage.
  • Speed Dating Design Testing: Design concepts were presented through storyboards, and feedback and feasibility were discussed with multiple stakeholders, including social workers, project analysts, and managers.
  • The study specifically suggests leveraging artificial intelligence and natural language processing tools to improve label selection accuracy and content filtering.

Research Outcomes

• Specific Outcomes:

  • Proposed 15 design concepts, including redesigning data labels for greater granularity, creating dashboards to visually display the impact of data on service performance, and providing social workers with real-time label definitions and training support.
  • Suggested the development of user-centered tools, such as AI-recommended relevant labels and filtering features, to improve the efficiency of data label recording.

• Advantages and Comparisons:

  • The redesigned data labeling system fosters both intrinsic and extrinsic motivation among social workers, reducing the interference of data collection with service quality.
  • It helps organizations better reflect the work of social workers, providing a richer data foundation for performance evaluation, in contrast to existing systems that focus solely on outcome-oriented labels.

• Experimental or Evaluation Results:

  • All participants emphasized the importance of aligning data labeling with social workers' goals.
  • Case studies showed that some proposals, such as reconciling the granularity of label concepts, were particularly well-received, though concerns about the complexity introduced by an excessive number of labels were also noted.

• Limitations and Future Directions:

  • Limitations:
    • The study only explored the needs and data systems of a single organization, which may not generalize to other fields.
    • The proposed design concepts were not field-tested, so their actual effectiveness remains unverified.
  • Future Directions:
    • Test these design concepts in a broader range of social service contexts.
    • Optimize AI-based data labeling systems to ensure accuracy and user-friendliness.
    • Explore mechanisms for multi-stakeholder collaborative design to reduce potential stress caused by data labeling systems.

This study provides profound insights into how data labeling design can better align with the goals of social service organizations and social workers, offering a fresh perspective for research in human-computer interaction.