Governor: Turning Open Government Data Portals into Interactive Databases

Title of the Paper

Governor: Turning Open Government Data Portals into Interactive Databases

Paper Information

• Subject Area: Data Management and Open Government Data
• Keywords: Open Data, Data Integration, Databases, User Interface, Data Visualization, Data Provenance

Research Background and Problem Statement

• Problems or Challenges:

  • Current Open Government Data Portals (OGDPs) have limited functionality, primarily offering search capabilities for dataset titles and descriptions. Users must download files and use third-party tools to analyze the data.
  • Non-programmer users face difficulties in efficiently finding and utilizing open data, and the data integration process is cumbersome.
  • Existing OGDPs fail to meet advanced user needs for data integration and data provenance, requiring users to independently manage the sources of raw data, which is error-prone and challenging.

• Significance of the Problem:

  • Open data aims to enhance government transparency, enabling citizens, researchers, and journalists to analyze data and monitor national policies and societal performance.
  • Addressing the limitations of OGDPs could provide significant societal value, such as identifying inefficiencies in government operations and uncovering corruption.

• Research Motivation and Related Work:

  • Existing OGDP search functions rely on metadata accuracy, and poor metadata quality can hinder users from discovering relevant data.
  • Dataset search engines (e.g., Google Dataset Search) and existing interactive data integration systems (e.g., Voyager, Ronin) fail to meet the non-programming needs of OGDP users and lack support for union, join operations, and data provenance.
  • No system currently provides non-technical users with database management system (DBMS) functionalities to effectively unlock the potential of open data integration.

Solution

• Proposed Method or Solution:

  • Introduced a browser-based open-source system named Governor, which models OGDPs as interactive databases.
  • By extending existing OGDP functionalities, Governor supports record-level search within tables and direct data table previews without requiring file downloads.
  • Provides intelligent recommendations, identifying and suggesting unionable and joinable tables for data integration.
  • Offers a color-coded visualization interface, enabling users to interactively manage data integration sources and trace back to the original tables in OGDPs.

• Innovations:

  • Introduced core DBMS functionalities to OGDPs, such as record-level search, data integration, and provenance management.
  • Designed an interactive system with a low technical barrier, addressing the pain points of non-programmer users.
  • Provided clear visualization and provenance features, allowing users to quickly understand the composition and sources of integrated tables.

• Implementation Steps and Key Technologies:

  1. Search Functionality:
     • Offers two modes: metadata search and record-level search.
     • Handles search requests using MongoDB and Elasticsearch.
  2. Data Integration:
     • Extends table rows via union operations or adds new columns via join operations.
     • Performs interactive SQL processing of integrated tables in the browser using DuckDB-Wasm.
  3. Data Provenance and Color Coding:
     • Uses color coding to indicate the sources of merged or joined data in integrated tables.
     • Allows users to trace value origins back to source columns and original tables via a visual interface.
  4. System Architecture:
     • Fetches OGDP metadata and table data via the CKAN API.
     • Compresses data using the Parquet format to reduce transmission load.
     • Supports integration with various government data portals.

Research Outcomes

• Specific Achievements:

  • Designed and implemented the Governor system, equipping OGDPs with interactive DBMS functionalities.
  • Evaluated the system through two user studies, demonstrating its efficiency and user-friendliness in data search and integration tasks.

• Advantages over Existing Solutions:

  • Compared to existing tools like Google Dataset Search, Auctus, and TODS, Governor supports deep record-level searches and direct multi-table integration.
  • Automated integration recommendations significantly reduce the time and error rates associated with manual downloading, organizing, and merging of tables.
  • The visual data provenance feature greatly improves the efficiency of data auditing and verification.

• Experimental or Evaluation Results:

  • In a controlled task and an open-ended task user study:
    • Most users successfully completed the tasks with good performance in terms of time and accuracy.
    • NASA TLX questionnaire results indicated low workload perception and high task success rates when using Governor.
    • Users generally found Governor easy to learn, intuitive, and powerful, particularly in table integration and data provenance functionalities.
  • Compared to existing tools, Governor provided a superior user experience, especially for non-programmer users, who showed high acceptance of the system.

• Limitations and Future Directions:

  • Limited filtering and data cleaning functionalities; users expressed a need for additional features like column-based filtering and data aggregation.
  • Current data integration is restricted to recommended tables; future work could allow users to freely select tables for integration.
  • Academic validation did not include direct baseline comparisons (e.g., traditional OGDP tools); further research in real-world scenarios is needed.