Filling the Gap: LLMs as Scaffolds for Competency Question Instantiation

Knowledge graphs (KGs) are a powerful way of representing information for digital humanities. However, non-technical users often struggle at the outset of exploration, a challenge defined as the Initial Exploration Problem. The Tús Maith framework addresses this issue through curated natural language questions and answers (CuQAs) created from Competency Questions (CQs) that aim to convey the scope of a KG and provide meaningful entry points into it. While prior work has explored using large language models (LLMs) for CQ template generation, the template-filling step, where questions and answers are instantiated with entity information, remains a key challenge. In this paper, we evaluate whether LLMs have the capacity to support domain experts in this stage, focusing on the Virtual Record Treasury of Ireland (VRTI) KG, where accuracy, provenance, and robustness are crucial for practical use. Using structured JSON inputs derived from popular search terms and expert-authored templates, we generated and assessed 24,900 question-answer pairs across four LLMs (GPT-5, DeepSeek-V3.1, Gemini 2.0 Flash, Qwen-2.5-72B) under two provenance conditions (basic vs. full). Our evaluation considers slot fidelity, semantic similarity, completeness, hallucination rates, and runtime efficiency, with statistical tests conducted per run per LLM, and additional batch-level analysis (n = 68) to isolate provenance requirement effects. We further show that a lightweight JSON validation check is an effective proxy for ground truth semantic evaluation of factual question-answer pairs. These LLM-generated, validated questions form an intermediate step in the lifecycle from abstract CQ templates to filled-in questions and answers intended to be reviewed and refined by the VRTI KG’s domain experts (historians) to produce the final user-facing questions (CuQAs). To demonstrate the practical impact, we present a prototype (TMv1) of the Tús Maith framework and highlight the design implications for curator-facing interfaces: provenance-transparent interaction, validation-integrated workflows, and performance-transparent model selection.