IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth

We present IKIWISI, an interactive visual pattern generator for assessing the reliability of vision-language models in multi-object recognition tasks with arbitrary, user-defined objects in video data, where ground truth is often unavailable. The name IKIWISI is an acronym for the phrase "I know it when I see it" and reflects the tool’s grounding in human visual perception research, leveraging intuitive pattern recognition capabilities to facilitate trust and interpretability. IKIWISI employs an easily interpretable binary heatmap, where columns represent video frames and rows represent user-defined objects. Cells are color-coded green or red to indicate an object's presence or absence. Using a research-through-design approach, we refined IKIWISI through several iterations. A final study with 15 participants demonstrated that IKIWISI is easy to use and enables reliable model performance assessments that correlate with true performance, when available. Furthermore, users only need to inspect a tiny fraction of heatmap cells to reach conclusions. IKIWISI promotes transparency through visual interpretation, allowing users to quickly detect anomalies and focus on critical areas for further analysis. This makes it a valuable tool for evaluating vision-language models on user-defined objects in real-world scenarios.