Semantic See-through Goggles: Wearing Linguistic Virtual Reality in (Artificial Intelligence)

When language is used as a medium for sensory information—as when one describes a scene in words—a kind of virtual reality emerges: realities projected into the same sentence become virtual/equivalent. We call this Linguistic VR (LVR). LVR is constituted by codes shaped chiefly by majority cultural norms. In today’s world—where AI, largely built on linguistic data and processes, is deeply entangled with the everyday mediation of sensory information—it is necessary to critically re-examine the virtuality of this VR. We propose Semantic See-through Goggles, a system that makes manifest, as a first-person experience, the LVR latent in the linguistic mediation of scenes, enabling intuitive understanding and analysis of its properties and issues. The system inserts a serial image-to-text and text-to-image transformation pipeline between the camera and head-mounted display (HMD) of video see-through goggles: the live view is converted into a single line of text, then re-generated as an image, and only this mediated image reaches the wearer’s eyes. We built a prototype and validated its basic properties, followed by a qualitative analysis of wearer experiences. The results suggest that this method enables subjective experience—and thus understanding—of the transformations of environmental information, and of the attendant issues, induced by the linguistic mediation of vision. We also obtained preliminary insights into both AI-driven linguistic mediation of sensory information and problems intrinsic to linguistic mediation itself.