General AI Systems and Interaction Design / GUI Semantic Modeling and Task Representation

How can semantic representations simultaneously capture task context, interaction information, and layout patterns of GUI screens?