How To Draw Commands? An Elicitation Study for Sketching on Spreadsheets

Research Background and Issues

• Issues and Challenges:
  • Modern user interfaces (UIs) rarely leverage people's natural sketching abilities for command input, relying instead on traditional keyboards, mice, or touch gestures.
  • Particularly in complex data operations such as spreadsheets, users must navigate intricate menus or input formulas via keyboards, which is especially inconvenient on touch devices.
• Significance:
  • Humans intuitively sketch, which can be used to directly express commands as visual information, reducing the interaction barriers with complex UIs.
  • Supporting direct spreadsheet manipulation through sketches can enhance user experience on touch devices and alleviate the inconvenience of using on-screen keyboards.
• Research Motivation and Related Work:
  • The authors identified a conflict between the complexity of spreadsheets (e.g., formulas, data visualization) and the demand for direct interaction on modern devices, especially touch devices.
  • While numerous studies have explored data input through natural language, visual operations, or gestures, few have focused on "expressing commands directly through sketches."

Solution

• Approach and Solution:
  • The authors conducted an "elicitation study," recruiting 36 participants to express various commands in sketch form on static spreadsheet images.
  • A set of 20 different tasks was provided, and participants' sketch-based inputs, including graphics, text, and other interaction methods, were collected.
• Innovations:
  • Compared to traditional keyboard input and complex UI operations, direct sketching allows for a more natural expression of multidimensional complex data operations.
  • The authors analyzed users' visual expression strategies in sketches, such as "selection," "arrows/connections," and "implicit information," proposing a set of potential universal visual languages.
  • They highlighted the potential for expressing ambiguous or repetitive commands and inferring incomplete commands in complex operations through "context."
• Implementation Steps and Techniques:
  • Tasks were categorized into five major types (data editing, structural changes, formatting, visualization, and mathematical calculations) to cover a wide range of spreadsheet functionalities.
  • Static spreadsheet screenshots and a custom application were used to record participants' sketch expressions, which were then categorized and analyzed based on the "action-parameter decomposition" principle.
  • The final classification system included categories such as arrows (CON), selection (SEL), and text symbols (TXT), forming a framework for deconstructing meanings.

Research Outcomes

• Specific Findings:
  • Identified common patterns and expression strategies used by users when sketching commands: for example, arrows to indicate direction for selection/operations, and cell selection achieved through boxing or underlining.
  • Summarized the potential of using specific graphical methods, such as arrows or brackets, to express "data repetition" (e.g., copying rows or calculating column sums).
  • Recognized the widespread phenomenon of expressing "implicit commands," such as inferring unspecified parameters through context and spreadsheet structure.
• Advantages Over Existing Solutions:
  • The visual "direct sketching" approach is more intuitive than traditional keyboard input on touch devices and allows for more precise expression of complex commands compared to touch gestures.
  • This method addresses issues of screen keyboard obstruction and operational inconvenience in large-screen interactive environments or touch devices.
• Experimental or Evaluation Results:
  • "Command consistency" (agreement rates) showed that while users tended to adopt similar expression strategies, there were significant variations in visual details, indicating the need for further abstraction of a universal visual language.
  • Overall consistency was moderate to high (average agreement rate: ~0.29); consistency was higher for simple and repetitive tasks, while innovative tasks exhibited greater diversity.
• Limitations and Future Directions:
  • Limitations:
    • Participants were aged 18-40, lacking data from older users.
    • The absence of real-time feedback during testing limited the simulation of dynamic interaction environments.
  • Future Directions:
    • Explore visual sketch expressions for more complex formulas and develop "interpretation engines" capable of real-time processing of such inputs.
    • Investigate the potential of combining sketches with voice and multimodal interactions like touch.
    • Build and evaluate prototype interfaces based on sketching to test usability and learning curves.

Through this study, the authors challenged traditional paradigms of spreadsheet operations, providing a solid foundation for expressing complex operations through visual sketches. Future research and development directions will drive broader applications, including sketch recognition supported by machine learning and the definition of a universal sketch language across devices.