Just-In-Time Objectives: A General Approach for Specialized AI Interactions

Paper Title

Just-In-Time Objectives: A General Approach for Specialized AI Interactions

Publication Info

• Topic area: Adaptive AI systems for user-specific interactions
• Keywords: Large language models, user-specific objectives, adaptive interfaces, generative AI, human-AI interaction, tool generation, expertise generation, evaluation, task-specific optimization, interactive systems

Background and Problem

• Problem / challenge: Large language models (LLMs) often produce generic outputs due to pre-defined training objectives that fail to adapt to specific user needs at interaction time. Users struggle with underspecified prompts and lack control over AI behavior.
• Significance: Addressing this limitation enables LLMs to produce more specialized, useful, and contextually relevant outputs, enhancing user productivity and satisfaction.
• Motivation and related work: Prior work highlights cognitive barriers in human-LLM interaction, such as difficulty in envisioning goals and steering model behavior. Studies on adaptive interfaces and dynamic UI generation provide foundational insights but lack generalizable methods for user-specific specialization across domains.

Solution

• Proposed approach: Just-in-time (JIT) objectives—a method for inferring user goals from observed context and dynamically steering LLM behavior to align with these objectives.
• Novelty:
  1. A generalizable architecture for inducing and applying JIT objectives across task domains.
  2. Integration of JIT objectives into generation and evaluation components of LLM systems.
  3. Implementation of the Poppins system, which generates tailored tools and expert responses based on user-specific objectives.
• Procedure and key techniques:
  • Objective induction: Infers user goals from interaction traces (e.g., screenshots, text) and translates them into JSON specifications with name, description, and importance weight.
  • Generation: Applies JIT objectives to optimize LLM outputs, such as feedback, tools, or expert responses.
  • Evaluation: Uses JIT objectives to assess and refine generated outputs, enabling iterative improvement.
  • System implementation: Poppins integrates JIT objectives to produce interactive tools and expert feedback tailored to user tasks.

Results

• Concrete findings:
  • JIT objectives achieve high accuracy (M = 2.04, SD = 0.5; Study 1) and usefulness (M = 2.18, SD = 0.55; Study 1) on a 7-point Likert scale.
  • Outputs generated with JIT objectives achieve win rates of 66–86% over baseline LLM outputs across experts, tools, and feedback.
  • Best-of-N sampling with JIT evaluators improves output quality, with diminishing returns beyond 10 samples.
• Advantage over baselines:
  • JIT objectives produce outputs that are more specific, goal-aligned, and preferred by users compared to generic LLM outputs.
  • Poppins-experts achieves higher ratings for overall quality (+0.59), user control (+0.35), usefulness (+0.23), and relevance (+0.11) compared to baseline systems.
• Experiments / evaluation:
  • Study 1 (N = 14): Evaluated accuracy and utility of JIT objectives on participant-submitted browser traces.
  • Study 2 (N = 205): Expanded evaluation across diverse tasks and domains, confirming generalizability.
  • In-lab sessions (N = 17): Explored qualitative user feedback on Poppins-generated tools and expert responses for writing tasks.
• Limitations and future work:
  • Time cost: Inducing and applying JIT objectives takes 1–3 minutes, limiting applicability for quick tasks.
  • Context visibility: Single snapshots may miss nuanced or long-term user goals.
  • Privacy risks: Observing user context raises concerns about data security and consent.
  • Generated artifacts: Outputs may lack proper attribution or reliability, especially for real-world expert references.

Summary

This paper introduces just-in-time objectives, a method for dynamically inferring and applying user-specific goals to steer LLM behavior. The approach is instantiated in the Poppins system, which generates tailored tools and expert responses based on observed user context. Evaluations demonstrate that JIT objectives improve output accuracy, utility, and user satisfaction compared to baseline LLMs. While the method addresses persistent challenges in human-AI interaction, future work should explore scalability, privacy safeguards, and broader applicability across domains.