Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability

Paper Title

Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability

Publication Info

• Topic area: Human-AI collaboration in cognitive tasks under time constraints
• Keywords: Large language models, critical thinking, time constraints, human-AI collaboration, cognitive performance, AI-assisted reasoning, decision-making, LLM access timing, time availability, experimental study

Background and Problem

• Problem / challenge: The impact of large language models (LLMs) on critical thinking performance remains unclear, particularly under varying time constraints. Existing studies lack systematic exploration of how LLM access timing and time availability interact to shape outcomes.
• Significance: Understanding the nuanced effects of LLMs on critical thinking is essential for designing effective human-AI collaboration tools and optimizing their use in real-world scenarios.
• Motivation and related work: Prior research has shown mixed results regarding LLMs’ influence on cognitive tasks, often relying on self-reports rather than objective performance measures. Studies have examined isolated aspects of time constraints but have not addressed their combined effects systematically.

Solution

• Proposed approach: A 4×2 factorial experiment investigating the effects of LLM access timing (Early, Continuous, Late, No LLM) and time availability (Insufficient vs. Sufficient) on critical thinking performance.
• Novelty:
  1. Empirical evidence on how LLM access timing and time availability interact to shape critical thinking outcomes.
  2. Insights into the mechanisms underlying these effects, including behavioral engagement and cognitive processes.
  3. Recommendations for designing temporally-aware LLM tools to support critical thinking tasks.
• Procedure and key techniques:
  • Participants (n=393) completed a critical thinking task requiring reasoned decision-making based on diverse documents.
  • Performance was assessed via essays, recall, evaluation, and comprehension measures.
  • Experimental conditions varied LLM access timing and time availability.
  • Behavioral engagement was analyzed through interaction logs, textual overlap, and self-reports.

Results

• Concrete findings:
  • Under Insufficient time, Early and Continuous LLM access improved Essay performance but impaired Recall.
  • Under Sufficient time, Late and No LLM access yielded better Essay performance, with Late LLM access reducing Myside Bias.
  • Participants with Early and Continuous LLM access showed limited improvement in Recall and Essay performance when given more time.
• Advantage over baselines:
  • Late LLM access under Sufficient time preserved argument quantity while reducing one-sided reasoning (Myside Bias).
  • Early and Continuous LLM access provided initial scaffolding under time pressure but constrained deeper deliberation when time was sufficient.
• Experiments / evaluation:
  • Participants completed a civic decision-making task using curated documents.
  • Essay scores were calculated based on valid arguments, document references, and trustworthiness evaluations.
  • Recall, Evaluation, and Comprehension scores captured additional cognitive activities.
• Limitations and future work:
  • The study focused on a specific task and framework, limiting generalizability to other domains.
  • Conducted in a controlled environment with crowdworkers; real-world dynamics may differ.
  • Future research should explore domain-specific tasks, naturalistic settings, and alternative time constraint configurations.

Summary

This study systematically examined the effects of LLM access timing and time availability on critical thinking performance. Results revealed a striking temporal reversal: Early and Continuous LLM access enhanced performance under time pressure but impaired it with sufficient time, while Late and No LLM access showed the opposite pattern. Behavioral analyses highlighted distinct engagement strategies, with Late LLM access supporting balanced reasoning under sufficient time. These findings underscore the importance of temporally-aware LLM designs and challenge simplistic narratives about LLMs’ universal benefits or harms. Future work should extend these insights to diverse domains and real-world contexts.