Karthik Sai Pasupuleti

Driver states such as drowsiness, cognitive overload, or inattention are critical to road safety, yet their recognition is often complicated by ambiguous or conflicting signals. Human behavior and states are inherently complex and uncertain, which is overlooked by traditional driver monitoring systems as they typically output a single predicted state. Recent advancements in large language models (LLMs) offer a unique opportunity to interpret such ambiguity by common sense reasoning over multimodal inputs, and provide multiple context-aware interpretations and interventions.

In this project, students will design and prototype an in-vehicle assistant that can detect and assess a chosen driver state (drowsiness, attention, workload) through ambiguity-awareness. The system will utilize multimodal inputs like facial expressions, posture, eye tracking, and driving behavior, along with contextual cues (time of day, traffic, driving duration) to resolve data conflicts and ambiguity via LLM reasoning. Based on its interpretation, the assistant will present its top inferred states combined with confidence levels, and appropriate next steps such as initiating an intervention or requesting further input. The system will also incorporate user feedback to refine its reasoning process and improve ambiguity resolutions. Students will evaluate system’s usability, perceived effectiveness in conveying ambiguous and complex states, exploring how LLM-driven reasoning can support more transparent and explainable human-vehicle interaction.