Prof. Dr. Ignacio Alvarez

Driver states such as drowsiness, cognitive overload, or inattention are critical to road safety, yet their recognition is often complicated by ambiguous or conflicting signals. Human behavior and states are inherently complex and uncertain, which is overlooked by traditional driver monitoring systems as they typically output a single predicted state. Recent advancements in large language models (LLMs) offer a unique opportunity to interpret such ambiguity by common sense reasoning over multimodal inputs, and provide multiple context-aware interpretations and interventions.

In this project, students will design and prototype an in-vehicle assistant that can detect and assess a chosen driver state (drowsiness, attention, workload) through ambiguity-awareness. The system will utilize multimodal inputs like facial expressions, posture, eye tracking, and driving behavior, along with contextual cues (time of day, traffic, driving duration) to resolve data conflicts and ambiguity via LLM reasoning. Based on its interpretation, the assistant will present its top inferred states combined with confidence levels, and appropriate next steps such as initiating an intervention or requesting further input. The system will also incorporate user feedback to refine its reasoning process and improve ambiguity resolutions. Students will evaluate system’s usability, perceived effectiveness in conveying ambiguous and complex states, exploring how LLM-driven reasoning can support more transparent and explainable human-vehicle interaction.

In this course, students will customize AI agents to exhibit emotionally-aware behaviors that align with predefined user personas. Leveraging conversational design principles, they will develop tailored conversational flows, focusing on tone, empathy, and adaptability of the AI agent. Usability testing will be conducted to assess the assistant’s emotional relevance, user satisfaction, and the effectiveness of its personality in different in-car scenarios.

In this project, students will design and implement a prototype in-vehicle game that incorporates input from an AV safety model, such as an AV agent equipped with RSS in the CARLA driving simulator to provide feedback on safety-related actions. Students will evaluate their designed game for usability, engagement, and its effectiveness in increasing safety awareness among in driving scenarios. Deliverables will include a functional game prototype, an evaluation report, and insights into the application of gamification in AV safety education.

As vehicles become increasingly intelligent and assistive, the challenge of designing intuitive, safe, and low-distraction interaction methods becomes more critical. Voice, visual, and touch interfaces, although common and widely used, can be intrusive or unreliable in certain situations. Electromyography (EMG) sensors offer a promising alternative by detecting subtle muscle activations that reflect a driver's intent or emotional state. These muscle-based gestures can enable silent, quick, and low-distraction communication between driver and vehicle systems, supporting a more natural, intuitive, and adaptive human-vehicle interaction.

In this project, students will explore how EMG-based gestures can be leveraged to improve in-vehicle interaction and feedback. They will design and investigate how such gestures can support communication and feedback integration with an LLM-based in-cabin assistant. This includes how drivers might express intent, preferences, or reactions to system assistance through adaptive gestures; and how LLMs can interpret or adapt to such inputs to enable natural and context-aware interactions. Students will analyze how this modality for interaction affects overall usability, trust, and satisfaction, and examine how it complements/interferes with other modalities like voice or touch.

Within this seminar, we will analyze aspects of aggression, vandalism, harassment, and abuse directed at autonomous robots (e.g. delivery service robots, automated vehicles). Our goal is to develop interface and technological solutions that mitigate these risks while ensuring the vehicles remain safe and functional for all users. Based on an individual literature review (related work), students (in groups of two or three) will identify key challenges and explore innovative concepts. Possible topics will be developed together in a dedicated brainstorming session and refined/substantiated by the individual teams (with feedback from the lecturer).

Emotions like anger, frustration, or anxiety can compromise driving performance by impairing situational awareness, reaction times, and decision making. Recent advancements in large language models (LLMs) have shown reasoning capabilities in detecting complex emotions using multiple modalities. By integrating multimodal sensor data and contextual driving information, LLMs can also be leveraged to detect vehicle occupant emotions and trigger actions to mitigate their impact on driving performance.

In this project, the students will design and prototype an automotive in-cabin assistant to detect driver emotions and respond through context-aware recommendations to support safer driving. The system will utilize multimodal inputs (voice, facial features, heart rate, driving behavior), contextual signal like traffic conditions, driving duration, weather conditions. Based on this information, the assistant will generate appropriate responses such as offering calming suggestions, initiating empathetic conversations. Students will evaluate system’s usability, perceived appropriateness and supportiveness of the system, and investigate how LLM-driven emotion recognition and context-aware responses can enhance human-vehicle interaction.