Research Directions

Our multimodal interaction research aims to integrate multiple input channels such as gestures, speech, and posture to create more natural and efficient human-computer interaction methods. This research primarily encompasses two directions: scripted multimodal interaction design and natural multimodal behavior understanding.

Scripted multimodal interaction refers to predefined interaction schemes designed by developers, such as using specific gestures combined with voice commands to accomplish target selection and task execution. In such interactions, users must follow established rules and use designated multimodal inputs to convey their intentions to the system. Our research in this direction focuses on exploring users' multimodal usage patterns and preferences across different scenarios and tasks, analyzing users' semantic understanding capabilities and encoding spaces for different modal information, thereby designing interpretable, learnable, and generalizable scripted multimodal interaction schemes for various interaction functions.

On the other hand, natural multimodal behavior understanding emphasizes allowing users to interact with systems using multimodal behaviors in a more free-form manner. Compared to scripted approaches, this method better aligns with user intuition, with natural and fluid interaction processes and minimal additional learning costs. However, this also places higher demands on multimodal behavior recognition and intent inference algorithms. To this end, we start from the collection and analysis of users' natural multimodal behaviors, combining traditional machine learning methods with large language models to build intelligent interaction models capable of understanding natural multimodal expressions, supporting more robust and generalizable human-computer interaction systems.

Our robot interaction research focuses on user-centered communication mechanisms, aiming to enhance the usability, comprehensibility, and trustworthiness of robotic systems through appropriate interaction and feedback modalities, making them better aligned with user needs and usage patterns.

In our specific research, we emphasize exploring users' core expectations regarding functional capabilities and interface information presentation when collaborating with robots to complete tasks, and summarizing the main difficulties and decision-making burdens users encounter during interactions. Through user studies, we analyze design considerations for robotic systems in task planning, execution feedback, operational transparency, and information presentation methods. We further investigate preference differences among different feedback modalities (such as visual and auditory) in robot control and interaction, as well as their impacts on interaction efficiency, workload, and user experience, thereby providing empirical evidence and design recommendations for more natural and efficient human-robot collaboration interfaces.

Additionally, we treat emotional factors as an important dimension affecting human-robot interaction quality, systematically studying the mechanisms through which robot emotional expression and emotion regulation strategies influence user feelings and behaviors. We focus on exploring how different emotion regulation approaches (such as empathetic versus non-empathetic regulation) affect user emotional changes, trust formation, advice adoption tendencies, and overall interaction evaluations across different contexts. Through experimental paradigms such as video and simulation tasks, we establish analytical frameworks to provide theoretical and experimental support for building robot interaction strategies with greater social intelligence and emotional adaptability.

Situation awareness refers to an individual's ability to perceive and understand critical information in dynamic environments, as well as predict future states. In high cognitive load, high information density application scenarios such as autonomous driving takeovers and UAV reconnaissance, whether users possess accurate situation awareness directly impacts their decision-making quality and task safety. Therefore, identifying and enhancing users' situation awareness levels has become a critical issue in intelligent system design. Our research primarily focuses on two directions: situation awareness state prediction and enhancing situation awareness through interface design.

First, we have constructed a comprehensive framework for situation awareness prediction that systematically integrates multiple influencing factors including user capabilities, user states, user behaviors, and situational contexts. Under this framework, we combine individual difference characteristics such as user age, gender, and driving experience, along with physiological and behavioral signals such as eye movements, electrocardiography, and skin conductance, with environmental features such as the size and color of critical targets in the scene, to build situation awareness state prediction models. These models can help systems identify situation information that users may miss or misinterpret, and further generate targeted prompts and assistance information, thereby supporting more timely and reliable decision-making.

Furthermore, we investigate the extent to which different types of user interfaces can help users more effectively grasp situational information. Related research covers multiple dimensions including content design for interface information presentation, presentation modalities (such as visual, auditory, or multimodal fusion), and information organization and expression methods. Through user experiments and statistical analysis, we strive to summarize interface design patterns across different task scenarios and propose generalizable design recommendations for situation awareness assistance interfaces, aiming to improve user comprehension efficiency, reduce cognitive load, and enhance task safety and overall interaction experience.

Our research in virtual and augmented reality interaction focuses on creating more comfortable and efficient immersive interaction environments for users, with particular emphasis on interface layout optimization, virtual button size and distribution design, and operational performance and usability across different interaction modalities. Through systematic user experiments and interaction data analysis, we evaluate users' behavioral characteristics and operational efficiency when completing fundamental tasks such as selection and clicking in virtual environments, proposing quantitative evaluation methods for virtual interaction tasks and providing verifiable design foundations and optimization directions for VR/AR interface control design.

Furthermore, our research emphasizes understanding interaction processes and cognitive state changes from users' multimodal behavioral data, and building adaptive user interfaces based on these dynamic information. We combine multimodal signals such as eye movements and hand gestures to analyze the impacts of bare-hand clicking and traditional control methods on users' visual search strategies, attention allocation patterns, and operational load. We also explore modeling methods that leverage multimodal features to predict users' attention states. This work establishes the foundation for providing real-time user state recognition and adaptive interaction support for immersive systems, advancing VR/AR interaction design from static interface optimization toward intelligent, personalized, and dynamic adaptation.