Towards Interactive Virtual Characters for Augmented Reality Applications

The Inria Centre at Rennes University is one of Inria's eight centres and has more than thirty research teams. The Inria Centre is a major and recognized player in the field of digital sciences. It is at the heart of a rich R&D and innovation ecosystem: highly innovative PMEs, large industrial groups, competitiveness clusters, research and higher education players, laboratories of excellence, technological research institute, etc.

Subject

Interactions with other persons are ubiquitous in our daily lives, ranging from social interactions in personal and professional situations, to physical interactions, e.g. in sports. For this reason, researchers have striven for years to understand, model, and repliquate humans to include them in digital replicas, with impressive advancements made in topics such as capturing, rendering, animating, etc. Realistic virtual humans are therefore now common in entertainment applications (e.g., movies, videogames), but have also demonstrated their value in other fields such as architecture, trainging, education, etc.

Recently, realistic virtual humans have also started to be more and more used in immersive Virtual Reality applications in order to study human behaviours. Because of the complexity of studying human behaviours in controlled real situations, being able to immerse users in virtual replicas populated with interactive virtual humans provides a novel paradigm to explore such questions. For instance, VR was used to compare pairwise interactions between real and virtual situations [Olivier et al. 2018], to explore when pedestrians choose to go through or around groups of characters [Bruneau et al. 2015], to explore the relation between gaze activity and collision avoidance during pedestrian interactions [Berton et al. 2019, 2020], or to explore exit choices during evacuations [Ríos and Pelechano 2020]. It was even recently used to crowd datasets by immersing only one single user, successively embodying several virtual agents by him/herself [Yin et al. 2022, 2024].

However, immersive VR applications can have limitations in the level of realism and variety of scenarios that can be explored, because of the time and budget constraints required to develop a variety of scenarios. Instead, Augmented Reality provides an interesting alternative to automatically deploy out-of-the-lab and in-the-wild applications directly in real situations. For instance, one could imagine using Augmented Reality to plan the organization of real events and evaluate the effects of specific arrangements on various crowd-related characteristics (e.g., evacuation, bottlenecks, flows), by automatically populating real places with virtual characters. Nevertheless, including virtual characters in AR applications, in particular involving large virtual crowds, is still challenging. It requires to include them in appropriate locations, depending on the topology of the environment that varies depending on the real scene. This becomes even more difficult when these characters are animated, as they should act according to the environment, and potentially interact with real humans that will be also present. While some attempts have been made at simulating virtual crowds in AR [Akaydın et al. 2013, 2014 ; Hürst and Geraerts 2019], these have only targeted extremely simple setups, and not complex real world situations, or for specific studies (e.g., pedestrian interaction with a virtual car in AR [Maruhn et al. 2020]). Furthermore, it is still unclear whether to which degree interactions with virtual humans in AR differ from interactions in real situations. 

The goal of this internship is therefore to explore, model and evaluate user interactions with virtual characters in AR applications. The first objective will consist in developing an experiment where users will be able to interact with a few animated characters in AR, in particular to evaluate potential differences to existing situations already studied in VR (e.g., dyadic interactions). Then, the second objective will be to explore the potential use of AR to study interactions with larger virtual crowds, which will introduce a number of constraints (in terms of the number of characters that can be animated, their inclusion in the real scene, the semantic of their interactions with the scene, etc.). The experiments and applications developed will be based on some of the tools already developed in the Virtus team for crowd simulation, character animation, and immersive virtual crowds, such as CrowdMP and Umans. 

This internship therefore involves

• Developing a framework for the animating and simulating virtual characters in AR, based on the Unreal Engine platform and the Meta Quest 3 Mixed reality HMD. 
• Developing a number of scenarios of increasing complexity involving user interactions with virtual characters in such AR scenarios
• Evaluating potential differences in user interactions with virtual characters in AR, in comparison to real or purely virtual situations.

References

[Akaydın et al. 2013] A. Akaydın, A. Aman, U. Güdükbay. Interactive Crowd Simulation for Augmented Reality Environments. Proceedings of 26th International Conference on Computer Animation and Social Agents (CASA), 2013.

[Akaydın et al. 2014] A. Akaydın, A. Aman, U. Güdükbay. Interactive Crowd Simulation on Mobile Devices in an Augmented Reality Environment. Proceedings of 27th International Conference on Computer Animation and Social Agents (CASA), 2014.

[Berton et al. 2019] F. Berton, A-H. Olivier, J. Bruneau, L. Hoyet, J. Pettre. Studying Gaze Behaviour during Collision Avoidance with a Virtual Walker: Influence of the Virtual Reality Setup. IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2019.

[Berton et al. 2020] F. Berton, L. Hoyet, A-H. Olivier, J. Bruneau, O. Le Meur, J. Pettre. Eye-Gaze Activity in Crowds: Impact of Virtual Reality and Density. IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2020.

[Bruneau et al. 2015] J. Bruneau, A-H. Olivier, J. Pettré. Going Through, Going Around: A Study on Individual Avoidance of Groups. IEEE Transactions on Visualization and Computer Graphics, 2015.

[Hürst and Geraerts 2019] W. Hürst, R. Geraerts. Augmented and Virtual Reality Interfaces for Crowd Simulation Software-A Position Statement for Research on Use-Case-Dependent Interaction. IEEE Virtual Humans and Crowds for Immersive Environments (VHCIE), 2019.

[Maruhn et al. 2020] P. Maruhn, A. Dietrich, L. Prasch, S. Schneider. Analyzing Pedestrian Behavior in Augmented Reality — Proof of Concept. IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR), 2020.

[Olivier et al. 2018] A-H. Olivier, J. Bruneau, R. Kulpa, J. Pettré. Walking with Virtual People: Evaluation of Locomotion Interfaces in Dynamic Environments. IEEE Transactions on Visualization and Computer Graphics, 2018.

[Ríos and Pelechano 2020] A. Ríos, N. Pelechano. Follower behavior under stress in immersive VR. IEEE Virtual Reality, 2020.

[Savenije et al. 2020] N. Savenije, R. Geraerts, W. Hürst. CrowdAR Table An AR system for Real-time Interactive Crowd Simulation. IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), 2020.

[Yin et al. 2022] T. Yin, L. Hoyet, M. Christie, M-P. Cani, J. Pettré. The One-Man-Crowd: Single User Generation of Crowd Motions Using Virtual Reality. IEEE Transactions on Visualization and Computer Graphics (IEEE Virtual Reality), 2022.

[Yin et al. 2024] T. Yin, L. Hoyet, M. Christie, M-P. Cani, J. Pettré. With or Without You: Effect of Contextual and Responsive Crowds on VR-based Crowd Motion Capture. IEEE Transactions on Visualization and Computer Graphics (IEEE VR), 2024

Requirements for candidacy

• Master in Computer Sciences, with profiency in C++ or C#
• General background in Virtual Reality or Computer Graphics
• Experience with Unreal would be beneficial