PhD Position F/M Multimodal workspace capture and inference for XR collaboration

The Inria Rennes - Bretagne Atlantique Centre is one of Inria's eight centres and has more than thirty research teams. The Inria Center is a major and recognized player in the field of digital sciences. It is at the heart of a rich R&D and innovation ecosystem: highly innovative PMEs, large industrial groups, competitiveness clusters, research and higher education players, laboratories of excellence, technological research institute, etc.

The candidate will integrate the Seamless team at the at the Centre Inria at Rennes University/IRISA. The Seamless team adopts a multidisciplinary approach in virtual/augmented reality (XR), human perception, human-computer interaction, and human factors, prioritizing the user at the center of the XR revolution. Their work addresses three key challenges: enabling smooth transitions between realities by bridging gaps in perception and interaction for a continuous experience; fostering equal collaboration across realities, ensuring shared awareness and uniform interaction capabilities regardless of individual differences; and advancing implicit, precise evaluation of user experience.

This PhD is part of the European Horizon Europe project OmenXR, which aims to develop a real-time, multisensory, and human-centered framework for hybrid collaboration in eXtended Reality (XR). To achieve this, OmenXR will first leverage state-of-the-art image-based rendering techniques to enable on-the-fly photorealistic visual fidelity using consumer-grade Mixed Reality (MR) hardware. Additionally, it will create a multisensory experience that anchors all collaborators in a sensory-coherent environment. Finally, OmenXR will utilize real-time reconstruction and understanding to enable novel forms of hybrid collaboration, integrating remote users, teleoperated robots, and intelligent virtual agents. This PhD project will partially focus on those objectives by enabling the capture and inference of physical spaces for AR/VR collaboration. 

Augmented Reality (AR) has been extensively investigated as a tool for remote assistance across various application domains, such as industrial maintenance and home support. In this context, a user operating within an AR-enabled system may solicit guidance from remote collaborators who possess visual access to the physical environment. To facilitate effective collaboration, it is imperative that remote users maintain a comprehensive understanding of the physical workspace, a requirement referred to as workspace awareness. Workspace awareness enables the remote collaborators to understand and assess the environment in which the co-located user is located. A wide range of works have explored to present AR workspaces to remote users [Assaf et al.]. Most of the works focuses on static 3D reconstructions [Kumaravel et al.], virtual proxies [Oda et al.], light fields [Mohr et al.] or real-time video feeds [Fages et al.]. All methods present several trade-offs, based on the size of the reconstructed workspace, the freedom that remote users have to navigate through the reconstructed workspace, the fidelity of the reconstructed workspace or the required preparation and instrumentation. While video provides high fidelity with minimal instrumentation, the navigation capability for remote users is limited and only provide a partial view of the workspace. However, due to the difficulty to ensure free exploration for remote collaborators, workspace awareness still remains an open problem, and it is typically supported either by virtual replicas/reconstructions or video feeds [4]. The appearance of 3D Gaussian Splatting (3DGS) methods [Kerbl et al.] [Meuleman et al.], real-time and high-fidelity reconstruction of physical workspaces is nowadays possible, there is still little research on how the online reconstruction process can be executed in real time without any instrumentation and still ensure a smooth collaboration among different actors. In this context, the first objective of the PhD is to evaluate collaborative incremental reconstruction methods to efficiently support workspace awareness even in the context of partial or uncertain reconstructions, and propose interaction methods to incrementally refine the dynamic reconstruction. Furthermore, workspace awareness is not solely limited to the visual modality, a second objective of the PhD is exploring methods to enrich the reconstruction inferring other sensory modalities, notably haptic information, using the capability of novel view synthesis and vision-based scene understanding methods [Park et al.]. In addition to the technical contributions, specific focus will be devoted the perceptual assessment of the reconstructed models to ensure that 3DGS methods do not introduce perceptual biases.

The first three months of the PhD candidate will focus on thorough analysis of the state of the art on asymmetrical collaboration (theoretical) and on the development of a proof-of-concept VR 3DGS-based rendering system using off-the-shelf algorithms and methods. The next twelve months will be focused on reconstruction fidelity assessment and explore perceptual issues of the 3DGS reconstruction (empirical, methodological). The next twelve months will be focused on the multimodal inference. The final six months will be focused on the writing of the manuscript and to the preparation of the defense, while finishing the initiated contributions.

Furthemore, the PhD student is also expected to participate on the different activities that will be organized around the project, project meetings and staff exchanges. 

References

• Assaf, R., Mendes, D. and Rodrigues, R.  Cues to fast‐forward collaboration: A Survey of Workspace Awareness and Visual Cues in XR Collaborative Systems, In Computer Graphics Forum, vol. 43, no. 2, p. e15066. 2024, doi.
• Bai, H., Sasikumar, P., Yang, J., and Billinghurst, M., A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. ACM CHI Conference on Human Factors in Computing Systems. 2020, 1-13, doi.
• Clark, H. H., and Wilkes-Gibbs, D. Referring as a collaborative process. Cognition 22, 1 (1986), 1–39, 1986, doi.
• Dix, A. Computer supported cooperative work: A framework. In Design issues in CSCW. Springer, 9–26. 1994, doi.
• Fages, A., Fleury, C., and Tsandilas, T., Understanding Multi-View Collaboration between Augmented Reality and Remote Desktop Users. ACM on Human-Computer Interaction. CSCW2, 2022, 549, 1-27. doi.
• Johnson, J. G., Danilo G., Tommy S., Evan S., and Nadir W. Do you really need to know where “that” is? enhancing support for referencing in collaborative mixed reality environments. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-14. 2021, doi.
• Kerbl, B., Kopanas. G, Leimkühler, T., & Drettakis, G., 3D Gaussians for Real-Time Rendering of Radiance Fields, ACM Trans. On Graphics, Volume 42, Issue 4 Article No.: 139, 1-14, 2023 doi.
• Kraut, R. E Miller, M. D., and Siegel, J. Collaboration in performance of physical tasks: Effects on outcomes and communication. In Proceedings of the ACM conference on Computer supported cooperative work. ACM, 57–66. 1996, doi.
• Kumaravel. B. T., Anderson, F., Fitzmaurice, G., Hartmann, B., and Grossman, T., Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence.” ACM Symposium on User Interface Software and Technology. 1-13, 2020 doi.
• Meuleman, A., Shah, I., Lanvin, A., Kerbl, B., & Drettakis, G. On-the-fly reconstruction for large-scale novel view synthesis from unposed images. ACM Transactions on Graphics (TOG), 44(4), 1-14, 2025, doi.
• Mohr, P., Mori, S., Langlotz T., Thomas, B. H., Schmalstieg, D., and Kalkofen, D. Mixed Reality Light Fields for Interactive Remote Assistance. ACM CHI Conference on Human Factors in Computing Systems., 2020, 1–12. doi.
• Oda, O., Elvezio, C., Sukan, M., Feiner, S., and Tversky, B., Virtual Replicas for Remote Assistance in Virtual and Augmented Reality. ACM Symposium on User Interface Software and Technology, 405–415, 2015. doi
• Oh, C. S., Bailenson, N. J. and Welch, G. F., A Systematic Review of Social Presence: Definition, Antecedents, and Implications. Frontiers on Robotics and AI, 5:114, 2018, doi.
• Park, S. Nam, J. Kim, U. Ju and S. Choi, "GenTouchVR: Generating a Touchable Virtual Reality Environment from a Single Image," in IEEE Transactions on Visualization and Computer Graphics, vol. 32, no. 5, pp. 4385-4395, May 2026, doi: 10.1109/TVCG.2026.3680620
• Piumsomboon, T., Dey, A., Ens, B., Lee, G. and Billinghurst, M., The Effects of Sharing Awareness Cues in Collaborative Mixed Reality. Front. Robot. AI, 2019 6:5. doi.

• Subsidized meals
• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking (after 6 months of employment) and flexible organization of working hours
• Professional equipment available (videoconferencing, loan of computer equipment, etc.)
• Social, cultural and sports events and activities
• Access to vocational training

2 300€ per month