PhD Position F/M Self-supervised learning for implicit shape reconstruction

The Inria Rennes - Bretagne Atlantique Centre is one of Inria's eight centres and has more than thirty research teams. The Inria Center is a major and recognized player in the field of digital sciences. It is at the heart of a rich R&D and innovation ecosystem: highly innovative PMEs, large industrial groups, competitiveness clusters, research and higher education players, laboratories of excellence, technological research institute, etc

Recent years have seen a surge in implicit neural shape representations for modeling 3D objects and scenes within deep learning frameworks. Thanks to their ability to continuously represent detailed shapes with arbitrary topologies in a memory-efficient way, these representations alleviate many of the shortcomings of the traditional alternatives such as polygon meshes, point clouds and voxel grids. In practice, these shape functions are typically multi-layer perceptrons mapping 3D points to occupancy or signed distance values. The zero level set of the inferred field can be rendered differentiably through variants of ray marching and tessellated into explicit meshes with Marching Cubes. Coupling these implicit neural functions with conditioning mechanisms allows generalization across multiple shapes. For instance, combining their inputs with local features generated from additional encoding networks [1,2,3,4] yields single forward pass inference models that can learn 3D reconstruction from various input modalities such as images [5,6] or partial point clouds [1,2,3,4].

These models are commonly trained using dense points sampled near the ground-truth surface. Hence, training them to perform reconstruction from images or point clouds requires typically substantial full 3D supervision that is hard to acquire. With the prospect of alleviating this expensive data dependence, we will explore in this project the extension of self-supervised methods to 3D implicit reconstruction. Existing self-supervised learning techniques in vision focus mostly on holistic 2D recognition tasks [7,8]. Our goal is to design self-supervised learning mechanisms that can reason locally [9] and benefit from inductive biases in 3D euclidean space.

As a primary application, we are interested in developing self-supervised deep learning based methods that can create accurate digital 3D replicas of people [10,11] from minimal input such as a single color image or sparse depth map, monocular color or depth videos, captured with a consumer grade camera. This research will contribute to the democratization of 3D people scanning and telepresence, among other human centered applications.

[1] “Few 'Zero Level Set'-Shot Learning of Shape Signed Distance Functions in Feature Space.” ECCV 2022.
[2] “Convolutional occupancy networks.” ECCV 2020.
[3] “POCO: Point Convolution for Surface Reconstruction.” CVPR 2022
[4] “Shape As Points: A Differentiable Poisson Solver.” NeurIPS 2021.
[5] “Disn: Deep implicit surface network for high-quality single-view 3d reconstruction." NeurIPS 2019.
[6] “Sdf-srn: Learning signed distance 3d object reconstruction from static images.” NeurIPS 2020.
[7] “A simple framework for contrastive learning of visual representations.” PMLR 2020.
[8] “Momentum contrast for unsupervised visual representation learning.” CVPR 2020.
[9] “Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.” CVPR 2021.
[10] “Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.” ICCV 2019.
[11] “Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.” CVPR 2021.

The PhD student will be tasked with:

• Examinating the state of the art of implicit shape reconstruction and self-supervised learning.
• Contributing new self-supervised deep implicit models for 3D reconstruction from images and point clouds.
• Achieving generalization to images, videos and point clouds of clothed people.

Candidates should preferably have a MSc or equivalent degree in computer science, applied mathematics, computer vision, computer graphics or machine learning. Proficiency in coding in Python and C++ is a plus.

• Subsidized meals
• Partial reimbursement of public transport costs
• Possibility of teleworking ( 90 days per year) and flexible organization of working hours
• Partial payment of insurance costs

Monthly gross salary amounting to 2051 euros for the first and second years and 2158 euros for the third year