PhD Position F/M Learning to Translate Freehand Design Drawings into Parametric CAD Programs

The Inria centre at Université Côte d'Azur includes 37 research teams and 8 support services. The centre's staff (about 500 people) is made up of scientists of different nationalities, engineers, technicians and administrative staff. The teams are mainly located on the university campuses of Sophia Antipolis and Nice as well as Montpellier, in close collaboration with research and higher education laboratories and establishments (Université Côte d'Azur, CNRS, INRAE, INSERM ...), but also with the regiona economic players.

With a presence in the fields of computational neuroscience and biology, data science and modeling, software engineering and certification, as well as collaborative robotics, the Inria Centre at Université Côte d'Azur  is a major player in terms of scientific excellence through its results and collaborations at both European and international levels.

This Ph.D. is funded by a joint ANR-NSF grant, in collaboration with Daniel Ritchie from Brown University

Context and Objective

Computer Aided Design (CAD) is a multi-billion dollar industry responsible for the digital design of almost all manufactured goods. It leverages parametric modeling, which allows dimensions of a design to be changed, facilitating physically-based optimization and design re-mixing by non-experts. But CAD’s potential is diminished by the difficulty of creating parametric models: in addition to mastering design principles, professionals must learn complex CAD software interfaces.

To promote effective modeling strategies and creative flow, design educators advocate freehand drawing as a preliminary step to parametric modeling. Unfortunately, CAD systems do not understand these drawings, so designers must re-create their entire design using complex CAD software. Can we automatically convert freehand drawings to parametric CAD models? Sketch-based modeling techniques do not produce parametric CAD programs [1]; classic CAD reverse-engineering techniques cannot handle drawings as input [2]; the newer field of visual program induction is promising but has been demonstrated only on simple shapes and programs [3]. By leveraging the visual vocabulary shared by drawing and CAD modeling, we will develop a system to translate from the natural language of drawing to the formal language of CAD.

References

[1] Symmetry-driven 3D Reconstruction from Concept Sketches. Felix Hähnlein, Yulia Gryaditskaya, Alla Sheffer, Adrien Bousseau. SIGGRAPH 2022.

[2] InverseCSG: automatic conversion of 3D models to CSG trees. T. Du, J. Inala, Y. Pu, A. Spielberg, A. Schulz, D. Rus, A. Solar-Lezama, W. Matusik. ACM Trans. on Graphics (SIGGRAPH Asia) 2018.

[3] Neurosymbolic Models for Computer Graphics. Daniel Ritchie, Paul Guerrero, R. Kenny Jones, Niloy Mitra, Adriana Schulz, Karl D. D. Willis, and Jiajun Wu. Eurographics 2023 STAR.

To handle drawings as input, we will treat them as timestamped sequences of strokes, allowing us to cast the problem as one of machine translation from drawing stroke sequences to CAD program token sequences. We observe that drawing strokes are grouped into coherent drawing operations that are correlated with CAD modeling strategies (e.g. first drawing construction lines and simple primitives shapes, then refining). We propose to extract these drawing operations as an intermediate representation, which helps disambiguate between the (potentially infinitely) many programs which can represent a single shape. Performing this extraction and then producing CAD programs are complex search problems; we will leverage novel deep neural networks to guide the search.

We will first create a dataset to study how designers draw parametric shapes. To do this, we will collect existing parametric shapes from public datasets and hire professional designers to draw these shapes. These drawings will be used to test our algorithms.

We will then develop a drawing analysis algorithm in order to identify the lines of a drawing which correspond to the same 3D parametric operation, then recognize this operation and deduce its parameters. To do this, we intend to use machine learning algorithms that we will train with synthetic drawings generated from reference parametric 3D shapes.

Finally, we will develop a program synthesis method which will combine all the estimated information (input design, recognized 3D operations, estimated parameters) and which will generate a valid program best reproducing the design while being modifiable in the existing 3D design tools. We plan to use a Transformer learning architecture, originally developed for text translation, to “translate” the drawing sequence into a sequence of 3D instructions.

The candidate must have experience in Python programming. Past experience in implementing 3D modeling or 3D analysis algorithms is a plus.

• Subsidized meals
• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking and flexible organization of working hours
• Professional equipment available (videoconferencing, loan of computer equipment, etc.)
• Social, cultural and sports events and activities
• Access to vocational training
• Social security coverage

Duration: 36 months
Location: Sophia Antipolis, France
Gross Salary per month: 2100€ brut per month (year 1 & 2) and 2190€ brut per month (year 3)