Internship: Combining Transformers and Normalizing Flows for Deep Surrogate Training

The Centre Inria de l’Université de Grenoble groups together almost 600 people in 23 research teams and 9 research support departments.

Staff is present on three campuses in Grenoble, in close collaboration with other research and higher education institutions (Université Grenoble Alpes, CNRS, CEA, INRAE, …), but also with key economic players in the area.

The Centre Inria de l’Université Grenoble Alpe is active in the fields of high-performance computing, verification and embedded systems, modeling of the environment at multiple levels, and data science and artificial intelligence. The center is a top-level scientific institute with an extensive network of international collaborations in Europe and the rest of the world.

 The length of the internship is 4 months minimum and the start date is flexible, but need a 2 month delay before starting the interhsip due to administrative constraints.

The candidate will integrate the Datamove team located in the IMAG building on the campus of Saint Martin d’Heres (Univ. Grenoble Alpes) near Grenoble. The DataMove team is a friendly and stimulating environement gathering Professors, Researchers, PhD and Master students. The city of Grenoble is a student friendly city surrounded by the alps mountains, offering a high quality of life and where you can experience all kinds of mountain related outdoors activities.

But there is also the possibility to pursue this internship being located at EDF R&D, Saclay, close to Paris. EDF is one fo the largest electricity supplier in Europe and their Saclay R&D EDF labs one of the largest industrial research center in France. EDF are long term collaborators actively involved in the development of Melissa and deep surrogate related investigation. EDF also brings industrial grade use-cases related to electrical machines (produced by Code_Carmel) and hydrological studies (produced by the open source code Open-Telemac).

Context

 Deep surrogates are deep neural networks trained from data produced by a numerical scientific simulation code, like fluids dynamics, weather forecast, molecular systems, etc. Deep surrogate are expected to be faster and smaller than the original simulation. There exists a wide variety of neural architectures used for deep surrogates, like U-Net, FNO, GNN, etc. Deep surrogate show different capabilities for generalization. Some are trained from a single simulation data, others from multiple simulation instances configured with different input parameters. A new trend is to train fundational models for scientific applications, leading to a neural network capable of supporting different types of simulations. Often these fundational models are based on a visual transformer architecture adapted for scientific data. The transformer architecture brings two key features for deep surrogates (1) the attention mechanism enables to capture correlations between simulation time steps; (2) the tokenization with positional encoding of input data into small data patches make the architecture more flexible to the resolution of the input data. In parallel, normalizing flow architectures, that project one known probability distribution to an other only partially known through data, have interesting properties (1) the are invertible and thus can be used for solving inverse problems; (2) they convey a measure of uncertainties through the learned probability distribution, a very important information for scientific computing.

Internship Goals

The goals of this internship is to investigate how effective the combination of transformer and normalizing flow architectures can be for training deep surrogates. We will consider as starting point several available architectures from the papers All-in-one simulation-based inference, Poseidon: Efficient Foundation Models for PDEs, ClimaX: A foundation model for weather and climate that will be analyzed, tested and eventually combined. For the purpose of experiments, we will integrate these models into the Melissa framework developed in our team. Melissa enables to train deep surrogates on supercomputers directly from the running simulations while they produce data. Melissa enables to train very efficiently on significantly more data that the classical offline approaches that store output data from simulations to files and then read them back for training. Melissa also simply makes it easier to train deep surrogates by combing data production and training in a unified workflow.

See above

This internship is open to students pursuing a master or an engineering degree.  We expect candidates with basic probability, deep learning, physics and numerical simulation knowledges, good Python programming skills (numpy, basic pytorch). The main communication language is English.

• Subsidized meals
• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking (90 days / year) and flexible organization of working hours
• Social, cultural and sports events and activities
• Access to vocational training
• Social security coverage under conditions

 €4.35 per hour of actual presence at 1 January 2024.

About 590€ gross per month (internship allowance)