Internship: Combining Transformers and Normalizing Flows for Deep Surrogate Training
Type de contrat : Convention de stage
Niveau de diplôme exigé : Bac + 4 ou équivalent
Fonction : Stagiaire de la recherche
A propos du centre ou de la direction fonctionnelle
The Centre Inria de l’Université de Grenoble groups together almost 600 people in 23 research teams and 9 research support departments.
Staff is present on three campuses in Grenoble, in close collaboration with other research and higher education institutions (Université Grenoble Alpes, CNRS, CEA, INRAE, …), but also with key economic players in the area.
The Centre Inria de l’Université Grenoble Alpe is active in the fields of high-performance computing, verification and embedded systems, modeling of the environment at multiple levels, and data science and artificial intelligence. The center is a top-level scientific institute with an extensive network of international collaborations in Europe and the rest of the world.
Contexte et atouts du poste
The length of the internship is 4 months minimum and the start date is flexible, but need a 2 month delay before starting the interhsip due to administrative constraints.
The candidate will integrate the Datamove team located in the IMAG building on the campus of Saint Martin d’Heres (Univ. Grenoble Alpes) near Grenoble. The DataMove team is a friendly and stimulating environement gathering Professors, Researchers, PhD and Master students. The city of Grenoble is a student friendly city surrounded by the alps mountains, offering a high quality of life and where you can experience all kinds of mountain related outdoors activities.
But there is also the possibility to pursue this internship being located at EDF R&D, Saclay, close to Paris. EDF is one fo the largest electricity supplier in Europe and their Saclay R&D EDF labs one of the largest industrial research center in France. EDF are long term collaborators actively involved in the development of Melissa and deep surrogate related investigation. EDF also brings industrial grade use-cases related to electrical machines (produced by Code_Carmel) and hydrological studies (produced by the open source code Open-Telemac).
Mission confiée
Context
Deep surrogates are deep neural networks trained from data produced by a numerical scientific simulation code, like fluids dynamics, weather forecast, molecular systems, etc. Deep surrogate are expected to be faster and smaller than the original simulation. There exists a wide variety of neural architectures used for deep surrogates, like U-Net, FNO, GNN, etc. Deep surrogate show different capabilities for generalization. Some are trained from a single simulation data, others from multiple simulation instances configured with different input parameters. A new trend is to train fundational models for scientific applications, leading to a neural network capable of supporting different types of simulations. Often these fundational models are based on a visual transformer architecture adapted for scientific data. The transformer architecture brings two key features for deep surrogates (1) the attention mechanism enables to capture correlations between simulation time steps; (2) the tokenization with positional encoding of input data into small data patches make the architecture more flexible to the resolution of the input data. In parallel, normalizing flow architectures, that project one known probability distribution to an other only partially known through data, have interesting properties (1) the are invertible and thus can be used for solving inverse problems; (2) they convey a measure of uncertainties through the learned probability distribution, a very important information for scientific computing.
Internship Goals
The goals of this internship is to investigate how effective the combination of transformer and normalizing flow architectures can be for training deep surrogates. We will consider as starting point several available architectures from the papers All-in-one simulation-based inference, Poseidon: Efficient Foundation Models for PDEs, ClimaX: A foundation model for weather and climate that will be analyzed, tested and eventually combined. For the purpose of experiments, we will integrate these models into the Melissa framework developed in our team. Melissa enables to train deep surrogates on supercomputers directly from the running simulations while they produce data. Melissa enables to train very efficiently on significantly more data that the classical offline approaches that store output data from simulations to files and then read them back for training. Melissa also simply makes it easier to train deep surrogates by combing data production and training in a unified workflow.
Our Related Publications
- MelissaDL x Breed: Towards Data-Efficient On-line Supervised Training of Multi-parametric Surrogates with Active Learning, SC AI4S 2024: https://hal.science/hal-04712480v1
- Training Deep Surrogate Models with Large Scale Online Learning, ICML 2023: https://hal.science/hal-04102400v1
- High Throughput Training of Deep Surrogates from Large Ensemble Runs, SC 2023, https://hal.science/hal-04213978v1
- Deep Surrogate for Direct Time Fluid Dynamics, Neurips 2023 Thirty-fifth Workshop on Machine Learning and the Physical Sciences. https://hal.science/hal-03451432v2
Principales activités
See above
Compétences
This internship is open to students pursuing a master or an engineering degree. We expect candidates with basic probability, deep learning, physics and numerical simulation knowledges, good Python programming skills (numpy, basic pytorch). The main communication language is English.
Avantages
- Subsidized meals
- Partial reimbursement of public transport costs
- Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
- Possibility of teleworking (90 days / year) and flexible organization of working hours
- Social, cultural and sports events and activities
- Access to vocational training
- Social security coverage under conditions
Rémunération
€4.35 per hour of actual presence at 1 January 2024.
About 590€ gross per month (internship allowance)
Informations générales
- Thème/Domaine :
Optimisation, apprentissage et méthodes statistiques
Calcul Scientifique (BAP E) - Ville : Saint Martin d'Hères
- Centre Inria : Centre Inria de l'Université Grenoble Alpes
- Date de prise de fonction souhaitée : 2025-02-01
- Durée de contrat : 6 mois
- Date limite pour postuler : 2024-11-21
Attention: Les candidatures doivent être déposées en ligne sur le site Inria. Le traitement des candidatures adressées par d'autres canaux n'est pas garanti.
Consignes pour postuler
CV + cover letter
Les candidatures doivent être déposées en ligne sur le site Inria.
Le traitement des candidatures adressées par d'autres canaux n'est pas garanti.
Sécurité défense :
Ce poste est susceptible d’être affecté dans une zone à régime restrictif (ZRR), telle que définie dans le décret n°2011-1425 relatif à la protection du potentiel scientifique et technique de la nation (PPST). L’autorisation d’accès à une zone est délivrée par le chef d’établissement, après avis ministériel favorable, tel que défini dans l’arrêté du 03 juillet 2012, relatif à la PPST. Un avis ministériel défavorable pour un poste affecté dans une ZRR aurait pour conséquence l’annulation du recrutement.
Politique de recrutement :
Dans le cadre de sa politique diversité, tous les postes Inria sont accessibles aux personnes en situation de handicap.
Contacts
- Équipe Inria : DATAMOVE
-
Recruteur :
Raffin Bruno / bruno.raffin@inria.fr
L'essentiel pour réussir
This internship will open the doors to the domain of AI-for-Science, giving the candidate the opportunity to gain skills on advanced neural networks architectures, multi-GPU training on supercomputers, conduct of numerical experiments, scientific writing.
A propos d'Inria
Inria est l’institut national de recherche dédié aux sciences et technologies du numérique. Il emploie 2600 personnes. Ses 215 équipes-projets agiles, en général communes avec des partenaires académiques, impliquent plus de 3900 scientifiques pour relever les défis du numérique, souvent à l’interface d’autres disciplines. L’institut fait appel à de nombreux talents dans plus d’une quarantaine de métiers différents. 900 personnels d’appui à la recherche et à l’innovation contribuent à faire émerger et grandir des projets scientifiques ou entrepreneuriaux qui impactent le monde. Inria travaille avec de nombreuses entreprises et a accompagné la création de plus de 200 start-up. L'institut s'efforce ainsi de répondre aux enjeux de la transformation numérique de la science, de la société et de l'économie.