Efficient and robust benchmarking for AI with benchopt

Le centre de recherche Inria de Saclay a été créé en 2008. Sa dynamique s’inscrit dans le développement du plateau de Saclay, en partenariat étroit d’une part avec le pôle de l’Université Paris-Saclay et d’autre part avec le pôle de l’Institut Polytechnique de Paris . Afin de construire une politique de site ambitieuse, le centre Inria de Saclay a signé en 2021 des accords stratégiques avec ces deux partenaires territoriaux privilégiés.

Le centre compte 40 équipes-projets , dont 32 sont communes avec l’Université Paris-Saclay ou l’Institut Polytechnique de Paris. Son action mobilise plus de 600 personnes, scientifiques et personnels d’appui à la recherche et à l’innovation, issues de 54 nationalités.

Le centre Inria Saclay - Île-de-France est un acteur essentiel de la recherche en sciences du numérique sur le plateau de Saclay. Il porte les valeurs et les projets qui font l’originalité d’Inria dans le paysage de la recherche : l’excellence scientifique, le transfert technologique, les partenariats pluridisciplinaires avec des établissements aux compétences complémentaires aux nôtres, afin de maximiser l’impact scientifique, économique et sociétal d’Inria.

Numerical evaluation of novel methods, a.k.a. benchmarking, is a pillar of the scientific method in machine learning. However, due to practical and statistical obstacles, the reproducibility of published results is currently insufficient: many details can invalidate numerical comparisons, from insufficient uncertainty quantification to improper methodology. In 2022, the benchopt initiative provided an open source Python package together with a framework to seamlessly run, reuse, share and publish benchmarks in numerical optimization. With this project, we aim at making benchopt a new standard in benchmarking by empowering researchers and practitioners with efficient and valid benchmarking methods.

The candidate will both contribute to the core benchopt library, and develop novel benchmarks for various AI fields, from optimization of large deep learning architectures to the evaluation of inverse problems resolutions. In particular, for core benchopt:

• Develop novel tools to better customize the HTML rendering of the benchmarks
• Improve the parallelization capabilities for the benchmarks

For the novel reference benchmark, a particular focus will be set on developing reference benchmarks for deep learning optimization, in particular with nanoGPT speed run optimization challenges, coupled with imagenet challenges.

Main activities:

• Participate in the development of the team's open source software benchopt
• Develop novel benchmarks in deep learning optimisation.

Additional activity: Participate to the team's research by providing support on how to evaluate novel methods on reference benchmarks.

• Strong mathematical background. Knowledge in machine learning is a plus.
• Good programming skills in Python. Knowledge of a deep learning framework is a plus.
• The candidate should be proficient in English. Knowing French is not necessary, as daily communication in the team is mostly in English due to the strong international environment.

• Subsidized meals
• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking and flexible organization of working hours
• Professional equipment available (videoconferencing, loan of computer equipment, etc.)
• Social, cultural and sports events and activities
• Access to vocational training

According to profile