M2 Internship: Zero-shot deepfake detection

The Inria center at the University of Rennes is one of eight Inria centers and has more than thirty research teams. The Inria center is a major and recognized player in the field of digital sciences. It is at the heart of a rich ecosystem of R&D and innovation, including highly innovative SMEs, large industrial groups, competitiveness clusters, research and higher education institutions, centers of excellence, and technological research institutes.

Deepfakes are synthetic media created using typically deep learning techniques, to manipulate or generate realistic audio, video, or images of people. While deepfakes have potential applications in entertainment and creative industries, they also pose serious threats. In case of videos, manipulation of people is particularly studied. Face swapping aims to replace the face in an original image or video with a selected one. It thus changes the identity (ID-Replaced). Face reenactment and lip synching are other types of video manipulation that aim at modifying the facial movements and expressions of a target person according to another one for example, or even according to some audio record. In this case the identity is not changed (ID-Remained).

While there is an increasing number of approaches for detecting whether an image or video has been the result of AI-based synthesis or manipulation, a key issue that detectors face is the generalization to unseen generative architectures. A significant hurdle in the development of robust deepfake detection systems is the limited availability of diverse datasets for training and evaluation purposes.

A possible solution to gain generalization and robustness is to shift to a different paradigm, where models are only trained on real videos, with the goal to detect manipulated videos based on their anomalous behavior [1]. Recent work has focused on verifying audio-visual consistency with different strategies [2-5].

On the other hand, few studies explore an approach that modelize not just real videos, but more precisely the real videos of a particular individual [6-9]. However those approaches have not been evaluated on same datasets, and do not provide sufficient details on performance for different types of manipulation (ID-replaced or ID-remained), robustness to degradation like compression or blurring, and false positives, that are a key issue of synthetic media detection methods because they compromise the trust of journalists and fact-checkers on their results.

The idea is to combine these 2 approaches, i.e. the possibility of modeling a person, and proposing a solution to detect manipulations independent of the modification technique.  The aim of the internship is to answer the following questions:

• To what extent are ID-Remained cases are more difficult than ID-Replaced ones?
• Are the behavioral characteristics proposed in the literature really relevant, discriminative and robust?
• How robust are these methods to low quality and high compression?
• What is the minimum number of videos needed to learn the model of an individual?

• Identify in the literature the most promising features and zero-shot methods
• Set up a robust evaluation framework to assess the various aspects of performance
• Evaluate the performances of selected methods, either existing ones or proposed ones
• Results analysis to pinpoint difficulties of both the dataset and the methods

• Master in Computer Sciences, with proficiency in python and its libraries for deep learning
• General background in computer vision and machine learning
• Understanding of deep learning methodologies and techniques;
• Proficiency in data handling, particularly in video processing

• Subsidized meals
• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking (after 6 months of employment) and flexible organization of working hours
• Professional equipment available (videoconferencing, loan of computer equipment, etc.)
• Social, cultural and sports events and activities
• Access to vocational training
• Social security coverage

Brut mensuel de 1766,92 euros.