2019-01676 - PhD Position F/M Integration of symbolic knowledge into deep learning

Contract type : Public service fixed-term contract

Level of qualifications required : Graduate degree or equivalent

Fonction : PhD Position


This PhD position is funded by the Inria Project Lab HyAIAI on “Hybrid Approaches for Interpretable AI”. The recruited student will benefit from the expertise of the Orpailleur team in pattern mining and symbolic reasoning and the Multispeech team in speech and audio analysis and deep learning.

As part of HyAIA, the student will have the opportunity to interact with the other Inria teams involved, namely Lacodam, Magnet, SequeL and TAU. He/she may also participate in the actitivities of the ANR project LEAUDS "Learning to understand audio scenes" involving Multispeech, the machine learning team at INSA Rouen, and Netatmo.


Current deep learning methods excel at supervised classification with a fixed set of classes, where information is provided at training time in the form of costly human labeling of training data. This approach faces several serious limitations: it does not scale with the thousands of fine-grained concepts experienced in daily life, it performs poorly when only a few labeled data is available for a given class, and it operates as a black-box and hence does not allow users to easily express desirable constraints. Humans, by contrast, reason and learn in a totally different way. Rather than treating each concept as a separate class, they understand concepts in relation to each other. This enables them to discover new concepts, to achieve good performance with little or no supervision, and to take specific constraints into account on the fly. This knowledge is typically represented as a knowledge graph, which encodes all known concepts along with their attributes and their relations.

We claim that integrating enriched knowledge graphs in the deep learning training and exploitation stages is a key towards addressing the above limitations. The resulting hybrid machine learning systems will not only understand human requirements, but also scale and perform better and be explainable by design. For example, considering the task of animal image recognition, knowing that “cats have whiskers and fur” and “sea lions have whiskers but no fur” shall help the machine learning system recognize whiskers even when the training images are labeled in terms of “cat” or “sea lion” only. Training a “whisker” classifier on images of multiple animals is then expected to improve the recognition accuracy for these animals, and to enable the discovery of other animals with whiskers which are neither a cat nor a sea lion and may be added to the knowledge graph after suitable interaction with the user.

A review of the various approaches undertaken to integrate symbolic reasoning and deep learning was provided in [1]. This includes designing neural networks whose units represent zeroth-order [2] or first-order logic formulas [3], or considering the outputs of a neural network as logic atoms and accounting for the probabilities of formulas derived from these atoms in the training cost [4]. Other efforts have seeked to represent each concept in a hierarchy via multiple units, i.e., to compute concept embeddings, by designing a network structure following that hierarchy [5, 6, 7]. It is also possible to design an end-to-end network that verifies whether a relation (specified in natural language) between its inputs is true or not [8]. Each of these approaches faces different limitations concerning the scalability to a large number of concepts and relations, the modeling of non-hierarchical relations, and the ability to compose the learned relations.

The goal of this PhD position is to design a scalable hybrid machine learning method that will enable the exploitation of arbitrary relations between concepts in an existing ontology, and the learning of new concepts on the fly that will enrich this ontology.

[1] A. S. d'Avila Garcez, T. R. Besold, L. De Raedt, P. Földiák, P. Hitzler, T. Icard, K.-U. Kühnberger, L. C. Lamb, R. Miikkulainen and D. L. Silver. Neural-symbolic learning and reasoning: Contributions and challenges. In 2015 AAAI Spring Symposia, 2015.

[2] S. N. Tran and A. S. d'Avila Garcez. Deep logic networks: Inserting and extracting knowledge from deep belief networks. IEEE Transactions on Neural Networks and Learning Systems, 29(2):246-258, 2018.

[3] G. Sourek, V. Aschenbrenner, F. Zelezný, S. Schockaert and O. Kuzelka. Lifted relational neural networks: Efficient learning of latent relational structures. Journal of Artificial Intelligence Research, 62:140-151, 2018.

[4] E. van Krieken, E. Acar and F. van Harmelen. Semi-supervised learning using differentiable reasoning. Journal of Applied Logics, in print.

[5] H. Wang, D. Dou and D. Lowd. Ontology-based deep restricted Boltzmann machine. In Proc. DEXA, 2016, pp. 431-445.

[6] E. Choi, M. Taha Bahadori, L. Song, W. F. Stewart and J. Sun. GRAM: Graph-based attention model for healthcare representation learning. In Proc. KDD, 2017, pp. 787-795.

[7] W. Goo, J. Kim, G. Kim and S.-J. Hwang. Taxonomy-regularized semantic deep convolutional neural networks. In Proc. ECCV, 2016, pp. 86-101.

[8] A. Santoro, D. Raposo, D. G. T. Barrett, M. Malinowski, R. Pascanu, P. Battaglia and T. Lillicrap. A simple neural network module for relational reasoning. In Proc. NIPS, 2017, pp. 4974-4983.

Main activities

The PhD student will be responsible for designing, developing, and evaluating this hybrid machine learning method. In order to achieve two-way interaction between an ontology and a deep learning system, he/she will seek to encode arbitrary relations between concepts into the neural network structure and to discover salient dependencies between its outputs.

Experimental evaluation will be conducted in particular for the task of audio event detection, using a deep learning-based system developed by Multispeech [9]. Symbolic knowledge will be integrated using the Audioset taxonomy, relevant linked open data, as well as new concepts and relations discovered from the audio, possibly using pattern mining methods.

[9] N. Turpault, R. Serizel and E. Vincent. Semi-supervised triplet loss based learning of ambient audio embeddings. In Proc. ICASSP, 2019.


Master degree in computer science or machine learning

Experience with programming in Python

Experience with audio processing and/or PyTorch is a plus

Benefits package

  • Subsidised catering service
  • Partially-reimbursed public transport
  • Social security
  • Paid leave
  • French courses


Gross Salary per month: 1982€ brut per month (year 1 & 2) and 2085€ brut/month (year 3)