PhD Position F/M Visuo-tactile control for manipulating deformable objects with robotic arms combining tactile perception and photometric visual servoing

The Inria Centre at Rennes University is one of Inria's eight centres and has more than thirty research teams. The Inria Centre is a major and recognized player in the field of digital sciences. It is at the heart of a rich R&D and innovation ecosystem: highly innovative PMEs, large industrial groups, competitiveness clusters, research and higher education players, laboratories of excellence, technological research institute, etc.

Conditions:

• The PhD student will be hosted in the Inria Rainbow team (https://team.inria.fr/rainbow) at Rennes.
• The PhD position is full-time for 3 years (standard duration in France). The position will be paid according to the French salary regulations for PhD students

Supervisors: Alexandre Krupa (Inria/IRISA - Rainbow), Youcef Mezouar (Institut Pascal - MACCS), Eric Marchand (Inria/IRISA - Rainbow)

Funding: This thesis is funded by the French PEPR Robotics Acceleration program as part of the DRMI project “Dexterous Robotic Manipulation for Industry”.

Starting date: October or November 2026.

Context:

A significant challenge in robotics is the ability to interact with deformable objects. Most existing robotic control frameworks are designed for manipulating rigid objects, limiting their applicability to complex tasks. To extend robotic manipulation capabilities, this PhD thesis aims to develop a novel control approach that leverages tactile sensors and depth (RGB-D) cameras to manipulate soft objects.

The core idea is to integrate visual and tactile feedback to track object deformations in real time and develop a visuo-tactile control strategy that enables one or more robotic manipulators to apply and regulate desired deformations. This capability is crucial for various applications, including robotic hand manipulation of soft materials, assembly and disassembly of flexible components, and precise handling of elastic materials. Within the French PEPR Robotics DRMI project, the research will focus specifically on industrial recycling, particularly the dismantling of equipment with flexible parts.

Controlling the deformation of soft materials requires understanding how robotic manipulations translate into material deformations. Existing approaches address this challenge using either:

• Data-driven methods, which estimate deformation behavior based on past visual observations [1], [12].
• Geometric-based models, such as ARAP [11] or LARAP [10].
• Physics-based models, such as Finite Element Models (FEM) [2-3] or mass-spring models [4-5].

These methods typically rely on extracting geometric features to represent the object’s deformation. However, their efficiency is often limited to specific object types and requires real-time feature extraction and tracking. Furthermore, most approaches assume prior knowledge of a geometric model (3D mesh) and a physics-based deformation model, which must be pre-built and calibrated for each object.

Envisaged Activities:

To overcome these limitations, this PhD thesis will elaborate a new approach that directly utilizes photometric and depth information from an RGB-D camera as visual features for control, eliminating the need for predefined geometric models and features extraction. While direct visual servoing has been successfully applied for rigid object positioning [6-7], its application to deformable object shaping remains unexplored.

This research will develop:

• A physics-based model generated online during the manipulation task, combining data from tactile sensors (mounted on robotic hands) and dense visual feedback.
• A hybrid visuo-tactile control scheme, fusing dense visual data with sparse tactile information (e.g., contact points, applied forces) to ensure stable and controlled deformation, particularly for handling fragile objects.
• A novel shape servoing framework, integrating photometric-based interaction models to predict and control deformation in response to robotic manipulations.
• Dimensionality reduction techniques to optimize the representation of photometric shape features using projection-based approaches [8-9]. 

Experimental Validation

The proposed methods will be implemented and validated on the recent TIRREX TRIAGo robotic platform, which consists of:

• Three robotic arms (7-DOF each) mounted on a mobile base.
• Two robotic hands (Allegro), each equipped with four Xela uSkin tactile sensors at the fingertips and 6-DOF force sensors at the base.
• An RGB-D camera mounted on the third robotic arm.

The system will be tested with deformable objects to evaluate its effectiveness in real-time manipulation and deformation control.

Expected Contributions:

This thesis will introduce:

• A real-time physics model generation technique fusing tactile and visual photometric data, eliminating the need for pre-built deformation models.
• A visuo-tactile interaction model to predict how photometric and tactile information evolve during manipulation.
• A novel robot control strategy that enables autonomous shaping of soft objects using multiple robotic arms.

This research will contribute to advancing soft object manipulation and open new possibilities in robotics for industrial automation, recycling, and dexterous manipulation of flexible materials.

References:

[1] R. Lagneau, A. Krupa, M. Marchal. Automatic Shape Control of Deformable Wires based on Model-Free Visual Servoing. IEEE Robotics and Automation, 5(4):5252-5259, October 2020.

[2] F. Ficuciello, A. Migliozzi, E. Coevoet, A. Petit and C. Duriez, "FEM-Based Deformation Control for Dexterous Manipulation of 3D Soft Objects," In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, IROS’18, Pages 4007-4013, Madrid, Spain, October 2018.

[3] M. Fonkoua Ouafo, F. Chaumette, A. Krupa. Deformation Control of a 3D Soft Object using RGB-D Visual Servoing and FEM-based Dynamic Model. IEEE Robotics and Automation Letters (also presented at ICRA 2025), 9(8):6943-6950, August 2024.

[4] F. Makiyeh, F. Chaumette, M. Marchal, A. Krupa. Shape Servoing of a Soft Object Using Fourier Series and a Physics-based Model. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, IROS'23, Pages 6356-6363, Detroit, USA, October 2023.

[5] F. Makiyeh, M. Marchal, F. Chaumette, A. Krupa. Indirect Positioning of a 3D Point on a Soft Object Using RGB-D Visual Servoing and a Mass-Spring Model. In Int. Conf. on Control, Automation, Robotics and Vision, ICARCV'22, Singapore, December 2022.

[6] C. Collewet, E. Marchand, F. Chaumette. Visual servoing set free from image processing. In IEEE Int. Conf. on Robotics and Automation, ICRA'08, Pages 81-86, Pasadena, Californie, Mai 2008.

[7] C. Collewet, E. Marchand. Photometric visual servoing. IEEE Trans. on Robotics, 27(4):828-834, August 2011.

[8] E. Marchand. Direct visual servoing in the frequency domain. IEEE Robotics and Automation Letters, 5(2):620-627, April 2020.

[9] E. Marchand. Subspace-based Visual Servoing. IEEE Robotics and Automation Letters, 4(3):2699-2706, July 2019.

[10] M. Shetab-Bushehri,  M. Aranda, Y. Mezouar.  Lattice-based shape tracking and servoing of elastic objects, IEEE Transactions on Robotics, 40, 364-381.

[11] M. Shetab-Bushehri,  M. Aranda, Y. Mezouar.  As-Rigid-As-Possible Shape Servoing," in IEEE Robotics and Automation Letters 7 (2), 3898-3905.

[12] M. Daniel, A. Magassouba, M.  Aranda, L.  Lequièvre, L., J.A Corrales, R.  Rodriguez, Y.  Mezouar, Y.  Multi Actor-Critic DDPG for Robot Action Space Decomposition: A Framework to Control Large 3D Deformation of Soft Linear Objects, IEEE Robotics and Automation Letters, 9 (2), 1318-1325.

Skills/Requirements

The candidate must have an excellent track of records and a Master Degree (or equivalent) in robotics and computer vision.

The candidate must have the following qualifications:

• Strong background in robotics
• Experience with computer vision, physical robots, or 3D simulation
• Excellent programming skills in C++
• Excellent written and oral English
• Ability to perform experimental validations
• Ability to work independently as well as collaboratively

• Subsidized meals
• Partial reimbursement of public transport costs
• Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)
• Possibility of teleworking (after 6 months of employment) and flexible organization of working hours
• Professional equipment available (videoconferencing, loan of computer equipment, etc.)
• Social, cultural and sports events and activities
• Access to vocational training
• Social security coverage

monthly gross salary 2300 euros