Learning Force Distribution Estimation for the GelSight Mini Optical Tactile Sensor Based on Finite Element Analysis
Introduction
Up-to-date contact-rich manipulation tasks remain one of the grand challenges of robotics. Equipping robots with a sense of touch promises to resolve many of the current difficulties. In particular, optical tactile sensors can provide high-resolution touch measurements at relatively low costs. Fundamentally, optical tactile sensors consist of a camera capturing the deformation of a soft silicone gel. To obtain a more meaningful physically grounded representation beyond raw images, this work focuses on predicting force distributions given the raw images. Specifically, the images should be mapped to shear and normal forces acting on the sensor’s contact medium. To obtain the ground truth force distributions corresponding to the images, we propose running a model-based finite element simulation using the CalculiX solver. Building the dataset and running the computationally expensive finite element simulations requires significant computational power, which can be efficiently achieved by leveraging HPC clusters to run these tasks in parallel. This results in a labeled dataset of force distributions and images, forming the basis for training a neural network that maps images to force distributions. Fitting neural network models is appealing as they offer efficient inference at test time compared to the computationally expensive FEA.
Methods
For the estimation of force distributions on a GelSight Mini sensor from raw RGB images, we follow a supervised machine learning approach. A U-net architecture is trained on a dataset of tactile images and corresponding force distributions derived from Finite Element Analysis (FEA). To record tactile images, real world indentation experiments were performed with the GelSight Mini and different 3D printed indenters, using a CNC milling machine. The corresponding force distributions are obtained by simulating the indentation movements using FEA. The U-net model predicts shear and normal force distributions from the raw images, trained by minimizing the Mean Squared Error between predicted and FEA-derived labels.
Results
The U-net effectively predicts force distributions from GelSight Mini images with a mean absolute error (MAE) of less than 1 N. It performs better for shear forces than normal forces due to the richer feature set for shear estimation. Trained on 24 × 32 × 3 force distributions, the model achieves the best balance between accuracy and resolution, outperforming alternative configurations. Generalization tests on unseen indenters and a different GelSight Mini sensor show reduced accuracy. Inference speed averages 4.2 ms, supporting real-time applications at 25 Hz.
Discussion
This study introduces a machine learning approach for estimating force distributions using the GelSight Mini tactile sensor. By training a U-net model on FEA-derived data, we achieved accurate predictions of both shear and normal forces from raw sensor images, particularly excelling in shear force estimation. The model shows potential for real-time applications and generalization to different sensors, despite shifts in contact localization. This method offers an efficient alternative to traditional approaches like inverse FEA, with potential for improving tactile perception in robotic manipulation tasks. Future work will focus on enhancing the model’s robustness to different tactile scenarios and refining its generalization capabilities across a broader range of objects and sensors.
