Hand motion estimation in the wild via IMUs and gaze-tracking: Can an efficient linear model suffice?
One of the major current challenges in robotics is the simulation-to-reality (sim2real) gap for training robot controllers particularly due to the limited ability of simulations to recreate real physics. This limitation could be partially mitigated by training controllers on human recordings. Transhumeral prostheses, meant to accurately recreate the motion of the missing part of the limb, could also benefit from such controllers. In this work, we propose a controller for transhumeral prostheses that utilizes a network of sensors worn by the user to enable naturalistic and intuitive control of the prosthesis, and demonstrate proof-of-concept hand motion predictors trained on an open-access dataset, based on a) linear regression and b) a UNet. The training data included 8 subjects, and we tested using a 9th subject. Our results demonstrate that the upper arm position is an important input for hand motion prediction, and adding additional sensor modalities can add benefit. Both methods generalized well to the test subject, yet surprisingly, the UNet’s prediction error was only ∼24% lower than the efficient linear model’s while requiring substantially more computation. Future work will extend this towards a sensor network to help address dynamics and hand kinematics during grasping and manipulation for a proposed transhumeral prosthesis controller.