Estimating the Pose of a Medical Manikin for Haptic Augmentation of a Virtual Patient in Mixed Reality Training

Virtual medical emergency training provides complex while safe interactions with virtual patients. Classical non-virtual medical training includes the use of medical manikins. Haptically integrating such a manikin into virtual training has the potential to improve the interaction with a virtual patient (e.g. regarding social presence) and the resulting training experience. This work analyzes user requirements and several technological approaches. The presented system estimates the pose of a medical manikin in order to haptically augment a 3D human model in a virtual reality (VR) training environment, allowing users to physically touch a virtual patient. The system uses a Convolutional Neural Networks-based (CNN) body keypoint detector to locate relevant keypoints of the manikin in the images of the stereo camera built into a head-mounted display (HMD). The system retrieves manikin position, orientation and joint angles using non-linear optimization. A manual precision analysis reports a mean error distance of 48 mm. Unless further optimized, we recommend applying this method for haptically augmented social interaction with virtual patients but not for practicing precise medical treatments.