A novel upper-limb tracking system in a virtual environment for stroke rehabilitation

Sounds cool, now lets get a provisional protocol written up on it and distributed to all 10 million yearly stroke survivors.

A novel upper-limb tracking system in a virtual environment for stroke rehabilitation

• Kuan Cha,
• Jinying Wang,
• Yan Li,
• Longbin Shen,
• Zhuoming Chen &
• Jinyi Long 

Journal of NeuroEngineering and Rehabilitation volume 18, Article number: 166 (2021) Cite this article

• 17 Accesses

• Metrics details

Abstract

Background

The transfer of the behaviors of a human’s upper limbs to an avatar is widely used in the field of virtual reality rehabilitation. To perform the transfer, movement tracking technology is required. Traditionally, wearable tracking devices are used for tracking; however, these devices are expensive and cumbersome. Recently, non-wearable upper-limb tracking solutions have been proposed, which are less expensive and more comfortable. However, most products cannot track the upper limbs, including the arms and all the fingers at the same time, which limits the limb parts for tracking in a virtual environment and may lead to a limited rehabilitation effect.

Methods

In this paper, a novel virtual reality rehabilitation system (VRRS) was developed for upper-limb rehabilitation. The VRRS could track the motion of both upper limbs, integrate fine finger motion and the range of motion of the entire arm and map the motion to an avatar. To test the performance of VRRS, two experiments were designed. In the first experiment, we investigated the effect of VRRS on virtual body ownership, agency and location of the body and usability in 8 healthy participants by comparing it with a partial upper-limb tracking method based on a Leap Motion controller (LP) in the same virtual environments. In the second experiment, we examined the feasibility of VRRS in upper-limb rehabilitation with 27 stroke patients.

Results

VRRS improved the users’ senses of body ownership, agency, and location of the body. The users preferred using the VRRS to using the LP. In addition, we found that although the upper limb motor function of patients from all groups was improved, the difference between the FM scores tested on the first day and the last day of the experimental group was more significant than that of the control groups.

Conclusions

A VRRS with motion tracking of the upper limbs and avatar control including the arms and all the fingers was developed. It resulted in an improved user experience of embodiment and effectively improved the effects of upper limb rehabilitation in stroke patients.

Trial registration

The study was registered at the First Affiliated Hospital of Jinan University Identifier: KY-2020–036; Date of registration: June 01, 2020.

Introduction

Stroke is a common health-care problem that results in obvious muscle weakness on one side of the body [1]. It is estimated that 50% to 75% of stroke patients have persistent impairment of the affected upper limb and must undergo repetitive physical training to recover their motor function [2]. To provide a more enjoyable and personalized motor rehabilitation experience, the use of virtual environments (VEs) as a tool is gradually becoming popular in the field because it offers richness of experience and is interesting to patients [3]. The virtual nature of the environment allows behaviors that are impossible or very expensive in reality to be implemented in a low-cost way. Bortone et al. [4,5,6] found that the use of a virtual environment and wearable devices offers a viable alternative to conventional therapy for improving upper extremity function in children with neuromotor impairments. Additionally, previous works indicate that the strength and ability of the affected side of the body can be effectively improved during rehabilitation that involves controlling an avatar’s upper limbs to interact with objects in VEs [7].

In many applications, avatars are used as the interface that allows people to interact with VEs [8]. Movement tracking is one of the key technologies for avatar control [9]. For partial upper-limb tracking, consumer devices, including hand-held controllers such as Oculus Touch or HTC VIVE controllers, and motion sensing devices, such as Kinect and Leap Motion controllers, are widely used [10]. Granqvist et al. [11] used HTC VIVE controllers with inverse kinematics for partial upper-limb tracking. However, the hand-held controller can only track the position of the hand and cannot provide information about the position of the fingers. Collingwoode-Williams et al. [12] built a system to study the effect of lip and arm synchronization on the feeling of body ownership. They used a Kinect device for body tracking and an Oculus Rift device for head rotation measurement. However, the Kinect device can only track the arms and a few key points on each hand (one key point on the fingertip, one key point on the thumb, and one key point on the whole hand). Complete upper-limb tracking enhances the realism of an avatar’s upper-limb behavior, which influences patients’ cognition and may be beneficial for rehabilitation [13]. Generally, for high-quality upper-limb tracking including the arms and all the fingers, it is necessary to use marker-based tracking systems that require the user to wear tracking suits, such as that employed by the OptiTrack system [13, 14]. These suits are expensive and cumbersome. The use of consumer devices is another less expensive solution. Lin et al. [15] used the Oculus Rift headset with a Kinect sensor, a Leap Motion controller and a dance pad to allow users navigate and manipulate objects inside synthetic scenes. With this method, only one Kinect was used, and it could not recognize which side of the body was facing the device. Wu et al. [16] introduced a setup that integrated multiple Kinects for robust and accurate full-body 3D skeleton tracking and a Leap Motion controller for tracking the hands. However, the tracking area of the Leap Motion controller was limited to a small range above the device.

In this paper, we propose a method that can control avatars’ upper limbs by tracking the movements of both arms and all fingers in a large tracking space. Using this method, we developed a virtual reality rehabilitation system (VRRS) that can map the movement of a user’s upper limbs to an avatar’s upper limbs to improve the user’s cognitive and affective experience during interaction in VEs. The difficulty of tasks can be conveniently modified to dynamically match the patient’s motor function, which is important for motor learning in general [17] and for rehabilitation in particular [18]. Hence, we hypothesized that VRRS can provide an effective rehabilitation training method to improve stroke patients’ recovery of their upper-limb motor function by enhancing their sense of body ownership, agency, and location of the body.

To test our hypothesis, we first performed an experiment with 8 healthy participants to compare VRRS with the partial tracking method based on Leap Motion [19]. In the experiment, we built VEs customized for upper-limb motion and assessed the senses of body ownership, agency, and location of the body and the usability of the systems. Then, to evaluate the feasibility of using the VRRS for rehabilitation training, another experiment was performed. For that experiment, 27 stroke patients were recruited, and the Fugl-Meyer (FM) scores for the patients’ upper limb motor functions were evaluated.

More at link.