A Preliminary Study of Using Kinect-based Physical Rehabilitation System to Perform Tai Chi Exercises with FLS Evaluation

Corresponding Author:

Jiann-Der Lee
Department of Electrical Engineering, Chang Gung University, Taoyuan, 333 Taiwan
Tel: 03-2118800, Ext: 5316

Abstract

In this study, a physical rehabilitation system based on Kinect sensor is proposed to assist patients with movement disorders to perform the “Tai Chi” exercises at home. With the aid of skeletal tracking of Kinect sensor, each movement of the patient while performing Tai Chi exercises is well recognized and validated. More specifically, patient’s movements are compared with the prerecorded movements form a coach, and further evaluated by using fuzzy logic system (FLS). The experiment was carried out according to an ABAB sequence, i.e., A represents the baseline and B represents intervention phases. Experimental results also show that the participants significantly increased their motivation for physical rehabilitation, thus improving exercise performance during the intervention phases.

Keywords

Tai Chi exercise, Physical rehabilitation, Kinect, Fuzzy logic system

Introduction

Parkinson’s disease (PD) is one of the best known movement disorders. It happens when there is a problem with certain nerve cells in the brain. The major symptoms of Parkinson’s disease involve the motor nervous system, including tremor, rigidity, bradykinesia, shuffling gait, and postural instability. It can also lead to a higher risk of falling, which can cause serious injuries [1-3].

In Chinese martial arts, Tai Chi is a functional exercise. By striding out and shifting the body’s center of gravity, it is useful for the patients with movement disorder to train the ability of maintaining balance [4]. In past, Yeh, et al. [5] showed that performing regular Tai Chi exercise program for twelve weeks can effectively enhance functional mobility, and further increase T cell mediators transforming growth factor β and interleukin 10. As described in the article of Hackney and Earhart [6], thirty-three people with PD were randomly categorized into two groups, Tai Chi group and control group. These participants in Tai Chi group received twice weekly one-hour lessons and completed twenty lessons within thirteen weeks. The other group received no intervention. Their conclusion indicated that PD participant who takes Tai Chi lessons twice per week has significant improvements in the terms of gait, balance and functional mobility. Later, a research article was published in New England Journal of Medicine by Dr. Fu-Zhong Li [7], in which 195 patients with Parkinson’s disease were randomly categorized into three groups; each group performed 60 minutes of an exercise program twice a week and lasted for 24 weeks. The group of patients who performed Tai Chi as their excises were better than those who performed resistance-training or stretch-training in the tests of balance. Their frequency of falling decreased, and their stride length during walking increased than before. The effect of practicing Tai Chi maintained three months after the end of exercise program. From these studies, it is obvious that Tai Chi exercise is a very good exercise for the patients with movement disorders to enhance their balance ability, or even further improve patient’s health status [8-11].

Traditionally, in order to practice Tai Chi exercise by accurately following the instructions of physical therapists, the patients have to go to a rehabilitation center or hospital. However, sometimes this becomes inconvenient for a patient who lives in suburb areas since additional time and money on transportation are required. Furthermore, if a physical therapist has to take care more patients at the same time, then individual treatment quality on each patient may decreased. One way to overcome this problem is to hire a home therapist, but obviously it was not economically viable [10]. An alternative solution is the tele-rehabilitation, which can deliver rehabilitation services over telecommunication network or internet. By this way, patients can follow the instructions from remote rehabilitation therapist or message from rehabilitation system to perform rehabilitation exercise or training at home for every day [11,12].

Recently, a popular device, i.e., Microsoft Kinect Sensor, has been widely used in various applications of computer vision. Comparing to other 3D scanners, it uses a relatively lowcost infrared RGB-D sensor to obtain depth information of the environment in realtime. Basically, its functions just like as a 3D structured-light depth scanner, and the 3D coordinate of an object in the field of view of a camera can be easily obtained. Although Kinect sensor is originally for Xbox 360 video game, it has been widely utilized in different fields of studies, such as object reconstruction [9] or AR applications [13,14]. Moreover, there are also some applications in developing health-care, rehabilitation or in-home activity monitoring of elders. For example, Lange, et al. [15] developed a low-cost balance rehabilitation tool using Kinect and Stone and Skubic [16] also presented a Kinect-based system to monitor elderly adults to avoid fall risk while performing in-home activity. The system developed by Metcalf, et al. [17] is used to capture and measure hand kinematics, and can be employed for home-based upper limb rehabilitation. Besides, Kitsunezaki, et al. [18] reported four types of Kinect-based physical applications to measure useful parameters such as walking time or joint angle ranges, to determining training effectiveness.

In this study, a rehabilitation system based on Kinect sensor is proposed. This system can assist the patients with movement disorders to perform the Tai Chi exercise program at home. The rehabilitation system shows a Tai Chi exercise video recorded from the Tai Chi coach on the screen, and the patients follow the instruction showed in this video to perform Tai Chi at home. Furthermore, the system records and recognizes patient’s postures of each movement, and compare the similarity of the movements performed by the coach. Fuzzy logic system (FLS) is utilized here to judge the comparison result, further determine the patient’s movement is “Perfect”, “Good”, “Fair” or “Need to be improved.” The result of this system let the patients know the condition of their movement, and what movement needed to be improved. Using this system, the patients can achieve the same rehabilitation effect as well as using expensive equipment in the past, and then greatly reduce the consumption of resources.

Methods

▪ System overview

A system flowchart of the proposed Kinect-based rehabilitation system is shown in Figure 1. We invited a Tai Chi senior instructor to perform the correct action in a Tai Chi course, and utilized Kinect sensor to record their movements as movement database. These movements performed by the coaches are considered as standard movement, and the skeleton of these movements are then extracted for later posture comparison.

Figure 1: The flowchart of the proposed system.

When a patient uses this rehabilitation system, the system first displays a series of standard movements on the screen. The patient follows the information on screen to perform Tai Chi exercise, and his/her movements are then compared to the movements of instructors by their corresponding skeletons. Finally, the results indicated the similarity score of their movements with the standard movements will show on screen, and the user can practice how to improve their movement if the score is unsatisfied.

▪ Kinect Sensor and Skeletal Tacking

Figure 2 shows the hardware architecture of a Kinect sensor. From this figure, it is seen that Kinect sensor mainly consists of color sensor, Infrared (IR) emitter and IR depth sensor. The IR depth sensor can measure depth in the IR camera field of view. By applying provided software development kit (SDK) [19], the depth information can further be utilized for skeletal tracking.

Figure 2: Kinect sensor and Skeletal Tracking Modes (Default/Seated).

Skeletal tracking of Kinect has two different modes-default mode and seated mode. Default mode tracks twenty skeletal joints of the full body, reporting their position as “tracked” or “inferred”. The seated mode only tracks the ten upper-body joints (shoulders, elbows, wrists, arms and head) and reports the lower-body joints (spine, hips, knees, ankles, and feet) as “not tracked” [19].

▪ Tai Chi rehabilitation exercises video

Considering the safety of patients with movement disorder and the elderly, Taiwan Movement Disorder Society (TMDS) thus devised two shorter style of Tai Chi based on the original Yang-style Tai Chi. First one is the “Tai Chi Regimen”, which is for healthy people or patients who can stand and walk independently. And the second one “Seated Tai Chi”, on the other hand, is designed for patients who are more challenged maintaining a standing posture [20].

In this system, we invite Master Yun-Chin Hsu, national grade coach of Yang-style Tai chi, to record these instruction videos of Tai Chi exercise. In Tai Chi, a “form” is a combination of several movements and postures. In the proposed system, the “Tai Chi Regimen” is divided into ten forms: (1) Commencing form, (2) Wave hands like clouds, (3) Grasp the bird’s tail, (4) Single whip, (5) Hand strums the lute, (6) White crane spreads wings, (7) Brush knee and twist step, (8) Repulse monkey, (9) Play arms like a Fan, (10) Closing form. These standard movements are shown in Figure 3.

Figure 3: Tai Chi Regimen ten strokes. The top row from left to right is form (1) to form (5). The bottom row from left to right is form (6) to form (10).

The “Seated Tai Chi” is divided into eighteen forms: (1) Commencing form, (2) Grasp the bird’s tail, (3) Single whip, (4) Jade lady works shuttles, (5) Hand strums the lute, (6) White crane spreads wings, (7) Beating tiger form, (8) Sideways swing fist, (9) Wave hands like clouds . (10) Flying oblique, (11) Single whip, (12) High pat on horse, (13) Fist under elbow, (14) Repulse monkey, (15) Shooting tiger form, (16) Apparent closing up, (17) Cross hands, (18) Closing form. Also, these standard movements are shown in Figure 4.

Figure 4: Seated Tai Chi eighteen strokes. The top row from left to right is form (1) to form (6). The middle row from to right is form (7) to form (12). The bottom row from left to right is form (13) to form (18).

▪ Skeleton Normalization and Similarity Comparison

In practical application, it is observed that people may stand at any position with various postures. Therefore, the space coordinates of the subject’s movement need to be normalized to homogenous coordinate system [21] before comparing the extracted skeleton. That is, we firstly calculate the translation of the geometry conversion in 3D Cartesian coordinates and define torso as the origin. In computer graphics, it always applies the homogeneous coordinate system to simplify the operation of the matrix. Here, in similar manner, we record the translation, scaling and rotation in the matrix form. The translation of geometry conversion in 3D Cartesian coordinates is shown as Figure 5, and the formula is expressed as Eq. (1),

Figure 5: Translation of geometry conversion.

(1)

Therefore, the torso center of the skeletons is aligned for normalization before movement comparison. After normalization, both of the skeletons used for movement comparison have a fairly consistent head-body-arm ratio. To compare these two skeletons extracted from standard movement and patient’s movement, it is necessary to find the difference between these two skeletons and give it a quantitative score. To achieve this goal, we firstly calculate the position difference of each joint between the skeletons of standard movement and patient’s movement. Assuming the skeleton extracted from the standard movement is named as skeleton A and skeleton of the user’s movement is named as skeleton B. The set of all skeletal joints of A can be denoted as P_A = {a₁,a₂,a₃,..,a₁₀}. Also, the set of all skeletal joints of B is denoted as P_B = {b₁,b₂,b₃,..,b₁₀}. Thus, we obtain 10 human skeleton joints with three dimensional coordinates and calculate their difference values. Here, the mean distance between the skeleton of standard movement and the skeleton of patient’s movement are obtained by Eq. (2):

(2)

Secondly, we consider the difference of angles between the skeletons obtained from the standard movement and the patient’s movement. Since two of these joints can form a line, and there are total nine lines for a seated skeleton. Next, we calculate the angle between two lines which have same connected joint, as shown in Figure 6. So, there are also nine angles for a seated skeleton. Assuming the set of angles in skeleton A is and the set of angles in skeleton B is thus, the mean difference of angles between skeleton A and B can be expressed by Eq. (3).

Figure 6: An example for illustration of the skeleton angles.

(3)

Fuzzy Logic System

Fuzzy set theory was originally introduced by Zadeh in 1965 [22]. Since the borders between each class are usually not clear in the problem of pattern classification, so fuzzy logic system (FLS) is proposed to make better decision than which human beings usually determine by experience. In general, fuzzy logic system is considered as a mapping relationship between input and output. The concept is to distribute the entire input space into overlapping intervals, then according some grouping methods output the fuzzy rules for classifying these values, and then finally defuzzify the results to obtain an output value. Basically, FLS can be divided into four parts: fuzzzifier, rules, inference system and defuzzifier [23,24], as shown in Figure 7.

Figure 7: Diagram of a fuzzy logic system.

The function of the fuzzifier is to map numerical data into fuzzy sets. The inputs are described by one or more fuzzy subsets defined by membership functions. In this system, each form of the Tai Chi consists of four key movements. Patients or elderly follow the teaching video to practice the exercise step by step. In each key movement, the joint distance d_joint and the difference of angle d_angle are calculated. Thus this fuzzy logic system has two inputs – mean of joint distance d_joint and mean of angle difference d_angle. For each input, four classes – “perfect”, “good”, “fair” and “poor” are adopted for grading. The membership functions of these two inputs are designed with Gaussian curve membership function and generalized bell-shaped membership function, and the numerical value of each input are mapped into these four classes. The membership functions of these two inputs are shown in Figures 8 and 9, respectively.

Figure 8: Membership function of joint distance.

Figure 9: Membership function of angle difference.

In general, the rules are a set of IF-THEN propositions, and the inference system is used to synthesize the solution of the rules to obtain the final membership function. In the proposed system, the output also has four classes as the inputs. The output membership function is also designed with Gaussian curve membership function and generalized bellshaped membership function, as shown in Figure 10. The final step of an FLS is to convert the fuzzy set into numerical data, which is called defuzzification. For defuzzification, the centerof- gravity method [24] is utilized because it is simple and efficient.

Figure 10: Membership function of output.

The system gives a score of the key movement according to the defuzzification result. If the results is “Perfect”, in this key movement the user gets 4 points. Else if the results are “Good”, in this key movement the user gets 3 points, and so on. After the end of a form of Tai Chi, it will summarize how many points the user has done and output the result, as shown in Figure 11.

Figure 11: Result grading after performing a Tai Chi form.

In this rehabilitation system, although we have an actual quantitative score to evaluate the patient’s movement, we don`t want the score to increase the patient stress. Therefore, instead of directly displaying the score, we divide the summarized score of a form into four levels of comments to inform the user: (1) Try it again, (2) Good work, (3) Excellent, (4) Perfect. The users can enjoy practicing Tai Chi by using this physical rehabilitation assistant system. Fig. 12 shows two examples of the comments produced by the system while the user practices “Seated Tai Chi”.

Figure 12: Implementation of the proposed system.

Experimental Results

▪ Participants

Since this system is designed for an individual with movement disorders, two patients are invited to participate the experiments. Neither of the participants had previous experience with Kinect. The profiles of these two patients are listed in Table 1.

Table 1: Participants’ Profile.

▪ Experiment setup

In order to validate the practicability of Kinectbased rehabilitation system, the experiment was designed according to an ABAB sequence [25]. More specifically, phase A represents the baseline phase and phase B represents intervention phase, and each phase includes the observation lasted for several days. In phase A, no assistive rehabilitation system was applied. The therapist instructed a participant the content of a rehabilitation program. The participant then repeated what she heard and started to perform Tai Chi exercise. On the other hand, in phases B, the proposed Kinect-based rehabilitation system was used. The participants used the cues provided by the system to complete the Tai Chi exercise, and the therapist is not involved in guiding patients. During the total experimental course, Tai Chi exercise was performed every day in each phase and the number of correct movements was recorded for performance comparison [26].

Results

Figure 13 shows the experimental result for patient 1. The numbers of correct movements in the baseline and intervention phases are shown, respectively. It is observed that during the first baseline phase, the number of correct movements was 22 on the average. And, during the first intervention phase, the number of correct movements increased to about 66. The number of correct movements decreased to 23 during the second baseline and increased again to 64 during the second intervention phase. The difference of numbers of correct movements between the baseline and the intervention was significant (p<0.05) on the Kolmogorov-Smirnov test [21].

Figure 13: Number of correct movements for patient1 completing the Tai Chi exercise.

Figure 14 shows the experimental result for patient 2. The numbers of correct movements in the baseline and intervention phases are shown, respectively. During the first baseline phase, the number of correct movements was 15 on the average. During the first intervention phase, the number of correct movements increased to about 49. The number of correct movements decreased to 14 during the second baseline and increased again to 51 during the second intervention phase. The difference of numbers of correct movements between the baseline and the intervention was significant (p<0.05) on the Kolmogorov-Smirnov test. These results show the significant potential of the proposed system in future applications.

Figure 14: Number of correct movements for patient2 completing the Tai Chi exercise.

Conclusions

This paper proposed a Kinect-based rehabilitation system for the patient with movement disorders by performing the Tai Chi exercises at home. We adopt a baseline/ intervention strategy to assess the effectiveness of this system. From the experimental results, it is shown that two patients have significantly improvement in performing the Tai Chi exercise.

Though this proposed system is a prototype, but the preliminary result is encouraged. In the future we will integrate virtual reality to enhance the entertainment effect. In additions, the skeleton can be replaced by an avatar to make it look more attractive. Because some users seemed to like the skeleton, the game can allow users to have an option of choosing an avatar or the skeletal image. Also, adding background music and interesting audio effects can make the game more appealing.

Acknowledgment

The work was partly supported by Ministry of Science and Technology, Republic of China, under Grant MOST 106-2221-E-182-025 and Chang Gung Memorial Hospital with Grant No. CMRPD2G0121 and CMRPD2C0043.

References

1. Jiann-Der Lee, Chung-Hsien Huang, Yi-Hsin Weng, et al. An automatic MRI/SPECT registration algorithm of using image intensity and anatomic feature as matching characters: Application on the evaluation of Parkinson’s disease. Nucl. Med. Biol 34(4), 447-457 (2007).
2. Jiann-Der Lee, Chung-Hsien Huang, Yi-Hsin Weng, et al. Improved accuracy for brain MRI/SPECT registration using a 2-cluster SPECT normalization algorithm and a combinative similarity measure: Application to the evaluation of Parkinson’s disease. Ann. Nucl. Med 21(4), 197-207 (2007).
3. Weng YH, Yen TC, Chen MC, et al. Sensitivity and specificity of 99mTc-TRODAT-1 SPECT imaging in differentiating patients with idiopathic Parkinson’s disease from healthy subjects. J.Nucl. Med 45(3), 393-401 (2004).
4. Han-Cheng Wang. Simplified Tai Chi and Sitting Tai Chi Teaching manual. Taiwan Movement Disorder Society, R.O.C. Taiwan, (2013).
5. Yeh SH, Chuang H, Lin LW, et al. Regular Tai Chi Chuan Exercise Enhances Functional Mobility and CD4CD25 Regulatory T Cells. Br. J. Sports. Med 40(3), 239-243 (2006).
6. Hackney ME, Earhart GM. Tai Chi improves balance and mobility in people with Parkinson disease. Gait & Posture, 28(3), 456-460 (2008).
7. Fu-Zhong Li, Peter Harmer, Katleen Fitzgerald, et al. Tai Chi and Postral Stability in Patients with Parkinson’s Disease. N. Engl. J. Med 366(1), 511-519, (2012).
8. Tsang WW, Wong VS, Fu SN, et al. Tai Chi improves standing balance control under reduced or conflicting sensory conditions. Arch. Phys. Med. Rehabil 85(1), 129-137 (2004).
9. Taylor-Piliae RE. Tai Chi as an Adjunct to Cardiac Rehabilitation Exercise Training. J. Cardiopulm. Rehab 23(2), 90-96 (2003).
10. Chien-Hsing Chou, Chi-Yi Tsai, Syue-Jyun Fang, et al. A Home Rehabilitation System Combined with Somatosensory Games for Stroke Patients. ICIC. Express. Letters 7(3), 1005-1010 (2013).
11. Holden MK, Dyar TA, Dayan-Cimadoro L. Telerehabilitation using a virtual environment improves upper extremity function in patients with stroke. IEEE. Trans. Neural. Sys. Rehab. Eng 15(1), 36-42 (2007).
12. Jia W, Yi WJ, Saniie J, et al. 3D Image Reconstruction and Human Body Tracking using Stereo Vision and Kinect Technology. IEEE. Int. Conf. Electro/Inform. Techn 1-4 (2012).
13. Placitelli AP, Gallo L. Low-Cost Augmented Reality Systems via 3D Point Cloud Sensors. Int. Conf. Signal-Image. Techn. Internet-Based. Sys 188-192, Naples, Italy (2011).
14. Santos ES, Lamounier EA, Cardoso A. Interaction in Augmented Reality Environments Using Kinect. Symposium on Virtual Reality, 112-121, Uberlandia, Brazil (2011).
15. Lange B, Chang CY, Suma E, et al. Development and evaluation of low cost game-based balance rehabilitation tool using the Microsoft kinect sensor. 33rd Ann. Int. Conf. IEEE. EMBS 1831-1834 (2011).
16. Stone EE, Skubic M. Evaluation of an Inexpensive depth camera for passive in-home fall risk assessment. 5th Int. Conf. Pervas. Comput. Techn. Healthcare, 71-77 (2011).
17. Matcalf CD, Robinson R, Malpass AJ, et al. Markerless motion capture and measurement of hand kinematics: validation and application to home-based upper limb rehabilitation. IEEE. Trans. Biomed. Eng 60(8), 2184-2192 (2013).
18. Kitsunezaki N, Adachi E, Masuda T, et al. Kinect applications for the physical rehabilitation. IEEE. Int. Symposium. Med. Measure. App. Proc 294-299 (2013).
19. Microsoft Developer Network, Kinect for Windows SDK.
20. Kuei-Min Chen, Wen-Ting Chen, Min-Feng Huang. Development of the simplified Tai Chi exercise program (STEP) for frail older adults. Complem. Therap. Med 14(1), 200-206 (2006).
21. Ching-Tang Hsieh, Ruei-Chi Chung, Yeh-Huang Wu, et al. Physical rehabilitation assistant system based on Kinect. Proc. Nat. Sympos. Sys. Sci. Eng 284-287(1), 1686-1690 (2012).
22. Zadeh LA. Fuzzy sets. Informat. Control 8(3), 338-353 (1965).
23. Mendel JM. Fuzzy logic systems for engineering: a tutorial. Proc. IEEE 83(3), 345-377 (2002).
24. Lin CT, Lee CSG. Neural-network-based fuzzy logic control and decision system. IEEE. Transact. Comp 40(12), 1320-1336 (1991).
25. Richards SB, Taylor RL, Ramasamy R, et al. Single subject research: applications in educational and clinical settings, New York: Wadsworth, USA (1999).
26. Siegel S, Castellan NJ. Nonparametric statistics for the behavioral sciences. New York: McGraw-HiU Book Company, USA (1998).