Multimodal assessment of upper limb rehabilitation in stroke patients based on cross-attention mechanism

 

What will it take to get thru your thick skulls that 'assessments' do nothing for recovery unless THEY POINT DIRECTLY TO EXACT RECOVERY PROTOCOLS?

This did nothing towards that, so useless!

Multimodal assessment of upper limb rehabilitation in stroke patients based on cross-attention mechanism

Author links open overlay panel, , , , , , 

https://doi.org/10.1016/j.asoc.2026.115199Get rights and content

Highlights

• •
  Non-contact 3D motion capture (Azure Kinect) joint angle MAE< 6° vs. NOKOV.
• •
  Res-Transformer feature extractor: 0.807–0.811 avg. acc (kinematics/EMG).
• •
  ReT-CA fusion model powers digital system: 89.2% avg. acc for stroke assessment.

Abstract

Stroke, characterized by high incidence, disability, and mortality rates, often causes brain damage and upper limb impairment, making objective, efficient, and personalized rehabilitation assessment crucial. This study developed a multimodal system for assessing upper limb motor function in stroke patients. First, a markerless motion capture system using the Azure Kinect DK was developed and validated against a gold standard NOKOV Motion Capture, achieving high accuracy (joint angle MAE < 6°). Second, a multi-modal feature set integrating kinematics and electromyography (EMG) data was constructed based on a customized Short-FMA test protocol. Third, an improved Res-Transformer model was proposed, demonstrating superior efficiency (40% faster training) and higher accuracy (avg. 0.807–0.811 for single modalities) compared to Transformer, LSTM, and TCN. Finally, a multimodal fusion model (ReT-CA), integrating Res-Transformer with a cross-attention mechanism, achieved a high classification accuracy (average 0.892). This model formed the core of a digital assessment system, featuring clinical score prediction, visualization, and report generation. The system offers a low-cost, accurate solution for objective stroke rehabilitation assessment, with the potential to be widely adopted in communities, hospitals, and other use scenarios.

Introduction

A stroke (also known as a Cerebrovascular Accident, or CVA) is a syndrome characterized by brain tissue damage resulting from the sudden rupture or blockage of cerebral blood vessels. It is distinguished by its high incidence, mortality, and disability rates, posing a significant global public health challenge [1]. According to data from the World Health Organization's Global Burden of Disease Study, stroke was responsible for 5.5 million deaths worldwide in 2016, accumulating 116 million Disability-Adjusted Life Years (DALYs) [2], [3]. Epidemiological statistics from 2019 confirm that stroke remained the third leading cause of global mortality and disability [4]. The burden is particularly pronounced in China. The number of stroke patients in China has reached 13 million [5], with both the incidence and mortality rates exhibiting a consistent upward trend. It projects that the incidence rate will reach 35% by 2050 [6].

Upper limb motor dysfunction is a typical clinical sequela of stroke, impairing patients' activities of daily living (ADL). Data indicate that approximately 85% of stroke patients experience upper limb motor impairment. Persistent dysfunction is observed in 60% of those undergoing rehabilitation, particularly in fine finger movements and wrist coordination control [7]. This dysfunction not only diminishes patients' quality of life but also imposes substantial economic and psychological burdens on both families and society [8]. Consequently, precise and objective assessment of upper limb motor function is paramount in stroke rehabilitation. It plays a crucial role in enhancing patient outcomes and mitigating societal burdens.

Currently, the most commonly employed clinical methods for assessing upper limb motor function in stroke patients rely on clinical scales. These include the Brunnstrom Recovery Stages, the Fugl-Meyer Assessment Upper Extremity Scale (FMA-UE) [9], and the Wolf Motor Function Test (WMFT) [10]. However, these traditional methods exhibit significant limitations: 1) Prolonged assessment duration – for instance, a single WMFT evaluation typically requires 30–45 min, substantially increasing healthcare costs; 2) Susceptibility to subjective interpretation by rehabilitation therapists, leading to considerable variability and difficulty in accurately capturing subtle neurological improvements during the rehabilitation process.

In recent years, the development of digital technologies has provided new research perspectives and tools for the rehabilitation field. Digital rehabilitation assessment enables precise quantitative evaluation of patient recovery progress through the use of various sensors, monitoring devices, and software platforms [11]. Multimodal data fusion technologies, such as motion capture and surface electromyography (sEMG), have gained significant attention due to their objectivity, precision, and real-time capabilities. However, current multimodal rehabilitation assessment methods predominantly rely on single-modality data analysis, limiting their ability to comprehensively and accurately reflect the complex upper limb functional status of patients.

Addressing these limitations, this paper aims to develop a multimodal data-driven upper limb rehabilitation assessment system for stroke patients utilizing the Azure Kinect DK device and sEMG technology. The integration of motion and EMG data enables more objective and accurate rehabilitation assessments. This approach contributes to enhanced efficiency and quality in stroke rehabilitation therapy. The outcomes of this research will not only serve as an objective clinical assessment tool, effectively reducing the workload of healthcare professionals and the economic burden on patients, but also provide innovative theoretical support and technical reference for the stroke rehabilitation field.

Multimodal technology has seen preliminary applications. However, existing methods still show notable limitations in data acquisition, model architecture, and fusion strategies. In motion capture, most studies rely on wearable sensors requiring physical contact [12], [13] or early-generation depth cameras with limited capabilities [14], [15]. For instance, while Lee et al. [16] achieved high scoring accuracy, their system still required force-sensitive resistors and employed rule-based classification algorithms.

At the model architecture level, existing approaches predominantly use traditional classifiers such as ANFIS [17] or DBN [18], which present limitations in feature representation capability and computational efficiency. Current multimodal fusion research primarily employs decision-level fusion [19] or simple feature concatenation methods, which cannot fully explore the deep interactive relationships across modalities. While these preliminary applications of multimodal technology demonstrate promise, further research is crucial to optimize fusion analysis methods for motion and EMG data, particularly to enhance their effectiveness in practical clinical assessments.

To address this need and the identified limitations, this study proposes a systematic and innovative solution: a multimodal rehabilitation assessment system based on the fusion of Azure Kinect DK and EMG data. This system aims to provide a more comprehensive and precise rehabilitation assessment solution for stroke patients by addressing the clinical need for an objective assessment of upper limb motor dysfunction. The main contributions of this paper include the following three aspects: (1) Development of an upper limb three-dimensional motion data acquisition system based on non-contact devices, enabling precise and convenient data collection. Comparison of upper limb joint angles with the gold-standard NOKOV system yielded absolute errors of less than 6°. (2) Proposal of a Res-Transformer model that processes raw data inputs directly, achieving a 7.8% average accuracy gain over feature-based methods. The model achieved average accuracies of 0.807 and 0.811 on motion and EMG modalities, respectively, while reducing training time by approximately 40%. (3) Construction of a multimodal fusion model and development of a digital upper limb rehabilitation assessment system. The system achieved an average accuracy of 0.892 in classification tasks for rehabilitation assessment, providing an objective, accurate, and intelligent assessment tool for stroke patients.

Access through your organization