Crapola like this DOES NOTHING to get survivors recovered!
Who gives a shit about inter-rater reliability you blithering idiots? Certainly not survivors!
Evaluating inter- and intra-rater reliability in assessing upper limb compensatory movements post-stroke: creating a ground truth through video analysis?
Journal of NeuroEngineering and Rehabilitation volume 21, Article number: 217 (2024)
Abstract
Background
Compensatory movements frequently emerge in the process of motor recovery after a stroke. Given their potential for unfavorable long-term effects, it is crucial to assess and document compensatory movements throughout rehabilitation. However, clinically applicable assessment tools are currently limited. Deep learning methods have shown promising potential for assessing movement quality and addressing this gap. A crucial prerequisite for developing an accurate measurement tool is ensuring reliability in assessing compensatory movements, which is essential for establishing a valid ground truth.
Objective
The study aimed to assess inter- and intra-rater reliability of occupational and physical therapists’ visual assessment of compensatory movements based on video analysis.
Methods
Experienced therapists evaluated video-recorded performances of a standardized drinking task through an online labeling system. The standardized drinking task was performed by seven individuals with mild to moderate upper limb motor impairments after a stroke. The therapists rated compensatory movements in predetermined body segments and movement phases using a slider with a continuous scale ranging from 0 (no compensation) to 100 (maximum compensation). The collected data were analyzed using a generalized-linear mixed effects model with zero-inflated beta regression to estimate variance components. Intraclass correlation coefficients (ICC) were calculated to assess inter- and intra-rater reliability.
Results
Twenty-two therapists participated in this study. Inter-rater reliability was good for the phases of reaching, drinking, and returning (ICC ≥ .0.75), and moderate for both phases of transporting. Intra-rater reliability was excellent for the drinking phase (ICC > 0.9) and moderate to good for the phases of reaching, transporting, and returning of our cohort. ICCs for smoothness and interjoint coordination were poor for both inter- and intra-rater reliability. The data analysis unveiled a wide range of credible intervals for the ICCs across all domains examined in this study.
Conclusions
While this study shows promising inter- and intra-rater reliability for the drinking phases within our sample, the wide credible intervals raise the possibility that these results may have occurred by chance. Consequently, we cannot recommend the establishment of a ground truth for the automatic assessment of compensatory movements during a drinking task based on therapists’ ratings alone.
Background
The Global Burden of Disease Study 2019 shows an increase in overall stroke incidents and exposes stroke as the third-leading cause of death and disability combined. About 80% of stroke survivors are affected by motor deficits, with approximately 50% experiencing persistent upper limb impairments [17, 27, 42].
Considering the involvement of arm and hand functions in activities of daily living (ADL), facilitating upper limb motor recovery following a stroke is essential. According to the current state of scientific consensus, motor recovery is distinguished by true recovery and motor compensation [8, 24, 32]. The Stroke Recovery and Rehabilitation Roundtable (SRRR) defines true recovery as regaining the same movement patterns as available before the injury and motor compensation as the development of new motor patterns by using intact body structures to accomplish an activity goal [8]. The definitions provided by Levin et al. [32] and Kleim [24] distinguish between motor compensation and true recovery across three different levels of the international Classification of Functioning, Disability and Health (ICF [62]) domains: health condition, body function/structure and activity. According to Levin et al. [32] changes at the Health Condition level occur through processes at the neuronal level. On this level, motor compensation is characterized by structural reorganization within the brain, where other brain regions assume the functions of damaged areas. At the body function/structure level, compensation is reflected in alternative movement patterns. At the activity level, compensation is evident when different limbs or end effectors are used compared to the premorbid status. Achieving a desired goal with the impaired arm can promote its use and prevent learned non-use, potentially enhancing functional capacity and independence in activities of daily living [21, 22]. However, compensatory movements can lead to musculoskeletal changes, increasing the risk of chronic pain [12, 13, 32]. Furthermore, compensation with the unimpaired limb can negatively impact neural reorganization, potentially hindering the true recovery of the impaired limb [22, 46]. Therefore, it is important to support the true recovery on the body function/structure level during rehabilitation, rather than allowing alternative movement patterns, to minimize adverse side effects. However, to facilitate the identification and treatment of motor compensation, a comprehensive assessment is necessary [10, 26]. Compensatory movements can be assessed by measuring the quality of upper limb movement patterns through motion analysis. Possible approaches to conduct this analysis include qualitative descriptions of movements based on visual observation, observer-based scoring using standardized assessment tools or kinematic motion analysis technologies, such as marker-based motion capture systems [50].
In research settings, marker-based motion capture systems are considered the gold standard for motion analysis. Unfortunately, its implementation in clinical practice is only possible to a limited extent because of high costs, duration of the assessment as well as requirement for high-quality instruments and specialized training [60].
Core measurement sets recommended to be applied in motor rehabilitation and recovery trials post-stroke describe the Fugl-Meyer Assessment (FMA, [16]) as essential tool for evaluating upper limb body functions, and the Action Research Arm Test (ARAT, [33]) for assessing motor function at the activity level [25, 41]. However, previous research indicated that these assessments simply focus on the ability to complete a task, without capturing the movement quality. Hence they do not differentiate between the recovery of movement patterns and the use of compensatory movement strategies [26, 32, 44]. In response to the need for an assessment specifically designed to measure motor compensation, Levin et al. [30] developed the Reaching Performance Scale for Stroke (RPSS). The RPSS [30] is an ordinal-scale assessment tool that evaluates compensatory movements of the upper extremity during two specific reaching tasks, excluding drinking motions. Recent studies by Subramanian et al. [54, 55] on the RPSS [30] provide promising results, describing it as a valid, reliable, and responsive scale for visually assessing motor compensation. However, the assessment is susceptible to ceiling effects due to its ordinal scaling. To our knowledge the RPSS is not yet commonly used in clinical practice. Consequently, therapists assess compensatory movements qualitatively through visual observation relying on their clinical expertise and experience [14, 28, 44, 47].
In recent years, considerable research has focused on developing assessments for motor compensation. However, due to the complexity of developing accurate and sensitive observer-based scoring assessments and the lack of consensus regarding the use of marker-based motion capture systems, further research is required to construct reliable, feasible and affordable measurement tools [26, 29, 47, 51].
Previous research has demonstrated that kinematic motion analysis technologies, such as marker-based motion capture systems, wearable sensors, marker-free vision sensors (e.g. simple cameras, or Microsoft Kinect depth sensors) or sensors embedded in rehabilitation training systems, provide objective, consistent, and precise detection of compensatory movements through automated processes, devoid of any floor and ceiling effects respectively [23, 28, 36, 60]. Recent reviews [39, 47] have highlighted the advantages of Machine Learning (ML) algorithms, which include high agreement levels, high accuracy, and low-cost, unobtrusive home based monitoring of patients. Supervised ML algorithms depend on labeled data, which serves as ground truth, to derive an optimal model capable of accurately predicting outcomes or classifying new, unseen data by leveraging learned patterns from the training dataset [5, 40]. In stroke rehabilitation, a promising domain for applying ML algorithms lies in the automated analysis of movement patterns in impaired limbs [39].
Therefore, we are currently conducting larger project to develop a ML-supported tool to analyze compensatory movements of the upper extremities and trunk on the body function/structure level, which can be used remotely with simple methods in the homes of patients after stroke [52, 56]. In this larger study, we plan to utilize simple webcam or smartphone cameras to assess compensation during a drinking task [1]. During the development of the ML algorithm, the completion of the task is recorded with a webcam and manually labeled by experienced therapists, who assess the extent of compensatory movements. This data was intended to be used to create a ground truth for the ML algorithm. While therapists are trained to assess movement quality in person (3-dimensional), it is unclear if they can do so when rating a drinking task based on videos (2-dimensional). The findings from Martinez et al. [35], Bernhardt et al. [6] and Bernhardt et al. [7] demonstrated good reliability in assessing reaching and object-lifting tasks using video analysis. However, these studies did not specifically evaluate the drinking task.
We seek to imitate human decision-making behavior using ML-supported models. To train an ML algorithm to imitate human intelligence, i.e. to build a ground truth that reflects human knowledge, we must make implicit therapeutic decisions, such as rating compensatory movements, explicitly available. This raises a broader discussion, which we intend to explore in this article. If we want ML to learn from human intelligence, a key challenge is determining whether humans can reliably assess movements during a drinking task from 2-dimensional video recordings in the first place. Depending on the findings, it remains uncertain whether basing a ground truth for assessment on human decisions would be a viable or effective approach.
Therefore, we conducted this study to investigate the inter-rater and intra-rater reliability of experienced therapists’ assessment of compensatory movement behavior based on video recordings and hence infer if such ratings are suitable for creating a ground truth for an ML application.
No comments:
Post a Comment