Freezing
of gait (FOG) is a common and debilitating gait impairment of
Parkinson’s disease (PD). Up to 80% of people with Parkinson’s disease
(PwPD) may develop FOG during the course of the disease [1, 2].
FOG leads to sudden blocks in walking and is clinically defined as a
“brief, episodic absence or marked reduction of forward progression of
the feet despite the intention to walk and reach a destination” [3]. The PwPD themselves describe freezing of gait as “the feeling that their feet are glued to the ground” [4].
Freezing episodes most frequently occur while traversing under
environmental constraints, during emotional stress, during cognitive
overload by means of dual-tasking, and when initiating gait [5, 6]. Though, turning hesitation was found to be the most frequent trigger of FOG [7, 8]. Subjects with FOG experience more anxiety [9], have a lower quality of life [10], and are at a much higher risk of falls [11,12,13,14,15].
Given the severe adverse effects associated with FOG, there is a large incentive to advance novel interventions for FOG [16].
Unfortunately, the pathophysiology of FOG is complex and the
development of novel treatments is severely limited by the difficulty to
objectively assess FOG [17]. Due to heightened levels of attention, it is difficult to elicit FOG in the gait laboratory or clinical setting [4, 6]. Therefore, health professionals relied on subjects’ answers to subjective self-assessment questionnaires [18, 19], which may be insufficiently reliable to detect FOG severity [20]. Visual analysis of regular RGB videos has been put forward as the gold standard for rating FOG severity [20, 21].
However, the visual analysis relies on labor-intensive manual
annotation by a trained clinical expert. As a result, there is a clear
need for an automated and objective approach to assess FOG.
The
percentage time spent frozen (%TF), defined as the cumulative duration
of all FOG episodes divided by the total duration of the walking task,
and the number of FOG episodes (#FOG) have been put forward as reliable
outcome measures to objectively assess FOG [22].
An accurate segmentation in-time of the FOG episodes, with minimal
over-segmentation errors, is required to robustly determine both outcome
measures.
Several methods have been proposed for automated FOG
assessment based on motion capture (MoCap) data. MoCap encodes human
movement as a time series of human joint locations and orientations or
their higher-order representations and is typically performed with
optical or inertial measurement systems. Prior work has tackled
automated FOG assessment as an action recognition problem and used a
sliding-window scheme to segment a MoCap sequence into fixed partitions [23,24,25,26,27,28,29,30,31,32,33,34,35,36].
For all the samples within a partition, a single label is then
predicted with methods ranging from simple thresholding methods [23, 26] to high-level temporal models driven by deep learning [27, 30, 32, 33, 36].
However, the samples within a pre-defined partition may not always
share the same label. Therefore, a data-dependent heuristic is imposed
to force all samples to take a single label, most commonly by majority
voting [33, 36].
Moreover, a second data-dependent heuristic is needed to define the
duration of the sliding-window, which is a trade-off between
expressivity, i.e., the ability to capture long-term temporal patterns,
and sensitivity, i.e., the ability to identify short-duration FOG
episodes. Such manually defined heuristics are unlikely to generalize
across study protocols.
This study proposes to reformulate the
problem of FOG annotation as an action segmentation problem. Action
segmentation approaches overcome the need for manually defined
heuristics by generating a prediction for each sample within a long
untrimmed MoCap sequence. Several methods have been proposed to tackle
action segmentation. Similar to FOG assessment, earlier studies made use
of sliding-window classifiers [37, 38], which do not capture long-term temporal patterns [39]. Other approaches use temporal models such as hidden Markov models [40, 41] and recurrent neural networks [42, 43].
The state-of-the-art methods tend to use temporal convolutional neural
networks (TCN), which have been shown to outperform recurrent methods [39, 44].
Dilation is frequently added to capture long-term temporal patterns by
expanding the temporal receptive field of the TCN models [45].
In multi-stage temporal convolutional network (MS-TCN), the authors
show that multiple stages of temporal dilated convolutions significantly
reduce over-segmentation errors [46]. These action segmentation methods have historically been validated on video-based datasets [47, 48] and thus employ video-based features [49]. The human skeleton structure that is inherent to MoCap has thus not been exploited by prior work in action segmentation.
To
model the structured information among the markers, this paper uses the
spatial-temporal graph convolutional neural network (ST-GCN) [50]
as the first stage of an MS-TCN network. ST-GCN applies spatial graph
convolutions on the human skeleton graph at each time step and applies
dilated temporal convolutions on the temporal edges that connect the
same markers across consecutive time steps. The proposed model, termed
multi-stage spatial-temporal graph convolutional neural network
(MS-GCN), thus extends MS-TCN to skeleton-based data for enhanced action
segmentation within MoCap sequences.
The MS-GCN was tasked to
recognize and localize FOG segments in a MoCap sequence. The predicted
segments were quantitatively and qualitatively assessed versus the
agreed-upon annotations by two clinical-expert raters. From the
predicted segments, two clinically relevant FOG outcomes, the %TF and
#FOG, were computed and statistically validated. To the best of our
knowledge, the proposed MS-GCN is a novel neural network architecture
for skeleton-based action segmentation in general and FOG segmentation
in particular. The benefit of MS-GCN for FOG assessment is four-fold:
(1) It exploits ST-GCN to model the structured information inherent to
MoCap. (2) It allows modeling of long-term temporal context to capture
the complex dynamics that precede and succeed FOG. (3) It can operate on
high temporal resolutions for fine-grained FOG segmentation with
precise temporal boundaries. (4) To accomplish (2) and (3) with minimal
over-segmentation errors, MS-GCN utilizes multiple stages of refinement.
More at link
