Deep learning-based detection of affected body parts in Parkinson’s disease and freezing of gait using time-series imaging

Abstract

We proposed a deep learning method using a convolutional neural network on time-series (TS) images to detect and differentiate affected body parts in people with Parkinson’s disease (PD) and freezing of gait (FOG) during 360° turning tasks. The 360° turning task was performed by 90 participants (60 people with PD [30 freezers and 30 nonfreezers] and 30 age-matched older adults (controls) at their preferred speed. The position and acceleration underwent preprocessing. The analysis was expanded from temporal to visual data using TS imaging methods. According to the PD vs. controls classification, the right lower third of the lateral shank (RTIB) on the least affected side (LAS) and the right calcaneus (RHEE) on the LAS were the most relevant body segments in the position and acceleration TS images. The RHEE marker exhibited the highest accuracy in the acceleration TS images. The identified markers for the classification of freezers vs. nonfreezers vs. controls were the left lateral humeral epicondyle (LELB) on the more affected side and the left posterior superior iliac spine (LPSI). The LPSI marker in the acceleration TS images displayed the highest accuracy. This approach could be a useful supplementary tool for determining PD severity and FOG.

Introduction

Parkinson’s disease (PD) is a neurodegenerative disease caused by the progressive loss of dopaminergic neurons in the substantia nigra pars compacta of the midbrain, which is critical for motor control^1,2. One of the motor symptoms of PD, freezing of gait (FOG), reduces the forward progression of the feet despite the intention to walk and increases the risk of falls³. FOG episodes are provoked while turning, gait initiation, walking through narrow passages such as doorways, or performing several tasks during walking. People with PD and FOG (freezers) are at a heightened risk of experiencing falls and must exercise greater caution when performing postural transitions or turn tasks in their daily activities⁴.

Turning is more complex and challenging than straight walking because it requires greater interlimb coordination, dynamic balance, posture and gait coupling, and cognition⁵. The unilateral onset and asymmetry of progressive motor symptoms are unique characteristics of PD and may affect the upper and lower extremities during turning tasks^6,7. Therefore, axial signs such as postural instability and asymmetric gait deficits, including repeated FOG episodes, can negatively affect overall mobility and daily activity^8,9. Thus, these factors are essential for assessing disease severity and long-term monitoring¹⁰.

FOG can be diagnosed through subjective or objective methods, which require a trained clinician to observe FOG signs using tools such as the Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) Part III scale and the New Freezing of Gait Questionnaire (NFOG-Q)¹⁰. However, these clinical evaluations are subjective and present challenges for tracking fine-grained changes in PD symptoms². The need for FOG classification extends beyond merely determining the presence of FOG in people with PD. Existing methods, such as video-based assessments by experts, may distinguish between FOG and non-FOG episodes. Nevertheless, the focus of our research is on developing classification approaches that go beyond binary classifications. These approaches are expected to provide a more nuanced understanding of FOG severity, potentially aiding early PD diagnosis and contributing to a comprehensive evaluation of motor symptoms¹¹. Therefore, we emphasize the necessity for objective and accurate technology to assess, diagnose, and classify the spectrum of PD symptoms, including FOG.

Recently, new methods have been developed for objectively measuring motor PD symptoms using artificial intelligence (AI) and signal processing technology¹². These methods focused on improving disease severity discrimination and classification using features and comprehensive gait characteristics of people with PD^13–15. Previous studies using various algorithms have been conducted to determine the optimal combination of gait parameters for identifying and classifying people with PD and healthy controls or subgroups such as freezers and nonfreezers^16–19. However, these studies used only limited gait features and were at risk of overfitting the data owing to their high correlation with multiple variables^15,20. This overfitting issue is exacerbated by small sample sizes and a lack of diverse datasets, leading to models that may not generalize well to new data²¹. These studies often fail to capture the complex, multidimensional nature of gait in people with PD, which reduces the robustness and applicability of the models in real-world scenarios¹⁵. Additionally, analyzing only a single wearable inertial sensor signal for specific movement data could limit the accurate evaluation of the extent of disease impairment. Moreover, analyzing single variables lacks reliability and sensitivity and cannot objectively identify body segments based on the severity of motor symptoms in people with PD²².

Signal data from wearable sensors, remote monitoring devices, and motion capture systems can be combined with an automated approach to overcome these limitations^23,24. Previous studies have shown that utilizing advanced machine learning (ML) and deep learning (DL) technologies can significantly improve precision and sensitivity^21,25. Mirelman et al.²⁵. used multiple wearable sensors to measure gait and mobility features across different stages of PD, achieving high discriminatory values with mean sensitivities ranging from 72 to 83% and specificities ranging from 69 to 80%²⁵. Similarly, Trabassi et al.²¹. demonstrated the effectiveness of using a selected set of inertial measurement unit (IMU)-derived gait features with various ML algorithms to classify PD, achieving a prediction accuracy higher than 80%. Their approach reduced the risk of overfitting and improved the interpretability of results²¹. These studies highlight the potential of AI in improving the accuracy and reliability of PD diagnosis and monitoring^21,25.

Furthermore, these technologies can also learn to model movement patterns, such as tracking the gait pattern or real-world motion of PD. They can facilitate continuous and objective measurement and monitoring of disease severity through accurate predictive modeling²⁶. Previous studies have suggested methods for capturing impaired gait patterns from lower back motion^27–29, describing gait complexity as a topological nonlinear dynamics system³⁰, exploring combinations of sensor locations^31,32, and using three-axis position and acceleration data^33–35, frequency, time-series (TS) of window lengths, and features^34,36 with ML and DL algorithms^24,34. These models, which have been trained to capture meaningful features associated with PD or FOG-related features, can help detect bradykinesia and FOG episodes² and differentiate medication “on” and “off” states³⁷ during activities of daily living. They can also predict postural instability/gait difficulty scores during the 2-min walk test³⁸ and characterize step and stride during the 10-m straight walk and timed up-and-go test at a self-selected speed^39,40. Although accuracy levels ranging from 82.7 to 96.7% have been achieved, the small sample size (8–31 patients) and class imbalance in most medical datasets remain a challenge. Moreover, the scarcity of data hinders its division into multiple sets for model training, validation, and testing³⁸.

Capturing TS gait patterns during turning tasks to observe axial signs, such as postural instability and asymmetry, may be necessary to address the abovementioned issues and provide objective measures for diagnosing and monitoring people with PD and freezers. Kwon et al.²⁴. reported that FOG is not limited to the lower extremities and involves significant upper extremity movements, potentially necessitating a holistic kinematic analysis of the entire body. Therefore, analytical and predictive models trained on overall body segment data to select optimal sensor attachment locations using the DL approach may be employed^9,26,36.

The present study aimed (i) to identify body segments potentially exhibiting symptomatic differences between people with PD and healthy controls as well as among those experiencing FOG (freezers), those without FOG (nonfreezers), and controls and (ii) to propose DL models capable of classifying these groups using TS position and acceleration data obtained from the identified body segments. Hence, we studied convolutional neural network (CNN)-based DL algorithms with images for TS gait patterns during the 360° turning task on the more affected side at a preferred speed in people with PD and freezers. Our findings may be used to suggest clinical measurements as a supplementary tool to discover novel kinematic biomarkers and develop objective severity scales of individualized PD and FOG for nonpharmacological treatments.

Methods

Ethics approval and consent to participate

This study was performed according to the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The study and its supplementary information files were approved by the Institutional Review Board of Dong-A University Hospital (IRB number: DAUHIRB-22-089) (see ethics approval letter). All patients provided written informed consent before data collection.

Participants

This study enrolled 60 individuals diagnosed with PD, comprising 30 individuals who experienced FOG (freezers) and 30 who did not experience FOG (nonfreezers), as well as 30 age-matched older adults as controls. The participants’ characteristics are summarized in Table 1, and a flow chart detailing their involvement is presented in Fig. 1. A neurology specialist diagnosed PD using the United Kingdom Parkinson’s Disease Society Brain Bank diagnostic criteria⁴¹. The inclusion criteria for individuals were as follows: aged between 55 and 85 years; had a modified Hoehn and Yahr stage of 2 or 3 (mild-to-moderate idiopathic PD)^42,43; scored more than 24 on the Mini-Mental State Examination (MMSE)⁴⁴; showed a stable response to antiparkinsonian medications; were classified as freezers or nonfreezers based on the NFOG-Q assessment for FOG⁴⁵; and could walk and stand unassisted during the clinical tests. Exclusion criteria included a history of cardiovascular, musculoskeletal, vestibular, or other neurological diseases; dependence on assistive devices for movement; and uncontrollable dyskinesia with drug therapy. The control group comprised healthy individuals without cognitive impairment or gait disturbance in the past 6 months⁴⁶.

Table 1.

Physical and clinical characteristics of participants.

	People with PD		Controls (n = 30)	p value
	Freezers (n = 30)	Nonfreezers (n = 30)
Sex (male/female)	19/11	16/14	14/16	0.466a
Age (years)	69.23 ± 5.33	69.23 ± 5.43	68.98 ± 5.57	0.943b
Height (cm)	159.36 ± 9.51	158.43 ± 8.37	159.74 ± 6.96	0.822b
Body weight (kg)	60.32 ± 8.49	60.96 ± 8.28	61.47 ± 7.33	0.857b
BMI (kg/m2)	23.73 ± 2.46	24.28 ± 2.67	24.08 ± 2.28	0.686b
MMSE (scores)	27.90 ± 1.92	27.10 ± 1.99	27.83 ± 1.53	0.112c
Disease duration (years)	8.56 ± 5.85	3.20 ± 2.44	-	< 0.001 d
Treatment duration (years)	7.80 ± 6.42	2.48 ± 2.31	-	< 0.001 d
L-DOPA equivalent dose (mg/day)	731.50 ± 329.98	437.18 ± 225.98	-	< 0.001 d
NFOG-Q (scores)	12.57 ± 7.30	-	-	-
Hoehn and Yahr scale	2.63 ± 0.41	2.27 ± 0.41	-	< 0.001 d
UPDRS total (scores)	53.43 ± 14.76	44.68 ± 11.21	-	0.012 e
UPDRS part III (scores)	34.92 ± 9.06	32.90 ± 6.85	-	0.335e
More affected limb (left/right)	20/10	27/3	All right-handed	< 0.001 a

The data are presented as the mean ± standard deviation, with significant differences between groups indicated in bold (p < 0.05); PD: Parkinson’s disease; BMI: Body mass index; MMSE: Mini-Mental State Examination; L-DOPA: Levodopa; NFOG-Q: New Freezing of Gait Questionnaire; UPDRS: Unified Parkinson’s Disease Rating Scale. Statistical analyses include ^a Fisher’s exact test, ^b One-way ANOVA, ^c Kruskal–Wallis H test, ^d Mann–Whitney U test, and ^e Independent samples t test.

Fig. 1.

Flowchart of the study participants. PD: Parkinson’s disease; NFOG-Q: New Freezing of Gait Questionnaire.

Test procedures

All people with PD underwent assessments in the “defined off” state, abstaining from antiparkinsonian medication for at least 12 h before measurements. After providing informed consent, the participants were evaluated using the UPDRS, modified Hoehn and Yahr scale, NFOG-Q, and MMSE (Table 1). Subsequently, they engaged in warm-up exercises and practiced three times before performing the 360° turning task. During this task, the more affected limb executed inner steps as participants turned around a cone at their preferred speed⁴⁴ (Fig. 2). The left and right limb features were labeled most affected side (MAS) and least affected side (LAS) depending on the neurologist’s diagnosis of PD symptom onset. MAS refers to the side that developed symptoms first^46–49.

Fig. 2.

Schematic illustrating the experimental setup and analysis phase for the 360° turning task.

The 3D motion capture system utilized six infrared cameras (Vicon MX-T10; Oxford Metrics, UK). A global coordinate system was defined with the positive X-axis to the right, positive Y-axis facing anteriorly, and Z-axis as the cross-product between the X- and Y-axes, with the positive Z-axis oriented superiorly (Fig. 2). Measurements, including height, body weight, shoulder offset, hand thickness, leg length, and elbow, wrist, knee, and ankle width, were collected bilaterally for estimating joint kinematics data⁴⁶. Using a modified version of the Helen Hayes marker set, 39 reflective markers in 14 mm spheres were placed according to the Plug-in Gait full-body model (Vicon Motion Systems Ltd., Oxford Metrics, UK)⁵⁰. As in our previous study⁴⁶, these markers were positioned at anatomical landmarks.

Data analysis

The 3D motion capture system recorded the raw data at a sampling rate of 100 Hz, which were subsequently processed using Nexus software (version 2.12.0, Vicon, UK) and MATLAB R2018b (MathWorks, Inc., Natick, MA). The data underwent filtration employing a fourth-order Butterworth low-pass filter with a 10 Hz cutoff frequency determined through frequency analysis. The initiation of the 360° turning analysis phase was defined as the point when the angle between the pelvic and mediolateral vectors exceeded 10°. The end of the rotation occurred when the two vectors completed a 360° turn, and the 350° turning phase was subsequently analyzed⁴⁶ (Fig. 2).

Data preprocessing

The participants completed the 360° turning task within an average 5–15 s duration. We performed the following preprocessing procedures. First, we set the trial length to 896 samples, and each participant performed three trials for the 360° turning task. Each trial corresponded to one sample for the training and testing of the classification. This length was determined by multiplying the pixel size of an input image by an even number factor of 4 (224 pixels × 4 = 896 samples, approximately 9 s). We used zero padding in trials lasting less than 8.96 s. Second, we normalized the data to eliminate the effect of differences in participants’ height and marker placement variations following reattachment. The normalization procedure involved centering the sensor data within each trial to achieve a median of 0 while retaining the original scale of the sensor data. Only the median value of the data was adjusted to align with the zero reference point. Third, we calculated the acceleration of each marker. The collected motion data were position-based. The position dataset was derived from a 3D motion capture system. This system placed 39 reflective markers on anatomical landmarks and included the center of mass (COM), resulting in 40 markers. The acceleration dataset in wearable applications is preferable because of its popularity and ease of access. The collected motion data were position-based, and to analyze the acceleration characteristics, we calculated the acceleration of each marker by performing the second numerical differentiation of the position data. This process involved computing the change in velocity over time for each marker’s 3D coordinates, providing acceleration values for each point in time.

Another additional step was to calculate the magnitude of the 40 markers in 3D coordinates to combine the information of each component of the X-, Y-, and Z-axes. The magnitude was calculated using the following formula: . In particular, the magnitude of the COM was calculated using the formula because we speculated that the z-component of the COM marker had no vital information for the classification. In data preprocessing, we applied no normalization to maintain intersubject variability, which may be an essential feature, for example, stride length. The entire network structure used in this study is presented in Fig. 3.

Fig. 3.

Workflow of the convolutional neural network approach based on time-series images. 3D: Three-dimensional; COM: Center of mass; PD: Parkinson’s disease; Cons: Control group; F: Freezers; NF: Nonfreezers. The co-authors have created the Fig. 3.

TS images and CNNs for classification

TS imaging methods

We expanded the perspective of the data from one-dimensional (1D) TS to 2D images to analyze the temporal aspect of the processed motion data using TS imaging methods: a recurrence plot (Rec)⁵¹ and Gramian Angular Field (GAF)⁵². These approaches offer two benefits: (i) they increase the dimension of the feature space, allowing for the inspection of motion data, and (ii) they exploit well-developed CNN-based DL architectures for classification. The following section provides further details on TS imaging methods.

Rec plot

Rec is a method used to convert a 1D TS to a 2D image by representing recurring states of the TS⁵¹. Given a TS, , with a length of , the trajectory at a discrete time point is:

where is the dimension of the trajectories and is the time delay. With these trajectories, the binarized recurrence matrix with being discrete time points are calculated as the pairwise distance between the trajectories, as shown below:

where is a norm operation, is the Heaviside function, and is the recurrence threshold. One limitation of this formation is that the calculated values are all discretized. The Heaviside function and recurrence threshold were excluded from the above equation to calculate the continuous matrix element of the recurrence matrix. Therefore, the recurrence matrix with continuous values is expressed as

We used the continuous recurrence matrix and set the dimension of the trajectories as 1 and the time delay as 1 to simplify the conversion of the TS motion data into 2D images.

GAF

GAF⁵² is also a TS data visualization method representing a TS as a 2D polar coordinate system. The general procedure of GAF with a given TS is as follows. First, min–max normalization was applied to TS .

Subsequently, the rescaled TS was projected into polar coordinates by turning the rescaled observations to the angle .

In the third step, we generated a GAF matrix using the calculated angle . There are two variations based on the type of trigonometric formula used to generate the GAF matrix. The Gramian Angular Summation Field (GASF), which uses the trigonometric sum with the cosine function to calculate the matrix, was chosen for data analysis. The following equation shows how the trigonometric sum was used to calculate the GASF matrix.

CNNs

To classify PD vs. controls and classify freezers vs. nonfreezers vs. controls using the processed TS images, three architectures were used: a residual neural network (ResNet)⁵³, a dense convolutional network (DenseNet)⁵⁴, and SqueezeNet⁵⁵. In this study, the input image size of the CNN model was set to 224 × 224 pixels. We used a ResNet model with 18 layers, a DenseNet-121 model consisting of four dense blocks with 6, 12, 24, and 16 channels, and the SqueezeNet v1.0 model.

ResNet

ResNet is a famous convolution-based deep neural network that classifies 2D images. It introduces a residual block, which creates a shortcut between the input and output layers to enable the training of deeper neural networks without vanishing or exploding gradient problems. This skip in the connection between layers helps propagate gradients through the neural network during training. The skip-in connection in ResNet is implemented by summing the outputs and the inputs .

where is an activation function with a weight parameter and a bias parameter similar to the output of a general neural network. We express the summation of the input as identity mapping. To perform identity mapping in the residual block, which means transferring the input to the output, there are two cases: (1) when the dimensions of the input and output layers are the same and (2) when they are different. The first case is solved easily using elementwise summation. The second case had two solutions, as proposed by He et al.⁵³. One is zero padding to match the input and output dimensions before elementwise summation. The other uses 1 × 1 convolutional layers for the projection from the input dimension to the output dimension.

DenseNet

DenseNet was proposed to improve performance using fewer parameters than ResNet⁵⁴. The critical concept of DenseNet is to use all previous feature maps for identity mapping. The authors focused on the possibility that the summation used in the residual block prevents information flow in the network. To solve this limitation, they introduced dense connectivity to concatenate all preceding feature maps for identity mapping. is the output of the th layer, and represents the input of the network. The concatenation operation is expressed as follows:

where is a composite function of batch normalization (BN), ReLU, and 3 × 3 convolution. receives the feature maps of all preceding layers, , and refers to concatenation.

In addition, the authors introduced two operations to reduce the number of feature maps: a bottleneck layer and compression. The bottleneck layers consist of BN-ReLU-1 × 1 convolution followed by BN-ReLU-3 × 3 convolution. The bottleneck layer contains a 1 × 1 convolution to reduce the number of parameters of the input feature maps. Compression is a way to reduce the dimensions of feature maps with a compression factor θ in a transition layer that acts as a downsampling layer. The transition layer consists of a 1 × 1 convolution layer and a 2 × 2 average pooling layer between the dense blocks.

DenseNet avoids vanishing gradient problems by connecting preceding feature maps, and it benefits from feature reuse because dense connectivity maintains information from the beginning to the end of the network. This advantage allows DenseNet to outperform ResNet using fewer parameters and fewer computational resources.

SqueezeNet

SqueezeNet aims to create a smaller neural network with fewer parameters while maintaining its performance, making it suitable for harsh computational environments⁵⁵. The authors stated that AlexNet had 240 MB of parameters, whereas SqueezeNet had only 4.8 MB. To achieve this, the authors propose three strategies:

Using 1 × 1 convolution instead of 3 × 3 convolution for fewer parameters.

Decreasing the number of input channels to reduce the parameter size.

Downsampling was performed in the latter part of the network to obtain large activation maps (maximizing performance with fewer parameters).

SqueezeNet includes a fire module comprising a squeeze layer (with only 1 × 1 convolutions) and an expand layer (with many 1 × 1 and 3 × 3 convolutions). For the second strategy, the authors set the number of hyperparameters in the squeeze layer to be less than that in the expand layer.

Statistical analysis

Data normality was assessed using the Shapiro–Wilk test. One-way analysis of variance and independent t tests or nonparametric statistics were used to analyze the means and standard deviations of all participants’ physical and clinical characteristics. All the statistical analyses were performed using IBM SPSS Statistics for Windows, version 22.0 (IBM Corp., Armonk, N.Y., USA). The significance level was set at p < 0.05.

Based on the CNN models and TS imaging methods, the classification analysis was performed as follows and is presented in Fig. 4.

Fig. 4.

Overview of sample sets and model evaluation procedures. PD: Parkinson’s disease; Cons: Control group; F: Freezers; NF: Nonfreezers; Acc: Accuracy; CNN: Convolutional neural network; s: Samples.

Five repeated random subsampling validation techniques were used to calculate the standard deviation, and t tests were used for statistical analysis.

For training and testing the CNN models, the dataset was split into training, validation, and test sets at percentages of 50%, 20%, and 30%, respectively.

For the two-class classification (PD vs. controls), the dataset initially consisted of 180 samples for PD and 90 samples for controls. Oversampling was performed to address data imbalance, resulting in 360 samples for the entire dataset. In the three-class classification (freezers vs. nonfreezers vs. controls), the dataset comprised 270 samples, with 90 samples per class. To perform an unbiased analysis, the imbalanced dataset of the control group was randomly oversampled using the imbalanced-learn (0.10.1) Python package.

To train the models, we set the hyperparameters as follows: a batch size of 32 or 500 epochs (with early stopping), utilization of the RMSProp optimizer with a learning rate of 1e-4, no momentum, and a smoothing constant of alpha = 0.99. To optimize the hyperparameters, the batch size (32) and epoch (500) were fixed, but the optimizers (Adam, SGD, and RMSprop) and learning rates (from 0.1 to 0.0001) were tuned for each CNN model. The models underwent a pretraining phase using the IMAGENET1K dataset and were subsequently fine-tuned using our dataset.

Results

Most relevant body segments used for classifying people with PD

We visualized 40 markers of body segments using TS images during the 360° turning task to identify the most prominent body segments for classifying the PD vs. controls (Fig. 5). Based on the position data, markers of interest were identified on the head, trunk, pelvis, and upper arm in the upper limb segment and on the femur, shin, and ankle in the lower limb segment. Furthermore, the neck, back, shoulder, wrist, and pelvis were identified in the upper limb segment, and the knee, shin, and heel were identified in the lower limb segment based on the acceleration data. The proposed algorithms, Rec and GASF, are based on CNNs.

Fig. 5.

Accuracy of convolutional neural network models for classifying people with PD and controls. The markers with the highest accuracy according to the TS images of position and acceleration data from 40 markers at each body segment (red); PD: People with Parkinson’s disease; Cons: Control group; TS: Time series. The co-authors have created the Fig. 5.

To classify PD vs. controls, markers identified using Rec on the TS image position dataset (Pos_Rec) were left on the front of the head, sternum, right lower third of the lateral shank_LAS (RTIB), and right lateral malleolus_LAS (RANK).

Markers identified using GASF on the TS image position dataset (Pos_GASF) were the left lower third of the upper arm_MAS, right third metacarpal head_LAS, right posterior superior iliac spine (RPSI), and right lower third of the lateral thigh_LAS.

The markers identified using Rec on the TS image acceleration dataset (Acc_Rec) were the seventh cervical vertebra (C7), right scapular medial border_LAS (RBAK), right shoulder_LAS (RSHO), right lateral femoral epicondyle_LAS (RKNE), and left lower third of the lateral shank_MAS (LTIB). Markers identified using GASF on the TS image acceleration dataset (ACC_GASF) were the left mediolateral styloid processes of the wrist_MAS (LWRA), right anterior superior iliac spine (RASI), and right calcaneus_LAS (RHEE).

The results were calculated from the test set, which was 30% of the entire dataset (50% for training, 20% for validating the trained models, and 30% for calculating the metrics for the results). Furthermore, the standard deviation of the results was calculated using the five repeated random subsampling validation techniques (the same for classifying “freezers”).

The confusion matrices of RTIB and RHEE in the classifications between PD and controls are presented in Fig. 6. Two markers were relevant to the identified body segments using the SqueezeNet model based on Pos_Rec and Acc_GASF with the highest accuracy. The SqueezeNet model achieved a sensitivity of 0.77 and a specificity of 0.76 for Pos_Rec and a sensitivity of 0.79 and a specificity of 0.78 for Acc_GASF.

Fig. 6.

Confusion matrices for the most accurate markers for distinguishing between people with PD and controls: freezers, nonfreezers, and controls. The number of samples represents the five repeated random subsampling validation techniques, with 54 and 27 samples, accounting for 20% of the test dataset. Pos: Position data; Acc: Acceleration data; Rec: Recurrence plot; Cons: Control group; PD: People with Parkinson’s disease; F: Freezers; NF: Nonfreezers.

Most relevant body segments used for classifying freezers

We visualized 40 body segment markers to classify freezers, nonfreezers, and controls into three groups (Fig. 7). Based on the position data, the body segment markers of interest were identified on the head, trunk, elbow, lower arm, wrist, and pelvis in the upper limb segment and the knee, shin, and heel in the lower limb segment. Furthermore, the head, trunk, back, elbow, wrist, finger, and pelvis were identified in the upper limb segment, and the knee, shin, ankle, and toe were identified in the lower limb segment based on acceleration data.

Fig. 7.

Accuracy of convolutional neural network models for classification between freezers, nonfreezers, and controls. The markers with the highest accuracy according to the TS images of position and acceleration data from 40 markers at each body segment (red) are referred to as follows: F: Freezers; NF: Nonfreezers; Cons: Control group; TS: Time series. The co-authors have created the Fig. 7.

For the classification of freezers vs. nonfreezers vs. controls, the markers identified using Pos_Rec were the left bilateral on the back of the head, sternum, left lateral humeral epicondyle_MAS (LELB), right lower third of the forearm_LAS (RFRM), right mediolateral styloid processes of the wrist_LAS (RWRA), left lateral femoral epicondyle_MAS (LKNE), RTIB, and RHEE. The marker identified using Pos_GASF was RPSI. The markers identified using Acc_Rec were the right bilateral on the back of the head, sternum, COM, right lateral humeral epicondyle_LAS, RKNE, RTIB, and left second metatarsal head_MAS. The markers identified using Acc_GASF were the tenth thoracic vertebra (T10), left third metacarpal head_MAS, LWRA, left posterior superior iliac spine (LPSI), and left lateral malleolus_MAS (LANK).

The confusion matrices of the LELB and LPSI for the freezer, nonfreezer, and control classifications are presented in Fig. 6. The two markers with the highest accuracy were related to the identified body segments using a SqueezeNet model based on Pos_Acc_Rec. The SqueezeNet model achieved sensitivity values of 0.54 for controls, 0.70 for freezers, and 0.42 for nonfreezers and specificity values of 0.84 for controls, 0.78 for freezers, and 0.79 for nonfreezers.

Discussion

This study used the DL algorithm CNN based on TS images to identify body segments that can be used to classify people with PD and freezers during a 360° turning task. The CNN model demonstrated moderate performance in identifying classifiable body segments using position and acceleration TS images of the 360° turning phase. The main findings of this study can be summarized as follows: (1) Based on the position and acceleration data converted into TS images, 40 markers for whole-body segments were identified using a CNN. These TS images of body segments classified PD vs. controls into two groups and freezers vs. nonfreezers vs. controls into three groups. (2) The most relevant body segment markers for classifying PD vs. controls included regions such as the head, sternum, and lower limbs (RTIB, RANK). For the classification of freezers vs. nonfreezers vs. controls, markers on the head, trunk (sternum), and limbs (LELB, RFRM, RWRA, LKNE, RTIB, RHEE) were significant. These markers are associated with key regions involved in postural control and balance, which are often compromised in PD^8,9. (3) Acceleration data revealed relevant markers on C7, T10, RBAK, RSHO, RKNE, and LTIB. These regions are crucial for dynamic movements and rapid adjustments, highlighting their importance in tasks requiring quick changes in direction, such as turning⁵. (4) Body segment identification using a position and acceleration TS image-based CNN algorithm during the 360° turning task could be a practical approach for objectively evaluating and classifying people with PD and FOG.

Body segments for classifying people with PD vs. controls

We identified body segments for classifying PD vs. controls based on the positions of 40 markers and acceleration TS images during the 360° turning phase. Our results revealed that head, trunk, and pelvis markers identified through position TS images could be linked to trunk and upper limb coordination ability, which is affected by the rigidity of the central body axis during turning^56,57. Compared to controls, people with PD have limited sequential head, shoulder, trunk, and pelvic movements during turning. Limited coordination could increase the risk of falls during turning, suggesting that increased postural instability in advanced disease may lead to considerable turning disturbances^58,59. Additionally, people with PD have lower activity in the cerebrum’s supplementary motor areas, affecting bilateral postural control and gait function, leading to dynamic instability when turning via the inner step of the MAS^60,61. This result indicated that people with PD encountered challenges when turning toward the more affected limb^48–51. Therefore, we suggest classifying people with PD and controls using the turning patterns of the head, trunk, arm, and pelvic segments related to upper limb coordination.

Our results identified wrist and heel markers as body segments from acceleration TS images of the distal body. These markers may indicate various motor symptoms of PD, such as tremors, bradykinesia, and FOG, especially during turning movements. They may also reveal asymmetries between the more and less affected sides of the body, typical in people with PD⁶². The acceleration patterns of these body segments showed greater variability in people with PD compared to controls, particularly in the distal body segments. These findings provide insights into the deterioration of distal segment motor control due to neurological damage, including the supplementary motor area receiving input from the damaged basal ganglia in people with PD⁶³.

Additionally, the progression of PD is associated with asymmetrical upper and lower limb movements⁶⁴. Previous studies have shown that people with early to moderate PD exhibit lower symmetry in their upper and lower limbs than healthy controls when walking at their preferred speed on a straight path. This was determined using three-axis inertial sensors attached to the hand and shank⁶³. Our findings demonstrated that the trunk (C7, RBAK, and RASI), arm (RSHO and LWRA), and lower limbs (RKNE, LTIB, and RHEE) were important body segments identified using acceleration TS images during turning. These findings suggest that wearable sensors could be valuable for assessing trunk and upper limb coordination and predicting falls due to postural instability and imbalances between the upper and lower limbs in people with PD during turning tasks^56,57.

In our CNN approach, the RTIB and RHEE markers emerged as the most important body segments in the position and acceleration TS images for classifying people with PD and controls, respectively. The Acc_GASF model achieved the highest accuracy for the RHEE marker in this study (78.3%). During turning, movement in lower limb segments, such as the shin and heel, is a sign of falls in early PD²³, and a decrease in turning performance from acceleration signals can be observed when evaluating PD severity¹⁶. Turning is an asymmetric task requiring the simultaneous creation of a stepping pattern between limbs on both sides, weight transfer, and support⁶¹. However, people with PD and deteriorated dynamic stability require greater bilateral coordination ability than controls. They perform careful movements, take short and slow steps and use a wider turning radius as a compensation strategy to prevent falls during turning⁶¹. Therefore, understanding the impairment characteristics of distal body segment movements during turning tasks is important for predicting and classifying potential falls in early PD patients^23,66.

Body segments for classifying freezers vs. nonfreezers vs. controls

We use a CNN model to identify body segments using position and acceleration TS images to classify freezers, nonfreezers, and controls. This model identified body segments associated with gait patterns that may accompany the characteristic movements of FOG during turning, such as short steps, ataxia, bradykinesia, and tremors. Our findings suggest that FOG is not limited to the lower limb and involves movements in the upper limb, further supporting the idea that FOG evaluation requires whole-body kinematic analysis²⁵.

Our results revealed that markers of the arms and legs, including both the MAS and LAS, may detect dynamic instability that can increase the risk of falls in freezers during 360° turning⁶⁰. In particular, arm (LELB) and leg (LKNE, RTIB, and RHEE) markers of the more and less affected sides may be associated with ipsilateral and contralateral temporal coordination, which could identify asymmetric characteristics of the upper and lower limbs during turning⁶⁷. Our previous findings support these results, where a delay in temporal coordination between the upper and lower limbs during turning was associated with lower interlimb coordination ability in freezers⁴⁸. These results also showed that markers of the trunk (sternum) and pelvis (RPSI) identified en-bloc turning characteristics while turning in freezers²⁶. En-bloc turning refers to a pattern lacking segmental movement dissociation typically observed in people with PD during turning tasks²⁴. This finding highlights the significance of these specific body segments in understanding the motor abnormalities associated with FOG in PD patients²². We identified markers attached to the trunk (sternum and COM), back (T10), and pelvis (LPSI) from acceleration TS images. The turning characteristics of these freezers may be caused by delayed reaction times, abnormal automatic postural responses, and trunk axial movements compared with those of nonfreezers and controls⁶⁸. Therefore, turning may threaten the stability of freezers and increase the risk of falls because it requires more precise and complex postural control than straight walking⁶⁰.

A CNN model based on position and acceleration TS images was proposed to identify body segments that can identify and classify gait disturbances while turning in people with PD and freezers. Despite extensive research on human activity recognition and fall or FOG detection, the use of TS patterns in the turning phase to improve disease severity and classification remains under-researched^27,69. Previous studies have predicted objective FOG and tremor severity scales using a single waist-worn triaxial accelerometer⁷⁰, wrist inertial measurement unit sensor⁷¹, and whole-body marker set²⁶ in a home or laboratory setting. They suggested that these monitoring systems could also be used in ambulatory or daily life to monitor FOG episodes or classify their severity⁷⁰. Our results revealed that the trunk, arm, and lower limbs were important body segments identified using acceleration TS images during turning, which could be used to classify freezers vs. nonfreezers vs. controls. These segments are consistent with those identified in previous studies focusing on people with PD, encompassing freezers and nonfreezers^16,64. This suggests that the body segments identified in our study represent those found in broader PD populations¹⁶. Our approach may be used for clinical evaluation and classification of people with PD and freezers to detect FOG and observe signs of disease progression. This may be achieved through wearable acceleration sensors attached to the trunk, wrist, waist, and ankle to capture time-series images in daily life^71,72.

Our results also showed that the position and acceleration TS images of the LELB and LPSI markers were the most important body segments for classifying freezers, nonfreezers, and controls. The LPSI marker showed the highest accuracy in this study (50.4%) in the Acc_GASF model. For the context of 3-class classification, random predictions result in an average accuracy of 33.33%. Investigating the gait patterns of LELB and LPSI markers provides insights into motor symptoms, particularly FOG, during turning in people with PD⁴⁹. The LELB marker pattern could indicate the patient’s ability to control the upper limb, influencing postural stability due to deteriorated upper limb movement for the MAS during turning⁵⁰. This could also offer information related to symptoms associated with FOG, such as short steps and irregular movements^49,50. Similarly, the movement of the LPSI marker reveals information regarding the patient’s posture and balance²³. Pelvic or waist movements during turning may be associated with symptoms such as FOG and postural instability⁷³. Therefore, a comprehensive evaluation of the turning gait patterns of the two markers may contribute to a more nuanced understanding of FOG, postural instability, and disease progression in people with PD⁴⁹.

This study has a few limitations. First, the effects of the “On” and “Off” medication states were not compared during the 360° turning tasks. Longitudinal studies are needed to assess various clinical characteristics in the medication “On” and “Off” states and to generalize the discrimination of disease severity using turning characteristics.

Second, one participant experienced an FOG episode while turning toward the inner step of their more affected limbs. However, these patients were excluded from the analysis. The classification algorithm demonstrated the ability to learn and model freezer turning patterns. Notably, there were no occurrences of freezing during the turning tasks, indicating successful differentiation between freezers, nonfreezers, and controls.

Third, this study analyzed TS data of the magnitude, , of 3D motions from 40 markers. The magnitude was calculated using the x-, y-, and z-components to simplify the analysis. Subsequently, a magnitude series representing a trial of the 360° turning task was transformed into a grayscale image. However, each component in the x-y-z coordinates may contain independent information for discriminating between PD, freezers and nonfreezers, and controls. Generating an RGB image instead of a gray image could help to reveal additional characteristics of the x-y-z components. The R-channel represents an image in the X-axis component of a 360° turning task trial, G is generated from the Y-axis, and B represents an image in the Z-axis component. Moreover, a larger checkerboard-like image that includes images generated from different markers and converted using different TS imaging methods may solve the problems described in this study. The performance of the informative checkerboard-like image is also an interesting question that was not scrutinized in this analysis.

Fourth, the acceleration data were calculated through numerical differentiation of the position data because the original dataset did not include acceleration information. Therefore, the acceleration results might differ from those of the accelerometer sensors. Nevertheless, we speculate that our acceleration results may explain the trends observed in accelerations measured by accelerometer sensors.

Finally, this study used three CNN models, ResNet, DenseNet, and SqueezeNet, to compare the performance of classifying TS images of 3D motion data of people with PD. However, many DL models in the literature can perform image classification. Therefore, future research should investigate advanced models and propose methods to improve performance.

In conclusion, our findings highlight potential methods for identifying body segments associated with motor symptoms during the 360° turning task in people with PD and FOG. Time-series images derived from these identified body segments may serve as a supplementary screening tool for assessing people with PD, including the FOG pattern. We anticipate that future research will pave the way for developing clinical applications involving assessment, diagnosis, and ongoing monitoring by integrating a broader range of time-series images obtained from wearable sensors placed on body segments directly correlated with PD severity.

Author contributions

HP, SS, CY, and SC conceived and designed the study. HP and SC recruited the participants. HP, SS, CY, and SC performed the data acquisition. HP, SS, and CY analyzed and interpreted the data. HP, SS, CY, and SC drafted the article. All authors read and approved the final version of the manuscript submitted.

Funding

This work was supported by grants from the National Research Foundation of Korea (NRF), funded by the Korean government (MSIT) (No. 2022R1A2C100933711; Changhong Youm). This research was also supported by the Basic Science Research Program through the NRF, funded by the Ministry of Education (No. 2022R1A6A3A0108756411; Hwayoung Park). This research received no specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The funding sources had no role in the study design; collection, analysis, or interpretation of the data; or writing of the manuscript.

Data availability

The datasets supporting this study’s findings are available from the corresponding author upon reasonable request.

Code availability

We do not have an available open-source code. The code for training and testing the deep learning models was written in Python 3.8 using PyTorch 1.9.1 and torchvision 0.10.1. The data management and feature processing scripts were written in Python 3.8 using pandas 1.3.3 and NumPy 1.21.2. The code used for the analysis may be requested by contacting the corresponding author.

Declarations

Competing interests

The authors declare no competing interests.