Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Jan 1.
Published in final edited form as: Am J Phys Med Rehabil. 2014 Jan;93(1):17–28. doi: 10.1097/PHM.0000000000000011

Peripheral Nerve Stimulation Compared to Usual Care for Pain Relief of Hemiplegic Shoulder Pain: A Randomized Controlled Trial

Richard D Wilson 1,2, Douglas D Gunzler 3, Maria E Bennett 4, John Chae 5,6,7
PMCID: PMC4038163  NIHMSID: NIHMS579755  PMID: 24355994

Abstract

Objective

This study seeks to establish the efficacy of single-lead, 3-week peripheral nerve stimulation (PNS) therapy for pain reduction in stroke survivors with chronic hemiplegic shoulder pain.

Design

Single-site, pilot, randomized controlled trial for adults with chronic shoulder pain after stroke. Participants were randomized to receive a 3-week treatment of single-lead PNS or usual care (UC). The primary outcome was the worst pain in the last week (Brief Pain Inventory, Short Form question 3) measured at baseline, and weeks 1,4, 12, and 16. Secondary outcomes included pain interference (Brief Pain Inventory, Short Form question 9), pain measured by the ShoulderQ Visual Graphic Rating Scales; and health-related quality of life (SF-36v2).

Results

Twenty-five participants were recruited, 13 to PNS and 12 to UC. There was a significantly greater reduction in pain for the PNS group compared to controls, with significant differences at 6 and 12 weeks after treatment. Both PNS and UC were associated with significant improvements in pain interference and physical health related quality of life.

Conclusions

Short-term PNS is a safe and efficacious treatment for shoulder pain. Pain reduction is greater than compared to UC and is maintained for at least 12 weeks after treatment.

Keywords: Electric Stimulation Therapy, Shoulder Pain, Peripheral Nerve Stimulation, Hemiplegia, CVA, Stroke

INTRODUCTION

Hemiplegic shoulder pain (HSP) is a common complication after stroke, affecting up to 60% of moderately to severely impaired stroke survivors.14 Many pathologies have been found in those with shoulder pain after stroke. No single etiology has been found as a cause for HSP, though it has been associated with a greater severity of motor impairment.5 Pain tends to occur with movement of the arm, particularly overhead activities, leading to poor rehabilitation outcomes, interference with daily activities6, and poor quality of life (QoL).7 Many who are diagnosed and treated promptly have improvement of symptoms, though a significant number do not respond to standard therapies and experience chronic pain. An estimated 20–30% of stroke survivors with moderate to severe stroke have shoulder pain at 4 years1, 2, an indication that current therapies are not always effective in treating chronic post stroke pain and its debilitating consequences.

Peripheral nerve stimulation (PNS) has been shown to be efficacious in treating chronic HSP. A multi-site randomized controlled trial (RCT) evaluated a four-lead approach in which the peripheral nerves innervating the trapezius, supraspinatus, middle deltoid, and posterior deltoid were stimulated through percutaneous electrodes for six-hours per day for six weeks.3, 4 Peripheral nerve stimulation was more efficacious than a humeral cuff-sling in reducing HSP, and the pain reduction was maintained for at least 12 months after end of treatment. Due to the technical difficulty and the pain associated with the 4-lead implantation procedure, and results that revealed maximum pain-relief prior to the end of stimulation, the procedure was modified to a single-lead approach with stimulation for six hours per day for a total of three weeks.8 In a case-series, treatment of chronic HSP with the revised approach was associated with a 63% pain reduction at 12 weeks after the end of stimulation, similar to results seen in the four-lead, six-week approach.9 However, the efficacy of the single-lead, three-week PNS treatment for chronic HSP has not been evaluated in a RCT.

The objective of this pilot RCT was to establish the initial efficacy of short-term single-lead PNS for the treatment of chronic HSP. We tested the hypotheses that compared to usual care, PNS would reduce pain; reduce pain interference with daily activities; and improve health-related quality of life (HRQoL).

METHODS

Participants

This was a single center, assessor-blinded RCT of PNS compared to usual care (UC) for HSP. Inclusion criteria were: ≥ 21 years old; ≥ 3 months after stroke with new or worsened shoulder pain on their affected side; HSP rated ≥ 4 out of 10 on the 11-point numeric rating scale (NRS) of the Brief Pain inventory Short Form, question 3 (BPI-SF3)10, 11; duration of HSP ≥ 3 months; and shoulder abduction weakness ≤ 4 (Medical Research Council Scale12). Exclusion criteria were: evidence of joint or overlying skin infection or history of recurrent skin infections; insensate skin; ≥1 opioid or nonopioid analgesic daily for shoulder pain; daily intake of pain medications for any other chronic pain; intra-articular or subacromial steroid injections to the shoulder in the previous 3 months; botulinum toxin injection to the trapezius, pectoralis or subscapularis muscle in the previous 3 months; currently receiving physical (PT) or occupational (OT) therapies for HSP; bleeding disorder or INR > 3.0 for those on warfarin; medical instability; pregnancy; uncontrolled seizures (>1 per month for the last 6 months); uncompensated hemi-neglect; severely impaired communication or cognition; moderate to severe depression (Beck Depression Inventory-Fast Screen13 13 or above); other confounding neurological conditions affecting the upper limb; other medical issues such as complex regional pain syndrome, bicipital tendonitis, myofacial pain syndrome; history of tachyarrhythmia with hemodynamic instability; any implantable stimulator such as demand pacemakers or defibrillators; or valvular heart disease including artificial valves. At the beginning of the trial, HbA1c>7.0 was an exclusion criteria to minimize the risk of infection among diabetics; however, this was subsequently removed due to lack of scientific support.

The study was conducted at an urban, academic rehabilitation center in the United States. The use of patient data for research purposes was approved by the committee on research ethics at the institution in which the research was conducted in accordance with the Declaration of the World Medical Association and informed consent from human subjects was obtained as required. The trial was registered at clinicaltrials.gov. Participants were randomized to PNS or UC by an adaptive randomization algorithm14 to maximize the chances of an even distribution of the following three characteristics across groups: time from stroke onset (≤ 18 months vs. >18 months)6, baseline subluxation (absence vs. presence)15, 16, and sensation (normal vs. abnormal)17. For each assignment, the algorithm calculated the balance of these three variables across the two groups and made an assignment to improve balance across groups. The study coordinator, who was not blinded, entered the variables, and the program output the group assignment.

Interventions

Those randomized to PNS received a single percutaneous electrode. The implantation procedure was previously described.9 Briefly, after identifying the target implantation site and depth on the deltoid muscle18 via monopolar needle stimulation, an insulated introducer loaded with a fine-wire percutaneous lead was inserted to the appropriate site and depth. Strong contraction of the middle and posterior deltoid muscles verified proper positioning. Pressure was maintained at the skin surface to anchor the lead’s barb in the belly of the muscle while the introducer was withdrawn leaving the lead in place. The lead was stimulated again to ensure proper placement.

After one week for electrode stabilization, an external stimulator (Rehabilicare ® NT2000, ® Empi, Inc., St. Paul, MN) was connected to the lead and parameters were set to stimulate at 12 Hz and 20 mA. Pulse duration (40–200µs) was adjusted to produce the strongest muscle contraction without discomfort. Participants were prescribed 6 hours of stimulation per day for 3 weeks, to be completed in single or divided doses, for a total of 126 hours of stimulation. The device has a compliance monitor that records the total time of stimulator usage. The stimulator completed a cycle every 30 seconds consisting of 5 seconds to ramp up, 10 seconds at maximum stimulation, 5 seconds to ramp down, and 10 seconds without stimulation. At the conclusion of the 3-week stimulation period, the lead was removed by gently pulling on its exposed end. All participants underwent anterior-posterior and scapular-Y view radiographs of the shoulder for surveillance for retained lead fragments.

Participants randomized to the UC treatment received 8 hrs of outpatient PT over a 4-week period from a licensed therapist, coupled with prescribed daily home exercises. Physical therapy has been identified as the most commonly prescribed first choice of treatment for HSP19 and the treatment program was based on published best practice guidelines and recommendations.2022 The therapist implemented an individualized treatment plan consistent with the needs of the participant and the following treatment principles: (1) proper positioning and handling, and the use of slings and supports to reduce the risk of trauma to the hemiparetic upper limb; (2) range of motion (ROM) and strengthening exercises within pain-free range and loads, respectively; (3) task-specific therapy for participants with residual hand function to reduce impairments and improve basic and instrumental activities of daily living (ADLs); (4) home exercise program on days participants do not receive PT. Participants were encouraged to use their hemiparetic upper limb for functional tasks as much as safely possible. Participants were asked to document their home program in a diary.

All participants underwent 5 blinded assessments. In order to facilitate assessor blinding during the treatment phase, UC participants wore a bandage covering a simulated electrode exit site. Outcome assessments were completed before randomization, at the beginning and end of the4-week treatment period, and at 6 and 12 weeks after treatment.

The primary outcome measure was the BPI-SF3. The BPI is a pain questionnaire which assesses both pain intensity (sensory dimension) and the interference (reactive dimension) of pain in daily activities. The BPI-SF3 asks participants to rate their worst shoulder pain in the last week on a 0 to 10 NRS, where “0” indicates “No pain” and “10” indicates “Pain as bad as you can imagine.” Secondary outcomes included the BPI-SF question 9 (BPI-SF9), the visual graphic rating scales (VGRS) of the ShoulderQ and the Short Form 36 version 2 (SF-36v2). The BPI-SF9 assesses the degree to which pain interferes with general activity, mood, walking ability, normal work, interpersonal relationships, sleep and enjoyment of life on a 0–10 NRS, where “0” indicates no interference and “10” indicates complete interference. The VGRS of the ShoulderQ is a structured questionnaire designed to assess severity of HSP at rest during the day, on movement, and at night on a 0–30 scale where higher numbers indicate greater pain 23. The SF-36v2 is a population-norm based HRQoL measure, presented in T-scores where population average equals a score of 50 with a standard deviation of 10.24

Concomitant Therapies

Many stroke survivors have multiple rehabilitation problems requiring additional rehabilitation therapy interventions; thus, due to ethical considerations, concomitant therapies, including PT, OT and pain medications were permitted and monitored during the entire study. However, in order to maintain internal validity to the extent that was possible, the study imposed several restrictions: 1) no PT or OT directed at the shoulder or experimental procedures involving the hemiparetic upper limb; 2) no intra-articular or subacromial corticosteroid injections to the affected shoulder; 3) may receive oral spasticity medications, but no neurolytic agents to shoulder adductors or internal rotators; 4) no change in dosing of analgesic or spasticity medications; and 5) no addition of analgesic or spasticity medications. Due to ethical concerns these restrictions could not be strictly enforced. Participants were queried weekly by telephone for concomitant therapies, including analgesic medications used in the prior week.

Sample Size

To detect a minimum clinically important difference of 2 points25 on the BPI-SF3 with an anticipated standard deviation for each mean of 2.5, estimated from a prior study4, with an alpha level of 0.05 and a power of 80%, for five waves of data, a sample size of 10 participants per group was necessary. With anticipated dropouts, a sample size of at least 12 participants per group was required.

Statistical Analysis

The effect of treatment group over time was analyzed using a linear mixed model for repeated measures for each outcome measure. This model is an extension of linear regression for longitudinal data, and is specifically designed to handle correlated repeated measurements, missing data and dropouts. In our model, we included a random intercept for each individual participant. We used a first-order antedependent covariance structure, since it is reasonable to assume that for each individual there is a greater correlation between assessments that are closer together and that variance might be different at different assessments. The model assessed whether the outcomes: 1) change in different ways over time between groups (group by time interaction term); 2) change over time (continuous time effect); or, 3) are different between groups under the assumption that the mean response profiles are parallel (group main effect). We also performed the same analyses with five discrete time points (baseline, start of treatment, end of treatment, and 6 weeks and 12 weeks post-treatment), since treatment is limited to the beginning of the study and there is interest in estimating differences between groups at different assessments. On one domain of the SF36, General Health, this discrete time model failed to converge with the covariance structure. Therefore, in this model we made no assumptions about the variances and covariances, allowing for the differences in variability of the measurements at each time point using an unstructured covariance structure. In all models, missing data were handled by a maximum likelihood algorithm under the assumption that the missingness is dependent on the data at hand (“missing at random (MAR)” assumption) and the analyses were conducted by the available-case, intention-to-treat method. When a significant treatment group by time interaction effect was found, pairwise comparisons between groups at the discrete time points of end of treatment (week 4) and follow-up timepoints (weeks 10 and 16) of the least squares means were computed from the model with discrete time points with Bonferroni correction (alpha=0.017) to control the familywise error rate. A per-protocol analysis limited to complete-cases was also conducted for the primary outcome. A secondary analysis was completed for global success of treatment of pain defined as ≥ 30% reduction in BPI-SF3 at end of treatment (week 4) that is maintained at weeks 10 and 16. An intention-to-treat analysis was completed with the available cases by multiple imputation of missing global success outcomes for those participants who did not complete all outcomes assessments. The multiple imputation was completed with a Markov chain Monte Carlo, single-chain method of 10 imputations. Each imputed dataset was analyzed by a generalized linear, log-binomial model that allows estimation of relative probability26 of successful outcome for each group. The dependent variable of global success and independent variable of treatment group were included in the model. Parameter estimates and corresponding covariance matrices for each imputed data set were then used to derive a valid univariate inference.27 The per-protocol analysis between groups was completed with a Fisher’s Exact test.

Differences in total cumulative hours of formal PT and OT were analyzed by Wilcoxon Rank sum test.

We define α = 0.05 for our level of significance in all statistical tests (before performing any Bonferroni correction, where appropriate). All statistical tests are two-tailed.

RESULTS

Trial Profile

Of the 88 stroke survivors screened, 35 (39.7%) met inclusion criteria for enrollment (see Figure 1). The most common reasons for exclusion were HbA1c greater than 7.0 (30.0%, this exclusion was changed during the study), the presence of heart valve disease (20.0%), and regular intake of analgesics for other pain syndromes (20.0%). Twenty-five participants were enrolled in the study with 13 randomized to PNS and 12 to UC. One participant in each group withdrew from the study prior to receiving either treatment due to medical illness and had no outcome assessments recorded after the first assessment. Two participants in the UC group were lost to follow-up after the second outcome assessment, and one participant in the UC group missed the outcome assessment at week 4 due to hospitalization. One participant in the PNS group completed the primary outcome (BPI-SF3) and one secondary outcome (BPI-SF9) by telephone because a face-to-face assessment was not possible within the specified window of the 16-week outcome assessment. Overall, five participants missed at least one follow-up assessment. Thus, 21 participants (84.0%) had complete data for the primary outcome. The data were missing in PNS group in 5.3% of outcomes, compared to 18.3% for the UC group. The primary analysis was intention-to-treat and involved all participants who were randomly assigned. The PNS group completed an average of 114 hours of a possible 126 hours (90.5%) of stimulation treatment. The baseline demographic and clinical data are presented in Table 1.

Figure 1.

Figure 1

Participant flow diagram.

Table 1.

Demographics

PNS Usual Care
n=13 n=12
Female (%) 46.2 58.3
Age (years, median +/− IQ range) 54.0 (50.0 – 68.0) 55.5 (50.0 – 62.5)
Subluxation (%) 61.5 66.7
Race/Ethnicity (%)
White 46.1 41.7
African American 53.9 50.0
Hispanic/Latino 0 8.3
Shoulder Pain > 18 mo. (%) 61.5 58.3
Time Since Stroke (years, median +/− IQ range) 2.6 (0.9 – 4.0) 2.3 (0.8 – 4.8)
Cortical Stroke (%) 66.7 54.5
Right Hemiplegia (%) 38.5 33.3
Hemorrhagic Stroke (%) 38.5 8.3
Comorbidities at Baseline (%)
Coronary Artery Disease 30.8 0
Congestive Heart Failure 0 0
Cardiac Arrhythmia 7.7 0
Diabetes Mellitus 38.5 41.7
Hypertension 76.9 100
Renal Dialysis 0 0
Pulmonary Disease 7.7 16.7
Peripheral Vascular Disease 0 8.3
Seizure Disorder 7.7 0
Osteoarthritis 23.1 0
Cancer 0 8.3
Analgesic Use at Baseline (%)
Narcotic 7.7 8.3
Non-Narcotic 23.1 25.0
Aspirin 54.0 41.7
NSAID 7.7 16.6
Anti-Epileptic 30.8 50.0

Demographics for PNS and Usual Care groups with chronic hemiplegic shoulder pain.

Abbreviations: PNS-peripheral nerve stimulation; IQ-interquartile; NSAID-Non-steroidal anti-inflammatory drug

Primary Outcome

The pain reduction with treatment for PNS group was significantly greater than the UC group (time by group interaction effect), although both groups experienced a significant pain reduction with treatment (time effect). We report least squares mean estimates (± sd) by group over time in Table 2 and hypothesis tests of model fixed effects results in Table 3. The mean severity rating at baseline was 7.5 (+/− 0.7) and 7.6 (+/− 0.7) for PNS and UC, respectively, which dropped to a 3.2 (+/− 0.7) and 6.1 (+/− 0.8), respectively, at 10 weeks, and remained a 3.0 (+/− 0.7) and 6.1 (+/− 0.8), respectively, at 16 weeks (Figure 2A). Pairwise comparisons revealed significant differences between groups of 2.9 (95% CI 0.8 – 5.0) at 10 weeks and of 3.1 (95% CI 1.0 – 5.2) at 16 weeks.

Table 2.

Outcomes Assessments of PNS vs. Usual Care for Hemiplegic Shoulder Pain

Week 0 Week 1 Week 4 Week 10 Week 16
Worst pain 7d (+/−SE)
PNS (n=13) 7.5 (+/−0.7) 6.1(+/−0.7) 2.6(+/−0.7) 3.2(+/−0.7) 3.0(+/−0.7)
Usual Care (=12) 7.6 (+/−0.7) 6.8(+/−0.7) 5.0(+/−0.8) 6.1(+/−0.8) 6.1(+/−0.8)
Pain Interference 7d (+/−SE)
PNS (n=13) 3.6 (+/−0.7) 2.3 (+/−0.7) 0.8 (+/−0.7) 0.8 (+/− 0.7) 1.1 (+/−0.7)
Usual Care (=12) 5.0 (+/−0.7) 4.6 (+/−0.7) 2.1 (+/−0.8) 3.0 (+/−0.8) 3.5 (+/−0.8)
ShoulderQ VGRS‡ (+/−SE)
PNS (n=13) 16.5 (+/−1.7) 14.8 (+/−1.8) 6.3 (+/−1.8) 6.7 (+/−2.1) 7.6 (+/−2.4)
Usual Care (=12) 16.5 (+/−1.8) 15.6 (+/−1.8) 11.0 (+/−2.0) 10.5 (+/−2.2) 13.1 (+/−2.6)
SF-36v2 (+/−SE)
Physical Component Summary
PNS (n=13) 28.0 (+/−2.7) 29.1 (+/−2.8) 33.0 (+/−2.8) 33.5 (+/−3.0) 34.1 (+/−3.2)
Usual Care (=12) 27.6 (+/−2.8) 28..9 (+/−2.9) 34.1 (+/−3.1) 33.2 (+/−3.2) 33.8 (+/−3.5)
Mental Component Summary
PNS (n=13) 58.1 (+/−4.0) 58.0 (+/−4.0) 57.6 (+/−4.0) 55.3 (+/−4.0) 58.6 (+/−4.3)
Usual Care (=12) 47.1 (+/−4.2) 45.8 (+/−4.2) 49.5 (+/−4.3) 47.3 (+/−4.4) 52.3 (+/−4.9)
Physical Functioning
PNS (n=13) 30.2 (+/−3.4) 30.6 (+/−3.5) 32.0 (+/−3.5) 31.5 (3.6) 30.1 (+/−3.8)
Usual Care (=12) 28.2 (+/−3.6) 28.1 (+/−3.6) 32.0 (+/−3.8) 30.4 (+/−3.9) 31.5 (+/−4.3)
Role-Limitations Physical
PNS (n=13) 30.5 (+/−2.9) 31.4 (3.0) 31.8 (+/−3.2) 33.0 (+/−3.5) 36.4 (+/−3.8)
Usual Care (=12) 25.0 (+/−3.0) 24.8 (3.2) 31.0 (3.6) 31.4 (+/−3.8) 33.4 (+/−4.3)
Bodily Pain
PNS (n=13) 36.1 (+/−1.9) 41.2 (+/−2.0) 47.9 (+/−2.2) 48.3 (+/−2.7) 49.3 (+/−3.3)
Usual Care (=12) 36.4 (+/−2.0) 38.3 (+/−2.1) 44.9 (+/−2.6) 41.2 (+/−3.1) 46.9 (+/−3.9)
General Health
PNS (n=13) 42.9 (+/−2.7) 40.2 (+/−2.8) 42.9 (+/−2.9) 39.9 (+/−3.2) 42.4 (+/−3.5)
Usual Care (=12) 39.0 (+/−2.8) 39.3 (+/−2.9) 41.1 (+/−3.1) 42.2 (+/−3.4) 39.4 (+/−3.8)
Vitality
PNS (n=13) 49.7 (+/−2.9) 48.0 (+/−2.9) 51.2 (2.9) 50.2 (+/−3.0) 53.2 (+/−3.4)
Usual Care (=12) 40.4 (+/−3.0) 40.3 (+/−3.0) 44.3 (3.2) 42.4 (+/−3.3) 47.5 (+/−3.9)
Social Functioning
PNS (n=13) 44.7 (+/−3.2) 47.1 (+/−3.2) 48.9 (+/−3.2) 48.0 (+/−3.3) 49.4 (+/−3.5)
Usual Care (=12) 35.5 (+/−3.3) 37.2 (3.4) 45.7 (+/−3.7) 41.5 (+/−3.7) 44.1 (+/−4.1)
Role-Emotional
PNS (n=13) 48.4 (+/−3.6) 49.2 (+/−3.7) 49.6 (3.6) 46.8 (+/−3.8) 50.5 (+/−4.2)
Usual Care (=12) 40.7 (+/−3.8) 40.3 (+/−3.8) 44.7 (4.0) 41.6 (+/−4.2) 45.8 (+/−4.9)
Mental Health
PNS (n=13) 53.9 (+/−3.9) 53.8 (3.9) 52.5 (+/−3.7) 51.3 (+/−3.5) 53.4 (+/−3.6)
Usual Care (=12) 45.3 (+/−4.0) 42.5 (+/−4.0) 44.6 (4.0) 44.7 (+/−3.8) 49.6 (4.0)

The effect of treatment group over time was analyzed using a linear mixed model for repeated measures for each outcome measure with five discrete time points (baseline, start of treatment, end of treatment, and 6 weeks and 12 weeks post-treatment). Least-squares means are presented. Treatment phase for both groups occurred between outcomes assessments at weeks 1 – 4.

Abbreviations: SE-Standard Error; PNS-Peripheral Nerve Stimulation; VGRS-Visual Graphic Rating Scales

Table 3.

Hypothesis Tests of Model Fixed Effects

Outcome Effect F value (df) p-value
Worst pain 7d, Intention to Treat group×time 4.4 (1,62) 0.040
time 13.4 (1,21) 0.001
group 0.9 (1,62) 0.335
Worst pain 7d, Per Protocol group×time 3.5 (1,60) 0.068
time 12.1 (1,18) 0.003
group 0.1 (1,60) 0.756
Pain Interference 7d group×time 0.7 (1,62) 0.398
time 15.4 (1,21) <0.001
group 3.9 (1,62) 0.054
ShoulderQ VGRS group×time 3.7 (1,62) 0.059
time 21.5 (1,21) <0.001
group 0.4 (1,61) 0.556
SF-36v2
Physical Component Summary group×time 0 (1,61) 0.983
time 18.4 (1,21) <0.001
group 0.01 (1,61) 0.923
Mental Component Summary group×time 0.9 (1,21) 0.357
time 0.6 (1,21) 0.439
group 4.2 (1,61) 0.046
Physical Functioning group×time 0.6 (1,61) 0.438
time 0.5 (1,21) 0.493
group 0.6 (1,61) 0.652
Role-Limitations Physical group×time 0.7 (1,61) 0.406
time 13.6 (1,21) 0.001
group 1.7 (1,61) 0.192
Bodily Pain group×time 0.4 (1,61) 0.543
time 17.2 (1,21) <0.001
group 1.0 (1,61) 0.316
General Health group×time 0.2 (1,61) 0.656
time 0.1 (1,21) 0.760
group 0.4 (1,61) 0.656
Vitality group×time 0.2 (1,61) 0.653
time 3.9 (1,21) 0.062
group 5.1 (1,61) 0.028
Social Functioning group×time 0.6 (1,61) 0.439
time 4.7 (1,21) 0.042
group 4.5 (1,61) 0.038
Role-Emotional group×time 0.3 (1,61) 0.567
time 0.7 (1,21) 0.415
group 2.6 (1,61) 0.116
Mental Health group×time 1.6 (1,61) 0.219
time 0.6 (1,21) 0.435
group 3.4 (1,61) 0.070

The effect of treatment group over time was analyzed using a linear mixed model for repeated measures for each outcome measure. The model assessed whether the outcomes: 1) change in different ways over time between groups (group by time interaction term); 2) change over time (continuous time effect); or, 3) are different between groups under the assumption that the mean response profiles are parallel (group main effect).

Abbreviation: df-degrees of freedom; VGRS-Visual Graphic Rating Scales

Figure 2.

Figure 2

Figure 2

Figure 2

Figure 2

Graphical comparison of: A) the worst pain in 1 week on 0–10 scale, Brief Pain Inventory, Short Form question 3; B) Pain interference on a 0 – 10 scale, Brief Pain Inventory, Short Form question 9; C) the summed Visual Graphic Rating Scale from the ShoulderQ questionnaire; and, D) the Bodily Pain domain of the SF-36v2. PNS is represented by the solid line and UC the dashed line. The period of treatment is represented by the shaded area. *=statistically significant differences between PNS and UC using Bonferroni correction for multiple comparisons.

The per-protocol analysis of 21 participants showed a significant reduction in pain in both groups (time effect) but no significant slope difference between groups during the study (time by group interaction effect). There was no significant main group effect.

Secondary Outcomes

Pain Interference

Both the PNS and UC group experienced a significant reduction in pain interference during the study (Figure 2B, time effect) though, the improvement was not significantly different between groups (time by group interaction effect). The mean pain interference rating at baseline was 3.6 (+/− 0.7) and 5.0 (+/− 0.7) for PNS and UC, respectively, which dropped to a 0.8 (+/− 0.7) and 3.0 (+/− 0.8), respectively, at 10 weeks, and remained at 1.1 (+/− 0.7) and 3.5 (+/− 0.8), respectively, at 16 weeks. There was no significant main group effect for pain interference.

Shoulder Q

There was a significant improvement in both PNS and UC for the summed VGRS of the Shoulder Q, though the improvement was not significantly different between groups (Figure 2C). There was no significant main group effect for the summed VGRS of the Shoulder Q.

SF36v2

There was a significant improvement in both the PNS and UC groups in the Physical Component Summary (PCS) score of the SF36v2, but no significant time by group interaction effect. There was no significant main group effect for the PCS score. The time and time by group interaction effects were not significant for Mental Component Summary (PCS) score, though the main group effect was significant (UC group was significantly lower at baseline). With respect to individual SF36v2 domains, there was a significant improvement for both groups (time effect) in the domains of Role-Limitations Physical, Bodily Pain (Figure 2D), and Social Functioning; however, there was not a significant slope difference between the two groups for any of the individual domains.

Global Success

The global rates of success (a 30% or greater pain reduction at end of treatment and all follow-up time points) in the intention-to-treat analysis revealed that there was not a significant difference in the proportion of successful outcomes in the PNS group compared to the usual care group (β = 0.8, t(98) = 1.4, p =0.175), with a relative probability of successful outcome for the PNS group compared to the UC group of 2.3 (95% CI 0.7 – 7.4). The per-protocol analysis revealed a success rate of 66.7% (8 of 12) for the PNS group and 25.0% (2 of 8) for the UC group (p=0.170), Figure 3. The relative probability of successful outcome for the PNS group compared to UC in the per-protocol analysis was 2.3 (95% CI 0.9 – 5.5).

Figure 3.

Figure 3

Successful outcomes (30% reduction in pain by end of treatment maintained at all remaining time points) for PNS and UC groups.

The secondary outcomes are presented in Table 2 and Table 3.

Safety

There were 14 electrodes implanted in 13 participants. One participant (7.7%) experienced dislodgement of the electrode and required re-implantation. Three participants (23.1%) experienced a retained electrode fragment (21.4%) that was identified on radiographic exam that did not result in an adverse event. There were no cutaneous infections during the study. Six participants (46.2%) experienced pruritis at the electrode or bandage site, and two (15.4%) had pain after implantation, both of which resolved without intervention.

Concomitant Therapies

Five subjects in the treatment group (38.5%), and 2 in the UC group (16.7%), received PT or OT outside of the study protocol. Twenty-three percent of the data for formal cumulative hours of PT and OT were missing. One participant in the PNS group was hospitalized between weeks 10 and 16 and received inpatient PT and OT, for which the total cumulative amount of which were not recorded. There was no significant difference in the recorded total cumulative hours of formal PT for the PNS group compared to the UC group (2.7 +/− 5.6 hours vs.3.9, +/− 9.8 hours, respectively, z=−0.2, p=0.8), though there was a significant difference in the recorded total cumulative hours of formal OT (4.8, +/− 6.5 hours vs. 0 hours, respectively, z=−2.0, p<0.05).

Unfortunately, the data for analgesic medication usage were not adequate for analysis due to missing data. Many participants had comorbid medical conditions and were taking multiple medications and, when asked about analgesic consumption for the prior week, they were often not able to recall which medications were used or the quantities of the medications consumed over prior week. The types of analgesics used by participants at baseline are listed in Table 1.

DISCUSSION

Key Findings

This trial provides evidence that may support our hypothesis that PNS results in a greater pain reduction than UC for chronic HSP, as measured by the primary outcome measure of BPI-SF3. Both PNS and UC were associated with reduction in pain during the treatment phase of the study, yet those participants who received PNS maintained their pain reduction through the end of the study. Conversely, pain increased after treatment in those who received UC such that a statistically and clinically important difference between the groups existed at weeks 10 and 16 (6 and 12 weeks after treatment, respectively.)

Improvements were found in all secondary outcomes, though the hypotheses that PNS would be associated with a greater reduction in pain interference and a greater improvement in HRQoL than UC were not supported in this trial. Both groups experienced a reduction in the interference with daily activities caused by pain, as measured by the BPI-SF9, and pain as measured by the VGRS of the ShoulderQ, with a greater improvement for those who received PNS that did not reach statistical significance. Clinically meaningful improvement measured as the global success rate, defined as a 30% reduction by end-of-treatment that was maintained at all remaining outcomes assessment, also showed better outcomes associated with PNS compared to usual care that did not reach statistical significance. Both groups experienced a similar improvement in a measure of their physical health, the Physical Component Score of the SF36v2, largely through an improvement in the Bodily Pain domain, though no significant difference between the two groups.

The results of this study also indicate that single-lead, 3-week PNS is a safe procedure for chronic HSP. The primary adverse event was a retained electrode fragment within the deltoid due to fracturing of the tip of the electrode during explantation, which occurred in 21.4% of participants. There is potential for further complications due to the retained electrode fragment of migration of the tip toward the skin and/or infection for which the treatments would be surgical excision or administration of antibiotics. In a study of over 850 electrode placement in the upper limbs in our laboratory, the risk for the complication of migration and/or infection from a retained electrode fragment is 1.5%.28 The risk of migration or infection in our study was 0.00321 per electrode. The improvement in pain experienced from the intervention outweighs this risk, though work to reduce the risk of electrode fragment retention is required.

Explanations

The mechanisms of action of PNS and UC for pain reduction in HSP are not known. The UC approach is based on proper positioning, support, improving biomechanics through range of motion and strengthening exercises, and improving functional use of the arm.2022 If biomechanical improvements are the pathway through which UC is effective, it is possible that any biomechanical improvement that was gained with treatment was lost after treatment, in spite of participants having been taught home exercises and encouraged to continue performance throughout the study period.

While PNS was initially designed to improve shoulder subluxation in those with HSP, it has since been found to improve pain in spite of lack of effect on subluxation or measures of biomechanics.3, 4, 9, 29 Whatever the mechanism, it appears that PNS differs from UC in that the improvements with treatment are maintained at least 12 weeks after treatment. It has been suggested that PNS may alter maladaptive neuroplastic changes associated with chronic pain4, though this has not yet been studied.

Comparison to Prior Work

Participants who received PNS in this trial had a 65.3% reduction in pain by the end of treatment that was maintained at a 60.0% reduction in pain at the end of the follow-up period 12 weeks after treatment. The current results are similar to the results of the prior assessor blinded multi-site, RCT of the 4-lead, 6-week PNS treatment4, in which there was a 65.6% reduction in pain by the end of treatment that was maintained at 59.4% reduction at 12-weeks after treatment. Two non-assessor blinded case-series have also found similar reductions in pain. A case-series of single-lead, 3-week PNS9 showed a 70% and 63% reduction in pain at the end of treatment and 12 weeks after treatment, respectively, whereas a case-series of 4-lead, 6-week PNS29 showed a 78.5% pain reduction at end of treatment, and 78.7% at 12 weeks after treatment.

There was also a significant improvement in pain interference (BPI-SF9) in both the PNS and UC groups. An improvement in pain interference was shown in the prior trial of 4-lead, 6-week PNS, though with a significant difference compared to treatment with a hemi-sling.4 In the present trial, there was an improvement in both PNS and UC, with a greater improvement for PNS that was not statistically significant.

Importantly, the samples differ between this and the prior trial of 4-lead, 6-week PNS. The prior trial limited recruitment to participants with a painful shoulder with subluxation, whereas the current trial enrolled participants with or without subluxation. There has been mixed evidence in the literature about the correlation of shoulder subluxation and HSP, with some studies suggesting a correlation exists 15, 16, 3032 whereas others have demonstrated the lack of correlation.1, 3337 That the results of the current trial in a sample heterogeneous for subluxation had similar improvement to a sample where all participants had subluxation provides further evidence that the mechanism of PNS is not related to subluxation.

Limitations

There are several limitations to this study. Even though more than 80% of the primary outcome data points were collected, there is imprecision in the estimate of the differences between groups due to variability in the data, the sample size, and missing data. This causes the confidence interval around the statistically significant difference in BPI-SF3 between groups at the final follow-up point of 16-weeks to span the range from 1.0, which is below the level of clinical significance25, 38, to 5.2 points, which represents a large clinical improvement in pain. In spite of imprecision around the estimate of the difference, our intention-to-treat analysis provides confidence that a difference in pain reduction between PNS and usual care does exist.

The secondary pain outcomes in this study did not reveal significant differences between PNS and UC, though the data seem to support a greater improvement in PNS than UC at end of treatment, and a greater sustained improvement at follow-up time points. The primary reason is likely related to sample size, incomplete data, and inherently higher variance associated with these secondary measures. The global success definition of a 30% reduction in pain at end of treatment and all follow-up time points is a conservative measure, though one of great importance to patients and providers. Future studies should be powered to find differences among these secondary outcomes, in addition to the primary outcome of worst-pain in the last 7-days.

As with all RCTs, there is a limit to the generalizability of the findings. Our study enrolled 30% of the people who were screened, which is higher than many RCTs, but still representing a minority of those with HSP. Further generalizability is lost by the controlled nature of the study that occurred at a single-site; however, that is appropriate for efficacy studies such as this one. Future effectiveness trials will need to be conducted to determine the pain reduction with PNS treatment in a clinical setting.

Finally, concomitant therapies may have compromised internal validity. There was a difference in the cumulative hours of OT in favor of the PNS group. However, since these therapies did not focus on the shoulder, they are unlikely to have contributed to the observed difference between groups. Unfortunately, we were unable to analyze analgesic medication use due to the poor quality of our data. Thus, the confounding effect of analgesic use cannot be ruled out.

Implications

This study adds to the evidence that PNS is an efficacious treatment for chronic HSP. It is a safe procedure that has been modified to be easier to deploy and with less discomfort to recipients, with similar results to the prior procedure. Effect size estimates from this trial will be used to inform the design of future trials to demonstrate effectiveness with respect to the primary and secondary outcome measures. While the mechanism of action remains unknown, it should be evaluated in future studies so that this treatment may be optimized.

CONCLUSION

This RCT provides evidence that single-lead, 3-week PNS is an efficacious and safe treatment for the reduction of chronic HSP, a reduction in pain interference, and an improvement in QoL. The reduction in shoulder pain is greater than that obtained from usual care and is maintained for at least 12 weeks after treatment. Future studies should determine mechanism of action, optimal stimulation delivery, and treatment effectiveness.

Acknowledgments

This work was sponsored by grant R01HD059777 and K24HD054600 from the Eunice Kennedy Shriver National Institute of Child Health and Human Development and the Clinical and Translational Science Collaborative of Cleveland, UL1TR000439 from the National Center for Advancing Translational Sciences (NCATS) component of the National Institutes of Health and NIH roadmap for Medical Research.

Footnotes

Disclosures:

The author and co-authors have the following disclosures: Maria Bennett is an employee of SPR Therapeutics, LLC. John Chae is a consultant and Chief Medical Advisor to SPR Therapeutics, LLC. Dr. Chae also owns equity in SPR Therapeutics, LLC. SPR Therapeutics, LLC. has a commercial interest in the device presented in this manuscript.

This work was presented, in part, at the INS 11th World Congress, Berlin, Germany. June 08–13, 2013.

Clinical Trials Registration: NCT01123382

Contributor Information

Richard D. Wilson, Department of Physical Medicine and Rehabilitation, Case Western Reserve University at MetroHealth Medical Center, Cleveland, OH; Cleveland Functional Electrical Stimulation Center, Case Western Reserve University, Cleveland, OH.

Douglas D. Gunzler, Center for Healthcare Research and Policy, Case Western Reserve University/MetroHealth Medical Center, Cleveland, Ohio.

Maria E. Bennett, SPR Therapeutics, LLC., Cleveland, OH.

John Chae, Department of Physical Medicine and Rehabilitation, Case Western Reserve University at MetroHealth Medical Center, Cleveland, OH; Cleveland Functional Electrical Stimulation Center, Case Western Reserve University, Cleveland, OH; Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH.

References

  • 1.Wanklyn P, Forster A, Young J. Hemiplegic shoulder pain (HSP): natural history and investigation of associated features. Disabil Rehabil. 1996;18(10):497–501. doi: 10.3109/09638289609166035. [DOI] [PubMed] [Google Scholar]
  • 2.Broeks JG, Lankhorst GJ, Rumping K, Prevo AJ. The long-term outcome of arm function after stroke: results of a follow-up study. Disabil Rehabil. 1999;21(8):357–364. doi: 10.1080/096382899297459. [DOI] [PubMed] [Google Scholar]
  • 3.Chae J, Yu DT, Walker ME, Kirsteins A, Elovic EP, Flanagan SR, et al. Intramuscular electrical stimulation for hemiplegic shoulder pain: a 12-month follow-up of a multiple-center, randomized clinical trial. Am J Phys Med Rehabil. 2005;84(11):832–842. doi: 10.1097/01.phm.0000184154.01880.72. [DOI] [PubMed] [Google Scholar]
  • 4.Yu DT, Chae J, Walker ME, Kirsteins A, Elovic EP, Flanagan SR, et al. Intramuscular neuromuscular electric stimulation for poststroke shoulder pain: a multicenter randomized clinical trial. Arch Phys Med Rehabil. 2004;85(5):695–704. doi: 10.1016/j.apmr.2003.07.015. [DOI] [PubMed] [Google Scholar]
  • 5.Yu DT. Shoulder Pain and Other Musculoskeletal Complications. In: Stein J, editor. Stroke recovery and rehabilitation. New York: Demos Medical; 2009. p. 437. [Google Scholar]
  • 6.Chae J, Ng A, Yu DT, Kirsteins A, Elovic EP, Flanagan SR, et al. Intramuscular electrical stimulation for shoulder pain in hemiplegia: does time from stroke onset predict treatment success? Neurorehabil Neural Repair. 2007;21(6):561–567. doi: 10.1177/1545968306298412. [DOI] [PubMed] [Google Scholar]
  • 7.Kalichman L, Ratmansky M. Underlying pathology and associated factors of hemiplegic shoulder pain. Am J Phys Med Rehabil. 2011;90(9):768–780. doi: 10.1097/PHM.0b013e318214e976. [DOI] [PubMed] [Google Scholar]
  • 8.Wilson RD, Bennett ME, Lechman TE, Stager KW, Chae J. Single-lead percutaneous peripheral nerve stimulation for the treatment of hemiplegic shoulder pain: a case report. Arch Phys Med Rehabil. 2011;92(5):837–840. doi: 10.1016/j.apmr.2010.11.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Chae J, Wilson RD, Bennett ME, Lechman TE, Stager KW. Single-lead Percutaneous Peripheral Nerve Stimulation for the Treatment of Hemiplegic Shoulder Pain: A Case Series. Pain practice : the official journal of World Institute of Pain. 2013;13(1):59–67. doi: 10.1111/j.1533-2500.2012.00541.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Tan G, Jensen MP, Thornby JI, Shanti BF. Validation of the Brief Pain Inventory for chronic nonmalignant pain. J Pain. 2004;5(2):133–137. doi: 10.1016/j.jpain.2003.12.005. [DOI] [PubMed] [Google Scholar]
  • 11.Cleeland CS, Ryan KM. Pain assessment: global use of the Brief Pain Inventory. Ann Acad Med Singapore. 1994;23(2):129–138. [PubMed] [Google Scholar]
  • 12.Compston A. Aids to the investigation of peripheral nerve injuries. Medical Research Council: Nerve Injuries Research Committee. His Majesty's Stationery Office: 1942; pp. 48 (iii) and 74 figures and 7 diagrams; with aids to the examination of the peripheral nervous system. By Michael O'Brien for the Guarantors of Brain. Saunders Elsevier: 2010; pp. [8] 64 and 94 Figures. Brain. 2010;133(10):2838–2844. doi: 10.1093/brain/awq270. [DOI] [PubMed] [Google Scholar]
  • 13.Beck AT, Steer RA, Brown GK. BDI-FastScreen for medical patients: Manual. San Antonio, TX: The Psychological Corporatio; 2000. [Google Scholar]
  • 14.Winstein CJ, Miller JP, Blanton S, Taub E, Uswatte G, Morris D, et al. Methods for a multisite randomized trial to investigate the effect of constraint-induced movement therapy in improving upper extremity function among adults recovering from a cerebrovascular stroke. Neurorehabil Neural Repair. 2003;17(3):137–152. doi: 10.1177/0888439003255511. [DOI] [PubMed] [Google Scholar]
  • 15.Roy CW, Sands MR, Hill LD. Shoulder pain in acutely admitted subjects. Clinical Rehabil. 1994;8:334–340. [Google Scholar]
  • 16.VanOuwenaller C, Laplace PM, Chantraine A. Painful shoulder in hemiplegia. Arch Phys Med Rehabil. 1986;67:23–26. [PubMed] [Google Scholar]
  • 17.Roosink M, Renzenbrink GJ, Buitenweg JR, van Dongen RT, Geurts AC, Ijzerman MJ. Somatosensory symptoms and signs and conditioned pain modulation in chronic post-stroke shoulder pain. J Pain. 2011;12(4):476–485. doi: 10.1016/j.jpain.2010.10.009. [DOI] [PubMed] [Google Scholar]
  • 18.Lee HJ, DeLisa JA. Manual of nerve conduction study and surface anatomy for needle electromyography. 4th ed. Philadelphia: Lippincott Wilkins and Williams; 2005. [Google Scholar]
  • 19.Snels IA, Beckerman H, Lankhorst GJ, Bouter LM. Treatment of hemiplegic shoulder pain in the Netherlands: results of a national survey. Clin Rehabil. 2000;14(1):20–27. doi: 10.1191/026921500668239146. [DOI] [PubMed] [Google Scholar]
  • 20.Teasell RW, Foley NC, Bhogal SK, Speechley MR. An evidence-based review of stroke rehabilitation. Top Stroke Rehabil. 2003;10(1):29–58. doi: 10.1310/8YNA-1YHK-YMHB-XTE1. [DOI] [PubMed] [Google Scholar]
  • 21.Bates B, Choi JY, Duncan PW, Glasberg JJ, Graham GD, Katz RC, et al. Veterans Affairs/Department of Defense Clinical Practice Guideline for the Management of Adult Stroke Rehabilitation Care: executive summary. Stroke. 2005;36(9):2049–2056. doi: 10.1161/01.STR.0000180432.73724.AD. [DOI] [PubMed] [Google Scholar]
  • 22.Khadilkar A, Phillips K, Jean N, Lamothe C, Milne S, Sarnecka J. Ottawa panel evidence-based clinical practice guidelines for post-stroke rehabilitation. Top Stroke Rehabil. 2006;13(2):1–269. doi: 10.1310/3TKX-7XEC-2DTG-XQKH. [DOI] [PubMed] [Google Scholar]
  • 23.Turner-Stokes L, Jackson D. Assessment of shoulder pain in hemiplegia: sensitivity of the ShoulderQ. Disabil Rehabil. 2006;28(6):389–395. doi: 10.1080/09638280500287692. [DOI] [PubMed] [Google Scholar]
  • 24.Keith RA. Functional status and health status. Arch Phys Med Rehabil. 1994;75(4):478–483. doi: 10.1016/0003-9993(94)90175-9. [DOI] [PubMed] [Google Scholar]
  • 25.Farrar JT, Young JP, Jr, LaMoreaux L, Werth JL, Poole RM. Clinical importance of changes in chronic pain intensity measured on an 11-point numerical pain rating scale. Pain. 2001;94(2):149–158. doi: 10.1016/S0304-3959(01)00349-9. [DOI] [PubMed] [Google Scholar]
  • 26.Spiegelman D, Hertzmark E. Easy SAS calculations for risk or prevalence ratios and differences. Am J Epidemiol. 2005;162(3):199–200. doi: 10.1093/aje/kwi188. [DOI] [PubMed] [Google Scholar]
  • 27.Rubin DB. Multiple imputation for nonresponse in surveys. New York: Wiley; 1987. [Google Scholar]
  • 28.Knutson JS, Naples GG, Peckham PH, Keith WM. Electrode fracture rate and occurrences of infection and granuloma associated with percutaneous intramuscular electrodes in upper-limb functional electrical stimulation application. J Rehabil Res Dev. 2002;39:671–684. [PubMed] [Google Scholar]
  • 29.Renzenbrink GJ, MJ IJ. Percutaneous neuromuscular electrical stimulation (P-NMES) for treating shoulder pain in chronic hemiplegia. Effects on shoulder pain and quality of life. Clin Rehabil. 2004;18(4):359–365. doi: 10.1191/0269215504cr759oa. [DOI] [PubMed] [Google Scholar]
  • 30.Najenson T, Yacubovich E, Pikielni SS. Rotator cuff injury in shoulder joints of hemiplegic patients. Scan J Rehab Med. 1971;3:131–137. [PubMed] [Google Scholar]
  • 31.Griffin JW. Hemiplegic shoulder pain. Phys Ther. 1986;66(12):1884–1893. doi: 10.1093/ptj/66.12.1884. [DOI] [PubMed] [Google Scholar]
  • 32.Ring H, Feder M, Berchadsky R, Samuels G. Prevalence of pain and malalignment in the hemiplegic's shoulder at admission for rehabilitation: a preventive approach. European Journal of Physical Medicine and Rehabilitation. 1993;3:199–203. [Google Scholar]
  • 33.Zorowitz RD, Hughes MB, Idank D, Ikai T, Johnston MV. Shoulder pain and subluxation after stroke: correlation or coincidence. Am J Occ Therapy. 1996;50(3):194–201. doi: 10.5014/ajot.50.3.194. [DOI] [PubMed] [Google Scholar]
  • 34.Bohannon RW, Andrews AW. Shoulder subluxation and pain in stroke patients. Amer J Occ Therapy. 1990 Oct 3;:507–509. doi: 10.5014/ajot.44.6.507. [DOI] [PubMed] [Google Scholar]
  • 35.VanLangenberghe H, Hogan BM. Degree of pain and grade of subluxation in the painful hemiplegic shoulder. Scand J Rehab Med. 1988;20:161–166. [PubMed] [Google Scholar]
  • 36.Arsenault AB, Bilodeau M, Dutil E, Riley E. Clinical significance of the V-shaped space in the subluxed shoulder of hemiplegics. Stroke. 1991;22(7):867–871. doi: 10.1161/01.str.22.7.867. [DOI] [PubMed] [Google Scholar]
  • 37.Joynt RL. The source of shoulder pain in hemiplegia. Arch Phys Med Rehabil. 1992;73(5):409–413. [PubMed] [Google Scholar]
  • 38.Dworkin RH, Turk DC, Wyrwich KW, Beaton D, Cleeland CS, Farrar JT, et al. Interpreting the clinical importance of treatment outcomes in chronic pain clinical trials: IMMPACT recommendations. J Pain. 2008;9(2):105–121. doi: 10.1016/j.jpain.2007.09.005. [DOI] [PubMed] [Google Scholar]

RESOURCES