Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Dec 1.
Published in final edited form as: J Psychiatr Res. 2016 Sep 9;83:176–183. doi: 10.1016/j.jpsychires.2016.09.007

The Premonitory Urge Revisited: An Individualized Premonitory Urge for Tics Scale

Joseph F McGuire a, Nicole McBride b, John Piacentini a, Carly Johnco b, Adam B Lewin b,c,d, Tanya K Murphy b,c,d, Eric A Storch b,c,d,e,f
PMCID: PMC5107126  NIHMSID: NIHMS817511  PMID: 27643476

Abstract

Background

Premonitory urge ratings have advanced our understanding of urge phenomenology among individuals with tic disorders (TD). However, these ratings have been limited by their reliance on a single global dimension of urge severity. This study examined the psychometric properties of a novel scale called the Individualized Premonitory Urge for Tics Scale (I-PUTS) that assesses urge severity across multiple dimensions (number, frequency, and intensity).

Method

Seventy-five youth with a TD and their parents participated. Clinicians assessed youth’s tic severity, depression severity, rages, and premonitory urges. Parents completed ratings of youth’s anxiety, affect lability, and general psychopathology. Youth completed self-report ratings of anxiety, urge severity, and distress tolerance.

Results

The I-PUTS identified that youth experienced an average of three distinct urges, but had an average of seven tics over the past week. Urges were primarily localized in the head/face, neck/throat, and arm regions. All I-PUTS dimensions exhibited excellent inter-rater reliability. The I-PUTS dimensions exhibited good convergent validity with global urge ratings and tic severity, and appropriate divergent validity from other clinical constructs. Youth who exhibited discrepant reports between clinician-administered and self-report urge ratings had less anxiety and tic severity, and greater inattention and externalizing problems compared to youth who exhibited good agreement.

Conclusions

The I-PUTS is a reliable and valid assessment of urge phenomena, which provides additional and complementary information to existing urge scales. It highlights the existence of multiple dimensions of urge severity, and presents particular utility when assessing urges in youth with TD who have inattention and externalizing problems.

Keywords: Tourette disorder, tic disorder, premonitory urge, tic severity, anxiety, attention deficit hyperactivity disorder

1. Introduction

Tics are sudden, rapid, and recurrent motor movements and/or vocalizations that occur in up to 21% of school-aged youth for brief periods (Cubo et al. 2011; Kurlan et al. 2002). When motor and/or vocal tics persist for more than one year, youth meet diagnostic criteria for either a persistent tic disorder or Tourette Disorder (henceforth collectively referred to as TD) (American Psychiatric Association, 2013). Youth with TD often experience co-occurring psychiatric symptoms [e.g., anxiety disorders, obsessive compulsive disorder (OCD), attention deficit hyperactivity disorder (ADHD)] (Freeman et al. 2000; Lebowitz et al. 2012; Specht et al. 2011), significant impairment (Conelea et al. 2011; Storch et al. 2007a), and suffer a poor quality of life (Eddy et al. 2010; Storch et al. 2007b).

While tics are the overt behavioral characteristic of TD, internal subjective somatosensory sensations play an important role in tic phenomenology. These somatosensory sensations have been described using various terminology over the past 20–30 years, with “premonitory urge” emerging as the dominant descriptor (O’Connor, 2002; Woods et al. 2005). Premonitory urges are unpleasant aversive sensations that precede tics, and are temporarily reduced or relived by the performance of tics (Leckman et al. 1993; Woods et al. 2005). Behavioral models suggest that this pattern of urge-relief develops a negative reinforcement cycle contributing to tic persistence, with the discontinuation of this cycle purported to play an important role in behavioral interventions (see Lewin et al. 2014 or Woods et al. 2008 for further detail). Retrospective reports suggest that tics generally onset around 6–7 years of age (Bloch and Leckman, 2009), but note that premonitory urges are first identified around 8–10 years of age (Banaschewski et al. 2003; Leckman et al. 1993; Woods et al. 2005). A majority of individuals with TD experience premonitory urges (Kwak et al. 2003; Leckman et al. 1993), with an average of 8 to 9 distinct urges reported over the previous week in largely adult samples (Leckman et al. 1993).

While interviews and self-reported checklists initially provided descriptive information about urge phenomenology, inconsistent methodology across studies limits comparability of findings (Banaschewski et al. 2003; Kurlan et al. 1989; Kwak et al. 2003; Leckman et al. 1993). Woods and colleagues developed a brief self-report scale called the Premonitory Urge for Tics Scale (PUTS) that assessed tic premonitory urges across 9 items (Woods et al. 2005). The PUTS has demonstrated good psychometric properties to assess an individual’s urge to tic across samples (McGuire et al. 2012; Reese et al. 2014; Steinberg et al. 2010; Woods et al. 2005). The PUTS total score has produced small-to-moderate associations with overall tic severity in some studies (r=0.21–0.33) (Crossley and Cavanna, 2013; Eddy and Cavanna, 2014; Steinberg et al. 2013; Woods et al. 2005), with others identifying weak or non-significant associations (Steinberg et al. 2010). When examining co-occurring symptoms, a moderate-to-strong relationship has been found between the PUTS total score and obsessive-compulsive symptoms (r=0.31–0.50) (Steinberg et al. 2010; Woods et al. 2005), with smaller and mixed associations observed with overall anxiety symptoms (r=0.33–0.35) (Eddy and Cavanna, 2014; Woods et al. 2005) and somatic/panic symptoms (r=0.18–0.40) (Rozenman et al. 2014). Although small associations with depressive symptoms have been identified (r=0.10–0.25) (Eddy and Cavanna, 2014; Steinberg et al. 2013), there has been inconsistent evidence regarding associations between ADHD severity and PUTS total score (Crossley and Cavanna, 2013; Eddy and Cavanna, 2014; Reese et al. 2014; Steinberg et al. 2010).

Although the PUTS has led to considerable advancements in understanding premonitory urge phenomenology, there are several considerations that warrant further examination. For instance, the PUTS measures premonitory urges as a unitary construct, across an undefined time period, and does not allow the respondent to distinguish between specific urges for different tics. This prevents the evaluation of subgroup analyses (motor tic versus phonic tic urges), and limits analyses to a single dimension across all experienced urges (Reese et al. 2014). As reports have identified that different tics and different individuals have a varying degree of premonitory urges (Leckman et al. 1993; McGuire et al. 2015), an individualized urge assessment may provide important complementary information and offer the chance to evaluate urges along multiple dimensions (Brabson et al. 2015; Reese et al. 2014). Such an evaluation could clarify the mixed associations between the PUTS total score and tic severity identified in prior reports (Crossley and Cavanna, 2013; Eddy and Cavanna, 2014; Steinberg et al. 2010; Steinberg et al. 2013; Woods et al. 2005), and further elucidate the contribution of premonitory urges in maintaining tic behaviors. Beyond examining multiple dimensions of urge phenomena, prior evaluations of premonitory urges have been largely restricted to common co-occurring psychopathological symptoms. Although highlighting consistent associations between the PUTS total score and internalizing symptoms, there is a need to expand evaluations to include related constructs that may contribute to urge phenomenology. Indeed, rage (Chen et al. 2013; Storch et al. 2012), dysregulation (McGuire et al. 2013), distress tolerance (Cougle et al. 2011), and emotional lability (Rizzo et al. 2014; Rosen et al. 2015) have been found to be important in TD and related conditions and may provide clarifying information on clinical construct associated with premonitory urges.

Accordingly, this study developed a novel clinician-administered measure called the Individualized Premonitory Urge to Tic Scale (I-PUTS) to assess tic urge phenomenology in a clinical sample of youth with TD and investigated its psychometric properties. First, we examined the presence, frequency, intensity, and dominant body regions associated with urges for individual tics using the I-PUTS. Second, we investigated the inter-rater reliability of the I-PUTS dimensions. Third, we explored the convergent and divergent associations between I-PUTS dimensions, PUTS total score, and clinical characteristics including tic severity, rage, dysregulation, distress tolerance, and affect lability. Finally, we examined youth who exhibited good agreement versus poor agreement in urge presence on the clinician-rated I-PUTS and self-reported PUTS.

2. Methods

2.1 Participants

Participants were 75 children and adolescents (60% male) between 6 and 17 years of age (Myears=10.68, SDyears=2.82) who had tics and presented at one of two specialty clinics for OCD and Tic Disorders in the southeastern United States. Participants met the following inclusion criteria: 6–17 years of age (inclusive); diagnostic criteria for a tic disorder with current tic symptoms present; and have at least one caregiver participate in the assessment. Based on an unstructured clinical interview, participants were not invited to participate if there was any concern of the following: active psychosis, mania, or active suicidal intent, intellectual disability, and/or other psychiatric conditions that would limit the youth or caregiver’s ability to complete study related procedures. Most participants had a diagnosis of Tourette Disorder (n=63), with the remaining participants having a chronic tic disorder (n=6) or a transient tic disorder (n=6). Twenty-six participants had co-occurring OCD, 29 participants had co-occurring ADHD, and 39 participants had a co-occurring anxiety disorder. Youth were mostly non-Hispanic Caucasians (84%) who had moderate tic severity on the Yale Global Tic Severity Scale (YGTSS: M=23.53, SD=8.39), with 33% currently taking a tic influencing medication (e.g., antipsychotic or alpha-2 agonist).

2.2 Measures

2.2.1 Yale Global Tic Severity Scale (YGTSS) (Leckman et al. 1989)

The YGTSS is a clinician-rated measure of tic severity that has excellent reliability and validity (Leckman et al. 1989; Storch et al. 2005), and acceptable internal consistency within this sample (α=0.79). The YGTSS includes a Symptom Checklist consisting of at least 42 common motor and phonic tics that are rated as absent/present over the past week. Afterward, clinicians rated motor and phonic tics separately on a 0 to 5 scale across five dimensions: number, frequency, intensity, complexity, and interference. These ratings are summed to produce a Total Tic score (range: 0–50). Clinicians also provided an Impairment score that reflected overall tic-related impairment over the past week (range: 0–50).

2.2.2 Individualized Premonitory Urge for Tics Scale (I-PUTS)

The I-PUTS is a clinician-administered measure that assessed the presence, frequency, intensity, and body region location of urges for individual tics endorsed over the past week using a symptom checklist that paralleled the YGTSS. Figure 1 presents sample items and a sample rating on the I-PUTS (contact the first or last author for the complete I-PUTS measure). For each endorsed tic, the clinician inquired whether youth experienced an urge prior to the tic. Next, the clinician inquired about the frequency of endorsed urges on a 4-point scale, which was similar to other urge categorization approaches (1=“Urge occurs 0–25% of the time you do the tic” to 4=“Urge occurs 75% –100% of the time you do the tic”) (Ganos et al. 2012). The clinician also inquired about the urge intensity on a 4-point scale (1=“minimal intensity/urge can be ignored for a considerable amount of time” to 4=“strong intensity/urge needs relief almost immediately”). When tics and/or urges were not endorsed, items received a rating of 0. Finally, the clinician inquired about the body region associated with each urge. Based on post-hoc examination, body regions were grouped into the following six categories: head/face, neck/throat, torso, arms, legs, and whole body/other. Items were summed to create a total number of distinct urges (I-PUTS Urge Number), total urge frequency (I-PUTS Frequency), and total urge intensity (I-PUTS Intensity).

Figure 1.

Figure 1

Open in a new tab

This figure presents the I-PUTS rating for a patient who has experienced a simple eye blinking tic in the past week, but no other simple eye movement tics. The clinician would endorse simple eye blinking as present, and inquire about urge presence, frequency, intensity, and body location. Ratings would be made using the provided descriptive anchors. Therefore, a patient who reported experiencing an urge in their eyes associated with the simple eye blinking tic, which occurred approximately 40% of time the tic occurred and had a moderate intensity would have the following rating on the I-PUTS.

2.2.3 Children’s Depression Rating Scale-Revised (CDRS-R) (Poznanski and Mokros, 1996)

The CDRS-R is clinician-administered scale that assessed depression severity. It consists of 17-items that are summed to produce a total depression severity score. The CDRS-R has demonstrated good psychometric properties in youth (Poznanski and Mokros, 1996).

2.2.4 Rage Outbursts and Anger Rating Scale (ROARS)(Budman et al. 2008)

The ROARS is a clinician-rated 0–9 point scale that measured frequency, intensity, and duration of rage and anger outburstsin the last seven days.

2.2.4 Child Behavior Checklist (CBCL) (Achenbach and Rescorla, 2001)

The CBCL is a parent-report questionnaire that assessed the frequency of behavioral and emotional problems within the past 6 months. The CBCL has well-documented psychometric properties and produced eight syndrome scales including Attention Problems, and two overall scores for Internalizing and Externalizing behaviors (total scores reported here) (Achenbach and Rescorla, 2001). Additionally, CBCL items can be used to create an index of parent-reported OCD symptoms (CBCL-OCS) and dysregulation (CBCL-DP) that has demonstrated good psychometric properties in previous studies and adequate to excellent internal consistency within this sample (α=0.67 and 0.93, respectively) (McGuire et al. 2013; Storch et al. 2006).

2.2.5 Children’s Affective Lability Scale (CALS) (Gerson et al. 1996)

The CALS is a 20-item parent report measure that assessed affect regulation in children. Internal-consistency, two-week test-retest reliability, and construct validity are excellent (Gerson et al. 1996).

2.2.6 Screen for Child Anxiety Related Emotional Disorders, Parent and Child Report(SCARED-P/C) (Birmaher et al. 1997)

The SCARED-P/C is a 41-item parent-and child-report measure that assessed the presence and frequency of anxiety symptoms. Items are rated on a 3-point scale, and summed to produce a total severity score. The SCARED-P/C has demonstrated good psychometric properties (Birmaher et al. 1999; Birmaher et al. 1997).

2.2.7 Premonitory Urge to Tic Scale (PUTS) (Woods et al. 2005)

The PUTS is a 9-item self-report questionnaire that measured premonitory urge sensations. Items are rated on a scale from 4-point scale, and summed to produce a total score. The PUTS has demonstrated good psychometric properties (Reese et al. 2014; Steinberg et al. 2010; Woods et al. 2005), and exhibited strong internal consistency within the sample (α=0.88).

2.2.8 Distress Tolerance Scale (DTS) (Simons and Gaher, 2005)

The DTS is a 15-item self-report measure of distress tolerance. Items are rated on a 5-point scale, with higher scores corresponding with greater distress tolerance. The DTS has demonstrated good psychometric properties (Simons and Gaher, 2005), and exhibited strong internal consistency within the sample (α=0.93).

2.3 Procedures

The local institutional review boards approved all study procedures. All youth and parents were interviewed by an experienced child and adolescent psychologist or board certified child and adolescent psychiatrist. After describing the study and obtaining written parental consent and youth assent, participants and their parents completed clinician-administered measures to assess tic severity (YGTSS), urge phenomenology (I-PUTS), rage (ROARS), and depression (CDRS-R). The I-PUTS administration was audio-recorded for quality assurance purposes. While parents completed parent-report rating scales (CBCL, SCARED-P, CALS), youth completed self-report rating scales (PUTS, SCARED-C, DTS). Psychiatric diagnoses were determined using best estimate procedures that involved consensus between two Ph.D. investigators guided by all available clinical information (Leckman et al. 1982). After study completion, 21% (n=15) of the I-PUTS assessments were randomly selected and rated by second rater to establish inter-rater reliability.

2.4 Analytic Plan

The proportion of missing data was minimal (less than 10%) and assumed to be missing at random. Although two participants were missing the I-PUTS, this data was not imputed due to the preliminary nature of the measure. The remaining missing data were addressed using expectation maximization (EM), which utilizes a two-step iterative process based on the likelihood estimation of obtaining missing values (Peugh and Enders, 2004). This approach provides accurate and efficient estimates for replacing missing data at the item level when less than 15% of item-level data are missing (Enders, 2003; Enders, 2010). Descriptive statistics characterized the sample and urge characteristics on the I-PUTS. As the I-PUTS dimensions (Kolmogorov-Smirnov, p<0.001) and PUTS total score (Kolmogorov-Smirnov, p<0.05) did not have a normal distribution, non-parametric statistics were employed as they are not based on normal distributions. Kruskal-Wallis tests compared differences in I-PUTS urge frequency and intensity across body regions. Intraclass correlation coefficients (ICCs) evaluated inter-rater reliability for each I-PUTS dimension. Spearman correlations examined the association between I-PUTS dimensions, PUTS total score, tic severity dimensions, and other clinical characteristics. Finally, youth were dichotomized into two groups: youth who exhibited good agreement on the I-PUTS and PUTS (i.e., reported the presence or non-presence of urges on both the clinician-rated I-PUTS and self-report PUTS), and youth who exhibited poor agreement on the PUTS and I-PUTS (i.e., reported no urges on one measure, but reported urges on the other measure). An independent sample t-test examined the difference in clinical characteristics between these two groups.

3. Results

3.1 Premonitory urge phenomenology

Although youth endorsed an average of seven tics over the past week (M=7.05, SD=4.48), only 79% of participants reported urges for endorsed tics on the I-PUTS. Youth had an average of 3 premonitory urges (M=2.86, SD=3.20, range: 0–16), experienced urges over 50% of the time they had the tic (M=2.37, SD=1.55), and felt a mild-to-moderate urge intensity on average (M=2.25, SD=1.44).

Premonitory urges were predominantly localized in the head/face region (35%), neck/throat region (22%), and arms (19%), with urges also reported in the legs (9%), torso (9%) and whole body/other regions (6%). Kruskal-Wallis tests revealed differences in urge frequency (χ2=12.83, p<0.03) and intensity (χ2=26.30, p<0.001) across body regions. Table 1 describes the average frequency and intensity for each rating, and pair-wise comparisons between body regions on average I-PUTS frequency and intensity ratings. Urges originating from the whole body/other region had a lower urge frequency score relative to all other body regions. However, urges originating from the whole body/other region had greater urge intensity scores compared to all other body regions. Additionally, urges originating from the neck/throat region also had greater urge intensity ratings compared to all other body regions (except for the whole body/other region). Finally, urges in the head/face region had a greater urge intensity rating compared to urges in the arm region.

Table 1.

Mean, Standard Deviation for I-PUTS Dimensions by Urge Body Region, and Pair-wise Comparisons for Average Frequency and Intensity Ratings on I-PUTS Using Kruskal-Wallis tests

Average I-PUTS Frequency Mean (SD) Range 1 2 3 4 5
1. Head and Face Region 3.11 (0.93) 1–4
2. Neck and Throat Region 3.37 (0.90) 1–4 2.97
3. Arms 3.18 (1.01) 1–4 0.41 0.84
4. Legs 3.17 (1.04) 1–4 0.19 0.58 0.00
5. Torso 3.33 (1.03) 1–4 1.52 0.00 0.47 0.38
6. Whole Body and Other Region 2.33 (0.89) 1–4 7.07** 10.34*** 6.64** 4.85* 6.56**
Average I-PUTS Intensity Mean (SD) Range 1 2 3 4 5
1. Head and Face Region 2.88 (0.84) 1–4
2. Neck and Throat Region 3.20 (0.78) 1–4 4.33*
3. Arms 2.45 (1.03) 1–4 4.61* 11.46***
4. Legs 2.50 (0.92) 1–4 2.66 7.74** 0.03
5. Torso 2.72 (0.89) 1–4 0.50 4.01* 0.89 0.54
6. Whole Body and Other Region 3.67 (0.89) 1–4 10.36*** 5.56* 12.00*** 9.99** 8.27**
Open in a new tab
*

p< 0.05,

**

p< 0.01,

***

p< 0.001

3.2 Inter-rater reliability of I-PUTS dimensions

Interclass correlations (ICCs) found that I-PUTS Urge Number (ICC=0.83, 95% CI: 0.48, 0.94), Urge Frequency (ICC=0.76, 95% CI: 0.30, 0.92) and Intensity (ICC=0.87, 95% CI: 0.62, 0.96) exhibited excellent inter-rater reliability.

3.2 Convergent Validity of the I-PUTS

The I-PUTS Urge Number (rs=0.28, p<0.02), Frequency (rs=0.23, p<0.05), and Intensity (rs=0.19, p=0.10) had small-to-medium associations with the PUTS total score (M=16.46, SD=6.03, range: 0–25). Table 2 presents the Spearman correlations between the I-PUTS dimensions, PUTS total score, and tic severity dimension. The I-PUTS Frequency had a small-to-moderate positive association with tic complexity. Additionally, the I-PUTS Intensity had a small-to-moderate positive association with both Tic Complexity and Total Tic score on the YGTSS. Meanwhile, the PUTS total score was not associated with any dimension of tic severity or overall tic severity.

Table 2.

Spearman Correlations between the I-PUTS (n=73), the PUTS (n=75), and Clinical Constructs

I-PUTS Total Number I-PUTS Total Frequency I-PUTS Total Intensity PUTS Total Score
Age .08 .02 .05 .24*
  Tic Severity & Impairment
YGTSS Total Motor Tic Score .15 .17 .21 .06
YGTSS Total Phonic Tic Score .17 .18 .20 .03
YGTSS Total Tic Score .18 .21 .23* .06
 Tic Number .10 .13 .15 −.03
 Tic Frequency .13 .15 .20 .13
 Tic Intensity .10 .13 .12 −.02
 Tic Complexity .22 .25* .26* .04
 Tic Interference .10 .13 .16 .09
YGTSS Impairment −.04 .03 .02 −.03
  Comorbid Symptom Severity
CDRS-R Total Score .02 −.06 −.04 .15
ROARS Total Score .10 .10 .10 −.12
SCARED Child Report Total Score .08 .07 .06 .48***
SCARED Parent Report Total Score .17 .14 .18 .26*
CBCL Obsessive Compulsive Scale .13 .12 .16 .19
CBCL Attention Problems −.05 −.05 −.00 −.01
CBCL Dysregulation Profile .02 .00 .05 .08
CBCL Internalizing Scale .10 .06 .11 .19
CBCL Externalizing Scale −.03 −.04 −.01 .00
  Distress Tolerance and Affect Lability
DTS Total Score .03 .04 .06 −.40***
CALS Total Score .08 .04 .06 .18
Open in a new tab
*

p< 0.05,

**

p< 0.01,

***

p< 0.001

Note: I-PUTS = Individualized Premonitory Urge for Tics Scale, PUTS = Premonitory Urge for Tics Scale, YGTSS = Yale Global Tic Severity Scale, CDRS-R= Children’s Depression Rating Scale-Revised, ROARS=Rage Outbursts and Anger Rating Scale, SCARED= Screen for Child Anxiety and Related Disorders, CBCL= Child Behavior Checklist, DTS= Distress Tolerance Scale, CALS= Child Affect Lability Scale

3.3 Divergent Validity of the I-PUTS

Table 2 displays the Spearman correlations between I-PUTS dimensions, PUTS total score, and other clinical characteristics. Notably, the I-PUTS dimensions of Urge Number, Frequency, and Intensity displayed small non-significant associations with common co-occurring psychopathology suggesting that these dimensions capture distinct phenomena. Meanwhile, the PUTS total score exhibited a small-to-moderate positive association with age, and a moderate-to-large positive association with anxiety symptom severity. Additionally, the PUTS total score also exhibited a moderate-to-large negative association with distress tolerance. No significant associations were observed with depression, rage, obsessive-compulsive symptoms, attention problems, dysregulation, and/or other internalizing or externalizing behaviors on either the I-PUTS dimensions or PUTS total score

3.4 Characteristics of youth who exhibited good and poor agreement on the clinician-rated I-PUTS and self-report PUTS

Although there was generally good agreement (76.7%) between clinician (I-PUTS) and self-report ratings (PUTS) regarding the presence of urges, some youth reported experiencing urges on the self-report but not the clinician-measure (17.8%), with only a few youth reporting urges on the clinician-measure and not the self-report (5.5%). An independent t-test compared youth who exhibited good (n=57) versus poor (n=16) agreement (see Table 3). Youth who had poor agreement between the I-PUTS and PUTS exhibited greater attention problems, dysregulation, and externalizing problems on the CBCL, but had a lower YGTSS Total Tic score and self-reported anxiety severity relative to youth who had good agreement between the two measures.

Table 3.

Comparison of Characteristics Between Youth with TD who have good (n=57) and poor (n=16) agreement on the I-PUTS and PUTS

Demographic and Clinical Characteristics Good Agreement (n=57) Poor Agreement (n=16)
Mean (SD) Mean (SD) d p
Age 10.77 (2.77) 10.88 (2.83) 0.04 0.90
 Tic Severity & Impairment
YGTSS Total Motor Tic Score 15.23 (5.51) 12.13 (6.74) 0.54 0.06
YGTSS Total Phonic Tic Score 9.30 (5.74) 7.63 (5.12) 0.30 0.30
YGTSS Total Tic Score 24.53 (8.20) 19.75 (8.61) 0.58 0.05
YGTSS Impairment 21.40 (11.87) 18.75 (10.88) 0.23 0.42
 Comorbid Symptom Severity Mean (SD) Mean (SD) d p
CDRS-R Total Score 27.78 (8.36) 30.81 (11.75) 0.33 0.25
ROARS Total Score 2.69 (3.04) 1.44 (2.34) 0.43 0.13
SCARED Child Report Total Score 23.74 (14.84) 14.47 (12.01) 0.65 0.03
SCARED Parent Report Total Score 18.87 (14.84) 15.88 (14.03) 0.20 0.47
CBCL Obsessive Compulsive Scale 3.51 (2.51) 3.91 (2.72) 0.16 0.58
CBCL Attention Problems 6.36 (5.06) 10.46 (4.96) 0.81 0.005
CBCL Dysregulation Profile 1.50 (0.99) 2.11 (1.10) 0.60 0.04
CBCL Internalizing Scale 12.33 (9.19) 13.51 (9.36) 0.13 0.65
CBCL Externalizing Scale 8.10 (7.30) 15.25 (10.80) 0.88 0.003
 Distress Tolerance &Affect Lability Mean (SD) Mean (SD) d p
DTS Total Score 49.98 (16.20) 48.07 (16.27) 0.12 0.68
CALS Total Score 16.92 (14.96) 20.71 (11.79) 0.26 0.35
Open in a new tab

Notes: I-PUTS = Individualized Premonitory Urge for Tics Scale, PUTS = Premonitory Urge for Tics Scale, YGTSS = Yale Global Tic Severity Scale, CDRS-R= Children’s Depression Rating Scale-Revised, ROARS=Rage Outbursts and Anger Rating Scale, SCARED= Screen for Child Anxiety and Related Disorders, CBCL= Child Behavior Checklist, DTS= Distress Tolerance Scale, CALS= Child Affect Lability Scale

4. Discussion

This study examined the psychometric properties of a clinician-administered individualized premonitory urge assessment tool called the I-PUTS. On the clinician administered I-PUTS, youth reported experiencing three distinct urges, described experiencing urges about 50% of the time they had a tic, and rated urges as mild-to-moderate in intensity. The I-PUTS revealed that premonitory urges were predominantly localized in the head/face, neck/throat, and arm regions—consistent with initial descriptive reports of urge phenomenology (Leckman et al. 1993). Although the whole body/other region had lower urge frequency, whole body/other urges had the greatest intensity compared to all other body regions, with urges located in the neck/throat region also exhibiting elevated urge intensity. These two body regions (typically associated with “whole body tics” and “vocal tics” respectively) may have unique sensory connections that contribute to youth’s perception of greater urge intensity.

The I-PUTS was found to have excellent inter-rater reliability across all urge dimensions. Additionally, there was support for the convergent validity of the I-PUTS urge dimensions with the PUTS total score and tic severity dimensions. Specifically, there were small-to-medium positive associations with the urge number and frequency, with a small-to-medium non-significant positive association with urge intensity. As the PUTS assesses the frequency of sensations prior to tics, the association between number and frequency is appropriate. However, identified associations may be greater if I-PUTS (past week) and PUTS (undefined time) had similar assessment intervals. For tic severity dimensions, there were medium positive associations between the urge frequency and tic complexity, as well as medium-sized positive associations between urge intensity and both Tic Complexity and Total Tic score. As complex tics are often associated with premonitory urges (McGuire et al. 2015), these associations provide additional support of convergent validity of the I-PUTS. Moreover, as premonitory urges are aversive and purported to develop a negative reinforcement cycle that maintains tic symptoms (Woods et al. 2008), the relationship between urge intensity and total tic severity is appropriate and provides further support for convergent validity of the I-PUTS.

The I-PUTS dimensions exhibited excellent divergent validity from other clinical characteristics including anxiety severity, obsessive-compulsive symptoms, and inattention. Moreover, there were no associations between rage, dysregulation, affect lability, and distress tolerance on the I-PUTS dimensions. This suggests that the I-PUTS is distinctly capturing urge phenomena, and its ratings are not significantly influenced by co-occurring psychopathology. Comparatively, the PUTS total score exhibited a large negative relationship with distress tolerance and a moderate-to-large positive relationship with anxiety severity. This provides descriptive support of urges as aversive, and suggests that youth who have lower distress tolerance experience greater urges. Meanwhile, the moderate positive relationship between urges and anxiety severity is consistent with other reports of urge phenomenology (Rozenman et al. 2014). The difference between the clinician-rated and self-reported urge ratings may be understood in at least two ways. Youth with TD who have low distress tolerance and/or greater anxiety severity may have inadvertently conflated other uncomfortable somatosensory or somatic anxiety sensations as premonitory urges during self-report ratings on the PUTS. Conversely, the difference between results of the clinician-rated and self-report urge scale may more likely be attributed to the broad assessment window shared by the PUTS, SCARED, and DTS. Indeed, experimental evidence has linked interoceptive awareness with premonitory urges (Ganos et al. 2015), suggesting that these relationships are not likely happenstance and attributable to conflated ratings. Given the importance of urge phenomena in TD, future research should investigate the causality of the relationship between premonitory urges and both distress tolerance and anxiety severity in a multiple time point study that includes both broad and time-specific measures.

Finally, when examining agreement between clinician-rated and self-report rated urges, there was generally good agreement across cases on the I-PUTS and PUTS (78%). However, some youth exhibited poor agreement (22%) as evidenced by reporting urges on one measure but not the other. Youth who had poor agreement exhibited greater attention problems, dysregulated behaviors, and externalizing problems on the CBCL relative to youth who exhibited good agreement. These youth may have experienced difficulty regulating their attention and/or behavior during the assessment, and inadvertently provided discrepant responses. This may account for the variable relationship observed between urge ratings and ADHD severity identified in previous reports. Conversely, these youth may have greater difficulty with urge awareness due to attention problems. In either case, youth with tics who present with attention problems, dysregulated behaviors, and/or externalizing behaviors may benefit from multiple forms of urge assessment to obtain accurate urge information. Additionally, youth with poor agreement also exhibited lower tic severity and anxiety severity. As urges are purported to maintain tic symptoms, youth with less urges might be expected to exhibited lower tic severity—as is observed here. Moreover, youth with less anxiety severity may have less somatic symptoms and in turn possess less interoceptive awareness, which has been linked to premonitory urges in experimental studies (Ganos et al. 2015).

Although providing important information regarding urge phenomenology, several limitations exist. First, the I-PUTS only assessed the number, frequency, and intensity of urges in an attempt to capture parallel dimensions of the YGTSS. There may have been unmeasured aspects of urge phenomenology that influenced findings. Second, the I-PUTS assessed urges for tics endorsed over the past week to parallel the YGTSS. There may have been urges absent from this window that would have been captured by the PUTS. Third, this study did not assess the test-retest reliability of the I-PUTS. Test-retest reliability of the I-PUTS should be examined in future studies. Finally, the significance value was set at p<0.05 for all analyses due to the preliminary nature of measuring different dimensions of premonitory urges on the I-PUTS. Although this may have impacted relationships identified as statistically significant, it would not have influenced the magnitude of associations.

In summary, premonitory urges play an important role in the behavioral model and behavioral treatment of tics. This study identified that the clinician-administered I-PUTS is a reliable and valid measure of urge phenomenology that captures multiple dimensions of premonitory urges. While the PUTS takes a global approach to premonitory urge phenomena, the I-PUTS employs a more individualized approach focused on each individual tic endorsed within the past week. The I-PUTS provides complementary information to the PUTS and assesses the body region associated with premonitory urges, which can further advance phenomenological understanding. While the administration of either self-report (PUTS) and/or clinician-administered (I-PUTS) urge measures can provide useful information, the I-PUTS may be particularly relevant for youth who have attention problems, dysregulated behaviors, and/or externalizing behaviors. Specifically, these youth had poor agreement between clinician-administered and self-report urge ratings, and may likely benefit from a more detailed urge assessment. While the I-PUTS serves as the first systematic evaluation of multiple urge dimensions (number, frequency, and intensity), it is important to consider that additional dimensions of premonitory urge may exist. Thus, there are several possible future directions for research. First, rigorous qualitative research on premonitory urge phenomena in youth and adults with TD may be useful to identify other possible dimensions of premonitory urges. Second, experimental studies could examine the relationship between I-PUTS dimensions and self-report ratings of urge frequency and/or intensity during experimental tic suppression and expression tasks in patients with TD. Finally, the relationship between premonitory urges, distress tolerance, and interoceptive awareness could be examined across multiple time points to better understand the time-course of these related constructs and its impact on tic outcomes.

Acknowledgments

The authors would like to acknowledge the contributions of Elysse Arnold, Amanda Collier, Brittney Dane, and Morgan King, and all the participating families.

Footnotes

Contributors

J. McGuire co-designed the study, assisted in data collection, analyzed the data, and wrote the first draft of the manuscript. N. McBride assisted in data collection, contributed to manuscript preparation, and revised/reviewed/critiqued the manuscript. J. Piacentini contributed to data analytic strategy, and revised/reviewed/critiqued the manuscript. C. Johnco contributed to data analysis, and revised/reviewed/critiqued the manuscript. A. Lewin co-designed the study, and revised/reviewed/critiqued the manuscript. T. Murphy assisted with the design of the study, and revised/reviewed/critiqued the manuscript. E. Storch co-designed the study, and revised/reviewed/critiqued the manuscript.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Achenbach TM, Rescorla L. ASEBA school-age forms & profiles: ASEBA Burlington. 2001. [Google Scholar]
  2. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5. Arlington, VA: American Psychiatric Association; 2013. [Google Scholar]
  3. Banaschewski T, Woerner W, Rothenberger A. Premonitory sensory phenomena and suppressibility of tics in Tourette syndrome: developmental aspects in children and adolescents. Developmental Medicine & Child Neurology. 2003;45:700–3. doi: 10.1017/s0012162203001294. [DOI] [PubMed] [Google Scholar]
  4. Birmaher B, Brent DA, Chiappetta L, Bridge J, Monga S, Baugher M. Psychometric properties of the Screen for Child Anxiety Related Emotional Disorders (SCARED): a replication study. Journal of the American Academy of Child & Adolescent Psychiatry. 1999;38:1230–6. doi: 10.1097/00004583-199910000-00011. [DOI] [PubMed] [Google Scholar]
  5. Birmaher B, Khetarpal S, Brent D, Cully M, Balach L, Kaufman J, et al. The screen for child anxiety related emotional disorders (SCARED): Scale construction and psychometric characteristics. Journal of the American Academy of Child & Adolescent Psychiatry. 1997;36:545–53. doi: 10.1097/00004583-199704000-00018. [DOI] [PubMed] [Google Scholar]
  6. Bloch MH, Leckman JF. Clinical course of Tourette syndrome. Journal of Psychosomatic Research. 2009;67:497–501. doi: 10.1016/j.jpsychores.2009.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Brabson L, Brown J, Capriotti M, Ramanujam K, Himle M, Nicotra CM, et al. Patterned Changes in Urge Ratings with Tic Suppression in Youth with Chronic Tic Disorders. Journal of Behavior Therapy and Experimental Psychiatry. 2015;50:162–70. doi: 10.1016/j.jbtep.2015.07.004. [DOI] [PubMed] [Google Scholar]
  8. Budman C, Coffey BJ, Shechter R, Schrock M, Wieland N, Spirgel A, et al. Aripiprazole in children and adolescents with Tourette disorder with and without explosive outbursts. Journal of Child and Adolescent Psychopharmacology. 2008;18:509–15. doi: 10.1089/cap.2007.061. [DOI] [PubMed] [Google Scholar]
  9. Chen K, Budman CL, Herrera LD, Witkin JE, Weiss NT, Lowe TL, et al. Prevalence and clinical correlates of explosive outbursts in Tourette syndrome. Psychiatry Research. 2013;205:269–75. doi: 10.1016/j.psychres.2012.09.029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Conelea CA, Woods DW, Zinner SH, Budman C, Murphy T, Scahill LD, et al. Exploring the impact of chronic tic disorders on youth: results from the Tourette Syndrome Impact Survey. Child Psychiatry & Human Development. 2011;42:219–42. doi: 10.1007/s10578-010-0211-4. [DOI] [PubMed] [Google Scholar]
  11. Cougle JR, Timpano KR, Fitch KE, Hawkins KA. Distress tolerance and obsessions: an integrative analysis. Depression and Anxiety. 2011;28:906–14. doi: 10.1002/da.20846. [DOI] [PubMed] [Google Scholar]
  12. Crossley E, Cavanna AE. Sensory phenomena: clinical correlates and impact on quality of life in adult patients with Tourette syndrome. Psychiatry Research. 2013;209:705–10. doi: 10.1016/j.psychres.2013.04.019. [DOI] [PubMed] [Google Scholar]
  13. Cubo E, y Galán JMTG, Villaverde VA, Velasco SS, Benito VD, Macarrón JV, et al. Prevalence of tics in schoolchildren in central Spain: a population-based study. Pediatric Neurology. 2011;45:100–8. doi: 10.1016/j.pediatrneurol.2011.03.003. [DOI] [PubMed] [Google Scholar]
  14. Eddy CM, Cavanna AE. Premonitory urges in adults with complicated and uncomplicated Tourette syndrome. Behavior Modification. 2014;38:264–75. doi: 10.1177/0145445513504432. [DOI] [PubMed] [Google Scholar]
  15. Eddy CM, Rizzo R, Gulisano M, Agodi A, Barchitta M, Calì P, et al. Quality of life in young people with Tourette syndrome: a controlled study. Journal of Neurology. 2010;258:291–301. doi: 10.1007/s00415-010-5754-6. [DOI] [PubMed] [Google Scholar]
  16. Enders CK. Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data. Psychological Methods. 2003;8:322. doi: 10.1037/1082-989X.8.3.322. [DOI] [PubMed] [Google Scholar]
  17. Enders CK. Applied Missing Data Analysis. New York: The Guilford Press; 2010. [Google Scholar]
  18. Freeman RD, Fast DK, Burd L, Kerbeshian J, Robertson MM, Sandor P. An international perspective on Tourette syndrome: selected findings from 3500 individuals in 22 countries. Developmental Medicine & Child Neurology. 2000;42:436–47. doi: 10.1017/s0012162200000839. [DOI] [PubMed] [Google Scholar]
  19. Ganos C, Garrido A, Navalpotro-Gómez I, Ricciardi L, Martino D, Edwards MJ, et al. Premonitory urge to tic in tourette’s is associated with interoceptive awareness. Movement Disorders. 2015;30:1198–1202. doi: 10.1002/mds.26228. [DOI] [PubMed] [Google Scholar]
  20. Ganos C, Kahl U, Schunke O, Kühn S, Haggard P, Gerloff C, et al. Are premonitory urges a prerequisite of tic inhibition in Gilles de la Tourette syndrome? Journal of Neurology, Neurosurgery and Psychiatry. 2012;83:975–978. doi: 10.1136/jnnp-2012-303033. [DOI] [PubMed] [Google Scholar]
  21. Gerson AC, Gerring JP, Freund L, Joshi PT, Capozzoli J, Brady K, et al. The Children’s Affective Lability Scale: a psychometric evaluation of reliability. Psychiatry Research. 1996;65:189–98. doi: 10.1016/s0165-1781(96)02851-x. [DOI] [PubMed] [Google Scholar]
  22. Kurlan R, Como P, Miller B, Palumbo D, Deeley C, Andresen E, et al. The behavioral spectrum of tic disorders A community-based study. Neurology. 2002;59:414–20. doi: 10.1212/wnl.59.3.414. [DOI] [PubMed] [Google Scholar]
  23. Kurlan R, Lichter D, Hewitt D. Sensory tics in Tourette’s syndrome. Neurology. 1989;39:731–734. doi: 10.1212/wnl.39.5.731. [DOI] [PubMed] [Google Scholar]
  24. Kwak C, Dat Vuong K, Jankovic J. Premonitory sensory phenomenon in Tourette’s syndrome. Movement Disorders. 2003;18:1530–3. doi: 10.1002/mds.10618. [DOI] [PubMed] [Google Scholar]
  25. Lebowitz ER, Motlagh MG, Katsovich L, King RA, Lombroso PJ, Grantz H, et al. Tourette syndrome in youth with and without obsessive compulsive disorder and attention deficit hyperactivity disorder. European Child & Adolescent Psychiatry. 2012;21:451–457. doi: 10.1007/s00787-012-0278-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Leckman J, Riddle M, Hardin M, Ort S, Swartz K, Stevenson J, et al. The Yale Global Tic Severity Scale: initial testing of a clinician-rated scale of tic severity. Journal of the American Academy of Child and Adolescent Psychiatry. 1989;28:566–73. doi: 10.1097/00004583-198907000-00015. [DOI] [PubMed] [Google Scholar]
  27. Leckman J, Walker D, Cohen D. Premonitory urges in Tourette’s syndrome. The American Journal of Psychiatry. 1993;150:98–102. doi: 10.1176/ajp.150.1.98. [DOI] [PubMed] [Google Scholar]
  28. Leckman JF, Sholomskas D, Thompson D, Belanger A, Weissman MM. Best estimate of lifetime psychiatric diagnosis: a methodological study. Archives of General Psychiatry. 1982;39:879–83. doi: 10.1001/archpsyc.1982.04290080001001. [DOI] [PubMed] [Google Scholar]
  29. Lewin AB, Wu MS, McGuire JF, Storch EA. Cognitive behavior therapy for obsessive-compulsive and related disorders. Psychiatric Clinics of North America. 2014;37:415–45. doi: 10.1016/j.psc.2014.05.002. [DOI] [PubMed] [Google Scholar]
  30. McGuire JF, Kugler BB, Park JM, Horng B, Lewin AB, Murphy TK, et al. Evidence-based assessment of compulsive skin picking, chronic tic disorders and trichotillomania in children. Child Psychiatry & Human Development. 2012;43:855–83. doi: 10.1007/s10578-012-0300-7. [DOI] [PubMed] [Google Scholar]
  31. McGuire JF, Piacentini J, Scahill L, Woods DW, Villarreal R, Wilhelm S, et al. Bothersome Tics in Patients with Chronic Tic Disorders: Characteristics and Individualized Treatment Response to Behavior Therapy. Behaviour Research and Therapy. 2015;70:56–63. doi: 10.1016/j.brat.2015.05.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. McGuire JF, Small BJ, Lewin AB, Murphy TK, De Nadai AS, Phares V, et al. Dysregulation in pediatric obsessive compulsive disorder. Psychiatry Research. 2013;209:589–95. doi: 10.1016/j.psychres.2013.04.003. [DOI] [PubMed] [Google Scholar]
  33. O’Connor K. A cognitive-behavioral/psychophysiological model of tic disorders. Behaviour Research and Therapy. 2002;40:1113–42. doi: 10.1016/s0005-7967(02)00048-7. [DOI] [PubMed] [Google Scholar]
  34. Peugh JL, Enders CK. Missing data in educational research: A review of reporting practices and suggestions for improvement. Review of Educational Research. 2004;74:525–56. [Google Scholar]
  35. Poznanski EO, Mokros HB. Children’s depression rating scale, revised (CDRS-R) Western Psychological Services; Los Angeles: 1996. [Google Scholar]
  36. Reese HE, Scahill L, Peterson AL, Crowe K, Woods DW, Piacentini J, et al. The premonitory urge to tic: measurement, characteristics, and correlates in older adolescents and adults. Behavior Therapy. 2014;45:177–86. doi: 10.1016/j.beth.2013.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Rizzo R, Gulisano M, Pellico A, Calì PV, Curatolo P. Tourette Syndrome and Comorbid Conditions A Spectrum of Different Severities and Complexities. Journal of Child Neurology. 2014;29:1383–9. doi: 10.1177/0883073814534317. [DOI] [PubMed] [Google Scholar]
  38. Rosen PJ, Walerius DM, Fogleman ND, Factor PI. The association of emotional lability and emotional and behavioral difficulties among children with and without ADHD. ADHD Attention Deficit and Hyperactivity Disorders. 2015:1–14. doi: 10.1007/s12402-015-0175-0. [DOI] [PubMed] [Google Scholar]
  39. Rozenman M, Johnson OE, Chang SW, Woods DW, Walkup JT, Wilhelm S, et al. Relationships Between Premonitory Urge and Anxiety in Youth With Chronic Tic Disorders. Children’s Health Care. 2014;44:235–48. doi: 10.1080/02739615.2014.986328. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Simons JS, Gaher RM. The Distress Tolerance Scale: Development and validation of a self- report measure. Motivation and Emotion. 2005;29:83–102. [Google Scholar]
  41. Specht MW, Woods DW, Piacentini J, Scahill L, Wilhelm S, Peterson AL, et al. Clinical characteristics of children and adolescents with a primary tic disorder. Journal of Developmental and Physical Disabilities. 2011;23:15–31. doi: 10.1007/s10882-010-9223-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Steinberg T, Baruch SS, Harush A, Dar R, Woods D, Piacentini J, et al. Tic disorders and the premonitory urge. Journal of Neural Transmission. 2010;117:277–84. doi: 10.1007/s00702-009-0353-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Steinberg T, Harush A, Barnea M, Dar R, Piacentini J, Woods D, et al. Tic-related cognition, sensory phenomena, and anxiety in children and adolescents with Tourette syndrome. Comprehensive Psychiatry. 2013;54:462–6. doi: 10.1016/j.comppsych.2012.12.012. [DOI] [PubMed] [Google Scholar]
  44. Storch E, Murphy T, Geffken G, Sajid M, Allen P, Roberti J, et al. Reliability and validity of the Yale Global Tic Severity Scale. Psychological Assessment. 2005;17:486–91. doi: 10.1037/1040-3590.17.4.486. [DOI] [PubMed] [Google Scholar]
  45. Storch EA, Jones AM, Lack CW, Ale CM, Sulkowski ML, Lewin AB, et al. Rage attacks in pediatric obsessive-compulsive disorder: phenomenology and clinical correlates. Journal of the American Academy of Child & Adolescent Psychiatry. 2012;51:582–92. doi: 10.1016/j.jaac.2012.02.016. [DOI] [PubMed] [Google Scholar]
  46. Storch EA, Lack CW, Simons LE, Goodman WK, Murphy TK, Geffken GR. A measure of functional impairment in youth with Tourette’s syndrome. Journal of Pediatric Psychology. 2007a;32:950–9. doi: 10.1093/jpepsy/jsm034. [DOI] [PubMed] [Google Scholar]
  47. Storch EA, Merlo LJ, Lack C, Milsom VA, Geffken GR, Goodman WK, et al. Quality of life in youth with Tourette’s syndrome and chronic tic disorder. Journal of Clinical Child and Adolescent Psychology. 2007b;36:217–27. doi: 10.1080/15374410701279545. [DOI] [PubMed] [Google Scholar]
  48. Storch EA, Murphy TK, Bagner DM, Johns NB, Baumeister AL, Goodman WK, et al. Reliability and validity of the child behavior checklist obsessive-compulsive scale. Journal of Anxiety Disorders. 2006;20:473–85. doi: 10.1016/j.janxdis.2005.06.002. [DOI] [PubMed] [Google Scholar]
  49. Woods DW, Piacentini J, Chang SW, Deckersbach T, Ginsburg GS, Peterson AL, et al. Managing Tourette Syndrome: A Behavioral Intervention for Children and Adolescents. New York: Oxford University Press; 2008. [Google Scholar]
  50. Woods DW, Piacentini J, Himle MB, Chang S. Premonitory Urge for Tics Scale (PUTS): initial psychometric results and examination of the premonitory urge phenomenon in youths with tic disorders. Journal of Developmental & Behavioral Pediatrics. 2005;26:397–403. doi: 10.1097/00004703-200512000-00001. [DOI] [PubMed] [Google Scholar]

RESOURCES