Development and Modification of an Outcome Measure to Follow Symptoms of Children with Sinusitis

Abstract

Objective:

To develop a parent-reported pediatric rhinosinusitis symptom scale (PRSS) that could be used to monitor symptoms of young children with acute sinusitis in response to therapy

Study design:

We developed an 8-item symptom severity scale and evaluated its internal reliability, construct validity, and responsiveness in children 2 to 12 years of age with acute sinusitis. Parents of 258 children with acute sinusitis completed the PRSS at the time of diagnosis, as a diary at home, and at the follow-up visit at day 10 to 12. Based on psychometric results and additional parent feedback, we revised the scale. We evaluated the revised version in 185 children with acute sinusitis.

Results:

Correlations between the scale and reference measures on the day of enrollment were in the expected direction and of the expected magnitude. PRSS scores at the time of presentation correlated with radiographic findings (P < .001), functional status (p <0.001), and parental assessment of overall symptom severity (p <0.001). Responsiveness (standardized response mean) and test–retest reliability of the revised scale were good (2.17 and 0.75, respectively).

Conclusions:

We have developed an outcome measure to track the symptoms of acute sinusitis. Data presented here support the use of the PRSS as a measure of change in symptom burden in clinical trials of children with acute sinusitis.

Resolution of symptoms is an important goal of antimicrobial therapy in children with acute sinusitis. The extent and rapidity with which antimicrobial therapy results in clinically meaningful improvements, as measured by a symptom scale, can be used as a standardized measure of efficacy in clinical trials.¹ In fact, because there are no reliable physical examination findings or laboratory tests that can be used objectively to follow the course of sinusitis in children, a symptom severity scale is one of the few ways to measure clinical outcome. Use of a symptom scale to assess outcome not only will facilitate comparisons within and between trials, but also will allow results across trials and of subjects within trials to be stratified according to symptom severity at baseline.² Because many children with sinusitis are <5 years of age, there is a need for a parent-reported outcome measure so that outcomes in children who are not able to reliably report their own symptoms can be assessed.³

Two parent-reported symptom scales have been developed,^{4, 5} and both were used in placebo-controlled therapy trials of children with sinusitis. Only one was psychometrically evaluated.⁴ The latter scale was developed by determining what 3 pediatricians considered important for the diagnosis of sinusitis. Because symptoms important for diagnosis may not be suitable for following the course of disease, and because the importance of these symptoms to parents was not assessed, we sought to develop a new scale for use as an outcome measure in studies of children with sinusitis.

The goal of this study was to develop and evaluate a parent-reported symptom scale for young children with acute sinusitis (pediatric rhinosinusitis scale, PRSS) that would enable clinicians and researchers to assess severity of symptoms at the time of diagnosis of acute sinusitis and more accurately document improvement or deterioration of symptoms during treatment.

STUDY DESIGN

An overview of the steps in this study is shown in Figure 1.

Figure 1.

Steps in the development of the Pediatric Rhinosinusitis scale (PRSS)

Conceptual Framework

We developed the scale specifically to be used as an outcome measure rather than as a diagnostic tool. Furthermore, we focused on measuring symptoms rather than overall quality of life because the latter is more prone to variability than measurement of symptoms.⁶, ⁷ In addition, because antimicrobial therapy most directly affects symptoms (rather than quality of life), we hypothesized that a scale focused on symptoms would provide a more direct and sensitive measure of treatment efficacy. We limited our study to children 2 to 12 years of age because scales developed for adults could reasonably be used in adolescents.

Scale Development

To determine which symptoms were most important to parents, we asked 30 parents of children with acute sinusitis and 18 children with acute sinusitis (diagnosed as described below) to complete a survey (written for parents, interview for children) about the presence or absence of these 15 symptoms attributable to acute sinusitis (headache, fever, daytime cough, nighttime cough, runny nose, stuffy nose, irritability, green or yellow mucus from the nose, trouble sleeping, eating less, feeling tired, playing less, bad breath, facial pain and facial swelling). If the symptom was present, we asked them to rate its severity (Did not bother me/my child = 0, Bothered me/my child a little = 1, Bothered me/my child a lot = 2). We also asked parents (and children) to list other symptom(s) that we had not already asked about and to rate its severity. Three new items were suggested resulting in a total of 18 items (“mucus down my throat,” “tearing of the eyes,” “puffy eyes”). We then used the methods developed by Juniper and Guyatt to determine the importance of each of 18 items. In this method, mean “importance” for each symptom is calculated by multiplying the prevalence of that symptom by its mean severity.⁸ Eight symptoms (eating less, bad breath, playing less, facial pain, and facial swelling, mucus down my throat, puffy eyes, and tearing of the eyes) were ranked lowest in importance by both parents and children and were omitted.

Accordingly, the pilot version of the PRSS included 10 symptoms (headache, fever, daytime cough, nighttime cough, runny nose, stuffy nose, irritability, green or yellow mucus from the nose, trouble sleeping, feeling tired) over the preceding 24 hours. We used a 3-point response scale (none = 0, a little = 1, a lot = 2) and obtained the total score by summing the scores on these 10 equally-weighted questions. We also conducted a teleconference with 4 experts on pediatric sinusitis to discuss the instructions, format, and choice of items for the scale. We then administered the 10-item pilot version of the scale to 10 patients with acute sinusitis. Based on the feedback we received during in-depth interviews with these patients, we further modified the formatting and wording of some of the questions.

Study 1 - Evaluation of Version 1.0

We prospectively enrolled 258 English-speaking children aged 2 to 12 years with clinically diagnosed acute sinusitis (see below for definition) presenting to 1 of 6 general ambulatory pediatric clinics in Pittsburgh (4 suburban, 2 urban) during 2 consecutive respiratory seasons (October 2008 to March 2010). The diagnosis of sinusitis was made according to stringently defined a priori clinical criteria consistent with the current guidelines from the American Academy of Pediatrics. Children with persistent upper respiratory tract symptoms [ie, 10-29 days of cough (must be present during the daytime) and/or nasal symptoms (rhinorrhea of any quality or congestion)] which were not improving, or worsening symptoms [substantial worsening of nasal symptoms or cough and/or fever after a period of improvement] were eligible. We excluded children who had received antimicrobial treatment within 7 days preceding presentation, had evidence of another presumed bacterial infection (i.e., acute otitis media or pneumonia), or who had underlying immune deficiency, cystic fibrosis, immotile cilia syndrome, or major developmental delay. Children with asthma were included only if they met inclusion criteria, were not wheezing upon examination, and had nasal symptoms that were worsening or persistent. Children with a history of allergic rhinitis who met the above criteria were included if their respiratory symptoms had worsened acutely. Children were managed at the discretion of their primary-care providers; 217 children (84%) were treated with an antimicrobial agent and the remainder (most of whom had relatively mild symptoms) were observed without immediate antimicrobial treatment. As described previously, all children had sinus radiographs performed on the day of enrollment.⁹ A follow-up visit was scheduled for day 10 to 12. Parents were asked to complete the PRSS at each study visit and as a once-a-day paper diary (in the evening). A score was computed for each day only if all of items on the scale were completed.

We also administered the following reference measures during both study visits: 1. Functional Status Questionnaire-IIR,¹⁰ is a 14-item scale that measures overall health status in children 0 to 16 years of age. The questionnaire asks parents about the presence or absence of key behaviors over the preceding 2-week period. Higher scores indicate more favorable status. For the present study, we modified the questionnaire to ask only about the preceding day, while leaving unchanged the wording, sequence and number of questions. 2. Child Assessment of Pain – Children were asked to assess the severity of their pain using the Bieri Faces pain scale following the developer’s instructions.¹¹ 3. Parental assessment of overall symptom severity – Parents rated the overall severity of their child’s symptoms using a rating scale (numbered 0 to 10) anchored at “Perfect Health” on one side and “Worst Imaginable Health” on the other.

The Institutional Review Board at the University of Pittsburgh approved this study before patient enrollment was initiated.

Modification of the scale

While evaluating version 1.0 of the scale, and despite our initial pilot study to assess the scale, we noted that some parents had difficulty answering the question regarding the color of their child’s nasal discharge. Accordingly, after the first study was completed, we conducted a pilot study to determine the extent parents understood the questions on the scale. Of the 25 parents interviewed, 4 had difficulty understanding the aforementioned question; no notable difficulties were identified with regards to the other questions on the scale. Accordingly, we replaced the question about green nasal discharge with the question, “Does your child have trouble breathing through the nose today”. Parents’ understanding was reassessed in 3 subsequent pilot studies (each with approximately 25 patients). The question: “Did your child have difficulty breathing?” was well understood by parents (perhaps because difficulty breathing is directly observable by parents whereas color of nasal discharge is not). In addition, difficulty breathing is directly related to the child’s quality of life, whereas the color of mucus is not.

We noticed in our first study that approximately 50% of children were rated as having “a lot” of cough, stuffy nose and runny nose (i.e., the highest ranking/at the ceiling). Accordingly, because these items were some of the most critical on the scale, in the revised versions of the scale, we increased the number of response options from 3 (none, a little, a lot) to 6 (none, very little, little, some, a lot, an extreme amount). We piloted and modified the response options to arrive at the wording in version 2.0 of the scale (Figure 2). In summary, version 2.0 differed from version 1,0 in the wording of one question (“colored nasal discharge” in version 1.0 vs. “difficulty breathing in version 2.0) and in the number of response options (3 in version 1.0 vs. 6 in version 2.0).

Figure 2.

Pediatric Rhinosinusitis scale (PRSS) version 2.0

Study 2 - Evaluation of version 2.0

To evaluate version 2.0 of the scale, we prospectively enrolled 186 children aged 2 to 12 years with clinically diagnosed acute sinusitis (as defined above) presenting to sites in Pittsburgh (PA), Madison (WI), Philadelphia (PA), or Bardstown (KY) during two consecutive respiratory seasons (February 2016 to Oct 2017). Inclusion and exclusion criteria were very similar in both studies except that in the second study we also excluded children with a PRSS score of <7 and children with an allergy to amoxicillin-clavulanate. Children were randomized to antibiotics or placebo 1:1. A follow-up visit was scheduled for day 12 to 18. Parents were asked to complete the PRSS at each study visit and as a once-a-day electronic diary (in the evening). This report does not address outcomes related to randomization as the trial is ongoing. The Institutional Review Board at the University of Pittsburgh approved this study before patient enrollment was initiated.

Statistical Analyses

Ceiling effects

We evaluated ceiling effects (defined as proportion of individuals choosing the highest response option at the enrollment visit) for each item and for the total score on each version of the scale.

Responsiveness

To evaluate responsiveness–the ability of the instrument’s scores to change in conjunction with changes in clinical status–we examined the change in scale scores from baseline visit to the follow-up visit. Generally, an instrument is considered responsive when the mean change in scores is large relative to the scores’ variability. We calculated the standardized response mean (SRM) by dividing the mean change in score by the standard deviation of the change. A SRM ≥0.7 usually indicates excellent responsiveness.^12-14 Because of the wide range in ages of children in the study, we examined whether responsiveness (and other psychometric properties of the scale) differed in children <6 and ≥6 years of age.

Reliability

To determine whether all items on the scale related to the same construct, we computed the overall Cronbach alpha for the scale. Only the baseline visit was used for this analysis. In general, a scale with a Cronbach alpha of ≥0.7 is considered to have good internal reliability. To assess test-retest reliability, we calculated the intraclass correlation coefficient¹⁵ between scores on Day 1 and 2 in children whose parent reported that their child was “the same” on the day 2 diary. An ICC of 0.40-0.74 and ≥0.75 indicate fair and good test-retest reliability, respectively.^16,17

Validity

To estimate cross-sectional construct validity, we examined the correlation between PRSS scores and scores on reference measures at the enrollment visit. To estimate longitudinal construct validity, we examined the correlation between change in PRSS scores from baseline visit to the follow-up visit and clinician’s assessment of child’s overall outcome at the time of follow-up. Possible outcomes (and their definitions) at this visit included: “cured” (almost all symptoms resolved), “improved” (better, but not completely), and “failed” (not much better, may need additional treatment).

Minimal Clinically Important Difference (MID), Resolution Score, and Factor Analysis

Because version 2.0 had better validity and responsiveness, we present results from this version in this paragraph. The MID of a scale is the smallest difference in score which parents perceive as beneficial. The MID helps investigators determine whether the observed changes in a particular trial are clinically meaningful. We estimated the MID by examining the median absolute and relative change in score in children whose parents rated them as being “A little better” from one day to the next during the follow-up period on the parental global assessment scale. Although using absolute change is well-established, in a paper on a similar outcome scale we developed for acute otitis media,¹⁸ we argued that relative change was more appropriate.

With each diary entry, we asked parents to rate whether their child was “back to normal health.” The mean score at the time when the parent first noted that their child was back to normal was used to determine the score that best corresponds to resolution of symptoms. This score may be used in the analysis of time to symptom resolution in future studies.

To explore how items could be grouped into subscales, we conducted principal components factor analysis using varimax rotation. The number of factors with an Eigen value of >1.0 generally indicates the number of underlying constructs and correlations coefficient of ≥0.5 are considered significant.

RESULTS

We enrolled 258 and 185 children with a diagnosis of acute sinusitis for studies 1 and 2, respectively. The demographic and clinical characteristics of the children in each study are shown in Table 1. Baseline PRSS scores at entry did not differ significantly by age, sex, race or ethnicity in either study.

Table 1.

Demographic and clinical characteristics of children with acute sinusitis in the population used for development (Version 1.0) and modification (Version 2.0) of the PRSS

Characteristic	Development Dataset (Study 1) N= 258	Modification dataset (Study 2) N= 185
	No. (%) of Children	No. (%) of Children
Mean age, yearsa	6.4 (2.9)	5.6 (2.7)
Sex
Male	131 (51.6)	98 (53.0)
Female	127 (48.1)	87 (47.0)
Race
Caucasian	162 (64.3)	98 (53.0)
African-American	74 (29.5)	63 (34.1)
Other	22 (6.2)	24 (12.3)
Ethnicity
Hispanic	11 (4.3)	19 (10.3)
Non-Hispanic	246 (95.7)	166 (89.7)
Maternal education
Less than high school	11 (4.3)	8 (4.3)
High school graduate/GED	67 (26.0)	54 (29.2)
Some college	75 (29.1)	72 (38.9)
College graduate	103 (39.9)	48 (25.6)
Unknown	2 (0.8)	3 (1.6)
Mean number of days with symptoms	14.3 (5.9)*	14.3 (5.9)*

^aMean (standard deviation), not number of patients presented

Responsiveness

“Headache” and “fever” had suboptimal responsiveness (standardized response means of 0.67 and 0.65, respectively). Because the responsiveness of the scale is of paramount importance in scales designed to follow symptoms, we eliminated these items from analysis in version 1.0; they were not included in Version 2.0. Thus, all data presented excluded these 2 items. Table II shows the responsiveness (as measured by the standardized response mean) for each item and for the scale as a whole for each version of the PRSS. Of note, responsiveness of the scale was similar in children <6 years and ≥6 years of age; SRM, 2.10 and 2.27, respectively for version 2.0).

Table 2.

Responsiveness and percent at ceiling of individual items and total score on the PRSS.

	Responsiveness (standardized response meana)		Percent at ceilingb at baseline
	Version 1.0 (Study 1)	Version 2.0 (Study 2)	Version 1.0 (Study 1)	Version 2.0 (Study 2)
Sleep	0.94	1.41	25.6	16.2
Cough AM	1.25	1.76	47.7	20.5
Cough PM	1.66	1.99	64.0	30.8
Green Mucus/trouble breathing	0.93	1.37	29.8	11.4
Stuffy Nose	1.35	1.27	53.9	10.8
Irritable	0.92	1.26	26.7	7.0
Tired	1.01	1.51	34.5	10.3
Runny nose	1.01	1.34	44.2	9.2
PRSS total score	1.97	2.17d	0.8	0.5e

^aStandardized response mean = change in score divided by the standard deviation of the change. An SRM >0.7 usually indicates excellent responsiveness

^bPercent at ceiling = percent at maximal score for item or scale

^cIn version 1.0 we asked about green mucus; in version 2, we asked about difficulty breathing

^d2.10 and 2.27, respectively in children <6 and ≥6 years of age

^e0.0% and 1.4% in children <6 and ≥6 years of age

Internal and test-retest reliability

Cronbach alpha of versions 1.0 and 2.0 were 0.58 and 0.79, respectively; this did not vary significantly by age (0.80 and 0.77 for children <6 and ≥6 years of age, respectively for version 2.0). Intraclass correlation between day 1 and day 2 scores among children rated as being unchanged by parents on the day 2 phone-call/diary was 0.52 and 0.75 for versions 1.0 and 2.0 of the scale; this did not vary significantly by age (0.78 and 0.70 for children <6 and ≥6 years of age, respectively).

Validity

Correlations between the PRSS and reference measures on the day of enrollment were in the expected direction and of the expected magnitude (Table 3). PRSS scores (version 1.0) at enrollment and radiographic findings were highly correlated (p = 0.002). PRSS scores (on version 1.0) in children who were judged as being “cured”, “improved” or “failed” decreased by 6.15, 3.92, and 3.86 points from baseline, respectively (n= 164, 65, and 13, respectively); there was a significant linear trend between clinician’s assessment at the time of the follow-up visit and the change in PRSS scores (p<.001). Only one child who “failed” was treated with a rescue antibiotic.

Table 3.

Correlations between PRSS scores (Version 1.0) with scores on reference measures at the time of enrollment compared with the predicted values of these correlations.

Reference Measures	Correlation predicted a priori	Correlation observed (V 1.0, Study 1)	Correlation observed (V 2.0, Study 2)
Functional status questionnaire	− 0.6	− 0.53	--
Pain score (child)	0.4	0.21	--
Radiography	N/Aa	0.23	--
Overall assessment (parent)	− 0.4	− 0.38	−0.41b

P<0.05 for all correlations

^aNo prediction made a priori

^b0.43 and 0.37 for children <6 and ≥6 years of age, respectively

MID, Resolution Score, factor analysis (Version 2.0)

The absolute and relative MID were 3.0 (1.0-5.0) or 18.2% (4.6-31.8), respectively. Absolute and relative resolution scores were 3.0 (1.0-5.5) and 87.2% (72.3-95.3), respectively. Factor analysis supported a three-factor solution with nasal symptoms (stuffy nose, difficulty breathing through the nose, and runny nose), cough (daytime cough, nighttime cough, sleeping difficulty) and malaise (tired, fussy).

DISCUSSION

The PRSS appears to be a valid and reliable measure of symptom burden in children with acute sinusitis. The scores correlated in the expected direction and magnitude with reference measures. Although radiographs were performed primarily to help address other aims of our first study,⁹ the strong association between the scores of the PRSS and results of the sinus radiograph further supports the validity of the scale. Responsiveness and test–retest reliability were good and comparable with other parent-reported symptoms scales.¹⁹ Psychometric properties of the scale were similar in children <6 years of age and children >6 years of age.

We propose several ways of analyzing PRSS scores. In the first and preferred approach, symptom burden over time would be compared between treatment groups using generalized estimating equations. In the second approach, time to achievement of a specified score could be compared between treatment groups. In the third approach, an improvement in symptom score by a value equal to the MID (e.g., a change of ≥20% in PRSS scores) could be compared between treatment groups at one or more designated time points.²⁰-²² In the latter 2 approaches, power to detect group differences is lower because the data are in essence dichotomized. As most therapies will affect both nasal symptoms and cough, the total score will be preferable for most analyses despite 2 to 3 factors being present.

Importantly, our results do not suggest using the PRSS score alone for diagnosing sinusitis. Rather, the scale was designed primarily to allow researchers to more accurately follow symptoms of groups of children over time. If the scale is used to follow symptoms of an individual child in a clinical setting, it would be preferable to use relative (rather than absolute) changes in score.

This study is limited in several respects. The total score on the scales was obtained by adding the scores for each item on the scale. This assumes that items on the scale have an equal weight or importance. Although this may be viewed as a limitation, previous studies suggest that differential weighting not only adds complexity, but also contributes relatively little to the predictive ability of the scale.²³

The PRSS appears to effectively measure both overall functional status and severity of symptoms in children with sinusitis. Changes in scores appear to be useful in the measurement of symptom improvement or deterioration. These results support future use of the PRSS as a measure of outcome in clinical studies of young children with acute sinusitis.

Abbreviations:

PRSS

Pediatric Rhinosinusitis Symptom Scale