Development and Content Validity Testing of Patient-Reported Outcome Items for Children to Self-Assess Symptoms of the Common Cold

Abstract

Background and objective

No pediatric patient-reported outcome instruments specific to the common cold are found in the literature. This study involved development and content validity testing of patient-reported outcome items (questions and response options) assessing cold symptoms in children aged 6–11 years.

Methods

Draft patient-reported outcome instructions, items, response scales, and recall periods were developed based on the literature and existing measures. Qualitative interviews were conducted with children (n = 39) who were currently (n = 31) or had recently (n = 8) experienced a cold and ten parents of a subset of children aged 6–8 years. The interviews were conducted over two rounds and included open-ended concept elicitation questioning, a free-drawing task, a card sorting task, and a task involving circling parts of the body, followed by cognitive debriefing of draft items. Thematic analysis of verbatim transcripts was performed to analyze the qualitative data. The findings were used to support revisions to the draft patient-reported outcome.

Results

Ten symptom concepts were reported by the children during concept elicitation. The creative tasks helped the children to describe their symptoms, generally using consistent language to do so, irrespective of age. Nineteen patient-reported outcome items were developed and subject to cognitive debriefing. Debriefing with both children and parents informed several small revisions and provided evidence that the majority of children found most patient-reported outcome items easy to understand, and that the items were mainly interpreted consistently and as intended.

Conclusions

This in-depth qualitative study has supported identification of relevant symptom concepts and the development and refinement of patient-reported outcome items to assess those concepts. The findings support the content validity of the items and suggest that they can be used with confidence in children aged 9 years and older. For children aged 6–8 years, it is recommended the items are administered with initial adult supervision to explain the more difficult concepts or through parent/interviewer administration.

Key Points

A pediatric patient-reported outcome has been developed to assess the symptoms of the common cold in children aged 6–11 years.

Combined concept elicitation and cognitive debriefing interviews employing creative methods were conducted with 39 children and ten of their parents.

The findings support the content validity of the final patient-reported outcome.

Introduction

The common cold, a viral infection of the upper respiratory tract, is the most frequently occurring human illness for which approximately 25 million individuals seek medical attention per year in the USA [1–3]. Children experience three to ten colds annually, accounting for 22 million missed days of school [4, 5]. The most frequently experienced symptoms are nasal, including rhinorrhea (runny nose), nasal congestion (stuffy nose), and sneezing. Sore throat can be present from the first day of symptoms, and a cough may develop on or around the fourth or fifth day. Low-grade fever, headache, and malaise can also be present [6]. Many children and their parents/caregivers lose sleep because the cough associated with the cold keeps them awake at night [3].

To conduct trials evaluating the efficacy of cold medications in children, assessments of common cold symptoms that can form trial endpoints are required and they should be appropriately developed and validated. Assessing cold symptoms in young children has the challenge that not all children will experience the same symptoms, which are generally mild and transient. Moreover, several symptoms (e.g., headache and chest congestion) are not directly observable, while others are highly variable (e.g., cough and sneezing). Consequently, assessment of symptom frequency or severity is best based on direct report from children through use of pediatric patient-reported outcome (PRO) instruments. Pediatric PRO instruments are available to assess conditions with symptoms that overlap with the common cold. These include the Strep PRO, which assesses sore throat [7] and the Pediatric Rhinoconjunctivitis Quality of Life Questionnaire [8], which assesses runny nose and sneezing. However, none providing a comprehensive assessment of cold symptoms are found in the literature.

Well-established methods for the development and validation of adult PROs are detailed in the PRO guidance for industry provided by the US Food and Drug Administration [9] and the reflection paper on health-reported quality of life by the European Medicines Agency [10]. Developing pediatric-focused assessments adds complications beyond those encountered in PRO development for adults [11, 12]. Foremost is the need to ensure, through rigorous qualitative research, that the concepts being assessed and the instrument instructions, items, response options, and recall periods are relevant and appropriate for the disease population of interest (context of use). Given the developmental changes in children’s physical, cognitive, and functional abilities, it is also important to conduct qualitative research within narrow age ranges to ensure that the PRO is appropriate for all intended ages. In the present research, the inherent difficulties in pediatric PRO development work have been addressed using concept elicitation and cognitive debriefing activities tailored to overcome some of those challenges.

The primary objective of this qualitative research was to conduct content validity testing of draft items that may comprise a pediatric PRO instrument assessing cold symptoms in children aged 6–11 years. The ultimate intention is that selected items from the pool of items could be included as endpoints in clinical trials to evaluate the efficacy of cough and cold medicines in children aged 6–11 years.

Methods

Sample and Recruitment

This was a qualitative interview study, conducted with 39 children aged 6–11 years, who were either currently experiencing a cold (n = 31) or had experienced a cold in the past 14 days (n = 8) but were otherwise healthy. The latter group was included because it is expected that by the end of the study period for a cold treatment trial, which could potentially use the PRO items, at least some of the participants will no longer be experiencing symptoms; therefore, it was considered important to test that children not experiencing symptoms were still able to understand and respond to the questions. Ten parents of participants aged 6–8 years were also interviewed (separately from their child) regarding how they thought their children would understand the items. Participants were recruited by a specialist patient recruitment agency from two locations in the USA: Los Angeles, California and Chicago, Illinois. The recruitment agency contacted parents in their database and informed them that if their child was experiencing a common cold (or had done so in the last few days) and would be willing and able to participate in an interview, they should respond. The recruitment agency then completed a screener with each parent/guardian to confirm eligibility. Written informed consent was then obtained from the parent/guardian and assent from the child prior to any other study activities.

The children had to have experienced at least one of the six symptoms of interest (headache, sinus pain/pressure, body aches/muscle aches, sore throat, cough, or chest congestion), in the past 14 days, as reported by their parent, to be considered for inclusion. The children were also required to have experienced at least one other symptom of the common cold (headache, sinus pain/pressure, body aches/muscle aches, sore throat, cough, chest congestion, nasal congestion [stuffy nose], runny nose, sneezing, or low fever). Diagnosis of cold symptoms was not confirmed by a clinician because parents do not usually take a child with the common cold to the doctor.

Children were excluded from the study if they had experienced cold symptoms continuously for more than 14 days; were experiencing severe cold symptoms such as high fever; were taking antibiotics; or had been diagnosed with sinusitis, otitis media, tonsillitis, strep throat, laryngitis, bronchitis, pertussis, or pneumonia. These children could be experiencing symptoms due to other conditions, thus their symptom experience could be different from that of the common cold. They were also excluded if they had mental, psychiatric, or cognitive conditions that might affect their ability to participate in the interview. Recruitment quotas ensured that the sample included children with a range of demographic characteristics and different combinations of the symptoms of interest, thus providing the ability to evaluate whether there were any age-related differences in comprehension or relevance of symptom concepts.

Interview Procedure

The interviews were split over two rounds to allow findings from the first round to be used to revise the original items for testing in the second round. Both children and parents were interviewed for approximately 1 h, but with breaks, particularly for the children. Parents were interviewed separately from their children. The interviews were conducted in either a research facility or in a room in a public building such as a library that was located conveniently for the participants. Each interview included concept elicitation activities (open-ended, exploratory questioning) and cognitive debriefing questions (testing of comprehension and relevance of the questions) [2, 13–15]. Experienced pediatric interviewers trained in qualitative interviewing techniques used a semi-structured interview guide to first elicit how children spontaneously talk about their cold symptoms to inform the development of the PRO items. Open-ended questions were designed to encourage spontaneous quotes regarding the children’s experience of cold symptoms using their own terminology [2, 14]. More direct questions were then asked, to ensure that feedback on all symptoms of interest was obtained.

Four creative tasks were employed during concept elicitation to encourage children to talk spontaneously about cold symptoms: free drawing, circling parts of the body on a diagram, card sorting, and a child–parent completion task. The free-drawing task asked children to draw what it feels like to have a cold and describe their drawing aloud. This method often helps nervous children to relax and draws out comments that might not be elicited from formal questioning [11, 16]. The ‘circle parts of your body’ task involved children circling areas of an outline of a human child’s upper body that felt different to them while experiencing the cold and describing how each felt.

The children were asked to sort a set of cards into “the order you think they should go in” that had the following response options associated with symptom severity printed on them: “not bad at all”, “a tiny bit bad”, “a little bad”, “bad”, “pretty bad”, “very bad”, and “really bad”. The interviewer presented the cards in a random order to establish whether children understood the severity continuum. The goal was to select five child-appropriate response options that align with the validated response scale in an adult PRO instrument for upper respiratory cold symptoms: none, very mild, mild, moderate, and severe [16]. Because the common cold is a generally mild condition, efforts were made to include more gradation toward the milder end of the response continuum (i.e., include an equivalent to very mild).

Following the concept elicitation questioning, children were asked to complete the draft PRO items using a think-aloud process during which they speak aloud their thoughts as they answer each question [17, 18]. This was followed by detailed cognitive debriefing questions to assess the content validity and clarity of each question, instruction, response scale, and recall period [2, 14, 19]. Ten parents of children aged 6–8 years were also asked whether they felt their children would be able to understand the instructions, questions, response options, and recall periods.

Generation and Refinement of Instrument

Draft instructions, items, response scales, and recall periods were based on the pediatric rhinitis literature, previously used measures of cold symptoms in adults, other pediatric PRO measures, and consideration of best practices in developing PRO items suitable for completion by children aged 6–11 years [9–11, 14, 17]. Efforts were made to keep items as short and simple as possible and avoid the use of double enquiry or overly clinical terminology in keeping with pediatric PRO best practice [11, 12, 15].

The children’s responses to the items during debriefing were recorded and summarized descriptively to provide insight on which items used the full response scale and may be more discriminating. Items with a strong floor (i.e., a high proportion of children choosing the lowest response) or ceiling (i.e., a high proportion of children choosing the highest response) effect and those with any one response option being overly favored were considered later with all qualitative evidence and their clinical relevance when considering whether any items might be candidates for deletion.

Qualitative Analysis

Interviews were audio recorded and transcribed verbatim. Qualitative analysis was performed by authors CM and KB with the oversight and guidance of RA. Qualitative analysis of verbatim transcripts involved sorting quotes by conceptual domain using thematic analysis and content analysis using Atlas.ti [20]. Thematic analysis involved reviewing transcripts and highlighting and coding quotes related to specific cold symptoms and concepts. Quotes related to a specific symptom/concept were then grouped from across transcripts during analysis and the findings summarized. This was the primary method of analysis. Content analysis was used where counts summarized findings. For example, the count of how many children depicted certain symptoms in the drawing task or the counts of how many children circled different parts of the body are examples of content analysis. Although assessing concept frequency is not a primary purpose of qualitative research, the number of children who mentioned a given concept was recorded to provide an indication of the relative importance of each concept. To ensure that the concepts elicited were fully explored, conceptual saturation was assessed: defined as the point at which no new relevant or important information emerges with the collection of more data [21]. Saturation was evaluated by splitting the child transcripts into four groups, each comprising ten or nine transcripts. The groups were based on the chronological order the interviews were conducted in, thus effectively random. The concepts reported in the first group of transcripts were then compared to those reported in the second group of transcripts. The concepts that emerged from the first two were then compared with the third and the concepts reported in the first three were compared with the final group. The point at which no new symptom concepts emerged was the point at which saturation was deemed to have been achieved.

Results

Demographics

Child Sociodemographic Characteristics

Thirty-nine children aged 6–11 years were included in the final study sample, with 19 participating in the first round of interviews and 20 in the second round. The interviews were conducted between February and June 2011 at a research facility or in quiet area of a public building (e.g., library). An additional 6-year-old child was unable to concentrate enough to participate. Sociodemographic characteristics for the total sample and sub-samples are presented in Table 1. Mean age was 8.2 years (range from 6 to 11 years), 19 (49%) were female. Just over half the sample were Caucasian (53.9%; n = 21), but other races and ethnicities were well represented.

Table 1.

Child sociodemographic characteristics and cold symptoms assessed by parent at screening

	Current cold (N = 31)	Recent cold (N = 8)	Total sample (N = 39)
Age (years)
Mean	8.2	8.4	8.2
Median	8	8	8
Range	6–11	6–11	6–11
Sex, n (%)
Male	16 (51.6)	4 (50.0)	20 (51.3)
Female	15 (48.4)	4 (50.0)	19 (48.7)
Ethnicity, n (%)
White/Caucasian	15 (48.4)	6 (75.0)	21 (53.9)
Black/African American	4 (12.9)	1 (12.5)	5 (12.8)
Hispanic/Latino	10 (32.3)	0	10 (25.6)
Multiracial	2 (6.4)	1 (12.5)	3 (7.7)
Current living status, n (%)
Living with both parents	29 (93.5)	7 (87.5)	36 (92.3)
Living with one parent	2 (6.5)	1 (12.5)	3 (7.7)
Number of children experiencing each cold symptoms of interest as reported by parents on screener, n (%)
Cough	29 (93.5)	6 (75.0)	35 (89.7)
Chest congestion	14 (45.2)	6 (75.0)	20 (51.3)
Headache	15 (48.4)	4 (50.0)	19 (48.7)
Sore throat	15 (48.4)	4 (50.0)	19 (48.7)
Body/muscle aches/weakness	15 (48.4)	3 (37.5)	18 (46.2)
Sinus pain/pressure	7 (22.6)	4 (50.0)	11 (28.2)
Number of children experiencing additional cold symptoms as reported by parents at screening, n (%)
Nasal congestion (stuffy nose)	18 (58.1)	6 (75.0)	24 (61.5)
Sneezing	19 (61.3)	5 (62.5)	24 (61.5)
Runny nose	14 (45.2)	2 (25.0)	16 (41.0)
Low fever	10 (32.3)	6 (75.0)	16 (41.0)

Child Cold Characteristics

The cold symptoms reported by each child’s parent during screening are shown in Table 1. Nearly all children had a cough as part of their cold (89.7%; n = 35). Four symptoms of interest (chest congestion, headache, sore throat, and body/muscle aches) were reported for approximately 50% of the sample. Only sinus pain/pressure was reported by a lower proportion of the sample (28.2%, n = 11). The most commonly reported symptoms after coughing were nasal congestion (61.5%; n = 24) and sneezing (61.5%; n = 24).

Sociodemographic Characteristics of Parents who were Interviewed

Ten parents of children with current colds were interviewed; mean age was 36.7 years (range from 31 to 49 years), and nine (90%) were female. Five parents (50%) were White/Caucasian, four (40%) were Hispanic/Latino, and one was Asian. Two parents (20%) had completed a college or university degree, while seven (70%) had completed some college.

Concept Elicitation Findings: Descriptions of Cold Symptoms by Children

The children were asked to describe their cold symptoms through open-ended questions such as “Tell me about your cold” and “What types of things happen when you have a cold?”. Table 2 summarizes the findings and descriptions used for each symptom across the sample, which were generally consistent. Conceptual saturation was achieved—all symptoms were reported in the first saturation checking group. Cough was the most commonly reported cold symptom, reported spontaneously by the majority (31/39) and by a further 4/39 only when probed directly. Chest symptoms, reported by 32/39 children (12 spontaneously and 20 only when probed directly), were most commonly described in terms of the chest feeling “full of mucus” (7/39), “hurting” (6/39), or a “squeezing” (3/39)/”tight” feeling (3/39). Twenty-eight of the 39 children spontaneously reported experiencing headaches, most often referring to their head “hurting”, despite only 18/39 parents reporting this symptom during screening. Thirty-six of the 39 children reported experiencing a sore throat (29 spontaneously, seven only when probed), most describing their throat “hurting”, while 25/39 children talked about feeling aches or weakness in their body (13 spontaneously, 12 only when probed). Four parents said their children (aged 6–8 years) would simply say that their body hurts.

Table 2.

Symptom concepts reported, number of reports, and example quotes

Symptom	Spontaneously reported (S)	Reported when asked directly (P)	Words or phrases used	Example quotes
Cough	31	4	‘normal/regular’ (S = 7), ‘dry’ (S = 3), ‘deep’ (S = 1), ‘loud’ (S = 1), ‘small’ (S = 1), ‘hacking’ (S = 1)	Q: “How bad has your cough been?”“It’s like a normal cough. How like everybody coughs” (7-year-old boy) “Um, it just - it hurts when I cough because of my throat’s dry.” (11-year-old boy). “I wrote it small because it was just a small cough.” (11-year-old)
Chest congestion	11	1	‘full of mucus’ (S = 7), ‘heavy’ (S = 3; P = 1), ‘congested’ (S = 1)	Q: “And you said that you have a lot of mucus?”“Yeah. Each time I cough a lot, like really hard, mucus just comes up my throat.”“OK. And where is the mucus to begin with?”“In my stomach, like my chest.” (9-year-old boy) Q: “What would you call that stuff in your chest?”“It’s like, it feels heavy and stuff” (8-year-old boy) “Because sometimes I get congested.”“Congested? Where do you feel congested?”“I feel congested, um, whenever I cough sometimes.” (8-year-old girl)
Chest pain	8	0	‘hurts’ (S = 6), ‘pain’ (S = 2)	“It feel - it hurts in my - I have chest pain.” (8-year-old girl) “Like it hurts and I have pain. I try to feel it and then breathe it in.” (7-year-old girl)
Chest: tightness	7	0	‘squeezing’ (S = 3), ‘tight/squished’ (S = 3), ‘pinched’ (S = 1)	Q: “Has your chest felt tight or heavy today?”“It feels like it’s tight, like it’s squeezing something.” (10-year-old girl) Q: “And does it ever feel, um, do you ever feel the mucus in your chest and your stomach?”“Yeah.”“OK, and what does that feel like?” “Like tightening up and then it just comes out …” (9-year-old boy)
Chest: difficulty breathing	3	0	‘hard to breathe’ (S = 1), ‘less lung capacity’ (S = 1), ‘had to take a big breath’ (S = 1)	Q: “You did circle the chest. How does your chest feel?”“Kind of like I’m being squashed in between two people. It’s like hard - not hard, like, really tight together. Hard to breathe.” (11-year-old boy) Q: “No? Did you ever feel like there is stuff in your chest?”“Not stuff in my chest, it’s like - like it’s less lung capacity. Or air.” (11-year-old boy)
Chest: other	2	0	‘feels weird’ (S = 2)	Q: “Your chest? Can you tell me how your chest feels right now?”“Sort of weird … It feels like really weird and it feels like if my heart’s going to pop out.” (7-year-old girl)
Headache	28	2	‘head hurts’ (S = 22), ‘pounding’ (S = 3), ‘dizzy’ (S = 3), ‘migraine’ (S = 1), ‘throbbing’ (S = 1), louder/blurrier (S = 1)	“My head hurts. I like, feel really bad.” (7-year-old girl) “It feels like your head is just crying. … You know how when you get a stomach ache, how it hurts. It’s like your head, but it doesn’t hurt as bad. And more it gives you like a dizzy feeling.” (9-year-old girl)
Sore throat	29	7	‘hurt’ (S = 18), ‘dry’ (S = 10), ‘itchy/scratchy’ (S = 7), ‘sore’ (S = 5), ‘burns’ (S = 5), ‘squeezing/pinching’ (S = 3)	“It felt like my throat - my throat was so dry. It felt like it was hurting a lot.” (9-year-old boy). “Uh, first, uh, I started out with my, uh, throat hurting.” (7-year-old boy)
Body/muscle aches or weakness	13	12	‘weakness’ (S = 8), ‘tired/asleep limbs’ (S = 5), ‘hurt’ (S = 5), ‘stinging’ (S = 1), ‘tingling’ (S = 1), ‘achy’ (S = 1)	“… sometimes when you have a cold you can just feel really weak, like achy.” (11-year-old girl) Q: “And how about your muscles? How do your muscles feel?”“Well, they feel a little weak because I have a cold.” (10-year-old girl)
Sinus pain/pressure	7	5	‘pushing’ feeling around eyes’ (S = 3), ‘hot/red’ (S = 3), ‘burned’ (S = 3), ‘eyes feeling ‘stretched’’ (S = 1), ‘eyes coming out’ (S = 1), ‘something was in there’ (S = 1), ‘stuff covering it’ ‘numb’ ‘dry (S = 1), ‘pluggish’ ‘squeezing’ (S = 1), ‘warm’ (S = 1)	Q: “Can you tell me how your face feels right now around your eyes and your nose?”It feels a little bit heavy … Like it feels like this part is pushing down these parts.”“So you’re kind of pointing at - by your eyebrows?”“Yeah. Because it - it feels like it’s pushing down my eyes.” (9-year-old boy) Q: “OK. Anything with how your body feel - felt?”“No. But my face - or my nose, it was like it burned. It felt like it was like on fire.” (11-year-old boy) Q: “So what about your eyes? Why did you circle the eyes?”“Because um, they were like hurting a lot and wouldn’t stop hurting … Like, sometimes like, in computer lab my head feels like it’s - my forehead feels like it’s thumping and that’s how my eyes feel, like they’re come - coming out.” (6-year-old girl)

Although 11/39 parents reported that their children had sinus pain/pressure during screening, only 8/39 children talked about sinus pain/pressure spontaneously, with a further five reporting the symptom when asked directly but did not elaborate, making it unclear whether they truly understood the concept. Children described their sinus pain with a wider variety of descriptors than for other symptoms, referring to their face feeling “pushed”, “stretched”, or “pluggish” or describing a “squeezing” sensation around their nose and eyes or a pain that was “burning” or “hot”. Four of the ten parents of children aged 6–8 years did not believe their child would be capable of explaining sinus pain.

Drawing Task

Of the 32 children that completed the drawing task, 13 spontaneously drew at least one symptom of interest. Twelve children (aged 6–11 years) drew themselves coughing, an example of which is shown in Fig. 1. Two children drew what happens during their cold when they get headaches: a 9-year-old girl showed herself with a “cold rag” on her head and an 11-year-old girl drew herself saying “leave me alone” because she did not want to play with other children when she had a headache. One 11-year-old boy drew himself with chest congestion, clearly showing mucus built up in his chest. One 9-year-old boy highlighted his throat in the picture, explaining: “Here’s the throat where the mucus is.”

Q: “Do you want to just explain to me what you did?” “Um, (inaudible) like, it goes through your throat and then up the mouth.” “OK and that’s the mucus right there?” “Yeah. And then the tissue on the nose, like blowing. That’s all I really think of, just like blowing your nose and coughing up stuff.”

Fig. 1.

Child’s drawing of coughing (11-year-old boy)

Circling Parts of the Body Task

Eighteen children (aged 6–11 years) circled the throat on the diagram, many saying it hurt during their cold. Thirteen children marked the head, explaining that they had experienced headaches. Nine children marked their chest on the diagram, and seven of nine children talked about coughing when asked why they identified the chest area. One 9-year-old child said he had marked his chest because he felt “mucus” in there. Six children marked the arms, reporting “weakness” and “achy” feelings. Two children marked the areas around their nose and eyes (two girls aged 6 and 11 years), with the 11-year-old child saying it felt “like a headache but wasn’t”, which suggests she may have been experiencing sinus pain but had difficulty articulating this.

Development of Draft Patient-Reported Outcome Items, Response Scales, Recall Periods, and Illustrations of Symptoms.

Item Content

Nineteen PRO items were tested in the first round of interviews with alternative wording used on showcards. Subsequent changes were tested in the second round. A tracking matrix was developed to document the changes made to the item content between interview rounds and present the rationale for the final items (Table 3). The items assessed the key cold symptom constellations reported and targeted for study including cough, chest congestion (chest congestion, chest tightness, chest pain, difficulty breathing), headache, sore throat, body/muscle aches, and sinus pain/pressure.

Table 3.

Item tracking

Item	Item tested in first round of interviews	Action following feedback from first round	Item tested in second round of interviews	Action following feedback from first round	Final item
Sinus: face hurt	“Right now, how much does your face hurt around your nose and eyes?”	No change	“Right now, how much does your face hurt around your nose and eyes?”	Delete item OR retain all three items for psychometric validation	N/A
Sinus: painful	“How painful was your face around your nose and eyes today?”	Changed to “How much pain did you feel in your face around your nose and eyes today?”	“How much pain did you feel in your face around your nose and eyes today?”	Retain item for pilot testing and psychometric validation	“How much pain did you feel in your face around your nose and eyes today?”
Sinus: feel full	“Right now, how much does your face feel full around your nose and eyes?”	Deleted	N/A	N/A	N/A
Sinus: how tight	“From when you woke up this morning until now, how tight did it feel around your eyes and nose?”	No change	“From when you woke up this morning until now, how tight did it feel around your eyes and nose?”	Delete item OR retain all three items for psychometric validation	N/A
Body ache: body hurt	“Right now, how much does your body hurt?”	Changed to “Right now, how much does your cold make your body hurt?”	“Right now, how much does your cold make your body hurt?”	Delete item	N/A
Body ache: body ache	“From when you woke up this morning until now, how much did your body ache?”	Changed wording from ‘body ache’ to ‘arms and legs ache’ because most children talked about their limbs aching in round 1. Reworded to include ‘does your cold’ for the same reasons as Q16 above	“Right now, how much does your cold make your arms and legs ache?”	Retain item for pilot testing and psychometric validation	“Right now, how much does your cold make your arms and legs ache?”
Body ache: body weak	N/A	N/A	“From when you woke up this morning until now, how much did your cold make your arms and legs feel weak?”	Retain item for pilot testing and psychometric validation	“From when you woke up this morning until now, how much did your cold make your arms and legs feel weak?”
Chest: how tight	“When you took a deep breath right now, how tight did your chest feel?”	No change	“When you took a deep breath right now, how tight did your chest feel?”	Retain item for pilot testing and psychometric validation	“When you took a deep breath right now, how tight did your chest feel?”
Chest: how full	“When you took a deep breath right now, how full of stuff did your chest feel?”	Deleted as did not appear to be as strong as the other items	N/A	N/A	N/A
Chest: breathe deep	“For all of today, how hard was it to breathe air deep into your chest?”	No change	“For all of today, how hard was it to breathe air deep into your chest?”	Retain item for pilot testing and psychometric validation	“For all of today, how hard was it to breathe air deep into your chest?”
Chest: heavy frequency	“From when you woke up this morning until now, how much of the time has your chest felt heavy?”	No change	“From when you woke up this morning until now, how much of the time has your chest felt heavy?”	Retain item for pilot testing and psychometric validation	“From when you woke up this morning until now, how much of the time has your chest felt heavy?”
Headache: head hurt	“Right now, how much does your head hurt?”	Order of Q18 and Q19 changed to test whether children would still have a preference for the wording of ‘headache’, vs ‘head hurts’	“Right now, how much does your head hurt?”	Retain item for pilot testing and psychometric validation	“Right now, how much does your head hurt?”
Headache: headache	“This afternoon, how bad was your headache?”		“This afternoon, how bad was your headache?”	Retain item for pilot testing and psychometric validation	“This afternoon, how bad was your headache?”
Throat: hurt	“Right now, how much does your throat hurt?”	No change	“Right now, how much does your throat hurt?”	Retain for pilot testing and psychometric validation	“Right now, how much does your throat hurt?”
Throat: bad	“Right now, how bad is your sore throat?”	No change	“Right now, how bad is your sore throat?”	Retain for pilot testing and psychometric validation	“Right now, how bad is your sore throat?”
Throat: swallow spit	“Right now, how much does it hurt to swallow your spit?”	Removed ‘spit’ due to parent feedback that it is not required in this item Recall period was also extended from ‘right now’ to ‘today’ to allow children to recall over a longer period of time	“Today, how much did it hurt to swallow?”	Retain for pilot testing and psychometric validation	“Today, how much did it hurt to swallow?”
Throat: how sore	“From when you woke up this morning until now, how sore has your throat felt?”	No change	“From when you woke up this morning until now, how sore has your throat felt?”	Delete as children prefer wording of ‘how bad’ or ‘how much your throat hurts’	N/A
Cough: how bad	“Right now, how bad is your cough?”	Changed recall from ‘right now’ to ‘today’ and moved to end of question so that it could be directly compared to the showcard item with the same wording	“How bad is your cough today?”	Retain for pilot testing and psychometric validation	“How bad is your cough today?”
Cough: how hard	“How hard have you coughed today?”	No change	“How hard have you coughed today?”	Retain for pilot testing and psychometric validation	“How hard have you coughed today?”
Cough: frequency	“From when you woke up this morning until now, how much have you been coughing?”	No change	“From when you woke up this morning until now, how much have you been coughing?”	Retain for pilot testing and psychometric validation	“From when you woke up this morning until now, how much have you been coughing?”
Cough: cough up stuff	“From when you woke up this morning until now, how much stuff did you cough up?”	Deleted as did not seem relevant to children and made some younger children think about vomiting	N/A	N/A	N/A
Cough: keep awake	“Last night in bed, how much did your cough keep you awake?”	Deleted given that ‘wake up’ and ‘fall asleep’ items ask about more specific concepts	N/A	N/A	N/A
Cough: wake up	“Last night in bed, how much of the time did your cough wake you up?”	No change	“Last night in bed, how much of the time did your cough wake you up?”	Retain for pilot testing and psychometric validation	“Last night in bed, how much of the time did your cough wake you up?”
Cough: fall asleep	“Last night in bed, how hard was it to fall asleep because of your cough?”	No change	“Last night in bed, how hard was it to fall asleep because of your cough?”	Retain for pilot testing and psychometric validation	“Last night in bed, how hard was it to fall asleep because of your cough?”

N/A Not applicable

Response Scales

Two pictorial scales were tested along with the verbal descriptors for each response option: circles of increasing size to indicate severity, and boxes that became gradually filled to indicate frequency or “how much” (Fig. 2). Similar diagrams have been found to aid understanding of the gradation in a response scale in pediatric PRO assessments [17].

Fig. 2.

Example of patient-reported outcome items tested during cognitive debriefing activities

Recall Period

Shorter recall periods (e.g., “since you woke up this morning”) are preferable for pediatric PROs as children can have difficulty recalling accurately over longer timeframes [11, 12]. Tying recall to a concrete event can also help younger children remember [12]. Clinical trials involving the assessment of the common cold may require multiple PRO assessments throughout a given day, thus requiring short well-defined recall periods. Several recall periods were tested for understanding and appropriateness for different symptoms, all for periods of less than 24 h. For immediate assessments of symptom severity, “right now” was tested with some symptoms.

Illustrations of Symptoms

An image of a child depicting the symptom accompanied each item to help children correctly think of the appropriate symptom. The child in the image was intended to be as neutral as possible in terms of sex, race, ethnicity, and emotion being portrayed, to reduce the risk that any child would not identify with the image or the response would be influenced by emotion [17].

Cognitive Debriefing Results and Refinement of the Draft Items

Cognitive Debriefing Results for the Item Content

Across the two rounds of cognitive debriefing, the items were largely well understood by children (Table 4 shows chest symptoms as an example). Items asking about cough, sore throat, and headache were the best understood and were most consistently interpreted without difficulty; however, some items, especially sinus pain, were less well understood or inconsistently interpreted and thus were deleted or reworded between the rounds of interviews. For example, based on evidence from the first round, one item on body aches was reworded from asking “how much did your body ache” to “how much does your cold make your arms and legs ache”. Reference to the child’s cold was added to help children separate this experience from other types of body aches (e.g., due to physical activity), while “body ache” was changed to “arms and legs ache” because most children identified aching in the arms and legs as due to their cold. The recall period of some items was adjusted to only retain the recall periods that were best understood. At least one item was retained for every symptom concept identified as relevant from the concept elicitation results.

Table 4.

Cognitive debriefing findings and subsequent changes to chest symptom items

Concept	Sub-concept	Item tested in first round of interviews	Cognitive debriefing findings	Decision regarding what to take into psychometric testing
Chest congestion	Tight chest	“When you took a deep breath right now, how tight did your chest feel?”	Well understood across all ages and both rounds of debriefing. Only 3/30 children (aged 6, 7, and 11 years) seemed to have some difficulty with ‘tight chest’ as they had never experienced it before. Only four children had some difficulty with reading some of the words in either the instruction or the item	Well understood and relevant—item retained unchanged: “When you took a deep breath right now, how tight did your chest feel?”
	Chest feeling full of mucus	“When you took a deep breath right now, how full of stuff did your chest feel?”	In round 1 of the debriefing, 7/15 children had some difficulty with this item as they did not understand what was meant by ‘stuff’ in their chest. Therefore, the item was deleted as was not as well understood as the other items	Item deleted
	Difficulty breathing	“For all of today, how hard was it to breathe air deep into your chest?”	23/28 debriefed on this item across both rounds had no difficulty understanding, and all but one understood the recall period	Well understood and relevant—item retained unchanged: “For all of today, how hard was it to breathe air deep into your chest?”
	Chest feeling heavy	“From when you woke up this morning until now, how much of the time has your chest felt heavy?”	21/28 children debriefed on this question seemed to understand ‘heavy chest’, and 17/21 were able to explain their understanding	Well understood and relevant—item retained unchanged: “From when you woke up this morning until now, how much of the time has your chest felt heavy?”

Cognitive Debriefing of the Illustrations Depicting Symptoms

Across both rounds of interviews, 8/12 children said the illustrations depicting a child experiencing each individual symptom helped them to answer the questions. Five parents also commented on the usefulness of the pictures. However, three children misinterpreted the illustrations. One child thought that lines coming from the child’s mouth to depict coughing showed the child speaking, one child thought that areas highlighted in red depicted blood (the intent was to highlight the areas of the body where symptoms occur), and one child thought that the images looked like an alien rather than a child. Based on these findings, the images were revised by a professional artist to improve interpretation.

Card Sorting Task

Thirty-six of the 39 children sorted the cards containing response phrases into an order by which they were satisfied. Three 6- and 7-year-old children did not complete the task; two because of reading ability and one because of time constraints. Six children (aged 6–8 years) placed the cards in an order that represented how their cold had progressed chronologically, rather than in terms of severity. 33 of the remaining 30 children sorted the cards into an order of increasing severity. While children understood the continuum, the severity level of “pretty bad” and “really bad” was unclear, thus these response terms were not selected for use going forward.

Parent–Child Completion Task

When asked to complete the questions with their parent, 4/39 children (aged 8–11 years) answered the questions entirely by themselves without any input from their parents. The remaining 35/39 children were read the questions by their parents. Only one child, aged 11 years, asked his parent for clarification on a question during the completion task. During the task, 7/39 parents questioned their child’s answer on at least one item; however, only one 6-year-old girl changed her response. The other six children kept their original answer despite being challenged. Example quotes from this exercise are provided in Table 5.

Table 5.

Example quotes from the parent-child completion task relevant to the children’s ability to answer the questionnaire

Issue during task	No. of children	Example quote
Child read and answered questions by themselves	4/39	Q: “I mean, just - do you think you - he - he was OK with answering the questions on his own?” DAD: “Oh, yeah. Yeah.” “OK. Do you think your - he - he needed any help from you for any of the questions?” “DAD: “Hmm, no.” (9-year-old boy)
		Q: “And do you think that your child would be OK to answer these questions on their - on his own?”MOM: “Yeah.”“OK. Do you think that there is, um, any questions that he actually needed help with?””MOM: “No, I don’t think so.” (11-year-old boy)
Child read questions by parent	35/39	Q: “OK. Um, so how did you feel about that, when she was able to help you?”“Um, I felt like I understand it better.”“OK. So would you prefer to answer these questions by yourself or with your mom’s help?”“My mom’s help.” (9-year-old girl)
		Q:“So we just want to see how you would normally interact. If you were to do this at home, how you would do this questionnaire with him?” MOM: So like if we were handed this and then we sat down together kind of thing? Um, well I would probably just read them to you.”
Parent challenges child’s answer	7/39	Q: “On these questions especially your body hurt and your arms and legs ache, I noticed that when she answered some, you sort of corked your eyebrow at her.”MOM: “Yeah, well because we’re talking about today. And … with all that she did today, I didn’t see her body hurting.” (208P-RC-F-6)
		MOM: “Right now, how much does your throat hurts? Not at all, a tiny bit, a little, some -?” “Not at all.” MOM: “Your throat’s not hurting right now?” “Uh-uh. (no)” (7-year-old girl)

In total, 25/34 parents believed their children would be able to answer the questions independently, but nine parents of children aged 6–8 years said it would be better if they either explained the items or reminded the children of the recall period. Six parents pointed out an inconsistency with what their children were reporting and what they had observed during the day, while 15 parents reminded their children of a specific time they may have experienced a symptom during the day. In addition, ten parents rephrased items to ensure their children understood, but none of this paraphrasing changed the intended meaning of the question. Most children (24/39) stated that they preferred their parents being there when answering the questionnaire (Table 5).

Item Response Distributions

Distributions of responses were reviewed for each item to provide additional insight into whether the items would discriminate among children differing in symptom severity. Cough and headache showed a good spread of responses with less spread for chest and sinus symptoms (see Table 6 for example response distributions for a few items).

Table 6.

Item response distributions for example questions

Item	Number of responses
	Current n (%)	Recent n (%)	Total n (%)
Cough: frequency (n = 39)
“From when you woke up this morning until now, how much have you been coughing?”
0 = Not at all	4 (14.8)	4 (50.0)	8 (22.9)
1 = A tiny bit	5 (18.5)	3 (37.5)	8 (22.9)
2 = A little	5 (18.5)	0	5 (14.3)
3 = Some	9 (33.3)	1 (12.5)	10 (28.6)
4 = A lot	4 (14.8)	0	4 (11.4)
Missing data			4
Chest: heavy frequency (n = 39)
“From when you woke up this morning until now, how much of the time has your chest felt heavy?”
0 = None of the time	10 (52.6)	6 (85.7)	16 (61.5)
1 = A tiny bit of the time	4 (21.1)	1 (14.3)	5 (19.2)
2 = A little of the time	2 (10.5)	0	2 (7.7)
3 = Some of the time	1 (5.3)	0	1 (3.8)
4 = All of the time	2 (10.5)	0	2 (7.7)
Missing data			13
Headache: head hurt (n = 39)
“Right now, how much does your head hurt?”
0 =  = Not at all	10 (47.6)	5 (71.4)	15 (53.6)
1 = A tiny bit	4 (19.0)	1 (14.3)	5 (17.9)
2 = A little	1 (4.8)	1 (14.3)	2 (7.1)
3 = Some	4 (19.0)	0	4 (14.3)
4 = A lot	2 (9.5)	0	2 (7.1)
Missing data			11
Sore throat: hurt (n = 39)
“Right now, how much does your throat hurt?”
0 = Not at all	9 (34.6)	4 (50.0)	13 (38.2)
1 = A tiny bit	7 (26.9)	2 (25.0)	9 (26.5)
2 = A little	1 (3.8)	2 (25.0)	3 (8.8)
3 = Some	5 (19.2)	0	5 (14.7)
4 = A lot	4 (15.4)	0	4 (11.8)
Missing data			5
Sinus pain: face hurt (n = 39)
“Right now, how much does your face hurt around your nose and eyes?”
0 = Not at all	12 (75.0)	4 (66.7)	16 (72.7)
1 = A tiny bit	1 (6.25)	2 (33.3)	3 (13.6)
2 = A little	1 (6.25)	0	1 (4.5)
3 = Some	1 (6.25)	0	1 (4.5)
4 = A lot	1 (6.25)	0	1 (4.5)
Missing data			17

Discussion

Developing pediatric PROs and conducting concept elicitation and cognitive debriefing interviews with children, particularly those as young as 6 years of age, is widely recognized to bring additional challenges not encountered in adult PRO development [11, 12, 22]. To minimize such difficulties and overcome the challenges, the interviewers took time to build rapport and used creative tasks to engage the children and help them describe their symptoms. Because interviews were conducted in research facilities and public buildings, these steps helped the children feel comfortable in the less familiar and slightly more formal research setting. If the interviews had been conducted in the children’s homes, the children may have felt more at ease, but they also may have been more likely to become distracted and lose focus. The free drawing and circling of upper body parts with symptoms worked well in identifying symptoms and confirming comprehension, and provided support for the concepts elicited during open-ended questioning. The card-sorting task provided strong evidence that the wording chosen for the response scales was well understood by children of all ages and that they understood the severity continuum. The benefit of using these creative tasks to help make the children comfortable and to aid in eliciting meaningful information is consistent with the success other researchers have had in using creative techniques in concept elicitation [16, 23]. Similarly, the illustrations in the items helped children to focus on the correct symptom. As others have reported previously, the response scales of graduated size or fill encouraged distinction between different response levels across the sample [17].

Children aged 6–8 years had some difficulty with the more complex concepts of sinus pain/pressure and chest congestion. Very few children aged 9 years and older had any problems with comprehension. A few younger children also had difficulty reading some words—this was addressed through revisions following the first round of cognitive debriefing. Other researchers have noted that limitations in reading ability and attention can result in some children in the 6- to -8-year-old age range having difficulty completing PROs, which can impact the validity and reliability of measurement [11, 12, 24–26]. However, others have provided evidence that valid and reliable assessment in this age range is feasible [11, 12, 22]. The International Society for Pharmacoeconomics and Outcomes Research Pediatric Task Force and others have highlighted that children’s ability to provide valid and reliable responses depends first on the concept being assessed, but also the language and reading ability of the individual child [11]. Variation by concept was observed in our study, with the complex concept of sinus pain less well understood than simpler and more commonly experienced symptoms such as sore throat and stuffy nose. In the validation of the PRO questions developed here (to be published in a subsequent article), we took account of individual differences in the abilities of children of the same age by requiring a certain reading level as part of the inclusion criteria of the study. Future research might also consider showing the questionnaire items to kindergarten and first-grade teachers for their professional opinion on whether typical children in their classes would be able to independently complete the questions. Parent administration of a child-completed PRO is one solution to overcome barriers to collecting self-report data from younger children with reading/attention limitations. This study provides evidence that such an approach should not threaten the validity of the endpoints as most children kept their answers even when challenged by parents. However, providing parents with clear instructions and training that they should not paraphrase or influence their child’s responses is a further step that can be taken to ensure the report remains a self-report from the child.

One limitation of the study was the reliance on parent/caregiver reports to confirm participants met the inclusion criteria, which could have introduced bias. Nevertheless, the interview findings did consistently provide corroborating evidence that the children reported cold symptoms that their parents also reported. Another limitation of the study is that diagnosis of the common cold was not verified by a trained physician. As parents typically do not take a child to the doctor’s office for a common cold, this approach is acceptable and reflects the real-world management of the common cold. Another limitation is that the study was only conducted in the USA. Further research in other countries would be valuable to confirm that the findings reflect the way other cultures describe cold symptoms.

Children were able to describe all symptoms of the common cold, using consistent language to do so, irrespective of age. Feedback from children and their parents suggests that the majority found most PRO items were easy to understand, interpreted consistently, and as intended. The items have strong face and content validity for severity level, symptom description, and various recall periods. Responses from children with current colds were spread across the scale demonstrating that children used the scale appropriately; however, most selected options in the middle and lower end of the 5-point scales. This is expected considering that the common cold is a relatively mild, self-limiting condition. Evidence of items having strong floor or ceiling effects was considered along with all qualitative findings when making decisions around deletion of items. It is acknowledged as a limitation that the presence of floor or ceiling effects could, to some degree, have been influenced by the inclusion criteria of the study. However, this evidence was primarily considered when trying to choose between items measuring the same symptom rather than in consideration of deleting symptoms, i.e., to identify which wording resonated best with children for characterizing a given symptom. Moreover, as this was a relatively small qualitative sample, caution was employed in giving too much weight to evidence of floor or ceiling effects. Item response distributions were examined further in later psychometric evaluation in a larger sample.

Conclusions

This in-depth qualitative study has supported the identification of relevant symptom concepts and the development and refinement of PRO items to assess those concepts. The findings support the content validity of the items and suggest that they can be used with confidence in children aged 9 years and older. For children aged 6–8 years, it is recommended the items are administered with initial adult supervision to explain the more difficult concepts or through parent/interviewer administration. Subsequent to the conduct of this research, a psychometric validation study was conducted to evaluate the reliability and validity of the items in a larger sample of children with colds. Those findings will be reported elsewhere.

Author contributions

All authors contributed to the design of the study and development of the items and response options. RA, KB, and CM were responsible for conducting the research, including development of study documents, data collection, data analysis, and interpretation. PH, BZ, and CG contributed to the review of study documents and clinical interpretation of the data. All authors were responsible for reviewing and revising the manuscript and have given approval for this version to be published.

Funding

This research was funded by McNeil Consumer Healthcare, a Division of Johnson & Johnson Consumer, Inc. Editorial support was provided by Rebecca Hall from Adelphi Values, and funded by members of the Consumer Health Products Association Pediatric Cough and Cold Task Group.

Data availability

The qualitative data described in this article are not publicly available in further detail.

Compliance with Ethical Standards

Conflict of interest

Rob Arbuckle is an employee of Adelphi Values, a health outcomes agency commissioned to conduct this research. At the time of the study, Chris Marshall and Kate Bolton were also employees of Adelphi Values. Patricia Halstead, Brenda Zimmerman, and Cathy Gelotte are employees or former employees of McNeil Consumer Healthcare, a Division of Johnson & Johnson Consumer, Inc., and hold stock or stock options in Johnson & Johnson. Cathy Gelotte has received consultant fees and an honorarium from the Consumer Health Products Association. The authors have no other conflicts of interest regarding the content of this article.

Ethics approval

The study was approved and overseen by Copernicus (MAP2-11-129), a centralized independent review board in the USA. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Consent to participate

All children provided written assent, and all parents provided written informed consent for the child to be involved. Separate written informed consent was obtained from those parents who were interviewed.