Initial Development and Pilot Testing of Observer-Reported Outcomes (ObsROs) for Children with Cystic Fibrosis Ages 0–11 Years

Abstract

Purpose

Patient-reported outcomes are important clinical trial endpoints. Young children may not be able to reliably report on how they feel or function, so observer-reported outcomes (ObsROs) may be more appropriate for them. The purpose of this study was to develop and pilot field test electronic parent-reported observational instruments for children with cystic fibrosis (CF) 0–6 and 7–11 years of age.

Methods

We performed concept elicitation interviews with parents of children with CF ≤ 11 years of age to elicit the respiratory signs they could observe at baseline and during an acute respiratory illness. The resulting instruments were refined based on interviews with parents and clinicians. We conducted a pilot field test to evaluate test-retest reliability and the ability of items to distinguish well and sick periods.

Results

The instruments consist of 17 items assessing respiratory signs and observable CF-related impacts. Test-retest reliability was acceptable for both age groups but discrimination was low for ages 7–11, likely reflecting less direct observation of older children by their parents.

Conclusions

An ObsRO for children with CF ages 0–6 appears promising, while self-report may be more appropriate for children >6 years of age. Next steps for the 0–6 year old instrument will be utilizing it as an exploratory endpoint in clinical trials to enable item reduction, scale development, and further reliability and validity testing. Ultimately, this ObsRO could be a promising endpoint for early intervention trials in young children with CF.

Introduction

In cystic fibrosis (CF), structural lung damage, inflammation, and infection begin early in life, frequently in infancy, and suboptimal nutrition remains a concern despite newborn screening. Thus, there are a growing number of clinical trials of CFTR modulators, pulmonary therapies, and nutritional interventions in infants and young children in order to improve short- and, ultimately, long-term outcomes¹. To date, clinical investigations in this critical young population have been hindered by a lack of standardized outcome measures.

In the context of clinical trials and the regulatory process, clinical outcome assessments can be classified as: a) biomarkers based on blood, tissue analysis, or computerized readings, and b) reported outcomes that may be patient-reported (PROs), clinician-reported (ClinROs), or observer-reported (ObsROs)². Currently available clinical trial endpoints in infants and young children with CF are generally surrogate endpoints such as growth, lung function, chest imaging, respiratory microbiology and fecal elastase. In terms of reported outcomes, progress is underway in developing a patient reported outcome (PRO) measure specifically for children ages 3–6, but psychometric data are not yet published³. In addition, many children in this age range may be too young to report for themselves in a valid and reliable manner on a range of outcomes, so an observer-reported outcome (ObsRO) may be an appropriate alternative.

PROs are reports coming directly from the patient about a health condition or its treatment without interpretation of the patient’s response by a clinician or anyone else⁴. In 2009, the U.S. FDA formally recognized the importance and clinical utility of PROs by releasing guidances for industry⁴ and on the qualification process of PROs for drug development tools⁵. ObsROs are needed when patients, because of age or other reasons, are unable to broadly report direct effects of their medical condition or its impact. Self-report of symptoms and feelings for youth under age 11 has had variable success in demonstrating acceptable measurement properties of reliability and validity⁶. Since children ≤6 years of age may not reliably report their own symptoms, the FDA recommends that parent-report of signs substitute for child report of symptoms in this very young age range⁴. Signs are patient phenomena that can be observed by another individual, such as a caregiver or health care provider, whereas symptoms are those patient-reported phenomena that are reported but not necessarily observed by anyone other than the patient. In the context of US regulatory practice, observer-reported outcome measures are recommended over proxy measures as suitable for drug approvals, due to documented reliability problems with proxy measures of phenomena that are not observable.⁷

A substantial body of literature suggests that disease-specific measures are more sensitive to change and provide information that is more relevant for clinical interventions than generic measures^8–10. The CF Questionnaire-Revised (CFQ-R) is a well-validated, internationally recognized disease-specific PRO¹¹. More recently, the CF Foundation has also funded Project Breathe (Chris Goss, MD, and Donald Patrick PhD, University of Washington, co-PIs), to develop and test a CF-specific patient-reported symptom diary for patients >11 years of age that has been evaluated in several clinical trials^12–14. Few ObsROs have been developed and published in any chronic disease in early childhood¹⁵ and, to our knowledge, no ObsRO specific to CF exists. Such measures are essential for clinical outcome assessment and trial endpoints that assess direct treatment benefit to the child and thus to their parents.

Our long-term objective is development of valid, reliable and sensitive CF-specific observer-reported instruments for use as clinical trial endpoints in infants and young children with CF to capture respiratory and disease impact signs. As a first step, this paper reports results of content development and pilot field-testing to evaluate preliminary measurement properties of these instruments, the Cystic Fibrosis Respiratory Symptom Diary - Ages 0–6 Version (CFRSD_0–6) and Ages 7–11 Version (CFRSD_7–11).

Methods

This study consisted of: 1) qualitative concept elicitation interviews with parents of children with CF ages 0 to 11 years and content development; 2) a) Cognitive interviews and focus groups with parents to assess the comprehensibility of the items and b) CF clinician review of their clinical relevance; and 3) Pilot field-testing of the measurement properties of the CFRSD_0–6 and CFRSD_7–11 (test-retest reliability and ability to distinguish sick vs. well periods). The Seattle Children’s Hospital Institutional Review Board approved the studies, and written informed consent was obtained from all parents.

Participants

All study participants were recruited from the CF Center at Seattle Children's Hospital. Eligibility requirements included confirmed diagnosis of CF; age 0–11 years; and parent/guardian able to speak English.

Concept elicitation interviews and content development

Participating parents were instructed to schedule a telephone interview within 30 days of contact with the CF clinic for an acute respiratory illness. The interviewer asked about signs parents observed at the time their child was experiencing a worsening of their respiratory status. Interviews were audio-recorded and transcribed by a professional transcriptionist. Interviews were conducted until no new signs or impacts were mentioned in three consecutive interviews (i.e., concept saturation). An initial code list was developed for identified themes in the interviews and refined over the course of coding. Pairs of coders compared their results periodically using ATLAS.ti qualitative analysis software¹⁶ and reconciled their codes. The investigators generated a long list of items based upon the coded text.

The items retained for field-testing were rated by all investigators as having met the following criteria: represented a theme that was important to parents; were clearly observable through sight, hearing, smell, or touch; wording reflected language of the parents; judged by the clinical investigators to be directly relevant to CF; likely to change with successful treatment; and likely to discriminate by disease severity. We also included items assessing sleep, eating, activity, and emotional impacts, the purpose of which was to contextualize the results from the respiratory symptoms and also to provide information relating to more distal impacts.

Cognitive interviewing and CF clinician review

Parents and clinicians critically reviewed draft items for comprehensiveness, comprehension, and relevance through cognitive interviewing¹⁷. This method identifies items that have problematic language, format, or produce confusion in the respondent. We elicited options from parents in order to revise areas of comprehension difficulty and then tested these changes¹⁸.

The draft instrument content was also critically reviewed by multiple CF clinical specialists including 5 enlisted nationally and 11 from Seattle Children’s Hospital (from pediatric pulmonology, nursing, respiratory therapy, social work) (all named in acknowledgement section). The national and local CF clinical experts were asked to identify missing content that in their opinion was central to CF in infants and young children as well as unnecessary content.

Pilot field-testing

Field-testing was performed in a separate cohort. Parents received text or email reminders to complete the age-appropriate electronic instrument daily for 7 days during 2 separate weeks when their child was well and during 1 week when their child was sick. Sick periods, defined as a period when the child was experiencing a respiratory exacerbation treated with oral antibiotics, were identified based on responses to a weekly antibiotic log. Well and sick periods needed to occur within 6 months of each other, and if the sick period occurred first, then subsequent well periods that occurred at least 1 month after the end of the sick period were selected for analyses. If more than one well period was available for comparison to the sick period, one of them was selected at random for analysis. Completion of at least 3 surveys on different days was required for a well or sick period to be included in the analysis. The study used REDCap™ to provide secure, web-based electronic data capture of parent responses to weekly antibiotic logs and to web-based versions of the CFRSD_0–6 and CFRSD_7–11 during well and sick periods.

Statistical analysis

Demographic and clinical characteristics of index children during concept elicitation and field-testing were summarized using means and standard deviations for continuous variables, and counts proportions for categorical variables. All analyses were stratified by age group (0–6 and 7–11). To assess test-retest reliability, one daily diary was chosen at random from each of two well periods. Diary item responses were dichotomized into present versus absent. 2 × 2 tables were used to describe the agreement between the two well period diary entries, and agreement calculated with corresponding 95% confidence intervals. To assess item discrimination between well and sick periods, one daily diary entry from a sick period and one daily diary entry from a well period were chosen at random. Again, for each sign, 2 × 2 tables are presented to describe the agreement between the responses. McNemar’s test for paired categorical data was used to compare the proportion of patients with the sign reported as present during the sick and well periods and test the null hypothesis that there is no difference in reporting of the presence of the signs during sick and well periods. A significant p-value of the McNemar test suggests a change in the distribution of sign presence between sick and well periods. Statistical analyses were performed using Stata software (version 14.1, 2011, StataCorp LP, College Station, TX).

Results

Participants

Concept elicitation interviews were conducted with 44 parents (25 ages 0–6, 19 ages 7–11) in 2012–2013. Individual cognitive interviews were conducted in 2014 with 4 parents (ages 0–6) and group cognitive interviews with 6 parents for the CFRSD_0–6 and 7 parents for the CFRSD_7–11. Thirty seven parents (17 for children 0–6, 20 for 7–11) participated in the field-testing in 2014–2015. Table 1 displays the demographic and clinical characteristics of the index children in the concept elicitation interviews and the index children enrolled in the field-testing within each age stratum. For field-testing, 52 parents were approached with two declining participation because in one case the child lived between two homes, and in the other the family was not interested in research participation at that time. In the two families that declined, one child was Non-Hispanic Caucasian and one was Native American.

Table 1.

Characteristics of Index Children for Concept Elicitation and Field-Testing

	0–6 years		7–11 years
	Concept Elicitation (n=25)	Field-Testing1 (n=17)	Concept Elicitation (n=19)	Field-Testing1 (n=20)
Age (years) at interview/enrollment, mean (SD)	3.2 (1.9)	2.8 (2.0)	9.1 (1.4)	9.6 (1.6)
Female, n (%)	11 (44.0)	9 (52.9)	9 (47.4)	7 (35.0)
White, n (%)	25 (100)	17 (100)	18 (94.7)	20 (100)
Genotype: F508del homozygous, n (%)	13 (52.0)	14 (82.3)	13 (68.4)	12 (60.0)
F508del heterozygous, n (%)	8 (32.0)	3 (17.7)	4 (21.1)	7 (35.0)
Other, n (%)	4 (16.0)	0 (0.0)	2 (10.5)	1 (5.0)
Pancreatic enzyme replacement therapy, n (%)	25 (100)	16 (94.1)	19 (100)	20 (100)
P. aeruginosa culture positive, n (%)2	1 (4.0)	1 (5.9)	2 (10.5)	1 (5.0)
BMI percentile, mean (SD)3	53.3 (25.6)	52.6 (26.3)	50.4 (26.3)	59.1 (27.9)
FEV1 percent predicted, mean (SD)4	----	----	96.0 (19.4)	95.1 (19.5)

¹Includes participants who had two usable well periods and/or a usable well period and sick period.

²Most recent respiratory culture prior to interview/enrollment.

³BMI percentile calculated for ages ≥ 2 years (Concept Elicitation, n=16 in the 0–6 age group; Field-Testing, n=11 in the 0–6 age group).

⁴FEV₁ percent predicted reported for the 7–11 age group only (Concept Elicitation, 1 participant with missing data).

Instrument items

The CFRSD_0–6 and CFRSD_7–11 each consist of 17 items: 13 respiratory signs and 4 CF-related impacts (see Appendix). Response categories for each item are presence/absence (yes/no).

Pilot field-testing

Twenty five participants were enrolled in each age group for pilot field-testing, however usable surveys were only available for the subset of participants whose well and sick periods met timing criteria and/or whose parents completed surveys on at least 3 days during each well or sick period. For the 0–6 year old group, 24 had usable data from two well periods (median 5, (IQR: 2 – 36)) days between periods) and 14 had data from a well and a sick period (median 105 (IQR: 67 –191)) days between periods). For the 7–11 year old group, 22 had usable data from two well periods (median 6, IQR: 2 – 24) days between periods) and 15 had data from a well and a sick period (median 101, IQR: 61– 182) days between periods). Because participants did not provide responses to all the diary items, the number of respondents for some of the items was less than the total number of participants with usable data. There were no differences in demographic characteristics detected between those with and without usable surveys.

For the 0–6 age group, Coughed While Asleep (9.0% of days) and Woke From Sleep Because of Cough (5.3% of days) had the highest proportion of days across all well and sick periods with the response “Don’t Know/Did Not Observe”. For the 7–11 age group, Coughed While Asleep (22.7% of days), Coughed Up Mucus (15.5% of days) and Decreased Activity To Catch Breath (6.0% of days) had the highest proportion of days across all periods with the response “Don’t Know/Did Not Observe". For the 0–6 age group, the following items were never endorsed during the sick period: Coughed Until Threw Up, Nostrils Flared With Breathing. For the 7–11 age group, Breathed Faster was never endorsed during the sick period and Coughed Asleep, Belly Moved, Nostrils Flared and Less Active were endorsed rarely.

Test-retest reliability

Table 2 displays, for each sign, whether it was present or absent at a random day during each of two well periods. Ideally, each sign would be absent during both well periods (upper left box of each 2 × 2 table). For both age groups, all items except Coughed While Awake were more frequently absent than present during each well period. Agreement was generally high in both age groups. For the 0–6 year olds, mean % agreement was >90% for 11 items and 80–90% for 4 items. The lowest agreement was for Ate Less and Stuffy Nose (mean 74% for both). For the 7–11 year olds, mean % agreement was >90% for 6 items and 80–90% for 9 items. The lowest agreement was for More Fussy (mean 62%).

Table 2.

Test-Retest Reliability of Item Responses from Two Well Periods

	Age 0–6 years				Age 7–11 years
Well period 1\2	n	No	Yes	% Agreement (95% CI)	n	No	Yes	% Agreement (95% CI)
Q1 cough awake				87 [66–97)				85.7 (64–97)
No	23	9	1		21	7	1
Yes		2	11			2	11
Q2 cough asleep				100 (83–100)				81.3 (54–96)
No	20	19	0		16	10	2
Yes		0	1			1	3
Q3 wet cough				82.6 (61–95)				90.5 (70–99)
No	23	19	1		21	15	1
Yes		3	0			1	4
Q4 cough woke up				91.7 (61–95)				80 (56–94)
No	24	20	1		20	14	1
Yes		1	2			3	2
Q5 cough till threw up				95.8 (79–100)				85.7 (64–97)
No	24	23	0		21	17	1
Yes		1	0			2	1
Q6 coughing fits				91.7 (73–99)				90 (68–99)
No	24	22	1		20	18	0
Yes		1	0			2	0
Q7 nostrils flared				95.7 (78–100)				85.7 (64–97)
No	23	22	0		21	17	1
Yes		1	0			2	1
Q8 belly moved				95.7 (78–100)				100 (84–100)
No	23	22	0		21	21	0
Yes		1	0			0	0
Q9 rib area sucked in				91.3 (72–99)				76.2 (53–92)
No	23	21	1		21	14	3
Yes		1	0			2	2
Q10 breathed faster				91.3 (72–99)				100 (85–100)
No	23	21	1		22	22	0
Yes		1	0			0	0
Q11 wheezed				91.3 (72–99)				81.8 (60–95)
No	23	21	1		22	17	1
Yes		1	0			3	1
Q12 stuffy nose				73.9 (52–90)				86.4 (65–97)
No	23	17	3		22	19	2
Yes		3	0			1	0
Q13 fever				91.7 (73–99)				95.5 (77–100)
No	24	22	2		22	20	1
Yes		0	0			0	1
Q14 less active				87 (66–97)				86.4
No	23	19	1		22	19	1
Yes		2	1			2	0
Q15 slept more				95.7 (78–100)				86.4 (65–97)
No	23	21	1		22	19	2
Yes		0	1			1	0
Q16 ate less				73.9 (52–90)				100 (82–100)
No	23	16	3		19	19	0
Yes		3	1			0	0
Q17 more fussy				82.6 (61–95)				61.9 (38–82)
No	23	19	2		21	10	4
Yes		2	0			4	3

Discrimination between well and sick periods

Table 3 displays, for each sign, whether it was present or absent at a random day during one well and one sick period. Ideally, each sign would be absent during the well period and present during the sick period (lower right box of each 2 × 2 table). For the 0–6 age group, 5 signs were distributed significantly differently between sick and well periods; for 3 of these, the sign was most often present when sick and absent when well. For the 7–11 age group, no sign was able to significantly differentiate sick from well periods and none of the signs was most likely to be present when sick and absent when well.

Table 3.

Ability of Signs to Discriminate Sick and Well Periods¹

	Age 0–6 years					Age 7–11 years
			Present when well					Present when well
Symptom	Present when sick	n	Yes	No	p-value	Present when sick	n	Yes	No	p-value*
Q1 cough awake	Yes	14	7	6	0.014	Yes	15	10	4	0.18
	No		0	1		No		1	0
Q2 cough asleep	Yes	14	0	8	0.005	Yes	10	2	0	0.99
	No		0	6		No		0	8
Q3 wet cough	Yes	14	3	9	0.002	Yes	15	4	5	0.257
	No		0	2		No		2	4
Q4 cough woke up	Yes	14	0	3	0.317	Yes	12	3	2	0.157
	No		1	10		No		0	7
Q5 cough till threw up	Yes	14	0	0	0.999	Yes	16	1	5	0.103
	No		0	14		No		1	9
Q6 coughing fits	Yes	14	0	4	0.18	Yes	16	1	3	0.317
	No		1	9		No		1	11
Q7 nostrils flared	Yes	14	0	0	0.317	Yes	16	1	1	0.99
	No		1	13		No		1	13
Q8 belly moved	Yes	14	0	2	0.999	Yes	14	0	2	0.564
	No		2	10		No		1	11
Q9 rib area sucked in	Yes	14	0	2	0.564	Yes	15	4	3	0.083
	No		1	11		No		0	8
Q10 breathed faster	Yes	14	2	2	0.157	Yes	16	0	0	0.99
	No		0	10		No		0	16
Q11 wheezed	Yes	14	1	5	0.025	Yes	16	0	4	0.18
	No		0	8		No		1	11
Q12 stuffy nose	Yes	14	0	9	0.011	Yes	16	0	3	0.99
	No		1	4		No		3	10
Q13 fever	Yes	14	0	1	0.317	Yes	16	1	3	0.317
	No		0	13		No		1	11
Q14 less active	Yes	14	0	3	0.317	Yes	16	0	1	0.317
	No		1	10		No		0	15
Q15 slept more	Yes	14	0	3	0.083	Yes	16	1	3	0.083
	No		0	11		No		0	12
Q16 ate less	Yes	14	1	3	0.317	Yes	13	13	0	0.99
	No		1	9		No		0	0
Q17 more fussy	Yes	14	1	5	0.103	Yes	14	14	0	0.99
	No		1	7		No		0	0

¹Comparison between one daily diary completed during a sick period and one daily diary completed during a well period, each chosen at random. Presence/absence of signs compared using McNemar’s test.

Discussion

We report the initial development and initial pilot field-testing of the first observational sign diaries (ObsROs) for children with CF 0–6 and 7–11 years of age. The instruments were developed with systematic input from parents of children with CF in these age ranges. In working closely with our clinical partners we also found that it was feasible to identify how parents describe CF-related signs in relation to diagnostic criteria commonly used in clinical practice.

These ObsROs were developed as part of a larger project to create age-appropriate, disease-specific reported outcomes across the life span of CF patients. To date we have developed and validated a self-report symptom diary for ages 12 years and older^12–14. Given our current results, we will plan to continue to develop the CFRSD_0–6 but do not plan to develop the CFRSD_7–11 further; instead, we will turn our attention towards developing an age-appropriate patient-reported outcome for the 7–11 year old age range.

The field testing of the CFRSD_0–6 reported here involved a small sample size from a single center. Our planned future development work for the CFRSD_0–6 will be to embed it as an exploratory endpoint in multicenter clinical trials that include diverse children in this age range, following a similar development strategy as we did for the symptom diary for ages 12 and older. This additional testing will enable us to perform item elimination, scale development and further reliability and validity testing.

In the 0–6 year old age group, we found that children had few respiratory signs during well periods (except daytime cough) and that test-retest reliability was generally high. We found good sick-well discrimination for only 4 items, though our ability to detect a significant shift in the distribution of sign presence between sick and well periods was limited by our small sample size. Thus, the instrument seems appropriate for continued development and testing. In contrast, we found that the proportion of responses of “Don’t know/did not observe” was much higher for the 7 to 11 years olds, and that sick-well discrimination was much lower in this age range. Thus, our ObsRO does not appear to discriminate in this age range, likely because children in this age range are not as closely observed by their parents. Children in this age range may be able to report for themselves and given the findings of this study, developing a self-reported signs instrument seems a promising path.

A limitation of this study is that the parent sample to develop the diary was recruited from a single clinical site in the Pacific Northwest, though with a five-state catchment area (Washington, Wyoming, Alaska, Montana, and Idaho). In addition, the draft instruments were reviewed by CF clinical experts dispersed throughout the U.S. The sample size for the pilot field study was small, and not all participants were able to provide sufficient well and sick period data for analysis. In the pilot study there were challenges related to the time requirements imposed by the design (two well periods within 1 month of each other for well-well comparisons, and a well and sick period within 6 months of each other for well-sick comparisons). In some instances, children did not experience any sick periods during their time on study, which reduced the number of well-sick periods completed. In other instances, children became sick during the interval when families were trying to complete two well periods (where the goal was to not have intervening illness), which reduced the number of well-well periods completed. The willingness of families to routinely complete sign diaries at home on a chronic basis is unknown and will be further explored in the next stage of instrument development..

In conclusion, we have completed initial development of the first observational sign diary for children with CF ages 0–6. Given the growing emphasis on clinical trials of early intervention strategies and disease-modifying therapies in infants and preschool children, we hope that this instrument, after significant further refinement and validation, could serve as a clinical trial endpoint in this population. In contrast, we found that our observational sign diary for ages 7 to 11 did not perform well, and we will turn our attention to developing an age appropriate patient-reported outcome for this age range.