Validation of an Asthma Knowledge Tool to Assess the Efficacy of Comprehensive Asthma Education

ABSTRACT

Introduction

Asthma is a leading cause of chronic disease in children, and poor control is often associated with a lack of asthma knowledge, which can be improved through education. Thus, it is important to have a validated tool that measures asthma knowledge. We aimed to develop an asthma knowledge tool to assess the efficacy of comprehensive asthma education for caregivers of children with asthma.

Methods

An asthma knowledge tool was designed using existing questionnaires and pilot tested amongst 10 asthma experts and 20 parents who had previously received asthma education. The Children's Hospital of Eastern Ontario Asthma Knowledge Tool (CHAT) consists of 35 true/false questions. The CHAT was then prospectively administered to caregivers between June 2021 and 2023 before receiving education, < 7 days following education, and < 2 weeks after completing posttest 1. Internal responsiveness of the CHAT was measured by comparing pre‐ and posttest scores using paired t‐tests and determining effect size. Test−retest reliability was calculated using the two‐way mixed effect Intraclass correlation coefficient (ICC) for posttest scores.

Results

Mean CHAT scores amongst 129 participants increased by 6.2 points between pretest to posttest 1 (95% CI 5.4–7.0), with an estimated effect size of 1.2, indicating a large change (p < 0.001). Test−retest reliability was “good” to “excellent” with an estimated ICC of 0.85 between posttests.

Conclusion

The CHAT exhibited good discrimination, and reliably measured a significant improvement in knowledge scores following asthma education. The CHAT is a validated asthma knowledge tool for caregivers of children with asthma, having implications for future quality improvement and research studies.

1. Introduction

Asthma is the leading cause of chronic disease in children, with an estimated prevalence of approximately 15% in many western jurisdictions [1, 2]. Childhood asthma impacts quality of life for children and families, and poor asthma control is associated with recurrent emergency department visits and hospitalizations [3, 4]. Many factors contribute to poor asthma control, especially a lack of knowledge around self‐management of the disease [5, 6, 7, 8, 9]. It has been shown that pediatric asthma care that includes education (for both caregivers and patients) results in greater understanding of the disease, subsequently leading to improvement in asthma control, quality of life, and health outcomes [4, 10, 11, 12, 13]. For example, in an American inner‐city study, consistent follow‐up that included asthma education resulted in fewer hospital visits and admissions [10]. More recently, Safi et al. showed that a comprehensive pediatric asthma management program involving a pediatric respirologist and nurse asthma educator also resulted in decreased emergency department visits and hospitalizations [4]. As caregivers play the most significant role in managing their child's asthma, particularly for younger children, educating caregivers about asthma is crucial for improving health outcomes.

A wide variety of questionnaires has been developed to assess asthma knowledge in caregivers and/or patients with asthma, some of which focus on measuring the effectiveness of asthma education programs [13, 14]. However, many of these questionnaires are outdated with respect to current asthma management concepts that have evolved over the past two decades [12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]. Furthermore, many of the published asthma questionnaires were designed to assess confidence and perceptions of self‐management skills rather than solely disease knowledge, thereby resulting in subjective and inconclusive results. Many tools have also been validated in adult asthma patients [19, 22] and are therefore not necessarily generalizable to a pediatric population. There is currently a paucity of up‐to‐date validated asthma knowledge tools available for use among caregivers of children with asthma [20, 22]. In this study, we aimed to develop and assess the validity of a new tool to measure asthma knowledge in caregivers of children with asthma, before and after participation in a comprehensive pediatric asthma education session.

2. Methods

2.1. Questionnaire Development

We developed an asthma knowledge questionnaire for parents/caregivers by extracting and adapting questions from the most relevant three previously published asthma knowledge tools: (i) the patient‐completed asthma knowledge questionnaire (PAKQ) developed by Beaurivage et al. in 2015 in Quebec, Canada for use in adults [22]; (ii) the 21‐item asthma knowledge questionnaire by Franken et al. in 2018 validated in the Netherlands in caregivers of children with asthma [21]; and (iii) the parent/caregiver knowledge tool validated in 1990 in Australia by Fitzclarence and Henry [17]. Using the most relevant questions from each of these questionnaires based on current management guidelines, with modifications to language and content to optimize applicability for a pediatric population, a draft version of the Children's Hospital of Eastern Ontario Asthma Knowledge Tool (CHAT) was created and subsequently evaluated for content and face validity.

Content validity was established by ensuring that the questionnaire comprehensively covered relevant and important theoretical concepts related to asthma self‐management knowledge, based on review by a group of content experts [23]. These experts comprised eight pediatric respirologists and two nurse asthma educators at the Children's Hospital of Eastern Ontario (CHEO), a tertiary care pediatric hospital in Ottawa, Ontario, Canada. The experts completed the tool to ensure consistency of answers and provided feedback on the content of questions.

Face validity refers to whether a test appears to measure what it is intended to measure, which can be partially assessed by administering the items to a representative sample of the target population for whom the tool is intended for to solicit their feedback and understanding [23, 24]. We assessed face validity in 20 parents of children aged 0−18, who were managed by a respirologist at CHEO for their asthma, and who had previously received comprehensive asthma education at CHEO. The tool was administered through the online Research Electronic Data Capture (REDCap) platform, which is a secure and encrypted database system that is widely used to support research [25]. Participants were given instructions to complete the tool independently without referring to source material and encouraged to provide written feedback on each item. Participants provided additional verbal feedback on the content, clarity of each question, and ease of electronic tool completion via qualitative interviews conducted with a research assistant (E.K.). All feedback from the asthma experts and the target population sample was used to further modify the tool.

The final version of the CHEO Asthma Tool (CHAT) consists of 35 knowledge questions, covering asthma pathophysiology, triggers, and management (Supporting Information S1: E‐Table 1), as an online survey administered via the REDCap platform. The answer options include “True,” “False,” and “Not Sure.” All questions are equally weighted, with a score of 1 for a correct answer, and a score of 0 for an incorrect or “Not Sure” answer, for a maximum total score of 35 points. The CHAT also includes a section to gather child and caregiver demographic information.

2.2. Participant Recruitment & Asthma Education

Eligible participants included all parents/caregivers of children who attended a comprehensive virtual asthma education session at CHEO between June 2021 and 2023. Comprehensive asthma education is currently provided as standard of care for all patients who meet the following criteria: (i) Admission to CHEO for an asthma exacerbation, and (ii) two or more emergency department visits to CHEO for a moderate to severe asthma exacerbation (treated with oral corticosteroids) within the previous 12 months. Eligible patients are automatically identified and referred for an asthma education session delivered in either English or French through care pathways programmed within the hospital electronic medical record. Asthma education is typically scheduled within 2 weeks of referral and is delivered by a certified asthma educator. Due to the COVID‐19 pandemic, education was delivered virtually via the web‐conferencing Zoom platform, which has been shown to be equally as effective in improving knowledge as in‐person asthma education, with the added advantage of convenience for caregivers [26]. All education sessions cover the same content, including an explanation of the pathophysiology of asthma, signs and symptoms, diagnostic criteria, trigger avoidance, management techniques, and review of a personalized asthma action plan. CHEO's comprehensive asthma education has been shown to be effective in reducing repeat asthma hospitalizations, while improving caregivers' perceived knowledge and confidence in managing asthma [26, 27]. Exclusion criteria included caregivers of children already followed in the respirology clinic at CHEO, and caregivers who had previously attended comprehensive asthma education at CHEO.

2.3. Study Process

From June 2021 to 2023, all caregivers scheduled for an asthma education session were requested to complete the CHAT before attendance (i.e., pretest) to establish their baseline knowledge score. Those who attended the education session in full, and who met eligibility criteria were invited to participate in the study by a research assistant who obtained informed consent. Participants completed the CHAT again within 1 week following asthma education (posttest 1), and again 2 weeks later (posttest 2), allowing for an assessment of reliability (i.e., reproducibility) of responses (see Figure 1). This study was approved by the Research Ethics Board at CHEO.

Figure 1.

Study process and cohort accrual. [Color figure can be viewed at wileyonlinelibrary.com]

2.4. Statistical Analysis

All study analyses were carried out by two biostatisticians. Pre‐ and post‐asthma education CHAT scores were compared using paired t‐tests to assess internal responsiveness [28], and effect size was calculated to determine the magnitude of the difference between test scores [29]. An effect size of greater than 0.8 was considered a large difference [29]. p < 0.05 was considered statistically significant. The two‐way mixed effect Intraclass correlation coefficient (ICC) was used to calculate test−retest reliability, which measures the consistency and reproducibility of results between posttests [30]. An ICC value between 0.75 and 0.9 was considered a priori as indicating good reliability, while a value above 0.9 would indicate excellent reliability [30].

Demographic characteristics between responders (i.e., Phase II study participants who completed pre‐ and posttests) and nonresponders (those who completed a pretest but did not attend education, or attended education but did not complete a posttest) were compared using Fisher's exact test, with p < 0.05 considered statistically significant. This study was conducted in accordance with the reporting guidelines for reliability and agreement studies (GRRAS) checklist (Supporting Information S1: Appendix A) [31].

3. Results

There were 20 caregivers of children followed at CHEO for their asthma who participated in the pilot testing phase and provided feedback on the content, clarity, and delivery of the CHAT. Participants reported that the tool was quick to complete (within 10 min), and that the online REDCap format was easy to navigate on various devices including computers, tablets, and smartphones. Participants also expressed a preference for the online format due to the convenience and flexibility of completing the tool at their leisure.

There were 410 pretests completed, 78 of which were excluded due to being duplicates, for a total of 332 individual caregivers who completed a pretest. Of these, 222 caregivers attended asthma education at CHEO and a total of 129 completed either posttest 1 or posttest 2, yielding a response rate of 58% for completion of a posttest. Of the 129 participants, 126 completed posttest 1, with 53 also completing posttest 2. There were three participants who only completed posttest 2. This study process and cohort accrual are illustrated in Figure 1.

Of the 129 participants who completed a posttest, 88% were mothers and 12% were fathers of the child (Table 1). The majority of caregivers were over the age of 30 years (≈83%), with an average child age of 3 years old. Most caregivers (87%) had at minimum some postsecondary education. There were no significant differences between the 129 responders and 124 nonresponders in any of the demographic characteristics collected (Table 1).

Table 1.

Descriptive characteristics of the CHAT study participants & nonresponders.

Characteristic	Study cohort (N = 129)	Nonresponders (N = 124)	p value	Test used
Individual completing this questionnaire			0.5	Fishera
Mother	113 (88%)	104 (84%)
Father	16 (12%)	19 (15%)
Other	0 (0%)	1 (0.8%)
Missing	0 (0%)	0 (0%)
What is your age (in years)?			0.4	Fisher
< 20	5 (3.9%)	3 (2.4%)
20−30	14 (11%)	22 (18%)
30−40	83 (64%)	75 (60%)
40+	25 (19%)	24 (19%)
Missing	2 (1.6%)	0 (0%)
What is your highest education level?			> 0.9	Fisher
Primary or elementary school	1 (0.8%)	1 (0.8%)
High school	11 (8.5%)	13 (10%)
College or university or other professional education	112 (87%)	105 (85%)
Prefer not to answer	4 (3.1%)	5 (4.0%)
Missing	1 (0.8%)	0 (0%)
Are you a new immigrant or refugee to Canada within the last 5 years?			0.6	Fisher
Yes	5 (3.9%)	6 (4.8%)
No	121 (94%)	118 (95%)
Prefer not to answer	2 (1.6%)	0 (0%)
Missing	1 (0.8%)	0 (0%)
What is your preferred spoken language?			0.2	Fisher
English	119 (92%)	118 (95%)
French	9 (7.0%)	3 (2.4%)
Other	1 (0.8%)	3 (2.4%)
Do you or anyone in your household smoke?			0.5	Fisher
No	108 (84%)	99 (80%)
Yes	21 (16%)	25 (20%)
What is your child's age (in years)?			0.058	Wilcoxon test
Median (range)	2.00 (1.00, 14.00)	3.00 (1.00, 14.00)
Mean (SD)	3.31 (2.56)	3.78 (2.67)
What is your child's sex?			0.6	Fisher
Male	76 (59%)	77 (62%)
Female	53 (41%)	47 (38%)
Other	0 (0%)	0 (0%)

^a Fisher's exact test.

The mean pretest score was 24.7 (±5.2 standard deviation [SD]) points out of a maximum score of 35 (n = 129), and the mean posttest 1 score was 30.8 (±3.5 SD, n = 126) points (Figure 2a, Supporting Information S1: E‐Table 2.1). The paired t‐test showed a significant increase in the mean test score by 6.2 points (95% CI 5.4−7.0 points), with an estimated effect size of 1.2 (p < 0.001), indicating a large change in the mean score (Supporting Information S1: E‐Table 2.1). Similarly, the mean posttest 2 score was 30.7 points (n = 56), and paired t‐test showed an increase of 6.0 points (95% CI 4.9−7.2 points) from the pretest to posttest 2 mean scores, with estimated effect size 1.16 (Supporting Information S1: E‐Table 2.2).

Figure 2.

(a) Distribution of CHAT scores for participants who completed a pretest (n = 129), posttest 1 (n = 126), and posttest 2 (n = 53). (b) Distribution of CHAT scores for participants who completed all questionnaires (n = 56). [Color figure can be viewed at wileyonlinelibrary.com]

Similar results were observed among the 53 participants who completed all three questionnaires (Figure 2b). The mean pretest score was 24.4 points (±5.61 SD), which increased to 30.9 (±3.38 SD) on posttest 1. This improvement was sustained on posttest 2, with a mean score of 30.7 (±3.94 SD). This represents an increase in the mean score by 6.5 points from pretest to posttest 1, and 6.3 points from pretest to posttest 2.

There was improvement in mean scores for most questions from pretest to posttest 1, with 14 of 35 total questions showing improvement by 20% or greater (Table 2). This included 3/8 questions from the first section, “About Asthma,” 4/15 questions from the second section, “Asthma Triggers,” and 7/12 questions from the final section, “Treating and Managing Asthma” (Supporting Information S1: E‐Table 1).

Table 2.

CHAT questions which showed > 20% improvement in correct responses from pretest to posttest 1.

Item	Question	Pretest % correct responses	Posttest 1% correct responses	% Improvement from pretest to posttest 1
About asthma
2	If one child in the family has asthma, then all of his/her brothers and sisters have a higher chance of having asthma as well	41.1	72.2	31.1
7	Most children with asthma have an increase in mucus when they drink cow's milk	27.9	79.4	51.5
8	The best way to diagnose asthma is by a chest X‐ray	56.6	91.3	34.7
Asthma triggers: The following factors can trigger asthma symptoms in people with asthma
12d	Dairy products (e.g., milk)	24.8	75.4	50.6
12h	Gluten	38	74.6	36.6
12i	Laughter	36.4	79.4	43
12l	Heartburn (acid reflux)	26.4	48.4	22
Treating & managing asthma
13	If you are prescribed a controller inhaler for daily use, you should always take a reliever inhaler first	60.5	88.1	27.6
15	Antibiotics are an important part of treatment for most children with asthma	63.6	84.9	21.3
17	Daily inhaler use can damage the heart	53.5	85.7	32.2
18	It's not good if children need to use the reliever inhaler every day	48.1	79.4	31.3
19	With controller inhalers, it does not matter if some doses are missed or if you go on and off them	70.5	90.5	20
21	For older children, all inhalers work just as well without a spacer	45.7	74.6	28.9
24	Some inhalers for asthma don't work unless they're taken every day	60.5	83.3	22.8

The mean scores of each question between posttests were similar, with majority scoring within 5% (Supporting Information S1: E‐Table 1). The estimated ICC was 0.85 (95% CI 0.76−0.91) based on the 53 participants who completed both posttest 1 and posttest 2, indicating “good to excellent” reliability between the posttest scores.

4. Discussion

This study describes the development and initial assessment of the validity of a new up‐to‐date asthma knowledge questionnaire for caregivers of children with asthma. The CHAT was able to measure a clear improvement in knowledge scores after education, with knowledge retention 2 weeks later, when administered to caregivers of children with asthma managed at a tertiary care center. The CHAT demonstrated very good internal responsiveness and reliability for measuring knowledge gain following caregiver participation in comprehensive asthma education.

The CHAT performs similarly to previously developed asthma knowledge tools from which the CHAT was adapted. The reliability of the CHAT was comparable to that reported for the tools developed by Fitzclarence and Henry [17], as well as Martinez et al. [18], with our assessment including more than double the sample size. Both the Martinez et al. tool and the Franken et al. tool (adapted from Martinez et al.) found correlations between knowledge scores and parental education levels [18, 21], which we did not observe with the CHAT. This could be due to slight variation in the level of education among caregivers of children treated at CHEO, as evidenced by more than 80% of our sample reporting postsecondary education. Finally, the 2015 PAKQ developed by Beaurivage et al. in Quebec, Canada, for adults with asthma similarly demonstrated good reliability and responsiveness [22].

At the time of developing the CHAT, no other recent caregiver asthma knowledge tools had been published. However, there has subsequently been development of the Pediatric Asthma Knowledge and Management (P.A.N.D.A) questionnaires in British Columbia, Canada by Levivien et al. in 2021, which comprise three separate questionnaires aimed for children, teenagers, and parents [20], and include similar content domains as the CHAT. The P.A.N.D.A had good reliability, with an ICC of 0.7 (95% CI 0.5−0.9) for total scores and > 0.5 for each topic domain, although slightly lower than that of the CHAT (ICC 0.85). Relative drawbacks of the validation of the P.A.N.D.A includes: (1) responsiveness was not assessed, and (2) test−retest reliability was tested in 24 h, which may be an insufficient timeframe to adequately assess knowledge retention (vs. participants remembering individual responses). Another minor drawback of the tool itself is that it took longer for participants to complete (i.e., < 20 min for the P.A.N.D.A, as opposed to < 10 min for the CHAT).

The internal responsiveness of the CHAT was assessed by comparing knowledge scores before and after comprehensive asthma education at CHEO. The Treating & Managing Asthma section was the lowest scoring section before education, indicating that while asthma education must be comprehensive, special attention should be placed on disease management, and particularly on inhaler use. These findings are consistent with previous pediatric studies that have found an association between caregiver knowledge around inhaler usage and medication adherence in preventing asthma admissions in children [8, 32]. Additionally, a reduction in hospital visits and admissions among adults with asthma has been demonstrated following education emphasizing adherence and medication administration technique [10], though we did not assess this in our study.

Strengths of this study include a rigorous and iterative tool development process with the incorporation of feedback from asthma care providers and caregivers. Furthermore, the education intervention was standardized and delivered by certified asthma educators to ensure consistency and quality. The content of the CHAT is also comprehensive and covers key topics required for understanding the disease and optimizing asthma control. As the content reflects the most recent guidelines and knowledge of the disease, this suggests generalizability of the CHAT for use in other settings beyond tertiary care. Further strengths of the tool itself include that it can be completed within 10 min and is easy to navigate via the online format.

One potential limitation with our study was the apparent low response rate of 58% for completion of a posttest after asthma education. This exceeded the response rate of the P.A.N.D.A study (36% response rate with 162 of 486 consented individuals completing the first questionnaire) [20] was comparable to the response rate of 54.3% for the Franken et al. tool [21], and lower than the Fitzclarence and Henry tool (response rates of 98.6% and 85.7%) [17]. Direct comparison of response rate between studies may be misleading however, due to differing study methodologies, as the aforementioned studies did not measure asthma knowledge after receiving asthma education; whereas our study required that participants complete a pretest, attend asthma education, followed by a posttest. The response rate for the CHAT may also reflect challenges in recruiting study participants during the COVID‐19 pandemic, as widely reported in the literature [33, 34, 35, 36]. Furthermore, since study recruitment, test completion, and asthma education were all completed online (primarily for infection control purposes), the research team's ability to ensure completion of all questionnaires was limited. However, the similarity in baseline characteristics between nonresponders and study participants suggests that respondent bias likely did not significantly bias the generalizability of our main study findings.

Another limitation is that the CHAT was only validated in English, despite 10% of participants reporting French as their first language. However, most French first‐language speakers treated at CHEO are also fluent in English and participants agreed to participate in an English asthma education session (as opposed to an education session in French, which was offered). As the CHAT was assessed in caregivers of children with a recent history of severe exacerbation requiring an emergency department visit or hospital admission, it may not accurately reflect patients in the community with milder asthma. However, patients in the community may still be at risk for exacerbation in the future and would likely also benefit from comprehensive asthma education. It is also unlikely that there would be significant differences in the ability to absorb asthma education or complete the CHAT between caregivers of community asthma patients versus patients seen at CHEO. Finally, we have only assessed the CHAT when administered in an online format, and while participants expressed preference for this format in the pilot testing phase and in a previous study [26], our findings may not be generalizable for those without internet or computer access, which requires further assessment. Overall, the CHAT is a new validated caregiver asthma knowledge tool with numerous potential applications. In clinical practice, the CHAT can be incorporated into asthma education programs to establish caregiver baseline knowledge before an asthma education intervention, which can then be tailored to better address knowledge gaps. Furthermore, the CHAT can be used to evaluate the efficacy of asthma education programs and provide direction on how to optimize such programs. Future steps include assessment of the CHAT's longer‐term predictive validity by determining if measured improvement in CHAT scores translates to improved quality of life and fewer asthma exacerbations. The CHAT should also be assessed in other centers and health care settings, in other languages, and in adolescents for broader generalizability.

5. Conclusion

We developed a new pediatric asthma knowledge tool for caregivers that is responsive and reliable for measuring knowledge following comprehensive asthma education. We propose that the CHAT be routinely integrated into asthma education programs to target baseline knowledge gaps and ensure knowledge gains. The CHAT can also be used to ensure the consistent quality of asthma education programs over time.

Author Contributions

Dr. Dhenuka Radhakrishnan conceptualized and designed the study, critically reviewed the data collected and data analyses, and critically reviewed and revised the manuscript. Dr. Dhenuka Radhakrishnan oversaw all aspects of the study. Dr. Enxhi Kotrri was involved in the study design, data collection for all phases of the study, including conducting all patient interviews for the pilot testing phase, organizing the data and reviewing data analyses, and writing the manuscript. Dr. Madhura Thipse was involved in the study design, data collection for Phase II of the study, reviewing data analyses, and revising the manuscript. Andrea Higginson was involved in the study design, conducting the asthma education sessions for caregivers, data interpretation, and critically reviewing and revising the manuscript. Marc Tessier conducted the asthma education sessions for caregivers and was involved in data interpretation, critically reviewing and revising the manuscript. Han Mei and Dr. Nicholas Mitsakakis conducted all the data analyses, participated in results interpretation, and critically reviewed and revised the manuscript. Jamie Strain was involved in data organization, reviewing data analyses, and critically reviewing and revising the manuscript.

Ethics Statement

This study involving human participants was reviewed and approved by the CHEO Research Ethics Board, CHEOREB# 20/95X, and adhered to the Tri‐Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2). Written informed consent to participate in this study was provided by all participants or their legal guardian/caregiver.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Appendix.

CHAT.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.