Primary Care Clinicians’ Knowledge and Confidence About Newborn Screening for Sickle Cell Disease: Randomized Assessment of Educational Strategies

Abstract

Objective

In Massachusetts, primary care clinicians receive and act upon hemoglobinopathy newborn screening results. We assessed clinicians’ knowledge, confidence, and practices regarding hemoglobinopathy newborn screening, and the effect of mailed educational materials vs interactive seminar on knowledge and confidence.

Methods

A randomized educational intervention trial was performed at 15 community health sites. Practices were randomized to determine the order in which the educational interventions were administered: mailed educational materials first or interactive seminars on the management of hemoglobinopathy newborn screening results first. Clinicians’ demographics, knowledge, confidence, and practices were assessed by a survey. Posttests were administered soon after the intervention.

Results

Responses came from 85 of 170 eligible providers (50%). Twenty-nine percent of respondents provided both pretests and posttests. In respondents with paired data, knowledge on a 5-point scale improved by 1.4 ± 0.4 (mean ± standard error of the mean, p = .003), while self-efficacy on a 16-point scale increased by 1.3 ± 0.3, p = .002. There were no significant differences between seminar and mailed-materials groups.

Conclusions

Both educational strategies led to modest improvements in knowledge about newborn screening for hemoglobin disorders. Enhancing knowledge and confidence about newborn screening-related tasks may improve clinicians’ capacity to act upon newborn screening results for hemoglobinopathies.

INTRODUCTION

Newborn screening for sickle cell disease was recommended by a 1987 consensus conference to allow prompt initiation of penicillin.¹ Presently, all 50 states and the District of Columbia, Puerto Rico, and the US Virgin Islands screen all newborns for sickle cell disease.² The objective of sickle cell newborn screening is 2-fold: (1) identify and educate families with sickle cell disease, sickle cell trait, and other hemoglobin variants; and (2) provide appropriate interventions (penicillin prophylaxis and education) to decrease mortality.³ Key components of any newborn screening program are: (1) initial screening; (2) immediate follow-up of the positive newborn that includes location, follow-up, and referral for diagnostic testing; (3) diagnostic evaluation (ascertaining true positives vs false positives); (4) immediate and long-term disease management; and (5) evaluation of all the components of the system, including process and outcome measures.⁴

Roles and responsibilities in newborn screening systems vary among states, and within a state, vary by disease.⁵ Most states notify the primary care physician about positive screening results and the need for follow-up.⁵ As the recipients of the screening results, primary care clinicians are key stakeholders, whose “knowledge and practices are critical to appropriate screening.”^{6, 7}

Physician notification presupposes that primary care providers have necessary information about and confidence to implement strategies for screened conditions, including sickle cell disease. We aimed to test this supposition formally and to apply an interventional strategy to improve the knowledge and confidence of primary care providers (PCPs) in our region in light of anecdotal evidence of deficiencies apparent to pediatric hematologists receiving referrals.

The interventions used for this study included mailed educational materials and interactive educational sessions using local experts. These strategies are based on 2 models of behavior change: social learning theory and social cognitive theory. Social learning theory focuses on the ability of peers/role models to influence behavior change.^8–10 Outreach visits by experts to a group of clinicians as used in our study employ this theoretical framework.

Social cognitive theory defines human behavior as an interaction of personal factors, behavior, and environment.¹¹ Specifically, we focused on the concept of self-efficacy, which refers to a person’s confidence in performing a particular behavior or skill that can be achieved through interactive educational sessions where clinicians are able discuss cases and role play specific behaviors.¹¹ Strategies such as interactive training sessions with case discussions, educational outreach visits by experts to a group of clinicians (academic detailing), reminders, audit with feedback, and patient-mediated interventions have all been shown to be effective in changing physician behavior.^12–15 Combining intervention strategies has also been shown to be useful in changing physician behavior, although there have not been clear links to improvements in patient outcomes.^12,15 Traditional didactic continuing medical education (grand rounds, conferences, lectures, etc) and educational mailings have resulted in limited changes in physician behavior.^12,14,15

The objectives of this study were to describe clinicians’ knowledge, confidence (assessed by self-efficacy), and practices regarding newborn screening for sickle cell disease as part of a Maternal Child Health Bureau Sickle Cell Disease/Newborn Screening demonstration project. We assessed the respective impacts of mailed educational materials vs interactive seminars on change in individual scores of knowledge and self-efficacy (confidence), which are short-term measures of physician competence.¹⁵ We hypothesized that educational interventions can improve the knowledge and confidence of primary care clinicians to provide follow-up care for children with sickle cell disease and trait.

METHODS

Design

We conducted a prospective randomized assessment of educational interventions (mailed educational materials or interactive training seminars) to change clinicians’ knowledge and self-efficacy about management guidelines for positive newborn screens for sickle cell disease.

Setting and Study Population

In Massachusetts, results from infants with presumed sickle cell disease are reported directly to the primary care provider and tracked by the New England Newborn Screening Program (NENSP) to ensure an initial appointment with a hematologist. Current Massachusetts reporting sequence for abnormal results for hemoglobin disorders is: (1) New England Newborn Screening program notifies the listed primary care provider by telephone as soon as results are available; (2) a written report and fact sheet are subsequently faxed to the PCP; (3) NENSP requests from PCP a repeat filter paper specimen for diagnosis confirmation sample; and (4) infant is followed by NENSP until the family is notified, penicillin prophylaxis initiated, and initial hematology visit scheduled. After confirmatory test is completed, the family is contacted by NENSP and offered home visit for teaching and family testing.

The study was conducted in 15 community health centers in greater Boston. These eligible centers: (1) had a significant population of African American and Latino patients, (2) cared for newborns within pediatrics or family practice programs, (3) had 5 or more positive newborn screens for a hemoglobinopathy per year (including variant hemoglobins such as Barts hemoglobin and heterozygous screens such as sickle cell trait) and (4) were located within the catchment area of a New England Pediatric Sickle Cell Consortium (NEPSCC) clinician. NEPSCC represents all pediatric hematology centers in Massachusetts. A recruitment letter was sent to the director of each eligible health center inviting them to participate in the study.

The study population was primary care clinicians (physicians and nurse practitioners in pediatrics and family practice who were considered regular staff at the health centers. This study was exempted from institutional review board approval at Children’s Hospital Boston. Clinicians provided informed consent through completion of the questionnaire.

Practices were randomized first into 2 intervention groups that determined the order in which the educational interventions were administered: one group received mailed educational materials first and the other group received interactive training seminars first. All respondents received pretests in the mail at least 2 weeks prior to receipt of their respective interventions. For the mailed-materials group, they first received mailed educational materials at least 1 week prior, and then post-tests for this group were administered for completion at their practice site. After completion of the posttest, the educational seminar was presented to the participants on the same day.

For the seminar first group, they received the educational seminar first at their practice site. The posttest was administered directly after the seminar for completion. After completion of the posttest, this group then received the same educational materials as the practices that received the mailings first. (Table 1) Surveys took approximately 20 minutes to complete. Since this project was a federal demonstration project, both groups eventually received both interventions, but the timing of the receipt of these interventions and the administration of the posttest differed during the study period.

Table 1.

Randomization Schema

Time Period	Mailed-Materials Group	Seminar Group
Period 1	Mailed pretest	Mailed pretest
Period 2	Mailed educational materials	Seminar followed by posttest at practice site
Period 3	Posttest administered before the seminar at practice site	Educational materials handed out after posttest

INTERVENTIONS

We randomized practices to 1 of 2 intervention arms. Subjects provided their own baseline data. We did not blind clinicians to group assignment, but because randomization was by center, no clinician was exposed to the other intervention, and all clinicians received the same educational seminar.

Interactive Training Seminar

The seminar consisted of a 40-minute, interactive, case-based PowerPoint presentation (Microsoft Corp, Redmond, Washington) in which an NEPSCC “expert” clinician reviewed the newborn screening process along with management strategies for positive newborn screens for hemoglobin disorders. Breakfast or lunch was provided to participants.

Mailed Educational Materials

NEPSCC developed a kit consisting of educational materials, including flow sheets summarizing management strategies. (PowerPoint presentation and educational materials are available from the authors upon request.)

DATA COLLECTION AND INSTRUMENTS

Data were collected from July 2003 through January 2004. Survey items were based on literature review and key informant interviews. An initial round of cognitive interviews was performed with pediatric residents and pediatric hematologists to assess survey clarity, and the survey was revised based on these interview results. Pretest and post-test surveys were linked by random-digit identifier.

Multiple-choice, self-administered questionnaires explored practice patterns regarding follow-up of positive newborn screens for hemoglobin disorders, opinions on the follow-up process for positive newborn screens, barriers to follow-up of abnormal newborn screening results, knowledge of newborn screening management strategies for sickle cell disease, and confidence (as assessed by self-efficacy) in skills to manage newborn screening results. Respondents also recorded demographic information on their gender, years in practice, specialty, and type of practice. Principal component analysis was used to create summary scores for outcome variables, knowledge, and self-efficacy. The entire survey is available from the authors upon request.

VARIABLES

Knowledge Survey

Knowledge was assessed using 5 clinical vignettes (1 question for each vignette) with 1 correct answer for each question. The 5-point knowledge summary score ranged from 0, all items incorrect, to 5, all items correct. Vignettes focused on sickle cell trait follow-up, confirmation of sickle cell disease in the newborn, provision of penicillin prophylaxis in newborns with sickle cell disease, screening of an older child for sickle cell disease, and screening of parents for sickle cell trait. The internal consistency of questions assessing knowledge had a Cronbach α of 0.71.

Self-efficacy Survey

Self-efficacy (confidence) was assessed using 4 questions that asked the clinicians to rate their confidence (ranging from 1 to 4 for each of the 4 questions) in their own abilities to explain: (1) inheritance patterns of sickle cell disease along with (2) family planning options, (3) social opportunities and challenges for sickle cell patients, and (4) options for prenatal diagnosis and interpretation and management of newborn screening results for hemoglobin disorders. Social opportunities referred to opportunities available to patients to obtain education, find employment, and have good quality of life. Psychometric testing confirmed the reliability of this measure. The overall self-efficacy summary score is the sum of the 4 component questions, ranging from 4 (not confident at all: answer 1 for all 4 questions) to 16 (very confident: answer 4 for all 4 questions). The internal consistency of these 4 items had a Cronbach α of 0.79.

ANALYSIS

The primary outcome measure was change in individual scores (knowledge and self-efficacy) between the pretest survey and posttest survey for the 2 groups (mailed educational materials and seminar). To compare the 2 groups of sites, we employed mixed-model linear regression analysis with individual change as the dependent variable, group as the independent variable, and site nested within group as a random effect to account for correlated response. Covariates and interaction terms were added to the model to test for confounding and effect modification.

Estimates of mean pretest and posttest scores (± standard deviation) were taken from raw data; estimates of mean change (± standard error) were derived from the fitted mixed-effects model and thus adjusted for clustering within site. We used p < .05 as the criterion for statistical significance in all tests performed with SAS software (SAS Institute, Cary, North Carolina).

To establish comparability of the 2 intervention groups at baseline, we conducted χ² tests (2 × n) on all responses. The results were corroborated by multiple logistic regression analysis with trial arm as the dependent variable (seminar group, mailed-materials group) and the baseline responses as independent variables. Logistic regression was similarly used to test for biased dropout, using follow-up status as dependent variable and baseline characteristics as predictors.

Power Considerations

This study was designed as a prospective, randomized comparison of the 2 educational intervention strategies, with a primary outcome measure of change in individual scores of knowledge and self-efficacy. The study was initially designed to include 30 randomized sites (15 per arm) with a projected total of 290 clinicians. We calculated the detectable effect size, defined as difference in mean change between arms, divided by standard deviation of change among individuals in 1 arm, assuming the following parameters: 70% response, a plausible range for within-site correlation of the primary endpoints (5%–20%), 80% power, and p < .05 as the criterion for statistical significance (type I error rate). With those parameters, the detectable effect size was 0.43 to 0.56, conventionally considered “moderate.”

At study completion, we had data from 15 sites with 85 respondents at baseline and 49 paired pretest and postintervention responses. Accounting for the reduced sample size and the observed intrasite correlation (21% for knowledge, 11% for self-efficacy), the effects detectable in practice with 80% power were 0.97 for knowledge and 0.90 for self-efficacy—both “strong.” The attained effect sizes were 0.7 for knowledge, 0.1 for self-efficacy. Post hoc power calculations indicated that the power of the study to detect effects of the observed magnitude was 15% for knowledge and 3% for self-efficacy.

RESULTS

Demographics, Retention, and Missing Data

The flow of study participants is detailed in Table 2. From 170 potential respondents, 85 (50%) provided responses to either the pretest or posttest. Of these 85 respondents, 49 clinicians provided paired data. The characteristics of the 49 respondents with paired data are shown in Table 3. Among the 49 respondents with paired data, 39 (80%) were female and 44 (90%) were white. Forty-four respondents (90%) practiced at a health center in an urban setting (92%). The majority of clinicians were attending physicians (63% of respondents, all pediatricians). Baseline comparability of the responses between the 2 intervention groups did not reveal any significant differences.

Table 2.

Flow of Study Participants

Eligible	30 sites, 290 potential clinicians
Recruited	15 sites, 170 potential clinicians
Category		Total	Seminar	Mailed Materials
Randomized sites		15	8	7
Potential clinicians		170	67	103
Provided data		95	35	60
Pretest		85	32	53
Posttest		59	21	38
Both (paired data)		49	18	31

Table 3.

Demographic Characteristics of Clinician Respondents With Paired Data

	Seminar Group	Mailed-Materials Group	pa	All
Sample size	18	31		49
Practice at health center	15 (83%)	29 (94%)	.34	44 (90%)
Urban location of practice	18 (100%)	27 (87%)	.28	45 (92%)
Attending physician	11 (61%)	20 (65%)	.55	31 (63%)
Nurse practitioner	3 (17%)	2 (6%)		5 (10%)
Other (PA, resident, RN)	4 (22%)	9 (29%)		13 (27%)
≥10 y in practiceb	7 (41%)	16 (70%)	.11	23 (58%)
Female	11 (61%)	28 (90%)	.03	39 (80%)
White	17 (94%)	27 (87%)	.64	44 (90%)
Pediatrics	10 (56%)	21 (68%)		31 (63%)
Family practice	7 (39%)	8 (26%)		15 (31%)
Other specialty	1 (6%)	2 (6%)	.78	3 (6%)

Abbreviations: PA, physician’s assistant; RN, registered nurse.

^ap value testing equal distribution of baseline characteristics in the 2 trial arms among 49 respondents who completed the pretest and posttest. Those 49 respondents did not differ in baseline characteristics from clinicians who did not complete the posttest (N = 26) as assessed by logistic regression.

^bMissing data for number of years in practice for 9 respondents.

Clinicians’ Practices and Opinions on Newborn Screening Follow-up for Sickle Cell Disease

We asked clinicians how often they sought to obtain a copy of newborn screening results when reports were not available for their review. Among 85 respondents who provided baseline survey data, 71 (84%) reported that they hardly ever attempted to obtain a written copy of newborn screening results when reports were not readily available during a clinic visit. For infants with positive (abnormal) newborn screening results, 50 (59%) received written results within 2 weeks of the infant’s birth. Sixty-two clinicians (73%) reported that they were primarily responsible for notifying infants’ parents or guardians about positive newborn screen for sickle cell disease. Forty-four (55%) clinicians stated that a genetic counselor was most responsible for providing genetic counseling and education for sickle cell disease.

Of the 49 respondents with paired data, 33 (67%) referred their patients with sickle cell disease to a genetic counselor or hematologist for genetic counseling. In contrast, thirty-nine (80%) clinicians seldom referred their patients with sickle cell trait to a genetic counselor. Fewer than half of clinicians (N = 22, 45%) perform genetic counseling and education for families affected with sickle cell disease and trait themselves. Twenty clinicians (41%) report never discussing the newborn screening process and potential results with a pregnant mother of an existing patient in their practice.

Barriers to Newborn Screening Follow-up Care for Sickle Cell Disease

We asked clinicians if they perceived difficulty adhering to the newborn screening guideline recommending that infants with an abnormal hemoglobinopathy screen have the diagnosis confirmed within 6 weeks. Sixty-three percent of respondents found this “not difficult.” Forty-seven percent of clinicians (N = 23) noted 2 common barriers to adherence: difficulty finding the patient and inability for the family to come in for an appointment. Furthermore, clinicians indicated limited time during visits (41%, N = 20), limited administrative support to retrieve newborn screening results (33%, N = 16), limited expertise (45%, N = 22), and difficulty identifying referral sources for genetic counseling (31%, N = 15) as barriers to discussing abnormal newborn screening results.

Clinicians’ Knowledge of Newborn Screening Management Strategies

Knowledge increased in the seminar group from 2.1 ± 1.7 to 4.4 ± 0.8, p = .01, (mean ± SD; 0, all incorrect, to 5, correct) and in the mailed-materials group from 2.6 ± 1.6 to 3.8 ± 1.2, p = .03 (Figure 1a). Among the 49 respondents with paired data, regression analysis indicated that in the combined groups (ie, all respondents from seminar and mailed-materials groups), knowledge increased significantly by 1.4 ± 0.4 (mean ± SEM, p = .003), though the magnitude of the increase was not different between groups (p = .38).

Figure 1.

Scores (Mean ± Standard Deviation) of Primary Care Clinicians’ Preintervention and Postintervention^a

^a All baseline and follow-up responses are included (N = 85).

Clinicians’ Confidence in Newborn Screening Follow-up Care

Confidence increased from 7.1 ± 2.6 to 7.8 ± 2.1 in the seminar group, p = .04 (mean ± standard deviation; 4, not confident, to 16, very confident on all 4 items), and from 7.0 ± 2.1 to 8.7 ± 2.4 in the mailed-materials group, p = .008 (Figure 1b). Among the 44 respondents with paired data, regression analysis showed that in the combined groups, confidence increased by 1.3 ± 0.3 (mean ± standard error of the mean), significant at p = .002, though the magnitude of the increase was not different between groups (p = .9).

We explored demographic characteristics (Table 3) as possible predictors for the observed effects of the interventions on knowledge and confidence. Post hoc power calculations indicated that the differential improvement between the 2 intervention groups would have had to be at least 2 points on the knowledge or confidence scale in order to be detectable with 80% power. The small differences that we observed (0.7 for knowledge, 0.1 for confidence) carried only 15% and 3% power of detection, respectively.

DISCUSSION

In this study, we focused on primary care clinicians’ knowledge and confidence as an indication of capability to provide follow-up care for children identified by the newborn screening program with sickle cell disease and other hemoglobin disorders, including sickle cell trait. We demonstrated that clinicians’ knowledge of newborn screening management strategies was poor but that it could be modestly improved with simple educational interventions. These interventions may also have a moderate effect on improving clinicians’ confidence in performing skills related to newborn screening follow-up care.

We also identified differences from established recommendations in practice patterns in Massachusetts related to newborn screening follow-up for hemoglobin disorders.^16,17 National recommendations suggest that primary care providers should be notified of positive screens within 5 to 7 days of testing.⁴ We found that slightly more than half of our respondents received written results of an abnormal newborn screen within 2 weeks of the infant’s birth. Most clinicians in our study indicated that they are primarily responsible for notifying infants’ parents about a positive newborn screen for sickle cell disease. This finding is consistent with a previous study of pediatricians and family physicians.⁷ Our study also found that a majority of clinicians hardly ever attempted to obtain a written copy of newborn screening results if not readily available in the children’s record. This sharply contrasts to a recent national survey of primary care physicians, which found that only 28% of respondents reported making no attempt to obtain results of the newborn screen.¹⁸ The American College of Medical Genetics recently developed management guidelines known as ACT sheets for each condition included in state newborn screening programs.^19,20 The use of these ACT sheets by primary care providers may help to reduce some of the variation in practice we noted in our study.

A recent national survey primary care clinicians showed that pediatricians had a lower odds of referring patients with sickle cell trait to a genetic counselor.⁷ Our study confirms this finding. A large majority of respondents noted that they did not refer their patients with sickle cell trait for genetic counseling, and many did not provide such counseling themselves. This finding suggests a missed opportunity, as a positive screen for sickle cell trait provides a window to counsel families about their risk for hemoglobin disorders in future pregnancies and evaluate parental genotypes, and provides access to resources for genetic counseling and family testing.

Our study also confirmed several barriers to short-term follow-up of positive newborn screens. Overall, these results reveal multiple areas for systems improvement in addition to disease-specific education.

We note a few potential limitations of this study. First, we had limited power to detect a difference between the seminar and mailing groups. This study was part of an evaluation of a federal demonstration project. Participation in this study was voluntary at the discretion of the director of the health site. Consequently, the number of sites recruited was much lower than anticipated. Second, participants were chosen in proximity to several pediatric hematology programs. Thus, the results may not be generalizable to clinicians far from expert resources, although the survey tools provided here could be used to replicate the study in those settings. Third, there was no assessment of retention of knowledge after the study period. Fourth, potential links between the intervention and patient outcomes were not assessed. Lastly, we relied on self-reports of practices and opinions about newborn screening, rather than detailed record reviews. Consequently, these results might be subject to recall and social desirability bias. It is unlikely that real-life practice is much better than self-reports.

The widespread use of tandem mass spectrometry has lead to increases in the number and complexity of disorders being identified nationally through newborn screening. This increase in the number of disorders identified through newborn screening has direct implications for primary care clinicians. It is well known that significant state-to-state variability exists in the process of reporting and tracking newborn screening results for several disorders.⁵ In this study, we highlight some notable trends in the follow-up of newborn screening results for hemoglobin disorders in Massachusetts. These initial results provide some useful insights about newborn screening follow-up in general and have implications for the follow-up of other disorders identified through state newborn screening programs.

CONCLUSIONS

We found that clinicians in Massachusetts most likely to be the recipients of hemoglobinopathy newborn screening positive results had relatively poor understanding of and only moderate confidence about relevant hemoglobinopathy management. Knowledge and confidence could be modestly improved with simple educational interventions. Further study would need to be performed to assess which educational strategy would lead to greater improvements in knowledge and confidence to follow-up newborn screening results for hemoglobin disorders. Nonetheless, we propose that modest effort expended on hematologist outreach to community providers for education might be useful for other states where primary care providers are responsible for receipt of newborn screening results. Targeted outreach to primary care providers may prove to be more efficient given those affected by sickle cell disease and other hemoglobin disorders tend to seek care at a limited number of institutions and are clustered geographically in the United States.