Canadian Paediatric Surveillance Program evaluation: An excellent report card

The Canadian Paediatric Surveillance Program (CPSP) is an active paediatric surveillance network providing data on conditions that, despite their low frequency, have high morbidity, disability and mortality, and place heavy emotional and financial burdens on families of affected children. In 2003, the CPSP underwent a formal evaluation to assess whether it fulfilled its objectives and satisfied the Centers for Disease Control and Prevention’s (CDC) surveillance evaluation criteria (1). The present article will provide a review of the CPSP, present the evaluation process and current results, and look at future challenges.

PROGRAM HISTORY

The CPSP was established in 1996 as a joint project of the Canadian Paediatric Society and Health Canada’s Centre for Infectious Disease Prevention and Control to improve the health of Canadian children and youth by national surveillance of uncommon high impact conditions. Since its inception, the CPSP has surveyed 24 conditions and expanded participation to more than 2300 paediatricians and paediatric subspecialists caring for a population of approximately 7.5 million children younger than 18 years of age. The CPSP promotes ‘global village’ collaborative research as a founding member of the International Network of Paediatric Surveillance Units (2), which is currently comprised of 13 national surveillance units worldwide.

Study proposals submitted for consideration by the CPSP Steering Committee are assessed according to six criteria (Table 1); however, preference is given to studies with fewer than 250 cases per year, of strong public health importance and which cannot be undertaken by any other means. Studies must conform to high standards of scientific rigour, ethics, confidentiality and practicality. To educate and ensure a standardized reporting basis, participants receive a protocol, a case definition and a brief description of the condition at the start of each study.

TABLE 1.

Study inclusion criteria

Rarity – fewer than 1000 cases per year

Paediatric and public health importance

Scientific importance

Uniqueness

Quality of proposal

Workload for paediatricians

The program uses a two-tiered active reporting process (Figure 1). Each month, the initial reporting form (Figure 2) lists conditions under surveillance. Respondents indicate either new cases or ‘nil’ reports, which are essential to confirm that cases were actively sought. For each case report, participants complete a detailed follow-up questionnaire, which is then analyzed and interpreted by investigators. Case ascertainment is verified by investigating duplicates and comparing data with sources such as the Canadian Association of Paediatric Health Centres, the Canadian Paediatric Decision Support Network, the Immunization Monitoring Program ACTive Centres (IMPACT), the Notifiable Diseases Reporting System and the Canadian Institute for Health Information. The CPSP also initiates one-time survey questions to capture the essence of a specific problem.

Figure 1).

Reporting process summary. CPSP Canadian Paediatric Surveillance Program

Figure 2).

Initial reporting form

Timely feedback to participants and other health care professionals, which is essential to the success of any surveillance program (1,3), is accomplished by publishing quarterly summaries of compliance rates, as well as case reports, study highlights, program news, educational resources and annual results. Also, investigators publish study results in peer-reviewed journals, and present at national and international scientific forums.

METHODS: EVALUATION OF THE CPSP

The evaluation process was initiated with the development of logic models (4) to gather background material and identify critical questions. These models illustrated short- and long-term outcomes in three key areas: the initiation of a study, the surveillance process and the impact of information dissemination. An expert advisory group (EAG) was convened to oversee the evaluation and formulate recommendations. Members of the EAG (Table 2) included a former senior policy advisor at Health Canada, a community paediatrician, a provincial epidemiologist, and international experts from the World Health Organization and the Wisconsin Division of Public Health. Four anonymous surveys, adapted from the 1997 Australian Paediatric Surveillance Unit program evaluation (5), were mailed to the following key players: 2326 participants, 53 investigators, 34 current and past Steering Committee members, and 56 public health professionals including decision makers at Health Canada, chief medical officers of health, provincial epidemiologists and the Working Group on Polio Eradication. Several nongovernmental organizations such as the Canadian Public Health Association, the Canadian Infectious Disease Society, the College of Family Physicians of Canada, and the Society of Obstetricians and Gynecologists of Canada, were also surveyed. The questionnaires incorporated both qualitative and quantitative measures. The CDC’s “Updated guidelines for evaluating public health surveillance systems” (1) were used as a template for analysis (Table 3). For selected studies, data from alternate sources were used to validate case ascertainment and to provide estimates of sensitivity.

TABLE 2.

The expert advisory group

Dr Robert McMurtry (Chair), former Assistant Deputy Minister, Health Canada

Dr Margaret Berry, Department of Neonatology, Montreal Children’s Hospital

Dr Jeff Davis, Department of Health and Family Services, Wisconsin Division of Public Health, USA

Dr Philippe Duclos, Project Leader, Immunization Safety Priority Project, World Health Organization

Dr Monika Naus, Associate Director, Epidemiology Services, British Columbia Centre for Disease Control

TABLE 3.

Centers for Disease Control and Prevention criteria for evaluating surveillance systems

Describe the surveillance system to be evaluated

Describe the public health importance of the health-related event under surveillance

Describe the purpose and operation of the system

Describe the resources used to operate the system

Gather credible evidence regarding the performance of the surveillance system

Indicate the level of usefulness

Describe system attributes

Simplicity

Flexibility

Data quality

Acceptability

Sensitivity

Predictive value positive

Representativeness

Timeliness

Stability

Adapted from reference 1

RESULTS

The overall survey response rate was 48.2% (48% participants, 45% investigators, 71% CPSP Steering Committee members and 46% public health professionals). The EAG independently reviewed all data and interviewed key people during a one-day, face-to-face meeting. The Chair of the EAG presented evaluation findings and recommendations to the CPSP Steering Committee at its November 2003 meeting.

PUBLIC HEALTH IMPORTANCE

The public health implications of CPSP study results are numerous, having documented missed prevention opportunities for congenital rubella syndrome (CRS), hemorrhagic disease of the newborn and vitamin D deficiency rickets; verified the importance of universal varicella vaccination for necrotizing fasciitis prevention; and confirmed that baby walker injuries (6) are still occurring and that a total ban of the devices is needed. Interestingly, the CPSP baby walker survey results were published in July 2002 and Health Canada did announce a total ban of baby walkers in April 2004. CRS surveillance monitors progress towards the goal of eliminating indigenous rubella infection during pregnancy. Acute flaccid paralysis and subacute sclerosing panencephalitis surveillance fulfills Canada’s commitment to both the global polio eradication initiative and the elimination of measles. The rarity of subacute sclerosing panencephalitis cases, two in four years, is a tribute to the success of the measles immunization program and the safety of the measles vaccine (7).

OPERATION OF THE SYSTEM

The CPSP mailing list is regularly updated and can be expanded according to specific studies. To further improve the 83% monthly response rate, a 2002 year-end letter appealing for active participation was sent to nonresponders and their numbers decreased by 86 (21.5%). The EAG commended the CPSP on being current, proactive and on target for achieving set objectives, and providing quality information and timely data on a national, collaborative basis. The EAG also commended the program for making good use of its resources in an economical way.

USEFULNESS

Does the system detect trends signalling changes in the occurrence of disease?

Although the CPSP usually cannot detect outbreaks or epidemics as they occur because of the reporting delay between the time of the monthly mail-out and its return by participants, the CPSP did capture the hemolytic uremic syndrome outbreak in May 2000 (8). Because CPSP studies run for several years, they can monitor trends in disease incidence, management and outcome over time. The EAG acknowledged the value of the one-time survey questions, noting that their use as a rapid national emergency response mechanism is an asset and a potential tool to be further explored.

Does the system provide estimates of the magnitude of morbidity and mortality related to the health problem under surveillance?

The CPSP affords a unique opportunity to ascertain the incidence and advance the epidemiology of uncommon conditions where Canadian data are often unknown. The studies on CHARGE association/syndrome (A/S), necrotizing fasciitis and Smith-Lemli-Opitz syndrome established Canadian incidence data and provided clinical knowledge on acute and chronic morbidity which was not available from other sources. The hemolytic uremic syndrome study reported epidemiological results comparable with the Australian data (2), while an international comparison of late hemorrhagic disease of the newborn incidence (1995–2000) showed Canada to have the lowest rate (0.37 per 100,000) (9). Studies on CHARGE A/S, neonatal herpes simplex infection and Smith-Lemli-Opitz syndrome enabled researchers to establish three independent cohorts to provide data on long-term outcomes.

Does the system stimulate epidemiological research likely to lead to control or prevention?

CPSP data, such as the data on prevaccine neonatal herpes, can better define the burden of illness in Canada, promote prevention, develop program strategies and enhance future research. For example, the lap-belt syndrome survey (10) confirmed the importance of first determining if these injuries were frequent enough to necessitate a review of child restraints in motor vehicles, and then, if prevention strategies needed to be re-evaluated.

Does the system identify risk associated with disease and/or lead to identification of prevention strategies?

The rickets study (11) identified a subset of Canadians at risk for nutritional rickets. Further analysis is needed to help develop public health policies and adapt prevention measures. The study on CRS (12) identified the importance of both herd immunity and standing orders for vaccination of all rubella-susceptible women in the immediate postpartum period. The necrotizing fasciitis study (13) reinforced the need for universal varicella vaccination.

Does the system lead to improved clinical practice by health care providers who are the constituents of the surveillance system?

Of the participants who responded to the survey, 68% found study protocols helpful and 63% found educational resources helpful. Importantly, 17% reported that the material changed their clinical practice through increased awareness of the conditions under surveillance (47%) and the availability of diagnostic criteria (13%). The CHARGE A/S study (14) resulted in improving diagnostic acumen and dramatically decreasing the mean age at diagnosis from 22.5 months (1994–1998) to 4.8 months (2000–2002). The nutritional rickets study (15) highlighted the need for further education of clinicians by determining that current clinical practice guidelines for vitamin D supplementation of all breastfed infants are not being universally implemented. At the time of the evaluation, CPSP study-related publications included 30 peer-reviewed articles, two annotations, 37 posters and 27 highlights.

Has the system led to changes in public health policy?

Of the public health survey respondents, 88% indicated that they had heard of the program and 86% knew of the study results. These results were used by 32% to evaluate public policy, by 47% as a basis for future research, by 70% as a guide for immediate action and by 60% for continuing professional development.

Has the CPSP provided a mechanism for national collaborative research?

Six of the 11 studies in 2002 had co-investigators from different centres. While 95% of investigators needed national case ascertainment to answer their research question, 68% felt that their research could not have been undertaken nationally without the CPSP. Sixty-five per cent of investigators indicated that the CPSP facilitated collaboration with other International Network of Paediatric Surveillance Units investigators. The CPSP conducted international comparisons for three studies: progressive intellectual and neurological deterioration/Creutzfeldt-Jakob disease (16), hemorrhagic disease of the newborn and CRS.

SIMPLICITY

The monthly reporting form is quick and easy to complete with participants indicating only the number of cases, including nil reports. Forty-one per cent of clinicians said they would reply at their own expense if postage-paid return envelopes were not provided. Ninety-six percent of clinicians returned most or all monthly reporting forms. Almost 50% reported at least one case and 47% of these had multiple reports. The follow-up study questionnaire was considered easy to complete by 80% of clinicians, with a few negative comments on the length and the amount of information required. Though access to hospital records hindered timely completion, 83% felt that the case-specific information was generally available.

FLEXIBILITY AND TIMELINESS

On average, the amount of time between the first submission of a new study proposal and implementation is 10 months. However, for public health emergencies, changes can be made to the monthly reporting form within days and 92% of clinicians were willing to report cases by phone or fax. Sixty-seven per cent of clinicians expressed interest in electronic reporting. Researchers valued the one-time survey question as a flexible option for obtaining timely data.

ACCEPTABILITY

The average initial response rate in 2002 was 83% with some provinces below the national average. The detailed questionnaire completion rate was 95% and 90% of physicians who reported a case had no hesitation in providing clinical information to the CPSP. At the time of the survey, 11 conditions were under surveillance and 70% of respondents thought that the number should stay the same. Ten per cent of clinicians had considered conducting a study through the CPSP. The majority of investigators (94%) stated that the CPSP study met their research objectives.

SENSITIVITY

Only 3% of participants knew of a case and returned the form without reporting it. An even smaller number (2%) knew of a case but did not return the form. To verify case ascertainment and estimate the sensitivity of specific studies, external validation was conducted from alternative sources such as the Notifiable Diseases Reporting System, the Health Canada’s Creutzfeldt-Jakob Disease Surveillance System and the Canadian Institute for Health Information. Sensitivity ranged from 16% (hepatitis C virus infection) to 89% (CRS) and 100% (cerebral edema in diabetic ketoacidosis, Creutzfeldt-Jakob Disease and acute flaccid paralysis).

POSITIVE PREDICTIVE VALUE

As duplicate reports and errors can impact on the positive predictive value (PPV), three calculations were performed (Table 4). Using the most liberal method (PPV3), all conditions had a PPV above 70% except two – hepatitis C and hemorrhagic disease of the newborn. In the latter instance, the acronym was sometimes confused with hemolytic disease of the newborn secondary to ABO/Rh incompatibility.

TABLE 4.

Positive predictive value (PPV) of cases reported to the Canadian Paediatric Surveillance Program (January 1999 to December 2002)

Conditions under surveillance	Total reports (n)	Valid reports	Invalid reports		Pending (n)	PPV1 (%)	PPV2 (%)	PPV3 (%)
		Confirmed (n)	Duplicates (n)	Discards (n)
Acute flaccid paralysis	402	218	149	28	7	54	86	91
Anaphylaxis	747	645	7	69	26	86	87	87
CHARGE association/syndrome	137	78	38	20	1	57	79	85
Cerebral edema in diabetic ketoacidosis	44	23	12	9	0	52	72	80
Congenital rubella syndrome	17	5	7	5	0	29	50	71
Creutzfeldt-Jakob disease	5	1	4	0	0	20	100	100
Hepatitis C virus infection	115	58	15	25	17	50	58	63
Hemolytic uremic syndrome	228	140	64	24	0	61	85	89
Hemorrhagic disease of the newborn	8	1	1	5	1	13	14	25
Necrotizing fasciitis	43	24	13	4	2	56	80	86
Neonatal herpes simplex virus	103	45	37	16	5	44	68	80
Neonatal hyperbilirubinemia	79	47	10	17	3	59	68	72
Neonatal liver failure/perinatal hemochromatosis	22	10	6	6	0	45	63	73
Progressive intellectual and neurological deterioration	99	61	14	24	0	62	72	76
Smith-Lemli-Opitz syndrome	86	35	32	19	0	41	65	78
Subacute sclerosing panencephalitis	3	2	1	0	0	67	100	100
Vitamin D deficiency rickets	33	24	5	3	1	73	86	88

PPV1 = all valid reports/total reports; PPV2 = all valid reports/(total reports – duplicates); PPV3 = all valid reports + duplicates/total reports

RESOURCES USED TO OPERATE THE SYSTEM

The CPSP contract includes the salaries of the CPSP Senior Coordinator (full-time), Medical Affairs Officer (part-time), CPSP Administrative Assistant/Clerk (full-time), as well as the cost of the scientific steering committee, postage, printing and other administrative work. The funding also covers the cost of maintaining the CPSP database and the promotion of the program both nationally, to increase participation and awareness of its contribution to public health, and internationally, to encourage collaboration with other paediatric surveillance programs. Lastly, the funds facilitate dissemination of educational materials on CPSP studies and timely transfer of study results.

DISCUSSION

The EAG findings demonstrated how well the CPSP met its objectives and performed against CDC surveillance guidelines, confirming the program’s value as a national, active surveillance tool.

For the present evaluation, one-time mail surveys were used to allow anonymous feedback from stakeholders. At first glance, an overall mean survey response rate of 48.2% may seem low, but a review of 321 distinct published physician mail surveys (17) showed a mean response rate of 54%. Furthermore, a steady decline has been documented in physician response rates for traditional mail surveys from an average of 60% in the 1970s to below 20% in the late 1990s (18). Still, there remains a possibility of bias because non-responders may differ in the importance with which they regard the CPSP. An argument to the contrary could be made that the paediatric community is committed to the CPSP, as shown by the 83% participants’ monthly response rate and the 95% completion rate of detailed questionnaires in 2003. Evidence of public health interest is shown by the 46% survey response rate of professionals in this sector, many of whom were in the midst of a SARS outbreak in Ontario and British Columbia.

The review process identified three main areas where documentation proved somewhat difficult. It was not possible to establish sensitivity for each respective study. By definition, the CPSP inclusion criteria favour studies where information is either limited or unavailable. For some studies, external case ascertainment was too costly, while for other studies, inaccurate records, such as the The International Statistical Classification of Diseases and Related Health Problems, tenth revision (18) hospital discharge codes, were found to be too imprecise.

Second, all studies undertaken by the CPSP must have medical and/or public health importance. Documentation on the impact of study results beyond the surveillance process was a challenge but it is an essential task that will need improvement in the future.

Thirdly, the ability to acquire epidemiological data and to stimulate simultaneous national collaborative research on 11 paediatric conditions is unique and was deemed to be of excellent value for the cost. However, a comparison of operating costs with the other 12 national paediatric surveillance units was impossible because each unit functions differently. Further, an attempt was made to compare cost-effectiveness with the Canadian Paediatric Society’s other key surveillance system, IMPACT, an active sentinel surveillance system for adverse reactions to immunizations and hospitalizations resulting from vaccine preventable diseases. However, such a comparison was not possible, because while the two systems are highly complementary, their models of surveillance are significantly different. The CPSP collects incidence and burden of illness data from all Canadian paediatricians on a wide range of uncommon childhood conditions, while IMPACT, based in 12 of the 16 paediatric hospitals, reviews charts of hospitalized cases. It should be stated that IMPACT has also undergone external reviews and has been found to be highly effective and cost-efficient. The EAG noted that while a cost comparison was impossible because of the two distinct methodologies, the CPSP provides excellent value for the money spent.

Several CPSP investigators did, however, independently calculate the costs of either preventing or managing patients with these high-impact conditions. It can be argued that financial savings can occur through increased awareness and education, resulting in earlier detection and treatment of patients with these conditions.

What lies ahead for the CPSP? Challenges for the future include:

• maintaining high interest and participation;

• adjusting the participant list to include other relevant health professionals;

• exploring the national capacity of the one-time survey question as an emergency response mechanism for public health threats;

• revisiting the possibility of electronic responses;

• disseminating important medical and public health surveillance results to specific target audiences for effective use in policy development; and

• continuing international cooperation and collaboration.

CONCLUSION

According to the EAG, the CPSP represents excellent value for the money spent, an achievement that was seen as exceptional and unsurpassed by any comparable program known to the group. The CPSP is unique in Canada, representing an important collaborative tool for surveillance, research and policy development. It is a robust program with a strong economical infrastructure, a well-established national collaborative network, a rapid real-time reporting rate, and a high degree of sensitivity and predictive value. The CPSP documents scientific data to improve the health of children and youth from coast to coast.