Assessment and Evaluation of Primary Prevention in Spinal Cord Injury

Abstract

Although the incidence of spinal cord injury (SCI) is low, the consequences of this disabling condition are extremely significant for the individual, family, and the community. Sequelae occur in the physical, psychosocial, sexual, and financial arenas, making global prevention of SCI crucial. Understanding how to assess and evaluate primary prevention programs is an important competency for SCI professionals. Assessing a program’s success requires measuring processes, outcomes, and impact. Effective evaluation can lead future efforts for program design while ensuring accountability for the program itself. The intended impact of primary prevention programs for SCI is to decrease the number of individuals who sustain traumatic injury; many programs have process and outcome goals as well. An understanding of the basic types of evaluation, evaluation design, and the overall process of program evaluation is essential for ensuring that these programs are efficacious. All health care professionals have the opportunity to put prevention at the forefront of their practice. With the current paucity of available data, it is important that clinicians share their program design, their successes, and their failures so that all can benefit and future injury can be prevented.

With time, talent, and treasure increasingly valuable, good intention alone does not suffice to determine whether any given prevention program is worth the effort. Evaluation produces information about the performance of a prevention program in achieving its objectives, most directly answering the questions of whether the program is working as intended and why. Successful programs have clear and explicit criteria for success. Their evaluation, which compares performance to plan, helps leaders improve and refine any given undertaking. Evaluation has often been a weak point in injury surveillance systems. Even rules and standards for evaluation, such as the 2001 World Health Organization (WHO) injury surveillance guidelines, falter in community use due to the lack of conceptual and terminological clarity. ¹ Conclusions were unable to be drawn about the effectiveness of motorcycle rider training on crash, injury, or offense rates due to the quality of 23 studies reviewed, including 3 randomized trials, from the Cochrane database.² Politics and values that frame a program undertaking can be challenged or reinforced by evaluation results.

In the midst of health and health care delivery fashions, interest in prevention has stood the test of time. Most people believe preventing illness is better than curing it, and experts have opined that much disease can be prevented. Russell notes that many people assume that prevention saves health care dollars, though that may not be the case.³ Prevention efforts can be thought of as the 3 es: education, enforcement, and environment. The latter 2 interventions generally require societal commitment, so many prevention efforts fall in the education category, either in the clinical context of education in the exam room or health promotion and behavioral change models to encourage lifestyle change. ⁴ That is not to say those education efforts are the most efficacious. The 2001 report of the Task Force on Community Preventive Services recommends 11 interventions to increase use of children safety seats, to increase use of safety belts, and to reduce alcohol-impaired driving. Most of these strategies fall in the enforcement category. The task force found insufficient evidence to recommend education-only programs to improve child safety seat use. ⁵

Evaluation Purpose and Content

Evaluation must be a key part of any program, from origin through execution. Programs that have not been evaluated do not carry much weight. Lack of evaluation leaves sponsors and donors to guess whether the initiative worked. Weak evaluation leads to careless conclusions about the need to continue or discontinue programs. Successful evaluation measures the program’s processes, outcomes, and impact; informs future program planning and design; provides important internal lessons for those conducting programs; ensures transparency and accountability; and teaches broad lessons about good practice. ⁶

Recent research demonstrates the utility of evaluation, while recognizing the challenges. Most people would agree that achievement of improved road safety is an important public goal, since 50 million people worldwide are injured or disabled annually in road traffic injuries. Hyder⁷ points out that in addition to political will and capacity building, “rigorous evaluation” is a key characteristic of any action to improve road safety, so as to demonstrate effectiveness and to convince policymakers and the general public of the utility of road safety investments. Esperato et al⁸ demonstrated effective evaluation planning for a road safety project by reviewing the literature on road safety intervention effectiveness and estimating lives that should be saved when the 10-country project is completed. However their evaluation work will likely be limited, because they note the weak effectiveness evidence that is available from the low and middle income countries whose roads are most in need of improvement. An intervention is only as good as the science that supports the undertaking.

Assuming a sound match of program cause and effect theory and implementation, the specification of program objectives provides the basis for evaluation. Programs are designed to change something, and the program is the source of that change. The evaluation will compare accomplishments to expected outcomes in at least the following areas:

• Who will change?

• What will change?

• When will it change?

• How will it change?

• By how much will it change?

• What will be the duration of the change?

As an example, an SCI primary prevention program might be designed to decrease the percentage of young adults who drive home “buzzed” or intoxicated. The “who” would be people who complete the program, not just individuals who attended some sessions. “What” would be the self-reported or observed use of designated drivers. “When” is likely to be the short period immediately at the conclusion of the program. Objectives must be clear, easily measurable, and realistic. Modification of objectives during a program should be deliberate and intentional. Most effective programs have stop points for mid-course corrections, because making program modifications on the fly can confound evaluation.

Types of Program Evaluation

In general, there are 3 types of evaluation. ⁹ Process-based evaluation is used to understand how a program works and delivers its results. Outcomes-based evaluation measures changes immediately after program implementation and establishes whether these changes occurred in response to the program. Impact-based evaluation, the most complex and meaningful, examines the long-term effects of an intervention on participants. As evaluation is often a combination of qualitative and quantitative methods that are applied non-experimentally or quasi-experimentally, association rather than causation is the best effect that can be shown.

Process-based evaluation dissects the component part of evaluation prevention program and describes the resources that are actually used to deliver the program. Often the personnel who are running and receiving the prevention program are the foci of this type of evaluation, which serves to answer questions such as how were the individuals implementing the intervention trained, how were the participants selected and recruited, and what feedback did the participants provide. As an example, a rehabilitation hospital may implement a peer-mentor secondary prevention program that pairs long-term SCI patients with newly injured individuals, hoping to decrease the incidence of secondary complications of SCI such as urosepsis or pressure ulcers. Aside from the quantitative analysis of the outcome (absolute number and rate of urological and skin morbidities), the process-based evaluation would seek to understand the recruitment of mentors and their training and the participants’ perception of the program, all compared to the plan.

Outcome-based evaluations look at what health status, behavior, attitude, awareness, or knowledge change occurred in response to the program. Even though it is desirable to affect every participant positively, the wise program evaluator will anticipate that only a certain percentage of prevention program completers will undergo a change as a result of the intervention. The evaluation process demonstrates whether that number was reached. Measurement of change could be a biological marker, disease state, health interview answers, or physical examination. In an SCI primary prevention program, the measurement of change might be how many individuals stated that they used a designated driver after the intervention compared to before.

Impact-based evaluations are complex. For all the years that we have been creating primary prevention programs for SCI, we do not know the answer to the question, “How many cases of paraplegia/tetraplegia were prevented by these programs?” Impact-based evaluations are aspirational. Causality is the central issue in impact evaluations. The ability to make causal inferences requires close attention to experimental design so that validity threats do not occur and competing explanations for observed outcomes are ruled out.

Program Evaluator

Programs require personnel to complete the evaluation. There are several options for performing this evaluation:

• Internal evaluation by those doing the inter-vention

• Internal evaluation by others in the organ-ization

• Internal evaluation by those doing inter-vention or others in organization with external experts

• External evaluation management with internal support by those doing the intervention or by others in the organization

Independent evaluators who are external to the serving organization provide the best objectivity, but costs associated with their use can lead organizations to choose less expensive alternatives.

Internal evaluation tends to be less expensive, whereas external evaluation tends to be more credible. If the key evaluation audience is external to the intervening organization, then external evaluation would be more effective. Conversely an internal evaluation seems less like an audit.

Design of Evaluation

In general, evaluations are pre-experimental, quasi-experimental, or experimental.¹⁰ Pre-experimental designs have internal validity threats (selection, maturation, attrition), but they tend to be simple to execute. Quasi-experimental designs are in the middle of the internal validity continuum; although these designs lack randomization, they have many features that mitigate many threats to internal validity. Experimental designs require randomization, which is often done after the individuals have consented to participate rather than at recruitment. Table 1 gives examples of different types of evaluation designs. Compromises of evaluation methodology are often made as resource-stretched practitioners seek good rather than ideal.

Table 1.

Evaluation designs

Pre-experimental	Quasi-experimental	Experimental
One-group posttest-only design	Single time series	Pretest/posttest control group
One-group pretest/posttest design	Multiple time series	Posttest-only control group
Posttest-only comparison group design	Repeated treatment
	Pretest/posttest nonequivalent comparison group design
	Recurrent institutional cycle
	Regression discontinuity

Published Studies on Injury Prevention

Given the substantial morbidity, cost, and risk of hospitalization that comes from SCI, preventing impairment, activity limitation, and participation restriction that result from tetraplegia or paraplegia appears to be worthy of effort.^11,12 Unfortunately there is very little published data on the effect of SCI prevention; the most relevant body of work relates to prevention of motor vehicle crashes or sports injuries, certainly proximate causes of paraplegia or tetraplegia. Messonnier¹³ reviewed health condition–specific prevention effectiveness. Bicycle-related head (brain) injuries were on the list of intervention areas, but intervention to reduce paralysis from SCI was not; it is not known whether this was due to inclusion decisions made by the US Preventive Services Task Force or the authors’ discretion.

Kuthy¹⁴ evaluated a drinking and driving prevention program consisting of a 20-minute slide presentation with narration aimed at high school students with the goal of raising awareness and promoting safer driving habits. Analysis of a convenience sample of 274 students who viewed the presentation using a one-group pretest/posttest/1 month delayed posttest format found a statistically significant improvement in reported driving behavior as a result of the program, which was maintained but not increased at 1 month follow-up.

Cusimano¹⁵ evaluated a ski and snowboard injury brochure and video prevention program using a randomized pretest/posttest design. Those seventh graders who participated had a statistically significantly increase in knowledge, attitudes, and reported safe behaviors compared to controls, but there was no difference in injury rates between the 2 groups during 4 ski trips that followed the program. Using the WHO Safe Communities model, Istre¹⁶ found in a non-equivalent 2-group pretest/posttest design that a 2-year multifaceted approach increased observed use of child safety seats.

Zhang¹⁷ reported on a school-based program of in-class education; parental, police, and community member workshops; posters, flyers, and first aid training; and student design of short film on traffic safety using Adobe Flash for Beijing middle and high school students. The intervention occurred weekly for 90 minutes over one academic semester. Using a 2-group pretest/immediate posttest design, they found a significant increase in knowledge of traffic signs and awareness of traffic safety in all intervention subgroups. Not all groups had a statistically significant decrease after the intervention in self-reported unsafe behaviors. The authors point out that observation of behavior or rates or crashes would have been more desirable end points than reports of behavior.

Not all reported programs have had positive results. Bhide¹⁸ evaluated the distribution and use of a diving safety video in high schools. Fifty-nine of the 92 schools that received the video responded to the evaluation survey, although one-quarter reported not having a copy of the video. On average, only 16% of students at any given school saw the video, highlighting the fact that underuse of materials is a challenge for many prevention programs that do not own the distribution channels for their work. Rivara¹⁹ reported on a 3-component intervention consisting of designated taxi stands; presentation of Last Call coasters, cards, and posters at bars; and public education through transit, radio, and TV ads. Using a 1-group pretest/posttest design, the intervention did not have an overall effect on self-reported driving after drinking too much or use of designated drivers or taxis, although significant improvement in use of taxis and designated drivers was found in a subgroup analysis of people who binge drink more than once per week.

In evaluating a comprehensive juvenile prevention program to decrease subsequent citations for adolescents who had been involved in moderate to severe traffic violations (Drive Alive), Ekeh²⁰ et al found that offending juvenile drivers who completed multimodal sessions of 2.5 hours each for 4 weeks showed statistically significant reduction in driving-related offenses for the 6 months after completion of the program, but this effect was lost in the long-term. They recommended booster interventions as a possible way to improve long-term program performance. Important lessons from this study include the importance of different types of educational pedagogies and the focus on observed behavior (but not rate or number of crashes) rather than attitudinal change as the outcome measure. Toledo²¹ compared young Israeli drivers and their parents who participated in a 45-minute meeting at their home to review accompanied driving (where someone over 24 years old is in the vehicle with the novice driver) and found a statistically significant decrease in crash rates for those in the intervention group compared to a control group that did not have a meeting. Manno et al²² compared a hands-on mobile injury prevention intervention with traditional classroom verbal curriculum on fifth graders with the objective of increasing safety knowledge in urban, suburban, home, and bus environments. Using a pretest/immediate posttest/delayed (6 months) posttest design for the 2 non-randomized groups, they demonstrated that the interactive and hands-on mobile approach made an immediate and delayed posttraining improvement over the traditional approach. Successful prevention efforts are usually more than a lecture and a brochure.

Many professionals know of or have participated in Think First, a traumatic brain (TBI) and SCI prevention program. Wright²³ evaluated this program using a pretest/posttest/delayed posttest design on all 11- to 15-year-old students in the state of Washington who watched a short film on the consequences of TBI and SCI at an assembly, listened to a lecture by a staff speaker who provided information on the frequency and causes of these injuries, heard testimony from a catastrophic survivor, and participated in question-and-answer sessions. Use of seat belts and helmets, avoidance of drugs and alcohol while driving or participating in sports, and checking water depth before diving were recommendations to the participants. Little change in attitude and no consistent change in knowledge or behaviors were seen as a result of Think First in this population. Avolio²⁴ also evaluated Think First, comparing high school and junior high school students on attitude, knowledge, and behavior before and after the intervention. Both groups gained knowledge, although the baselines were different. Neither group reported behavior change as a result of the program.

Future Directions

The paucity of published data on the effectiveness of SCI primary prevention programs provides little guidance to individuals and organizations who are working in this area. Prevention programs must publish not only their program design and theory, but also their outcome success or failure so that others can reflect on these findings as they develop and implement services that respond to local needs and divergent populations. In general, prevention programs with multimodal (including social media) components that include long-term follow-up and “booster” interventions appear to be more effective than one-dimensional, one-time programs. Achievement of attitudinal changes and a decrease in rates of injury or disease remain the reference standards in evaluating primary SCI prevention programs.