Children’s hospitals are well-positioned to promote childhood safety. Children’s hospitals without a medical director were most likely to consider using an efficacious injury prevention program.
Keywords: Program adoption, Dissemination and implementation, Injury prevention, Organizational factors, Pediatric injury
Abstract
Most childhood injuries can be prevented with the correct use of safety devices and appropriate supervision. Children’s hospitals are well positioned to promote these behaviors with evidence-based programming; however, barriers exist to adopting such programs. The purpose of this study was to describe organizational and administrative factors related to the adoption of an efficacious injury prevention (IP) program by children’s hospitals in the USA. IP specialists at 232 U.S. children’s hospitals were invited to complete a baseline survey, and then offered Safe N’ Sound (SNS), an efficacious computer IP program targeting parents of young children. Following this promotion period, specialists were surveyed again to assess their level of SNS adoption. Organizational and administrative factors associated with SNS adoption were identified using conditional random forest models (n = 93). Random forests identified a set of six predictors with potential utility for classifying hospitals as having SNS adoption activity or not; the final pruned classification tree indicated that four of these were best able to differentiate hospitals with and without adoption activity—having a medical director, having other hospital units that provided IP programming, the number of requests the IP unit received within the past year, and the belief of administrative leaders in their responsibility to develop programming all influence decisions. Hospitals without a medical director were most likely to demonstrate adoption activity. Medical directors, or other organizational leaders, can facilitate the adoption process for evidence-based intervention, but may need to be engaged intentionally when disseminating new products, tools, or approaches.
Implications
Practice: IP departments are more likely to adopt efficacious IP programs if the program appears easy to integrate with the work of other IP programs in the hospital.
Policy: Coordinating efforts among all staff who might be engaged in similar efforts and providing training to medical directors and other leaders may also be important to increase adoption.
Research: Using random forest techniques and classification models, researchers can identify the factors most important in differentiating program adopters and nonadopters of innovations.
BACKGROUND
Because they are a stable and reliable provider of community services, health care organizations are often the target of dissemination efforts. Several external factors motivate hospitals to engage in injury prevention (IP) work. First, the American College of Surgeons (ACS) Committee on Trauma requires that hospitals provide efficacious IP programs and appropriately staff these programs to maintain accreditation [1]. Second, the Internal Revenue Service (IRS) requires (as of 2009) nonprofit hospitals to fully detail their community health needs and corresponding improvement plans to be granted the community benefit tax credit [2]. An additional incentive is the finding that trauma has surpassed cardiac costs for health care utilization [3].
Because of these influences and their mission goals, IP programming is often provided by community hospitals. However, despite their positioning to address injury rates in the community and the breadth of available promising programs, it is far from commonplace that evidence-based practices are incorporated into hospital-based programs. This disconnect, broadly, has prompted a surge of research attention; many efforts, both theoretical and empirical, have been launched to fully understand this well-cited “quality chasm” between scientific discoveries and corresponding best practices [4]. Although efficacy trials are certainly required to test interventions under tightly controlled conditions, implementation research and effectiveness trials are critical [5]. Specifically, to maximize the impact that IP programs have on injury morbidity and mortality, we need a more sophisticated understanding of the factors related to program adoption.
Several categories of factors have been examined for their association with program adoption [6], factors which are also explicated in various theoretical models [7, 8]. Organizational level predictors [9] include hospital characteristics such as urbanicity, size, and population served; community indicators such as the perceived or actual burden of a health problem on the population served; the presence of a champion and the importance placed on outreach; community benefit; accreditation; and other organizational metrics. Administrative-level predictors [9–11] may include staff latitude in program selection, the degree of staff engagement in prevention coalition or networks, number of staff involved in prevention efforts, the centrality of prevention to staff roles and job descriptions, budgetary support for programming, and the extent and nature of existing programming. Lastly, features of the program, or innovation, are also related to program adoption [12, 13], including relative advantage over other programs, and the level of investment required to adopt and maintain the intervention.
This article describes organizational and administrative factors related to the outcome of considering Safe N’ Sound (SNS) for adoption by children’s hospitals in the USA. SNS, developed by the project team and shown in previous work to be efficacious [14–16], delivers tailored information to parents of young children to promote the use of needed safety precautions. We describe an innovative approach to developing models for small sample sizes, which is often the case in dissemination research that focuses on organizations as the unit of analysis. We conclude with a discussion of how identified organizational factors can be considered in marketing plans for distributing efficacious programs to children’s hospitals.
METHODS
Overview
We conducted a natural experiment among members of the Children’s Hospital Association (CHA) by offering all IP specialists a toolkit for using SNS and assessing adoption and related factors before and after implementation.
Participants
Of the 218 members of the CHA, 187 met the inclusion criteria for this study: U.S.-based, full members from either a freestanding children’s hospital or a primary teaching hospital, or associate or supporter-level members. Invitations to participate were sent from the CHA, and questionnaires were completed by the IP specialist at each hospital.
Baseline assessment
The survey instruments, described elsewhere [17], assessed domains of adoption and implementation identified by previous research, as well as metrics of IP programming. Briefly, there were a total of 33 items on the assessment including 17 items asking participants to rate their agreement/disagreement or level of importance using a seven-point Likert scale with a middle neutral point. For the purposes of modeling adoption, every Likert type questions was transformed into a binary variable by taking any level of positive agreement or positive importance as the positive category and the neutral or any level of disagreement/lack of importance as the negative category. The type of “IP programming for 0–4 year olds” question was also simplified into a binary predictor by distinguishing whether or not hospitals exclusively offered only car seat programs. A brief description of the items can be found in Table 1.
Table 1.
Description of full set of predictor variables considered for modeling Safe N’ Sound (SNS) adoption activity status
| Variable description | Abbreviation |
|---|---|
| Hospital and IP program administrative/structural factors | |
| Offer IP programs for 0–4 year olds (car seat only; car seat and/or other programs) | progptipfor04a |
| IP program considered its own unit within the hospital (Yes/No) | hospip7aa |
| Other groups in the hospital also offer IP related programming (Yes/No) | hospip7dd |
| IP unit has a director of programming (Yes/No) | hospip7ee |
| IP unit has a medical or clinical director (Yes/No) | hospip7ff |
| Have IP staff tried new IP programs or strategies within the past 6 months (Yes/No) | hospippipnew |
| Patients asked to use computer at hospital (Yes/No) | progptcompuse |
| Where is IP programming administratively housed (in children’s hospital; outside of children’s hospital) | hospip4rc |
| Factors considered when choosing IP programs/activities | |
| The program is evidence-based (is an important factor; is not an important factor) | dec6arc |
| I have or can get the funding in my current budget for the program (is or is not an important factor) | decs6brc |
| I have partners who want to collaborate on the program (is or is not an important factor) | dec6crc |
| I have the staff capacity to meet all program components (is or is not an important factor) | dec6drc |
| I think people in the community will like the program (is or is not an important factor) | dec6frc |
| The program will reach a lot of people (is or is not an important factor) | dec6grc |
| It comes with safety supplies, materials, or other program resources (is or is not an important factor) | dec6hrc |
| The program has a champion (is or is not an important factor) | dec6irc |
| The program has been used by similar hospitals or in similar communities (is or is not an important factor) | dec6jrc |
| I can count the program as a “community benefit” for tax purposes (is or is not an important factor) | dec6krc |
| Environment for IP staff within the hospital | |
| IP staff able to stay up to date with research (agree; disagree or neutral) | about9arc |
| IP staff have influence over decisions (agree; disagree or neutral) | about9brc |
| Hospital leadership trusts IP staff professional judgement (agree; disagree or neutral) | about9crc |
| Hospital provides funds for IP staff to attend conferences and trainings (agree; disagree or neutral) | about9drc |
| Hospital staff encouraged to try promising new IP activities (agree; disagree or neutral) | about9erc |
| Hospital has people passionate about IP other than IP staff (agree; disagree or neutral) | about9frc |
| Hospital admin wants IP unit to be seen as community leader in IP (agree; disagree or neutral) | about9grc |
| IP reporting, budget, and staffing | |
| Hospital requests financial data from IP program for its community benefit reporting on IRS form 990 schedule H (Yes; No; Unsure) | staffing13rc |
| Expect total budget for injury prevention programming in the current year to be different than previous year (Yes, will probably increase; Yes, will probably decrease; No, will probably stay the same) | staffing14bc |
| Total number of FTEs for IP unit (number 0 or greater) | staffingfte |
| IP Unit activities within the hospital during the past 12 months | |
| How many times did IP staff meet with hospital admin to discuss IP (number 0 or greater) | about8a |
| How many times did IP staff teach hospital staff about IP (number 0 or greater) | about8d |
| How many times did IP staff receive requests from hospital administration for information related to IP (number 0 or greater) | about8b |
| How many times did IP staff receive requests from nonadministrative hospital staff for information related to IP (number 0 or greater) | about8c |
| How many times did IP staff review injury records/data to understand most common causes of injury in community (number 0 or greater) | about8e |
As an incentive for participating, we held a raffle for the CHA’s annual conference-fees and per diem (valued at $1,000 each). Electronic invitations and reminders were sent by the CHA, and all responses were collected using survey software supported by the CHA.
Program delivery
Following this assessment, we provided SNS to all member hospitals, along with a training manual, toolkit, and program materials (www.safensoundallaround.com). The program is available in both English and Spanish and offered as a stand-alone computer application to address any barriers of firewalls and network constraints. We also conducted IP webinars as part of the Injury Prevention Cooperative (www.childrenshospitals.net), presented the program at the annual meeting of the CHA, and listed SNS in the CHA list of IP programs in its monthly newsletter. Examples from the Carolinas Center for Injury Prevention which houses the program at their children’s hospital were included to demonstrate its usability. To address external validity, we disseminated the program the same way that most programs would be presented to children’s hospitals, only using the CHA resources as would be available to any program developer, so that we could study the natural process of adoption and implementation.
Follow-up assessment
Three months after SNS was offered to hospitals, a follow-up survey was sent to all 187 hospitals to assess adoption decision making and implementation; 93 follow-up assessments were completed. Because the time for hospitals to fully execute and adopt programs is typically longer than the period between our assessments, we classified hospitals as having evidence of “adoption activity” if they had either fully implemented the program or indicated that they had done at least one of the following activities: (a) making plans to use SNS as part of their IP programming, (b) downloaded SNS to any hospital computer, or (c) installed SNS on at least one hospital computer for patients/community member use as having evidence of “adoption activity.” All other hospitals were deemed as having “no adoption activity.” We henceforth refer to those hospitals with adoption activity as “adopters” and those without as “nonadopters.”
Analytic approach and statistical analyses
In total, 24 of the 93 hospitals were categorized as having evidence of adoption activity. The missing item rates for the predictor variables of interest for these 93 responding hospitals were very low ranging from a minimum of 0% to a maximum of 6%. The missing values were imputed using a nonparametric missing value imputation approach based on a series of iteratively fit random forest models [18] and optimally tuned to reduce imputation error.
Because the number of predictor variables of interest was large relative to the number of respondents and because some of the relationships between organizational, administrative, and personal factors of interest may relate to adoption activity in ways that go beyond marginal contributions (i.e., one or more factors may moderate the impact of other factors), we needed to balance the complexity of the number of candidate predictors and the sample size. We employed a two-step process to create a classification model SNS adoption. The first step identified a smaller set of candidate variables using a random forest method that has been shown to be a highly effective nonparametric method for the “small n large p” scenario [19] that is capable of identifying important main effects and interactions [20]. Because the predictor variables were a mix of both continuous and categorical variables that were potentially correlated, we implemented a series of five independent conditional random forests along with a conditional permutation accuracy importance measure to derive unbiased measures of predictor importance [21]. The permutation importance measures were averaged across the five forests to form a single importance measure. We note that because random forest models do not provide a table of coefficients or statistical tests of significance, we utilized a variable importance measure [21] to identify variables to include in our final models. Specifically, those predictors that had a positive average permutation importance score were used in the second step to create a binary classification tree model of adoption. The tree model was grown to full length and node splitting was based on the Gini index of node impurity [22]. The final model was extracted from this fully grown tree using cost-complexity pruning with the optimal value chosen by the “one standard error rule” [22] based on a series of 25 repeated 3-fold cross validations.
RESULTS
Summary statistics that describe the univariate relationship between each of the 27 categorical predictors included in our final assessment and adoption activity status are shown in Table 2. Adopters and nonadopters differed either significantly or marginally significantly for five of these variables. Most notably among them was whether or not the IP program had their own medical director (p = .014). Only 17% of adopting hospitals reported that their IP program had a medical director compared with 45% of nonadopting hospitals. All of the adopting hospitals reported that “people in the community will like the program” was an important factor in IP programming consideration compared with 87% of nonadopting hospitals (p = .062). Just under 80% of adopting hospitals reported that IP programs were offered by other groups within the hospital compared with 60% of nonadopting hospitals (p = .082). Nearly 93% of nonadopting hospitals reported that obtaining funding for IP programs was an important consideration compared with 83% of adopting hospitals (p = .179). Finally, roughly 92% of adopting hospitals reported that they agreed that their IP staff had influence over decisions compared with 80% of nonadopting hospitals (p = .181).
Table 2.
Summary statistics for categorical predictors by adoption activity status
| Predictor (categorical scale) | Nonadopters (n = 24) % | Adopters (n = 69) % |
|---|---|---|
| Percentage of hospitals reporting that… | ||
| IP programs for 0–4 year olds are offered | 66.67 | 75.00 |
| IP program considered its own unit within the hospital | 57.97 | 54.17 |
| Other groups in the hospital also offer IP-related programming | 59.42 | 79.17** |
| IP unit has a director of programming | 40.58 | 29.17 |
| IP unit has a medical or clinical director | 44.93 | 16.67 |
| Have IP staff tried new IP programs or strategies within the past 6 months | 62.32 | 66.67**** |
| Patients asked to use computer at hospital | 30.43 | 41.67 |
| Where is IP programming administratively housed | 69.57 | 62.50 |
| Percentage of hospitals who reported that the item was an important factor when considering programming… | ||
| The program is evidence-based | 89.86 | 87.50 |
| I have or can get the funding in my current budget for the program | 92.75 | 83.33* |
| I have partners who want to collaborate on the program | 82.61 | 91.67 |
| I have the staff capacity to meet all program components | 88.41 | 95.83 |
| I think people in the community will like the program | 86.96 | 100.00** |
| The program will reach a lot of people | 81.16 | 79.17 |
| It comes with safety supplies, materials, or other program resources | 60.87 | 66.67 |
| The program has a champion | 66.67 | 75.00 |
| The program has been used by similar hospitals or in similar communities | 65.22 | 58.33 |
| I can count the program as a “community benefit” for tax purposes | 24.64 | 20.83 |
| Percentage of hospitals who agreed that (vs. disagreed or neutral)… | ||
| IP staff are able to stay up to date with research | 75.36 | 83.33 |
| IP staff have influence over decisions | 79.71 | 91.67* |
| Hospital leadership trusts IP staff professional judgement | 84.06 | 91.67 |
| Hospital provides funds for IP staff to attend conferences and trainings | 65.22 | 58.33 |
| Hospital staff are encouraged to try promising new IP activities | 46.38 | 54.17 |
| Hospital has people passionate about IP other than IP staff | 76.81 | 83.33 |
| Hospital admin wants IP unit to be seen as community leader in IP | 71.01 | 79.17 |
| Percentage of hospitals reporting that the… | ||
| Hospital requests financial data from IP program for its community benefit reporting on IRS form 990 schedule H (options were: Yes, No, and Unsure) | Yes: 53.62 No: 20.29 | Yes: 50.00 No: 12.50 |
| Compared with previous year, total budget for injury prevention programming in the current year is expected to... (options were: Increase, Decrease or Stay the Same) | Increase: 15.94; Decrease: 24.64 | Increase: 16.67; Decrease: 25.00 |
The majority of variables were either dichotomous or were transformed to be dichotomous for the purposes of modeling. The percentages refer to the “yes/important/agree” category. The last two variables (financial related) each had three levels. IP injury prevention; IRS Internal Revenue Service; ****p < .025; **p < .10; *p < .20 comparing the percentage of hospitals reporting an affirmative status for the predictor across adoption activity status.
The summary statistics for the six continuous predictors by adoption activity status are given in Table 3. There were marginally significant differences between adoption categories in the mean number of Total IP full time equivalents (FTEs) (p = .071), the number of IP-related requests received within the past year from hospital administrators (p = .179), and the number of times the IP unit reviewed injury records (p = .120). The median number of requests received within the past year from nonadministrative staff also differed by adoption status with adopting hospitals reporting a median number of such requests to be 4 compared with 10 for nonadopting hospitals (p = .184).
Table 3.
Summary statistics for the continuous predictors by adoption activity status
| Predictor | Nonadopters (N = 69) | Adopters (N = 24) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Mean | 95% CI for the mean | SD | Median | Range | Mean | 95% CI for the mean | SD | Median | Range | |
| Total IP FTEs | 4.72 | [3.13, 6.31] | 6.62 | 2.00 | 39.50 | 2.95** | [1.8, 4.09] | 2.72 | 1.90 | 10.00 |
| Number of times in past 12 months… | ||||||||||
| Met with hospital admin to discuss IP | 8.04 | [5.25, 10.84] | 11.64 | 4.00 | 60.00 | 7.54 | [2.76, 12.32] | 11.33 | 4.00 | 52.00 |
| Taught staff about IP | 19.19 | [10.74, 27.63] | 35.15 | 12.00 | 200.00 | 14.54 | [5.39, 23.69] | 21.67 | 9.00 | 100.00 |
| Received requests from hospital admin for information | 6.26 | [4.01, 8.5] | 9.34 | 3.00 | 50.00 | 4.08* | [1.72, 6.45] | 5.60 | 2.00 | 24.00 |
| Received requests from nonadmin staff for information | 18.04 | [9.93, 26.15] | 33.77 | 10.00 | 250.00 | 18.79 | [−2.51, 40.09] | 50.44 | 4.00* | 250.00 |
| Reviewed injury records | 16.19 | [9.03, 23.36] | 29.83 | 8.00 | 200.00 | 9.08* | [3.38, 14.79] | 13.51 | 4.50 | 52.00 |
Tests for the equality of means for the continuous predictors across adoption activity status were performed using two-sample t-tests with adjustments for unequal variances. Tests of equality of medians for each predictor across adoption activity status were performed using Mann-Whitney U tests. IP injury prevention. **p < .10; *p < .20.
Based on the average conditional importance measures derived from the series of five independent conditional random forests, there were six predictors whose average importance exceeded zero as indicated by the darkened circles in Fig. 1. We note that Fig. 1 is to random forest what a table of coefficients and their statistical significance tests are to logistic and regression models in that it provides a metric to evaluate how impactful each of the variables is for predicting SNS adoption activity. The following variables all had positively valued average importance and thus formed the final subset of predictors for inclusion in our modeling step: whether or not the IP unit had a medical director, whether or not IP programming was offered by other groups in the hospital, whether or not a program under consideration is evidence-based, whether or not IP staff are able to stay up to date with research and have influence over IP-related decisions, and the number of times IP staff receive requests from nonadministrative hospital staff regarding IP information in the past year (Fig. 1).
Fig 1 |.
Average variable importance scores for each of the candidate predictors for predicting Safe N’ Sound (SNS) adoption activity. The figure uses the abbreviated variable names but more complete descriptions of the variables are included in Table 1.
Although the random forests identified a set of six predictors with potential utility for classifying hospitals as having SNS adoption activity or not, the final pruned classification tree (Fig. 2) indicated that four of these were best able to differentiate hospitals with and without adoption activity.
Fig 2 |.
Final tree model predicting Safe N’ Sound (SNS) adoption activity from the subset of foremost important predictors identified from the conditional random forest models. If the condition described under the numeric node indicators (numbers in small boxes) is satisfied, move to the left and down the tree; otherwise, move to the right and down the tree. In total, there were five terminal nodes in our final model: 2, 6, 14, 30, and 31. The numbers in each of the boxes below the node labels describe the most common class of adoption activity (0 = nonadopter, 1 = adopter); the proportion of nonadopters and adopters and the overall percentage of the data set (93 hospitals) that fell into that node. IP injury prevention.
According to the left-most node on the bottom of Fig. 2 (numeric label 2), roughly 38% of the 93 hospitals in our sample had a medical director and 89% of those had no evidence of adoption activity. Furthermore, those hospitals whose IP programs had their own medical director had an SNS adoption activity rate that was a full one-third less than that of those hospitals whose IP units did not have their own medical director (Rate Ratio [RR] = 0.33; 95% CI [0.12, 0.89]). However, for those hospitals that did not have a director, the likelihood of adoption activity varied by whether or not IP programming was offered by other units in the hospital, the number of requests the IP unit received within the past year from nonadministrative hospital staff for data, information or outreach about IP, and whether or not IP staff had influence over decisions. Specifically, the adoption activity rate for hospitals with IP units without their own medical director for which IP programming was solely provided by the main IP unit (Node 6) was about 1.7 times greater than that of those hospitals whose IP programs had a medical director (95% CI [0.5, 6.0]). The SNS adoption activity rate for those hospitals whose IP units did not have their own medical director, who had IP programming offered by other units, and who received nine or more requests for IP information from nonadministrative staff (Fig. 2; Node 14) was 1.9 times that of hospitals whose IP units had a medical director (95% CI [0.5, 6.9]). Those hospitals with IP programs that did not have their own director but had IP programming offered by other units, and whose primary IP unit received less than 9 requests for IP information from nonhospital administrators within the past year with staff who agreed that they had influence over their decisions (Fig. 2; Node 31), formed the group of hospitals with the highest percentage with adoption activity. In total, 17% of responding hospitals fell into this node and of those, 75% indicated SNS adoption activity. The hospitals that fell into this node were over 6.6 times more likely to have adoption activity compared with those hospitals whose IP programs had a medical director (Fig. 2; Node 2) (95% CI [2.5, 17.2]).
Model performance
The overall accuracy of the model was computed to be just under 83%, meaning that 83% of the hospitals in our analytic data set were correctly classified as either adopters or nonadopters by the classification tree depicted in Fig. 2. The model was slightly better at correctly classifying nonadopters compared with adopters as the specificity of the model was just over 94% and the sensitivity was 50%. The overall area under the receiver operating characteristic curve (which plots measures of sensitivity vs. 1-specificity at various discrimination threshold values) for our final model was just under 78%, indicating a fairly reasonable level of discrimination between adopters and nonadopters. The utility of the final model for identifying future children’s hospitals that are nonadopters was reasonable as the estimated negative predictivity rate (probability that a hospital predicted to be a nonadopter was actually a nonadopter) was about 84% and the cross-validated estimate of negative predictivity (as a proxy for future performance of the model for this metric) was 78%.
DISCUSSION
Using a highly externally valid design along with a rigorous modeling strategy, this study identified the most influential predictors of program adoption by U.S. children’s hospitals. The random forest model was useful because there were a large number of predictors indicated by theory or identified by previous research and a constrained number of organizations. This two-step strategy for first identifying predictors and then using a classification model can be a statistically efficient approach for dissemination research that involves organizations as implementation units and analytic units of analysis.
The most influential factor in the adoption of SNS was the presence of a medical director: IP programs with a medical director were the least likely to report adoption activity, regardless of other factors. In this case, hospitals with a medical director may require a more extensive vetting process for potential programs than those hospitals that do not have this administrative position. This vetting process may require a longer time from introduction to adoption and thus adoption activity may not have been observable within the follow-up period for this study. The presence of a medical director may also influence the type of programming that is explored by the unit. It may be the case that medical directors have their own preferences for programs and may direct a programmatic strategy that is consistent with these preferences, leaving less time for staff to explore other approaches. Further research might examine the decision-making process of medical directors and the degree to which they consider evidence-based practices and a public health approach.
For hospitals without a medical director, the next measure that most differentiated adoption was having other units in the hospitals that provided IP services. The influence of other units providing IP services suggests that coordination among hospital staff is important when initiating hospital-based initiatives. It could the case that IP departments, which largely focus their efforts on community-based initiatives, are seeking ways to provide a service within the hospital to parallel the work of other IP departments such as the poison center or sports medicine that provide more hospital-related services that could be viewed as more useful by hospital administration than community-based efforts.
The number of requests received by IP staff, if there is no medical director and there are others that are engaged in IP efforts, was another important consideration. This factor may suggest that programs that are well integrated into hospital activities, even when there is no medical director and other units are providing IP services, are more likely to try innovative approaches.
Whether or not IP staff believed they influenced decisions was the last factor that differentiated adoption among those IP programs without a medical director in hospitals where other units are offering IP programs who received a high amount of requests from nonadministrative units. Leaders who believe they are trusted to develop programming appear more likely to adopt new innovations. From all of the constructs capturing organizational culture, having influence over programmatic decisions appears to be most important. When IP specialists perceive that they have control over decisions, they may be more inclined to explore evidence-based programs rather than simply implement a program favored by leadership. Conversely, it may be more difficult to disseminate programs in organizations where staff have less control.
Although this work was conducted in partnership with specialists in pediatric hospitals using an IP program as a resource, these results can inform prevention programming in other settings and for other topics. Future research and efforts to disseminate efficacious or novel programs should acknowledge the influence that administrative structure has on the adoption process. Medical directors, or other organizational leaders, can facilitate the adoption process for evidence-based interventions, but may need to be engaged intentionally when disseminating new products, tools, or approaches. Coordinating efforts among all staff who might be engaged in similar efforts and providing leadership development to specialists may also be important to increase adoption. In the absence of a medical director, a staff member may be more inclined to assume programmatic responsibilities if they know that they are the only provider of such services. Knowing that, program developers can customize training opportunities for such staff members and evaluate the success of these initiatives. Integrating prevention services into routine clinical practice and attending to the organizational culture may increase adoption of efficacious programs.
A strength of this project is that we sought to maximize external validity by studying the natural adoption process by hospitals of an innovative program. Increased attention to external validity is critical for closing the gap between what is known to work and what is incorporated into routine care [23]; using natural experiments is a methodological approach for studying the real world implementation of novel tools [24]. Proximal outcome variables, in addition to those measuring implementation, also strengthened the study and allowed us to examine adoption activity even when such investigations had not yet resulted in program implementation.
Limitations of the study include the 3-month follow-up period for addressing adoption activity. Because adoption of evidence-based programming is a complex process in clinical organizations, future research should incorporate longer follow-up periods which would allow for more in-depth study of the factors related to adoption. Furthermore, we also note that our overall response rate to the follow-up survey was nearly 50% and as such the estimates of adoption have a potential for nonresponse bias. To examine this more closely, we compared response distributions for 31 of the 36 covariates that were available from hospitals that responded to either the baseline or the follow-up assessments. In total, there were 50 hospitals that responded to the baseline but not the follow-up assessment. We found significant differences for two of these variables when comparing the 50 nonrespondents and the 93 respondents to the follow-up assessment. Those hospitals who reported that other groups offered IP programming in their hospitals had nearly 2.7 times the odds of responding to the follow-up survey compared with hospitals who reported that there was only a single group offering IP programming (hospip7dd), χ2(1)=7.9, p = .002. This variable was also among one of the most significant variables for predicting SNS adoption activity. Take together these associations suggest that if nonresponse bias is present it is likely manifesting in a higher false negative rate, meaning that more hospitals might be adopting SNS programming than is predicted by the model. The other variable that exhibited significant differences between those hospitals that responded and did not respond to the follow-up assessment was the number of FTEs related to IP programming. We found that the median FTE for responding hospitals was double that of nonresponding hospitals (e.g., 2 vs. 1; Mann-Whitney U = 1257.0; p = .001). Although this difference was significant, its potential in terms of nonresponse bias appears small because of the small influence the FTE variable had on predicting SNS adoption activity as shown at the bottom of Fig. 1.
CONCLUSION
Using random forest techniques and classification models, researchers can identify the factors most important in differentiating program adopters and nonadopters of innovations. In this application, it appears that adoption is not simply associated with a single predictor but rather represents a combination of factors including programming being housed in a unit without a medical director and within hospitals that have other units that offer IP programming and where IP specialists have time and believe they also have the latitude and autonomy to respond to their community needs in selecting programs. Because nonadopters were most likely hospitals with a medical director, future work disseminating clinical or prevention services to hospitals might benefit from considering the needs of medical directors directly and studying ways to market innovation to supervisory personnel. Furthermore, recruitment messaging could be tailored by leveraging important contextual factors to improve adoption rate of such IP programming.
Compliance with Ethical Standards
Primary Data: The findings reported have not been previously published. The manuscript is not being simultaneously submitted elsewhere. The data have not been previously reported. The authors have full control of all primary data and they agree to allow the journal to review their data if requested. The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Conflict of Interest: All authors report that they have no conflicts of interest.
Authors’ Contributions: N.L.W. led the development of the Safe N’ Sound (SNS) program, served as principal investigator (PI) for the funded project, and led the development of the assessments and implementation protocols. N.L.W. drafted the paper and incorporated critical review from co-authors. T.D.B. assisted with the survey design and implementation and was the primary contributor to the overall analysis of study data including the two-step modeling process. T.D.B. also assisted with interpretation of resulting models and contributed to several sections of the manuscript. K.J. served as the research manager for the project. She assisted in the development of the SNS resources, the assessment, and the distribution of SNS to hospitals. J.W. consulted on all matters related to the project from the perspective of an injury prevention specialist. She contributed to the interpretation of the findings. All authors read and approved the final manuscript.
Ethical Approval: The study was approved by the institutional review board (IRB) of the corresponding author’s institution. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The protocol was approved by the Saint Louis University IRB. This article does not contain any studies with animals performed by any of the authors.
Acknowledgments
The authors thank Karen Seaver Hill, Nancy Hansen, and Stacy Biddinger from the Children’s Hospital Association for their collaboration and the participating injury prevention specialists for their contributions. This work was made possible through a grant (1R21CE001830-01; PI Weaver) from the Centers for Disease Control and Prevention. Research reported in this abstract was further supported by the Washington University Institute of Clinical and Translational Sciences grant UL1 TR000448 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH). The funding agency was not involved in the design of the study; in the collection, analysis, and interpretation of the data; or in writing the manuscript.
References
- 1. Committee on Trauma, American College of Surgeons. Resources for the optimal care of the injured patient, Chapter 18. Chicago IL: Committee on Trauma, American College of Surgeons; 2014: 139–142. [Google Scholar]
- 2. Hellinger FJ. Tax-exempt hospitals and community benefits: a review of state reporting requirements. J Heal Polit Policy Law. 2009; 34: 37–61. [DOI] [PubMed] [Google Scholar]
- 3. AHRQ. Total Number of People Accounting for Expenditures (Depublicated) by Site Visit: United States. 2007, Medical Expenditure Panel Survey Component Data. Rockville, MD: Agency for Healthcare Research and Quality; 2007. [Google Scholar]
- 4. Green L, Brownson R, Fielding J. How is the growing concern for relevance and implementation of evidence-based interventions shaping the public health research agenda?Annu Rev Public Heal. 2017; 38: i–iii. [DOI] [PubMed] [Google Scholar]
- 5. Brown CH, Curran G, Palinkas LA, et al. An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Heal. 2017; 38: 1–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Scholtes B, Schröder-bäck P, Mackay JM, Vincenten J, Förster K, Brand H. Facilitators and barriers for the adoption, implementation and monitoring of child safety interventions : a multinational qualitative analysis. Inj Prev. 2017; 23(3): 197–204. [DOI] [PubMed] [Google Scholar]
- 7. Dzewaltowski D, Glasgow R, Klesges L, Estabrooks P, Brock E. RE-AIM: evidence-based standards and a web resource to improve translation of research into practice. Ann Behav Med. 2004; 28: 75–80. [DOI] [PubMed] [Google Scholar]
- 8. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012; 43: 337–350. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Wandersman A, Duffy J, Flaspohler P, et al. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008; 41: 171–181. [DOI] [PubMed] [Google Scholar]
- 10. Aarons GA, Sommerfeld DHM, Walrath-Greene CM, et al. Evidence-based practice implementation: the impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implement Sci. 2009; 4: 83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Prism SM, Feldstein AC. A practical, robust implementation for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008; 34: 228–243. [DOI] [PubMed] [Google Scholar]
- 12. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006; 21: 688–694. [DOI] [PubMed] [Google Scholar]
- 13. Scott SD, Plotnikoff RC, Karunamuni N, Bize R, Rodgers W. Factors influencing the adoption of an innovation: an examination of the uptake of the Canadian Heart Health Kit (HHK). Implement Sci. 2008; 3: 41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Nansel TR, Weaver NL, Jacobsen HA, Glasheen C, Kreuter MW. Preventing unintentional pediatric injuries: a tailored intervention for parents and providers. Health Educ Res. 2008; 23(4): 656–669. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Tse J, Nansel TR, Weaver NL, Williams J, Botello-Harbaum M. Implementation of a tailored kiosk-based injury prevention program in pediatric primary care. Heal Promot Pr. 2014; 15(2): 243–251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Weaver NL, Nansel TR, Williams J, Tse J, Botello-Harbaum M, Willson K. Reach of a kiosk-based pediatric injury prevention program. Transl Behav Med. 2011; 1(4): 515–522. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Weaver NL, Kortlandt V, Williams J, et al. Assessing community-based injury prevention services in U.S. children’s hospitals. AIMS Public Health. 2014; 1(4): 199–210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Stekhoven DJ. Using the missForest Package. 2011: 1–11.
- 19. Strobl C, Malley J, Tutz G. An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychol Methods. 2009; 14(4): 323–348. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Mendez G, Buskirk TD, Lohr S, Haag S. Factors associated with persistence in science and engineering majors: an exploratory study using random forests. J Eng Educ. 2008; 97: 57–70. [Google Scholar]
- 21. Strobl C, Boulesteix AL, Zeileis A, Hothorn T. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics. 2007; 8: 25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. James G, Witten D, Hastie T, Tibshirani R.. An Introduction to Statistical Learning. New York: Springer; 2013. [Google Scholar]
- 23. Leviton LC. Generalizing about public health interventions: a mixed-methods approach to external validity. Annu Rev Public Heal. 2017; 38: 371–391. [DOI] [PubMed] [Google Scholar]
- 24. Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu Rev Public Health. 2017; 38: 39–56. [DOI] [PMC free article] [PubMed] [Google Scholar]